summaryrefslogtreecommitdiffstats
path: root/lib/bb/parse/__init__.py
AgeCommit message (Collapse)Author
2017-08-09cooker: Improve inotify overflow handlingRichard Purdie
Add a proper function for clearing the mtime cache. Clean up the inotify event overflow case to err on the side of caution and clear any potentially now out of sync caches. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16parse: Don't try to expand __base_depends/__dependsRichard Purdie
Trying to expand a variable which isn't a string doesn't make sense. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-18lib/bb/parse: properly handle OSError when updating mtime cachePaul Eggleton
If a file no longer exists, drop it from the cache silently instead of generating a traceback. This was visible in some cases when a recipe was deleted when bitbake was resident in memory. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01bb.parse: properly error out on filesystem errorsChristopher Larson
We've had a long-standing bug where a legitimate error reading a file (IOError or OSError) is always suppressed as though it was a 'file not found' case. As a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently not parse local.conf, rather than erroring to let the user know about the problem. Fix this by handling the ENOENT case specifically. Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-19bitbake: Add explict getVar param for (non) expansionRichard Purdie
Rather than just use d.getVar(X), use the more explict d.getVar(X, False) since at some point in the future, having the default of expansion would be nice. This is the first step towards that. This patch was mostly made using the command: sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *` Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-13cooker/cache/parse: Implement pyinofity based reconfigureRichard Purdie
Memory resident bitbake has one current flaw, changes in the base configuration are not noticed by bitbake. The parsing cache is also refreshed on each invocation of bitbake (although the mtime cache is not cleared so its pointless). This change adds in pyinotify support and adds two different watchers, one for the base configuration and one for the parsed recipes. Changes in the latter will trigger a reparse (and an update of the mtime cache). The former will trigger a complete reload of the configuration. Note that this code will also correctly handle creation of new configuration files since the __depends and __base_depends variables already track these for cache correctness purposes. We could be a little more clever about parsing cache invalidation, right now we just invalidate the whole thing and recheck. For now, its better than what we have and doesn't seem to perform that badly though. For education and QA purposes I can document a workflow that illustrates this: $ source oe-init-build-env-memres $ time bitbake bash [base configuration is loaded, recipes are parsed, bash builds] $ time bitbake bash [command returns quickly since all caches are valid] $ touch ../meta/classes/gettext.bbclass $ time bitbake bash [reparse is triggered, time is longer than above] $ echo 'FOO = "1"' >> conf/local.conf $ time bitbake bash [reparse is triggered, but with a base configuration reload too] As far as changes go, I like this one a lot, it makes memory resident bitbake truly usable and may be the tweak we need to make it the default. The new pyinotify dependency is covered in the previous commit. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-05-30event: Add SkipRecipe event to replace SkipPackageRichard Purdie
In the depths of time we were rather confused about naming. bb files are recipes, the event to skip parsing them should be SkipRecipe, not SkipPackage. This changes bitbake to use the better name but leaves the other around for now. We can therefore start removing references to it from the metadata. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-05-11parse: Improve file inheritance loggingRichard Purdie
The file inheritance logging has been here since the dawn of time. It duplicates output many times over and logs to debug level 2. When running with the debug option, its understandable the user may want to see the paths of files included in the build. These changes remove pointless/duplicate output and print the include paths clearly at debug level one in a form which users should be able to more easily parse. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-02-17parse: make vars_from_file return None for non-recipesPaul Eggleton
It doesn't really make sense to set PN from .conf files, for example. More concretely, this avoids the config hash changing unnecessarily within Hob due to PN effectively changing (since bblayers.conf is parsed first and then .hob.conf). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-11-29parse/ConfHander/BBHandler/utils: Fix cache dependency bugsRichard Purdie
Currently bitbake only adds files to its dependency list if they exist. If you add 'include foo.inc' to your recipe and the file doesn't exist, then later you add the file, the cache will not be invalidated. This leads to another bug which is that if files don't exist and then you add them and they should be found first due to BBPATH, again the cache won't invalidate. This patch adds in tracking of files we check for the existence of so that if they are added later, the cache correctly invalidates. This necessitated a new version of bb.utils.which which returns a list of files tested for. The patch also adds in checks for duplicate file includes and for now prints a warning about this. That will likely become a fatal error at some point since its never usually desired to include a file twice. The same issue is also fixed for class inheritance. Now when a class is added which would be found in the usual search path, it will cause the cache to be invalidated. Unfortunately this is old code in bitbake and the patch isn't the neatest since we have to work within that framework. [YOCTO #5611] [YOCTO #4425] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-24parse: Return IOError when including file with absolute pathMartin Jansa
* resolve_file was behaving different when relative and absolute paths were passed to it * include relative-path/non-existent-file.inc works correctly resolve_file throws IOError, BBHandler.py:handle() doesn't catch it, ConfHandler.py:include() catches IOError and shows: DEBUG: CONF file 'relative-path/non-existent-file.inc' not found * include /absolute-path/non-existent-file.inc was failing, because resolve_file just returns fn, BBHandler.py:handle() calls bb.parse.mark_dependency(d, abs_fn) which throws: OSError: [Errno 2] No such file or directory: '/absolute-path/non-existent-file.inc' and parsing fails. Ad isfile() test for absolute fn and throw IOError to make resolve_file behavior consistent for both paths. * I know we had some issues with -b relative-path-to-recipe.bb and absolute path, so consider this patch only as RFC and documentation of this problem * Catch OSError too in ConfHandler.py:include() e.g. in case the file exists, but user cannot read it or something like that. Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-18bitbake: data_smart.py and friends: Track file inclusions for bitbake -ePeter Seebach
This code adds inclusion history to bitbake -e output, showing which files were included, in what order. This doesn't completely resolve timing questions, because it doesn't show you which lines of a file were processed before or after a given include, but it does let you figure out what the path was by which a particular file ended up in your build at all. How it works: data_smart acquires a .history member, which is an IncludeHistory; this represents the inclusion of a file and all its inclusions, recursively. It provides methods for including files, for finishing inclusion (done as an __exit__), and for dumping the whole tree. The parser is modified to run includes inside a with() to push and pop the include filename. RP Modifications: a) Split Include and Variable tracking b) Replace deepcopy usage with dedicated copy function c) Simplify some variable and usage Signed-off-by: Peter Seebach <peter.seebach@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-19parse/cache/cooker: Preserve order in the file inclusion listRichard Purdie
The data returned by get_file_depends() may me used in contexts like checksums where order is important. The current usage of sets means that some of the checksums can change in circumstances they should not. This patch changes to use lists, thereby removing the problem. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-28bitbake: Added BBINCLUDED variable.Lianhao Lu
Added new variable BBINCLUDED indicating the file dependency information. It exposes the internal variable '__base_depends' and '__depends'. Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-23bitbake: add file and line number to ParseErrorPaul Eggleton
Ensure that a file and line number are reported for ParseError where possible. This helps particularly in the case of inherit and require which previously did not report either of these upon failure. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-27Update users of getVar/setVar to use the data store functions directlyRichard Purdie
(From Poky rev: affd0907253387cee0ba0c7e9f92bcc109895446) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-07Merge branch 'poky-sync'Chris Larson
* poky-sync: build: kill unneeded environment setup bits parse: save python functions into the metadata Include exported variables in task process env runqueue: start implementing quieterrors Overhaul environment handling runqueue: resurrect use of file objects for pipein/pipeout runqueue: clean up message log levels runqueue: use correct task ID when checking validity of setscene tasks Add support for 'noexec' tasks runqueue: implement cache checking from metadata runqueue: pass task hash information to tasks runqueue: avoid unnecessary delays Implement task signatures data: Add emit_func and generate_dependencies codeparser: Implement persistent cache data_smart: track variable references codeparser: add module for parsing shell/python runqueue: implement scenequeue fetch: add SRC_URI checksum Signed-off-by: Chris Larson <chris_larson@mentor.com>
2011-01-04parse: Use constants from stat instead of magic numbersBernhard Reutner-Fischer
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-12-29Implement task signaturesRichard Purdie
Includes functionality to find out what changes between two different signature data dumps. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-12-17parse: use bb.utils.whichChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-09-03Switch bitbake internals to use logging directly rather than bb.msgChris Larson
We use a custom Logger subclass for our loggers This logger provides: - 'debug' method which accepts a debug level - 'plain' method which bypasses log formatting - 'verbose' method which is more detail than info, but less than debug Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04cache: use a set() for __dependsBernhard Reutner-Fischer
to make updating depends easier/more intuitive/eventually faster Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-06-04Make the file not found error actually useable.Chris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-11Apply some 2to3 transforms that don't cause issues in 2.6Chris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Drop __all__ usageChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-04-09Stop using functions via 'bb' that were moved from there to other modulesChris Larson
Signed-off-by: Chris Larson <chris_larson@mentor.com>
2010-02-12[parse] Use bb.which in resolve_fileChris Larson
Signed-off-by: Chris Larson <clarson@kergoth.com>
2010-02-12[parser] Make resolve_file only resolve the pathHolger Freyther
Do not attempt to open the file in the resolve_file method (a lot like bb.which... maybe bb.which can be used). This way we don't need to open/close a file which we have already parsed.
2010-02-12[parser] Remove the "data" from feeder, evaluate after parsing a fileHolger Freyther
Evaluate the statements after having parsed one file. This is referred to as "entwirren" and we can remove the direct evaluation and postpone a bit, in the future we can use a cached copy instead of parsing the original.
2010-02-12[parse] Move vars_from_file from bb.parse.BBHandler into bb.parse.Chris Larson
Signed-off-by: Chris Larson <clarson@mvista.com>
2010-02-12[parse] Don't use relative import of parse_py.Chris Larson
Signed-off-by: Chris Larson <clarson@mvista.com>
2010-02-12[parser] Cary a Statement Node through the parsingHolger Freyther
When parsing we will collect a number of statements that can be evaluated...The plan is to be evaluate things twice (old+new) and then compare the result, it should be the same.
2010-01-21[parse] Unify opening a file...Holger Freyther
2008-09-30parse/__init__.py: Fix missing update_mtime function causing shell reparsing ↵Richard Purdie
breakage (from poky)
2006-12-08Add proper GPLv2 headers to all BitBake filesHolger Hans Peter Freyther
BitBake trunk is now GPLv2 only, no mix of MIT,FreeBSD License is left. Update GPL headers to point to the correct address of the FSF Update the list of authors. Uli Luckas, Seb Frankengul and Tim Amsell contributed to the sourcecode as well
2006-11-14cache.py: Share the parser's mtime_cache, reducing the number of stat syscallsRichard Purdie
2006-04-28 bitbake/lib/bb/parse/__init__.py:Holger Hans Peter Freyther
Bug 895. __depends is a single string with "filename@time" the string gets splitted by ' ' as it is assumed that "filename@time filename2@time2" is true. Basicly on PPC/Darwin we have "PowerPC Macintosh.conf" splitting by space leads to the error observed by koen. Resolution: As we use __depends only as a list, save it as a list. This avoids the int->str->int, and split, append, join operations. bitbake/lib/bb/cache.py: __depends is now a list, change the version of the cache and simplify the method.
2005-09-08parser: add function to update the mtime for one file, courtesy Justin PatrinMichael 'Mickey' Lauer
parser: move import bb and import os out of the commands (import in a function that's called often has a performance penalty) lib: increase revision shell: add 'reparse' and 'fileReparse' commands, courtesy Justin Patrin
2005-07-11Sorry for the inconvience and add the missing fileHolger Hans Peter Freyther
I wonder what would happen if I only commit after getting enough donations for a file... Currently it defaults to the python implementation
2005-07-10Prepare for the new tokenizer C-based parser as optional parser.Holger Hans Peter Freyther
For now import bb.parse will function the way it used to work
2005-05-26have bb.parse.handle() throw ParseError if the input file is not Phil Blundell
understood by any handler, rather than quietly returning None
2004-12-07Initial import.Chris Larson
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-27 17:14:35 +0000