summaryrefslogtreecommitdiffstats
path: root/lib/bb/cooker.py
Commit message (Collapse)AuthorAgeFilesLines
* cooker: fix watching directories with Python 3.6+Paul Eggleton2017-10-311-1/+11
| | | | | | | | | | | | | | | | | | | | | | In Python 3.6, glob.glob() was reimplemented to use os.scandir() (which itself appeared in Python 3.5), thus our monkey patching of os.listdir() here was no longer effective. The end result was not only that bitbake wouldn't notice added recipes or bbappends with BB_SERVER_TIMEOUT set when being run with Python 3.6 (the shipped Python version on Fedora 26 and some other distribution versions), it also broke devtool modify, devtool upgrade and devtool extract since they rely on the ability to create a bbappend on the fly and have bitbake pick it up. To fix it, do the same monkey patching for os.scandir(), which needs to be conditional upon that actually existing since we have to support Python 3.4 that doesn't have it. Long term we should probably look for a better way to handle this that doesn't involve monkey patching Python library code. Fixes [YOCTO #12185]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: ensure monkey-patching in collect_bbfiles() gets undone on errorPaul Eggleton2017-09-141-19/+19
| | | | | | | | | | | | In collect_bbfiles() we're monkey-patching os.listdir in order to find which directories to watch, and then undoing that when we're finished - however if an exception occurred for any reason there was nothing to ensure the latter occurred. This may not have caused any issues, but as this kind of thing really ought to be secured using try...finally just in case, so do that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: fix watching empty directoriesPaul Eggleton2017-09-141-3/+6
| | | | | | | | | | | | | | | | | The code that was supposed to watch directories along BBFILES for creation of new files wasn't working in the case where the directory did not initially contain any matching files - since in updateCache() we are passing the directory path to add_filewatch() and the latter function calls os.path.dirname() on the path on the assumption that it is a file path, and thus the parent of the directory got watched but not the directory itself. (If the directory wasn't empty everything worked fine since add_filewatch() was called elsewhere with the path to one of the files in that directory, and thus the directory got watched). Add a parameter to add_filewatch() to tell it we are passing it directory path(s) rather than file path(s). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process/cooker: Improve readypipe handlingRichard Purdie2017-09-111-5/+1
| | | | | | | | | | | Issues in start are not being correctly detected by the current readypipe code. Change it to use specific "ready" or "fail" messages to correctly determine the correct failure mode and avoid bitbake seeming to hang (it does currently timeout eventually). [YOCTO #12062] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: clear extra config settings and remote datastores on client disconnectPaul Eggleton2017-09-051-0/+2
| | | | | | | | | | | | | | When the UI disconnects, we can throw away any server-side remote datastores we created in response to calls from the UI, and we *must* drop everything in extraconfigdata or it will taint any future operations. Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors when running devtool.DevtoolTests.test_devtool_update_recipe_local_files within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker.py: Fix layer priority processingMark Hatle2017-09-051-7/+7
| | | | | | | | | | | | | | | If you have a layer with a blank BBFILE_PATTERN the layer was ignored when processing the list of layers with priorities. This list is not only used for processing recipes, but also by additional programs such as bitbake-layers show-layers. Without this change, a layer that provides configuration or classes only does now show up in show-layers, which is used by the yocto-compat-layer.py script. This causes a failures in the compatibility check. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Ensure parseConfiguration clears parsecache_validRichard Purdie2017-09-011-0/+2
| | | | | | | | | | BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers was failing and highlighted that since parseConfiguation clears data structures, it needs to also clear parsecache_valid as it no longer contains correct data. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Ensure buildFile doesn't have lasting side effectsRichard Purdie2017-08-311-0/+5
| | | | | | | | | | | | | | | | | | | | | | | BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_build_image fails, the reason is that internally, the limited_deps flag is set in the bitbake server and then never cleared. This causes the sysroots to be setup incorrectly (as per the limited dependency case) and builds break. There is also potential for corruption of recipecaches. Add shutdown/cleanup code to ensure these effects don't 'stick'. This bug is particularly nasty as you can destroy TMPDIR with large sysroots in build work directories which are prone to break. Also ensure mtime cache is cleared (to match buildTargets) and that no lasting changes are made to siggen either which ensures: BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_upgrade_git devtool.DevtoolTests.test_devtool_virtual_kernel_modify works. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: ensure we can run buildFileInternal() after cache is populatedPaul Eggleton2017-08-311-4/+4
| | | | | | | | | | | | | | | | | | | | | | | If you run some other operations that result in the cache being populated, and then call buildFileInternal(), then you can end up in a situation where the cache already contains information about the recipe. For example in OE this can now happen when you use devtool upgrade. Normally this doesn't cause any problems, unless you have a non-absolute path in BBLAYERS - in buildFileInternal() we are calling matchfile() which will convert the filename to absolute, but later when taskdata goes to find the providers of the recipe it finds the non-absolute path, sets up the task information using this and then the runqueue can't find any tasks matching the absolute path. To fix this, back out the optimisation I did earlier in bitbake rev ba53e067a2d448dd63b4ca252557ce98aa8e6321 to avoid calling parseConfiguration() again, which is unfortunate but does result in the cached information being that causes the problem being cleared out. This fixes "Task do_unpack does not exist for target ..." running devtool upgrade within intel-iot-refkit. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Allow changes in PRSERV_HOST to be responded toRichard Purdie2017-08-311-8/+11
| | | | | | | | | | | | When we reload the configuration, PRSERV_HOST can change. Therefore restart the PR Server depending on the new configuration at reparse. Note that the server has to be started from the right process, it can't be in the UI which shuts down as that shutdown triggers a shutdown of its children and the PR Server shuts down too. This is why we need pre_serve() which ensures its executed in the right context. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Change to consistent prefile/postfile handlingRichard Purdie2017-08-311-13/+10
| | | | | | | | | | | | Currently the original prefile and postfile passed when starting bitbake server are 'sticky'. With the new memory resident model this doesn't make sense as the server the system is started with isn't special. This patch changes the code so the prefile/postfile are used if specified on the commandline and not used otherwise. This makes the behaviour much more predictable and expected and as an added bonus simplifies the code. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Handle datastore tracking for showEnvironment server side onlyRichard Purdie2017-08-311-1/+10
| | | | | | | | The current passing of "tracking" backwards and forwards, client to server is ugly and complex and error prone. Instead, set this during showEnvironment commands triggering a reset there if/as required. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* prserv/cooker: Drop unused paramRichard Purdie2017-08-311-1/+1
| | | | | | Drop pointless unused function parameter. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Ignore common bitbake files for the parse cache invalidationRichard Purdie2017-08-091-0/+3
| | | | | | | | | | | | Writes to the cookerdaemon log and/or the lockfile were meaning the parser cache was always being invalidated and reparsed. This is unnecessary so spot accesses to these two common cases and ignore the files from a reparse perspective. This doesn't remove many sources of reparse but does improve several common cases. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Improve inotify overflow handlingRichard Purdie2017-08-091-1/+4
| | | | | | | | Add a proper function for clearing the mtime cache. Clean up the inotify event overflow case to err on the side of caution and clear any potentially now out of sync caches. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Ensure we handle inotify before running the next commandRichard Purdie2017-08-091-7/+10
| | | | | | | | | | | | The inotify watch events are handled "at idle" which could in theory mean a command could run before any preceeding inotify events have been processed. This leads to a theoretical race window where those events may have a signficicant effect on the command. Add a mechanism to allow us to ensure all pending events are processed before running commands. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Track directories searched for bbappend/bb filesRichard Purdie2017-08-091-3/+21
| | | | | | | | | | | | | | | Some of the directories searched by BBFILES are not currently being added to the inotify watch list. This can mean that added append files are not noticed leading to misleading metadata results when using BB_SERVER_TIMEOUT != 0. We use glob to expand the BBFILES references and without writing our own implentation, figuring out the directories it searches is hard. We use some horrible hacks here to intecept the listdir calls, I'm open to better ways to handle this but this does ensure we have the right watches set. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker/siggen: Reset siggen when reparsingRichard Purdie2017-08-091-0/+1
| | | | | | | | | | If we don't do this, we get basehash mismatch errors occurring from the reparse which would then set bitbake's error exit code. This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order to fail with a non-zero BB_SERVER_TIMEOUT. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Ensure all tasks are shown in task dependency treeRichard Purdie2017-08-011-3/+3
| | | | | | | | | | "bitbake -g m4-native -c do_unpack" doesn't list any dependencies in task-depends.dot. This is because no header describing the task was added unless a task has dependencies. Tweak the code to fix this. [YOCTO #10893] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process/cooker: Allow UI process to know if the cooker was started successfullyRichard Purdie2017-07-281-1/+5
| | | | | | | | | | | | | | | Currently if the server fails to start, the user sees no error message and the server will be repeatedly attempted to be started until some longer timeouts expire. There are error messages in the cookerdeamon log but nobody thinks to look there. Add in a pipe which can be used to tell the starting process whether the cooker did actually start or not. If it fails to start, no further attempts can be made and if present, the log file can be shown to the user. [YOCTO #11834] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker/process: Drop server_main functionRichard Purdie2017-07-241-22/+0
| | | | | | | | | | | | Now that there is only one server, this abstraction is no longer needed and causes indrection/confusion. The server shutdown is also broken with the cooker post_server calls happening too late, leading to "lock held" warnings in the logs if PRServ is enabled. Remove the abstraction and put the shutdown calls in the right order with respect to the locking. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: add internal mode to build_file() functionPaul Eggleton2017-07-211-9/+25
| | | | | | | | | | | | | | | In OE's devtool we want to repeatedly run build_file() without showing unnecessary messages and triggering buildhistory for each call. build_file() is just a wrapper around the buildFile command. Change the final "hidewarning" parameter of the buildFile command to "internal" and have this call a new buildFileInternal() function without triggering any of the normal build events, silencing the normal info messages from the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and avoiding calling parseConfiguration() which we've already done at this point. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cookerdata/cooker: Restore original datastore upon client disconnectRichard Purdie2017-07-211-0/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server: Rework the server API so process and xmlrpc servers coexistRichard Purdie2017-07-211-57/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes the way bitbake server works quite radically. Now, the server is always a process based server with the option of starting an XMLRPC listener on a specific inferface/port. Behind the scenes this is done with a "bitbake.sock" file alongside the bitbake.lock file. If we can obtain the lock, we know we need to start a server. The server always listens on the socket and UIs can then connect to this. UIs connect by sending a set of three file descriptors over the domain socket, one for sending commands, one for receiving command results and the other for receiving events. These changes meant we can throw away all the horrid server abstraction code, the plugable transport option to bitbake and the code becomes much more readable and debuggable. It also likely removes a ton of ways you could hang the UI/cooker in weird ways due to all the race conditions that existed with previous processes. Changes: * The foreground option for bitbake-server was dropped. Just tail the log if you really want this, the codepaths were complicated enough without adding one for this. * BBSERVER="autodetect" was dropped. The server will autostart and autoconnect in process mode. You have to specify an xmlrpc server address since that can't be autodetected. I can't see a use case for autodetect now. * The transport/servetype option to bitbake was dropped. * A BB_SERVER_TIMEOUT variable is added which allows the server to stay resident for a period of time after the last client disconnects before unloading. This is used if the -T/--idle-timeout option is not passed to bitbake. This change is invasive and may well introduce new issues however I believe the codebase is in a much better position for further development and debugging. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: fix whitespacePaul Eggleton2017-07-181-4/+4
| | | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Use multiple BuildStarted events for multiconfigRichard Purdie2017-07-071-12/+16
| | | | | | | | | | | | | | | | | | | | | | | | | Currently builds in multiple TMPDIRs with multiconfig can break since the BuildStarted event is used to create directory strutures in several cases (e.g. buildstats.bbclass) and there is only on BuildStarted event generated in a multiconfig build. We have two options, a) to add a new MultiConfigBuildStarted event which is generated once per multiconfig, or b) allow multiple BuildStarted events. Having reviewed the code and current users of BuildStarted, sending one event per multiconfig seems like its the best way forward and the existing code looks able to cope with the duplication of events. I did also check toaster and I think that can handle this issue too (multiconfig builds may have other issues there). I'm therefore proposing we send multiple BuildStarted events for multiconfig and for consistency, send multiple BuildCompleted events too. We need to ensure that BUILDNAME, BUILDSTART and DATE/TIME are set consistently in all the different multiconfig datastores. These events can write to the datastore so copies are not used. buildFile was also cleaned up to ensure it uses the right datastore in various places. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: ensure graceful exit after exception during BuildCompleted handlerPaul Eggleton2017-06-141-2/+4
| | | | | | | | | | | | If an event handler for bb.event.BuildCompleted fails, we still need to call finishAsyncCommand() or else BitBake will just exit immediately without showing any error summary, or worse in the case of memory resident mode BitBake will hang and if you Ctrl+C to break out, the command won't be marked as finished which means that no further commands will be able to be executed until the server is manually restarted. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: fix always loading cache on every UI start with memresPaul Eggleton2017-06-141-5/+12
| | | | | | | | | | | | | | | | | | | The main point of memory resident bitbake is to avoid loading data unnecessarily on every bitbake invocation. Unfortunately the code that updated options from the UI was simply treating the fact that either of the "prefile" or "postfile" options were in the list of options passed in as an indication that the configuration was invalid, which was bad because these are always passed in. We only need to mark the configuration as invalid and thus reload it (and thus reload the cache) if the option value has actually changed. At the same time, the recently handled "tracking" option needs to be treated in a similar manner since the configuration needs to be reparsed if that has changed. Also, add a few extra debug messages to aid debugging this code in future. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* command / cooker: drop Hob support commands and functionsPaul Eggleton2017-06-091-193/+0
| | | | | | | | | | | | | | | | | Drop a number of the commands and support functions that were created solely to support functionality in the now-removed Hob UI. In most cases we now have this functionality elsewhere e.g. to modify config files and recipes we have bb.utils.edit_metadata_file() and friends, and in OE we have oe.recipeutils.patch_recipe_file() and friends which build on top of the former. Additionally, some of it represented pretty egregious incursion of OE metadata-specific references into BitBake code. For now I have left in the find*File functions and commands as they are relatively generic and possibly still useful, but they might be removed in future. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: add BB_CMDLINE to enable access to UI command line with memresPaul Eggleton2017-06-091-1/+6
| | | | | | | | | | | | | | | | | | | | | In OpenEmbedded's buildhistory class we want access to the bitbake command line that launched the build, and up to now we were simply using sys.argv from within the event handler to get that. Unfortunately that doesn't work in memory resident mode, since the event handler is naturally executing within the server and thus will give you the command that launched the bitbake server which is much less interesting. Add a dynamic variable BB_CMDLINE to provide access to this, set from sys.argv within the UI process in updateToServer(). (Note that BB_CMDLINE isn't currently passed through to the worker, so this is only really readable from event handlers plus any explicit getVariable calls - in theory an observe-only UI could read it for example.) Part of the fix for [YOCTO #11634]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: set config valid flags in initConfigurationData()Paul Eggleton2017-06-091-6/+2
| | | | | | | | | | | | If we set these flags here then we can not only tidy up some of the code that calls initConfigurationData() (and reset() which simply calls the former), we also avoid calling initConfigurationData() an extra time unnecessarily during startup (since setting baseconfig_valid = False in updateConfigOpts() also resulted in initConfigurationData() being called from updateCacheSync() at the end of the command.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: fix file changes not triggering task re-execution with memresPaul Eggleton2017-06-091-1/+13
| | | | | | | | | | | | | | | | | | | | | | If you build a recipe, modify one of the files referred to in SRC_URI, then build it again, it should re-run do_fetch and the tasks that depend upon it to incorporate the modified file. With memory resident mode this was not working unless you restarted the server, because the mtime of the file was cached and thus we never checked the actual file. Clear out the mtime caches in buildTargets() to ensure this isn't an issue. I did a minor refactoring to break this out to its own function since we were already clearing out the build mtime cache here. (The test I used for this was . ./oe-init-build-env-memres, bitbake -c fetch mdadm, modify the comments at the top of one of the patches referred to in the recipe, and then bitbake -c fetch mdadm again and it should re-execute the fetch task ("... 0 didn't need to be re-run".) Fixes [YOCTO #10732]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: fix undefined dep in generatePkgDepTreeDataAndy Voltz2017-06-041-2/+2
| | | | | | | | The loop for populating the depends of the tree data should use dep instead of item. Signed-off-by: Andy Voltz <andy.voltz@timesys.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker.py: use correct multiconfig prefix in dependency graphPatrick Ohly2017-03-221-1/+1
| | | | | | | | | | The dependency graph in the bb.event.DepTreeGenerated and the corresponding pn-buildlist and task-depends.dot from "bitbake -g" contained entries like multiconfig:qemuarm.gcc (dot as last separator) instead of the correct multiconfig:qemuarm:gcc (colon as separator). Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* data/cooker: Sort output data shown by -eRichard Purdie2017-03-211-1/+1
| | | | | | | | Displaying the environment data in a sorted manner makes it easier to compare data between different setups and means you can know where to find specific entries. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: detect malformed BBMASK expressions which begin with a separatorAndre McCurdy2017-02-191-0/+5
| | | | | | | | When constructing an older style single regex, it's possible for BBMASK to end up beginning with '|', which matches and masks _everything_. Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Rewrite dependency dot file generationRichard Purdie2017-02-191-53/+45
| | | | | | | | | | | | | | | | | | | | | | The package-depends.dot and pn-depends.dot files are inaccurate, missing out key dependencies such those made via the [depends] flags. As such they can be misleading to the user. They mainly exist for historical reasons, coming from a time before we had task based execution. This commit removes the two dated file formats and replaces them with a recipe-depends.dot which is a flattened version of task-depends.dot. The old format files are removed if present so that the user can't get confused about why data might not match between files. The code is also rewritten to use 'with f: f.write()' syntax as is more commonly used now. Also update the docs to match the change. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker.py: run sanity checks for multiconfigJuro Bystricky2017-02-131-1/+2
| | | | | | | | | | Also run sanity check for all additional multiconfig configurations as listed in BBMULTICONFIG, not just the "default" configuration. [YOCTO#10810] Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib: Drop now unneeded update_data callsRichard Purdie2017-02-131-3/+0
| | | | | | | | Now that the datastore works dynamically we don't need the update_data calls so we can just remove them. They're not actually done anything at all for a while. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Add BB_LIMITEDDEPS supportRichard Purdie2017-01-191-0/+1
| | | | | | | | | When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited. We really need a way to know this from the metadata and this new variable provides this in worker context. This means existing code can stop having to guess. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Only add universe targets where the task existsRichard Purdie2017-01-191-0/+9
| | | | | | | | | | | Currently "bitbake universe -c fetch" will error if every task in the system doesn't have a fetch task. This seems unreasonable so filter the list based upon whether the task exists. This implementation does assume if a task exits for a given provider name, it can run, even if the preferred provider that ends up getting built might be the namespace without the task. This is unlikely to be a real world issue at this point. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker.py: new multiconfig '*' syntax supportJuro Bystricky2017-01-171-1/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently you cannot build a target for all the configured multiconfigs without specifying a list. The list can be quite long, requiring to type several lines of text. This enhancement is to support globbing so that you can do this, e.g. instead of: $ bitbake multiconfig:A:bash multiconfig:B:bash bash you can do: $ bitbake multiconfig:*:bash There are real world use cases where it is desirable to use multiconfig with two different tasks. For example: SDKs with multiple toolchains but also containing set of additional host tools, or multiconfig builds requiring one image for the main CPU(s) and a different co-image for a companion CPU. For this reason, two variations of the new syntax are supported. For example, the following: $ bitbake multiconfig:*:meta-toolhchain would expand to: $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain meta-toolchain However the following: $ bitbake multiconfig:*:meta-toolhchain hosttools would expand to: $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain hosttools In other words, if the user specified the "default" task explicitly, it replaces the implicit "default" task. [YOCTO#10680] Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bb/cooker: BBCooker stops notifier at shutdownAníbal Limón2017-01-061-0/+2
| | | | | | | | | | | At end of BBCooker needs to release fd's associated with pyinotify watchers to avoid: Too many open files (EMFILE) error in different scenarios like several instances of tinfoil. [YOCTO #10873] Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker/command: Drop expanded_dataRichard Purdie2016-12-161-14/+12
| | | | | | | | | | | | | Some of our metadata assumes that BuildStarted and BuildCompleted events see the same data store. This is the case for buildTarget but not for buildFile and recent changes mean this is now a problem. The update_data() call is now an empty operation and there is no difference between the expanded_data and data so we can simply remove the expanded_data and its references and use data everywhere. This has been inteded for a while but the above issue makes this more pressing to finally clean up. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: enable setVariable command to affect task executionPaul Eggleton2016-12-141-0/+1
| | | | | | | | | | Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: allow buildFile warning to be hidden programmaticallyPaul Eggleton2016-12-141-4/+5
| | | | | | | | If we want to use this function/command internally, we don't want this warning shown. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: rewrite as a wrapper around the UIPaul Eggleton2016-12-141-3/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rewrite tinfoil as a wrapper around the UI, instead of the earlier approach of starting up just enough of cooker to do what we want. This has several advantages: * It now works when bitbake is memory-resident instead of failing with "ERROR: Only one copy of bitbake should be run against a build directory". * We can now connect an actual UI, thus you get things like the recipe parsing / cache loading progress bar and parse error handling for free * We can now handle events generated by the server if we wish to do so * We can potentially extend this to do more stuff, e.g. actually running build operations - this needs to be made more practical before we can use it though (since you effectively have to become the UI yourself for this at the moment.) The downside is that tinfoil no longer has direct access to cooker, the global datastore, or the cache. To mitigate this I have extended data_smart to provide remote access capability for the datastore, and created "fake" cooker and cooker.recipecache / cooker.collection adapter objects in order to avoid breaking too many tinfoil-using scripts that might be out there (we've never officially documented tinfoil or BitBake's internal code, but we can still make accommodations where practical). I've at least gone far enough to support all of the utilities that use tinfoil in OE-Core with some changes, but I know there are scripts such as Chris Larson's "bb" out there that do make other calls into BitBake code that I'm not currently providing access to through the adapters. Part of the fix for [YOCTO #5470]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* command: provide a means to shut down from the client in memres modePaul Eggleton2016-12-141-0/+7
| | | | | | | | In memory resident mode we don't really want to actually shut down since it's only the client going away. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Handle inofity queue overflows more gracefullyRichard Purdie2016-12-081-0/+8
| | | | | | | | | If many files change and the inotify queue overflows, rather than print a traceback, invalidate the caches and warn the user. [YOCTO #10676] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Fix world taskgraph generation issueRichard Purdie2016-12-081-4/+10
| | | | | | | | | | | | The processing of the "do_" prefix to tasks is currently inconsistent and has resulted in "bitbake world -g" being broken as task prefixes don't get handled correctly. Make the "do_" task prefix handling consistent through various codepaths. [YOCTO #10651] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>