summaryrefslogtreecommitdiff
path: root/bin
Commit message (Collapse)AuthorAgeFilesLines
* bitbake-layers: Handle exception raised bytinfoil.prepare()Aníbal Limón2017-02-191-10/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The tinfoil.prepare method can raise Exceptions when is parsing initial data so add this call inside try, finally to avoid get blocked for don't call tinfoil.shutdown(). The tinfoil_init function was remove because isn't make sense now since tinfoil.prepare() needs to be inside try, finally closures. Example of raised exception and gets blocked: $ bitbake-layers add-layer ~/repos/meta-openembedded/meta-python/ Traceback (most recent call last): File "/home/alimon/repos/poky/bitbake/bin/bitbake-layers", line 83, in main tinfoil.prepare(True) ... File "/home/alimon/repos/poky/bitbake/lib/bb/tinfoil.py", line 268, in run_command raise TinfoilCommandFailed(result[1]) bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last): File "/home/alimon/repos/poky/bitbake/lib/bb/command.py", line 81, in runCommand result = command_method(self, commandline) ... File "/home/alimon/repos/poky/bitbake/lib/bb/cooker.py", line 1314, in handleCollections raise CollectionError("Errors during parsing layer configuration") bb.cooker.CollectionError: Errors during parsing layer configuration Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: remove cut and paste cruftbrian avery2017-02-131-12/+2
| | | | | | | | | | | There was a superfluous section in toaster start/stop script that was setting the TOASTER_DIR and had a set of comments around that setting. This was done in two places and only the last one was effective. This patch removes the spurious section to make it clearer what was happening and what TOASTER_DIR was actually getting set to. Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: remove cut and paste cruftbrian avery2017-02-131-1/+4
| | | | | | | | | | | There was a superfluous section in toaster start/stop script that was setting the TOASTER_DIR and had a set of comments around that setting. This was done in two places and only the last one was effective. This patch removes the spurious section to make it clearer what was happening and what TOASTER_DIR was actually getting set to. Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update version to 1.33.1Richard Purdie2017-02-071-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Bump version to 1.33.0 (develoment version)Richard Purdie2017-01-271-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: Add BB_LIMITEDDEPS supportRichard Purdie2017-01-191-0/+2
| | | | | | | | | When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited. We really need a way to know this from the metadata and this new variable provides this in worker context. This means existing code can stop having to guess. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: bin/toaster whitelist TOASTER_DIRbrian avery2017-01-171-0/+1
| | | | | | | | | | | | | Some of the tinfoil2 changes resulted in TOASTER_DIR being cleared by the memory resident bitbake server toaster starts up. This prevented toaster from being able to connect to its sqlite database. Adding TOASTER_DIR to the BB_ENV_EXTRAWHITE list stops the cooker from clearing out the TOASTER_DIR variable. [YOCTO #9252] Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: dry-run real tasks when BB_SETSCENE_ENFORCE is setPaul Eggleton2016-12-161-5/+7
| | | | | | | | | | | | | | | | | | | | | | For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used by the installation process for the extensible SDK), we don't actually need the whitelisted real tasks to execute - we just need to have them in the dependency tree so that we get all of the setscene tasks they depend on to run. Therefore we can actually dry-run those real tasks i.e. they won't be run (and thus we won't waste a significant amount of time doing so) and won't be stamped as having run either. We do already have a dry-run mode in BitBake (activated by the -n or --dry-run command line option), but it dry-runs the setscene tasks as well which we don't want here. Note that this has no effect on the checking we are doing with BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are scheduled to run - that's handled separately. Fixes [YOCTO #10369]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runqueue: enable setVariable command to affect task executionPaul Eggleton2016-12-141-2/+10
| | | | | | | | | | Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: Reference pip3 not pipRichard Purdie2016-12-081-1/+1
| | | | | | | | Now that we use python3, we should refer to pip3, not pip. [YOCTO #10774] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-worker: Further IO performance tweaksRichard Purdie2016-12-071-4/+9
| | | | | | | | | | | | Looking further at the CPU loads on systems running large numbers of tasks, the following things helps performance: * Loop on waitpid until there are no processes still waiting * Using select to wait for the cooker pipe to be writable before writing avoiding pointless 100% cpu usage * Only reading from worker pipes that select highlights are readable Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: runbuilds Write the pidfile in python rather than shell scriptMichael Wood2016-11-301-1/+1
| | | | | | | | | | | | Write the pid file out in the start up of this management command. This ensures this has happened instead of relying on the shell command having been run which may or may not be the case. This also makes it simpler for testing. Couple of clean ups of runbuilds as identified by pyflake Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: remove True option to getVarFlag callsJoshua Lock2016-11-301-1/+1
| | | | | | | | | | | | getVarFlag() now defaults to expanding by default, thus remove the True option from getVarFlag() calls with a regex search and replace. Search made with the following regex: getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: remove True option to getVar callsJoshua Lock2016-11-302-3/+3
| | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-worker: Handle cooker/worker IO deadlockingRichard Purdie2016-11-291-16/+36
| | | | | | | | | | | | | | | | | | | | | | I noiced builds where tasks seemed to be taking a surprisingly long time. When I looked at the output of top/pstree, these tasks were no longer running despite being listed in knotty. Some were in D/Z state waiting for their exit code to be collected, others were simply not present at all. strace showed communication problems between the worker and cooker, each was trying to write to the other and nearly deadlocking. Eventually, timeouts would allow them to echange 64kb of data but this was only happening every few seconds. Whilst this particularly affected builds on machines with large numbers of cores (and hence highly parallal task execution) and in cases where I had a lot of debug enabled, this situation is clearly bad in general. This patch introduces a thread to the worker which is used to write data back to cooker. This means that the deadlock can't occur and data flows much more freely and effectively. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-worker: print full traceback instead of message onlyMarkus Lehtonen2016-11-041-2/+3
| | | | | | | | | | | | Print full traceback instead of just the exception message in the child() function inside fork_off_task(). This makes debugging a lot easier as the function catches a generic "Exception" and the exception message alone might not give much information. [YOCTO #10393] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update version to 1.32.0Richard Purdie2016-10-111-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update version to 1.31.2Richard Purdie2016-10-051-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: stop modifying OEROOT in toaster scriptEd Bartosh2016-09-281-4/+6
| | | | | | | | | | | | | | Setting OEROOT in toaster script makes oe-init-build-env to break with error: bash: ../bitbake/bin/../../scripts/oe-buildenv-internal: No such file or directory This happens because OEROOT contains path relative to build directory. Renamed OEROOT to OE_ROOT and unset it after it's used. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: don't kill all runserver processesEd Bartosh2016-09-061-3/+0
| | | | | | | | | | | | | | | | Toaster script kills runserver process 2 ways: - sending signal to pid from .toastermain.pid. - sending signal to pids found by grepping ps output: ps fux | grep "python.*manage.py runserver" Second approach is redundant and harmfull as it kills all django development server running on the machine. [YOCTO #7973] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: remove handling of .toasterui.pidEd Bartosh2016-09-061-4/+0
| | | | | | | | | This file is not created anywhere, but handled in toaster script code. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: don't kill toaster on startEd Bartosh2016-09-061-8/+0
| | | | | | | | | | | | | | There is no point of trying to kill django development server when toaster starts because 'manage.py checksocket' command is already used in the script code to check if development server port is occupied. Even if Toaster is listening on another port, killing previous instance looks quite implicit and doesn't solve anything as there are other processes that might be still running. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-diffsigs/bitbake-layers: Ensure tinfoil is shut down correctlyPaul Eggleton2016-09-022-28/+31
| | | | | | | | We should always shut down tinfoil when we're finished with it, either by explicitly calling the shutdown() method or by using it as a context manager ("with ..."). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* bitbake-selftest: enable bitbake logging to stdoutMarkus Lehtonen2016-08-191-1/+19
| | | | | | | | | Now you get the bb logger output for failed tests. This helps debugging problems. Also, all stdout/stderr data for successful tests is silenced which makes for less cluttered console output. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-selftest: introduce BB_TMPDIR_NOCLEANMarkus Lehtonen2016-08-191-0/+1
| | | | | | | | Set this env variable to 'yes' to preserve temporary directories used by the fetcher tests. Useful for debugging tests. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-selftest: add help text for env variable(s)Markus Lehtonen2016-08-191-1/+13
| | | | | Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-selftest: utilize unittest.main betterMarkus Lehtonen2016-08-191-22/+8
| | | | | | | | | This simplifies the script, and, gives new features. It is now possible to run single test functions, for example. This is nice when writing new test cases. Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update version to 1.31.1Richard Purdie2016-08-181-1/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Initial multi-config supportRichard Purdie2016-08-171-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds the notion of supporting multiple configurations within a single build. To enable it, set a line in local.conf like: BBMULTICONFIG = "configA configB configC" This would tell bitbake that before it parses the base configuration, it should load conf/configA.conf and so on for each different configuration. These would contain lines like: MACHINE = "A" or other variables which can be set which can be built in the same build directory (or change TMPDIR not to conflict). One downside I've already discovered is that if we want to inherit this file right at the start of parsing, the only place you can put the configurations is in "cwd", since BBPATH isn't constructed until the layers are parsed and therefore using it as a preconf file isn't possible unless its located there. Execution of these targets takes the form "bitbake multiconfig:configA:core-image-minimal core-image-sato" so similar to our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND. Implementation wise, the implication is that instead of tasks being uniquely referenced with "recipename/fn:task" it now needs to be "configuration:recipename:task". We already started using "virtual" filenames for recipes when we implemented BBCLASSEXTEND and this patch adds a new prefix to these, "multiconfig:<configname>:" and hence avoid changes to a large part of the codebase thanks to this. databuilder has an internal array of data stores and uses the right one depending on the supplied virtual filename. That trick allows us to use the existing parsing code including the multithreading mostly unchanged as well as most of the cache code. For recipecache, we end up with a dict of these accessed by multiconfig (mc). taskdata and runqueue can only cope with one recipecache so for taskdata, we pass in each recipecache and have it compute the result and end up with an array of taskdatas. We can only have one runqueue so there extensive changes there. This initial implementation has some drawbacks: a) There are no inter-multi-configuration dependencies as yet b) There are no sstate optimisations. This means if the build uses the same object twice in say two different TMPDIRs, it will either load from an existing sstate cache at the start or build it twice. We can then in due course look at ways in which it would only build it once and then reuse it. This will likely need significant changes to the way sstate currently works to make that possible. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: Build datastores from databuilder objectRichard Purdie2016-08-171-10/+13
| | | | | | | | | | | | Rather than passing in a datastore to build on top of, use the data builder object in the cache and base the parsed recipe from this. This turns things into proper objects building from one another rather than messy mixes of static and class functions. This sets things up so we can support parsing and building multiple configurations. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: Remove old toaster config loading mechanismMichael Wood2016-08-111-15/+0
| | | | | | | | This has been replaced using django's inbuilt loaddata. Django command documented at: https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata Signed-off-by: Michael Wood <michael.g.wood@intel.com>
* toaster: checksettings call django's loaddata instead of custom commandMichael Wood2016-08-111-1/+2
| | | | | | | | | | | | | Call django's inbuilt loaddata command to load the appropriate fixtures. We also attempt to load a fixture called "custom" and fail silently if we don't have one. This is where initial customisations can be done to load particular settings or data into Toaster (for example layers or default values for variables) Make sure the value for TEMPLATECONF is available to checksettings so that we can have a go a working out which default data to load. Signed-off-by: Michael Wood <michael.g.wood@intel.com>
* toaster: --help now returns 0 instead of 1Randy Witt2016-08-111-0/+4
| | | | | | | | | If the user explicitly passes in "--help" then it should return 0. This is the convention follow by the typical application. This allows the user to check for options without triggering an error. Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eventreplay: rewrite the scriptEd Bartosh2016-07-081-125/+82
| | | | | | | | | | | Rewritten toaster-eventreplay to make code working as expected, more compact and readable. [YOCTO #9585] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eventreplay: reorganize importsEd Bartosh2016-07-081-11/+6
| | | | | | | | | | | | Cleaned up module imports: - Removed unused imports - Removed import of print_function - Removed duplicated imports - Splitted importing bb.lib to 2 lines Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eventreplay: replace MockConfigParameters with namedtupleEd Bartosh2016-07-081-12/+2
| | | | | | | | | | | | class MockConfigParameters has only one attribute and only __init__ method. Replacing it with namedtuple makes code less nested and more readable. [YOCTO #9585] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eventreplay: fix event loading codeEd Bartosh2016-07-081-1/+3
| | | | | | | | | | | | | | Event objects are represented by base64-encoded strings in the event file and can't be loaded by existing eventreplay code. Fixed the code of loading events from file by decoding base64 strings into the binary form and loading them with pickle.load. [YOCTO #9585] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eventprelay: implement setEventMask commandEd Bartosh2016-07-081-0/+6
| | | | | | | | | | | Stored event mask list as self.eventmask for future use. Fixed Exception: Command setEventMask not implemented. [YOCTO #9585] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* eventreplay: add MockConnection.getEventHandle methodEd Bartosh2016-07-081-0/+2
| | | | | | | | | | | Fixed AttributeError: 'MockConnection' object has no attribute 'getEventHandle' [YOCTO #9585] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-worker: don't reassign sys.stdoutEd Bartosh2016-07-081-2/+1
| | | | | | | | | | | | | | | | | | | | Worker needs input stream in binary mode as it reads binary content from it. Current code does it by detaching a buffer from sys.stdin and assigning it back to sys.stdin. Detached buffer is io.BufferedReader in binary mode. This operation is implicit as its purpose is not easily understandable from the code. Replacing it with fdopen(sys.stdin.fileno(), 'rb') should make the code more understandable. Assigning the buffer to sys.stdin is not needed as worker doesn't use sys.stdin. Moreover, it leads to difficult to debug issues down the stack. For example, devpyshell doesn't work without reopening sys.stdin in text mode. This is not needed anymore after this fix as sys.stdin is not changed in worker code and remains in text mode. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: bin Use python 3 for our django modules checkMichael Wood2016-06-141-1/+1
| | | | | | Explicitly use python3 so that the modules for python3 are checked. Signed-off-by: Michael Wood <michael.g.wood@intel.com>
* toaster: use python3 explicitlyEd Bartosh2016-06-011-3/+3
| | | | | | | | | | Explicitly used python3 as default python for oe builds will continue to be python2. [YOCTO #9584] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* goggle/image-writer: Drop since bitrotting and no longer usedRichard Purdie2016-06-011-127/+0
| | | | | | | | | The upgrade to python3 is the final nail in the coffin for image-writer and the goggle UI. Neither seem used or recieve patches and are based on old versions of GTK+ so drop them, and the remaining crumbs support pieces. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* image-writer/goggle: Disable pygtkcompat problemsRichard Purdie2016-06-011-1/+2
| | | | | | | | | Disable the problematic gtk usage for use with pygtkcompat. The following commit removes these tools/UIs entirely but we may as well leave this piece in the history in case anyone does want a starting point for reusing them. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Convert to python 3Richard Purdie2016-06-0110-43/+51
| | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: Remove DATABASE_URL being passed around as an environment varMichael Wood2016-05-201-2/+0
| | | | | | | | | | | | | | | | We don't need to pass the DATABASE_URL around and read it back if we setup the django framework in the correct way. We make the default sqlite database path a full path so that the database isn't being assumed to be in CWD. Also add some more useful comments on the database settings. This is preparation work to migrate the build tests and be able to trigger builds on differently configured databases. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Implement support for per-task exportsChristopher Larson2016-05-171-0/+10
| | | | | Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-layers: convert to plugin-basedChristopher Larson2016-05-171-1005/+60
| | | | | | | | This uses bb.utils.load_plugins, based on the plugin handling in recipetool and devtool in oe-core. Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: use print function in toaster scriptEd Bartosh2016-05-131-2/+2
| | | | | | | | | | Used print() function instead of print statement to make toaster script to work with both python 2 and python 3 [YOCTO #9584] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bin/bitbake-worker: Fix invalid bb.msg.fatal usageRichard Purdie2016-05-121-1/+2
| | | | | | | The logging domain specified to bb.msg.fatal was invalid. Replace with a logger.critical() call instead. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>