summaryrefslogtreecommitdiffstats
path: root/bin
AgeCommit message (Collapse)Author
2017-09-06bitbake-setup: Add WIP implementation of bitbake-setup [DO NOT MERGE]mgh/bitbake-setupMark Hatle
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
2017-09-06bitbake-selftest: Add layers module testsMark Hatle
The tests cover both the layers/layerindex and the layers/manager. Both sets of tests included network connection components. They honor the BB_SKIP_NETTESTS='yes' if networking is not available, to prevent test failures. Tests Implemented: layers/layerindex: - Branch, LayerItem, LayerBranch, LayerDependency, Recipe, Machine and Distro objects - LayerIndex setup using the layers.openembedded.org restapi - LayerIndex storing and retrieving from a file - LayerIndex verify dependency resolution ordering layers/manager: - manager basic functionality - manager and fetcher plugin setup/fetch/unpack of a remote repository - fetcher plugin generating a list of downloaded layers Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
2017-08-22Toaster: custom start and stop actionsDavid Reyna
Toaster needs the ability to allow custom extensions to execute when Toaster is started and stopped. Toaster will look for a custom extension script in the fixtures directory and execute any applicable hooks. [YOCTO #11938] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27bitbake-diffsigs: fix regression after recent server changesPaul Eggleton
We were bridging the gap between the server and UI here by calling a bb.siggen.find_siginfo, a function defined and set on that module from the metadata. This worked from the UI side before but since the recent server changes is no longer accessible. Create a new command so this can execute on the server side and return the result by way of a new event. (We're still running compare_sigfiles() on the signature generator but that isn't quite the same thing and does still work.) Fixes [YOCTO #11844]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21Update to version 1.35.0 (development version with server rework changes)Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21tinfoil: add a parse_recipes() functionPaul Eggleton
Python style recommends underscore based naming rather than camelCase, and thus the former has been used for most of tinfoil's functions. Add an underscored version of parseRecipes() for consistency and change the one place we call it to use the new version. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-27toaster: address Django-1.10 API deprecationsDavid Reyna
There are four main API deprecations in Django-1.10: (a) String view arguments to url() must be replaced by the explicit class reference (b) New TEMPLATES stucture in settings.py consolidates TEMPLATE_DIRS, TEMPLATE_CONTEXT_PROCESSORS, TEMPLATE_LOADERS, TEMPLATE_STRING_IF_INVALID, and TEMPLATE_DEBUG (c) patterns() wrapper in url() is removed, with urlpatterns now a simple list (d) NoArgsCommand in commands() must be replace by BaseCommand, and handle_noargs() changed to handle() Also, the Django version checker must be updated to accept two digit sub-version numbers (e.g. "1.8" < "1.10") [YOCTO #11684] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23bitbake-selftest: add bb.tests.event to bitbake-selftestJair Gonzalez
Tests for bitbake event module were created on bb.tests.event. This change is to include them on the default test list in bitbake-selftest script. [YOCTO #10368] Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-16toaster: noweb should init databaseDavid Reyna
When the 'noweb' option for Toaster is used, perform the database check/create if the Toaster database does not yet exist. This will allow Toaster to not fail if the first use is with 'noweb'. This avoids potentially clashing database updates if there are multiple overlaping 'noweb' sessions (for example with a CI system). If the user wished to update the database, they can either use the explicit "lsupdate" command or (re)start a web hosted Toaster session (which is gated by the webserver's PID). [YOCTO #11378] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05bitbake-layers: check layer dependencies before addingChang Rebecca Swee Fun
In the original implementation, "bitbake-layers add-layers <layer>" succeeded without error checking. This will further introduce failures in recipe parsing only when "bitbake" command is executed. Adding a meta layer without its dependency layer(s) should failed and exit the process gracefully. Added extra argument "-F" to force add a layer without checking layer dependency. [YOCTO #10913] Signed-off-by: Phoong Stanley Cheong Kwan <stanley.cheong.kwan.phoong@intel.com> Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-26git-make-shallow: add script to make a git repo shallowChristopher Larson
This script will be used by the git fetcher to create shallow mirror tarballs. usage: git-make-shallow [-h] [--ref REF] [--shrink] REVISION [REVISION ...] Remove the history of the specified revisions, then optionally filter the available refs to those specified. positional arguments: REVISION a git revision/commit optional arguments: -h, --help show this help message and exit --ref REF, -r REF remove all but the specified refs (cumulative) --shrink, -s shrink the git repository by repacking and pruning While git does provide the ability to clone at a specific depth, and fetch all remote refs at a particular depth, the depth is across all branches/tags, and doesn't provide the flexibility we need, hence this script. Refs (branches+tags) can be filtered, as the process of history removal scales up rapidly with the number of refs. Even the existing `git fetch --depth=` is extremely slow on an upstream kernel repository with all the branches and tags kept. This uses the same underlying mechanism to implement the history removal which git itself uses (.git/shallow), and the results, when configured similarly, are in line with the results git itself produces with `fetch --depth`. Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19bitbake: Update version to 1.34.0 for stable releaseRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12bitbake-layers: add signal hander to avoid exceptionZhixiong Chi
Fixed: bitbake-layers show-recipes | less press "q" to exit There will be a Broken pipe error output as follows: "BrokenPipeError: [Errno 32] Broken pipe" Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10bitbake-diffsigs: fix traceback with no argumentsPaul Eggleton
In the move over to argparse we've made the two signature file arguments optional and thus if -t is not in use we need to explicitly check if at least one signature file has been specified - and if not, show an error and the command-line help. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-07bitbake-diffsigs: colourise outputPaul Eggleton
If the output is a TTY, add colour to the output in order to make it easier to read. At the moment this is fairly basic, just add colour to the "titles" of each change and to the diff output. I tried to introduce this without changing the code too much - rather than moving everything over to the new python formatting style, I've introduced a color_format() function which takes care of the colour formatting, either accepting additional format arguments or alternatively leaving the caller to use the old-style formatting (%) to insert values. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-07bitbake-diffsigs: add an option to find and compare specific signaturesPaul Eggleton
With the -t option which recurses to find the ultimate cause of a signature change, it was hardcoded to take the last two executions of the specified task. On the other hand, if you have two specific task hashes (say from bitbake output, or some other tool) then you'll want to pick those, so provide an option to specify those as well. (Note, the new -s option needs to be specified alongside -t rather than instead of it.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-07bitbake-diffsigs: change to use argparsePaul Eggleton
Argparse is a bit easier to deal with than optparse, and since we're about to add some options, migrate this script over. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-07lib/bb/siggen: show a diff when dumping changes to multi-line valuesPaul Eggleton
When dumping changes to signatures e.g. output of bitbake -s printdiff, if for example a function has changed, it's much more readable to see a unified diff of the changes rather than just printing the old function followed by the new function, so use difflib to do that. Note: I elected to keep to one item in the returned list per change, rather than one line per line of output, so that the caller can still look at changes individually if needed. Thus I've added some handling to bitbake-diffsigs to split the change into lines so that each line is displayed indented. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-07bitbake-diffsigs: drop naive logic for removing duplicate filesPaul Eggleton
This logic doesn't work in practice, certainly not with current versions where sigdata files are preserved in the stamps directory and therefore there will often be multiple sigdata files - you can now easily get files for the same signature from sstate and the stamps directory with the result that bitbake-diffsigs reports nothing has changed. Instead, let's change the find_siginfo function in OE-Core to simply not return duplicates so we don't have to filter them out here. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-07bitbake-diffsigs: properly report which signature is missingPaul Eggleton
If just one of the two signatures we want to compare aren't available, report that one rather than misleadingly claiming both are missing. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-07bitbake-diffsigs: fix -t picking wrong files to comparePaul Eggleton
We weren't picking the right files to compare here - according to the order in which the list is sorted (by mtime), we need to be taking the last two items and not the first two. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-06Bump version to 1.33.4Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-30bitbake: Bump version to 1.33.3Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-27lib/bb/msg: introduce logger_create() functionPaul Eggleton
We use this code to set up a logger with colour in a number of different places, so create one function that does this and make some of bitbake's utility scripts use it. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22bitbake-diffsigs: Add debug supportPeter Kjellerstedt
Currently shows the name of the signature files that were found when --task is used. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22bitbake-dumpsig: Add debug supportPeter Kjellerstedt
Currently shows the name of the signature file that was found when --task is used. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22bitbake-dumpsig: Allow recipe and task to be specifiedPeter Kjellerstedt
Use --task <recipe> <task> to dump the signature info for a given recipe and task. This is similar to the --task option of bitbake-diffsigs. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-24bitbake: Update version to 1.33.2Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19bitbake-layers: Handle exception raised bytinfoil.prepare()Aníbal Limón
The tinfoil.prepare method can raise Exceptions when is parsing initial data so add this call inside try, finally to avoid get blocked for don't call tinfoil.shutdown(). The tinfoil_init function was remove because isn't make sense now since tinfoil.prepare() needs to be inside try, finally closures. Example of raised exception and gets blocked: $ bitbake-layers add-layer ~/repos/meta-openembedded/meta-python/ Traceback (most recent call last): File "/home/alimon/repos/poky/bitbake/bin/bitbake-layers", line 83, in main tinfoil.prepare(True) ... File "/home/alimon/repos/poky/bitbake/lib/bb/tinfoil.py", line 268, in run_command raise TinfoilCommandFailed(result[1]) bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last): File "/home/alimon/repos/poky/bitbake/lib/bb/command.py", line 81, in runCommand result = command_method(self, commandline) ... File "/home/alimon/repos/poky/bitbake/lib/bb/cooker.py", line 1314, in handleCollections raise CollectionError("Errors during parsing layer configuration") bb.cooker.CollectionError: Errors during parsing layer configuration Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-13toaster: remove cut and paste cruftbrian avery
There was a superfluous section in toaster start/stop script that was setting the TOASTER_DIR and had a set of comments around that setting. This was done in two places and only the last one was effective. This patch removes the spurious section to make it clearer what was happening and what TOASTER_DIR was actually getting set to. Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-13toaster: remove cut and paste cruftbrian avery
There was a superfluous section in toaster start/stop script that was setting the TOASTER_DIR and had a set of comments around that setting. This was done in two places and only the last one was effective. This patch removes the spurious section to make it clearer what was happening and what TOASTER_DIR was actually getting set to. Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07bitbake: Update version to 1.33.1Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-27bitbake: Bump version to 1.33.0 (develoment version)Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19cooker: Add BB_LIMITEDDEPS supportRichard Purdie
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited. We really need a way to know this from the metadata and this new variable provides this in worker context. This means existing code can stop having to guess. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-17toaster: bin/toaster whitelist TOASTER_DIRbrian avery
Some of the tinfoil2 changes resulted in TOASTER_DIR being cleared by the memory resident bitbake server toaster starts up. This prevented toaster from being able to connect to its sqlite database. Adding TOASTER_DIR to the BB_ENV_EXTRAWHITE list stops the cooker from clearing out the TOASTER_DIR variable. [YOCTO #9252] Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16runqueue: dry-run real tasks when BB_SETSCENE_ENFORCE is setPaul Eggleton
For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used by the installation process for the extensible SDK), we don't actually need the whitelisted real tasks to execute - we just need to have them in the dependency tree so that we get all of the setscene tasks they depend on to run. Therefore we can actually dry-run those real tasks i.e. they won't be run (and thus we won't waste a significant amount of time doing so) and won't be stamped as having run either. We do already have a dry-run mode in BitBake (activated by the -n or --dry-run command line option), but it dry-runs the setscene tasks as well which we don't want here. Note that this has no effect on the checking we are doing with BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are scheduled to run - that's handled separately. Fixes [YOCTO #10369]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14runqueue: enable setVariable command to affect task executionPaul Eggleton
Allow the client to set variables with the setVariable command and have those changes take effect when running tasks. This is accomplished by collecting changes made by setVariable separately and pass these to the worker so it can be applied on top of the datastore it creates. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08toaster: Reference pip3 not pipRichard Purdie
Now that we use python3, we should refer to pip3, not pip. [YOCTO #10774] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake-worker: Further IO performance tweaksRichard Purdie
Looking further at the CPU loads on systems running large numbers of tasks, the following things helps performance: * Loop on waitpid until there are no processes still waiting * Using select to wait for the cooker pipe to be writable before writing avoiding pointless 100% cpu usage * Only reading from worker pipes that select highlights are readable Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: runbuilds Write the pidfile in python rather than shell scriptMichael Wood
Write the pid file out in the start up of this management command. This ensures this has happened instead of relying on the shell command having been run which may or may not be the case. This also makes it simpler for testing. Couple of clean ups of runbuilds as identified by pyflake Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: remove True option to getVarFlag callsJoshua Lock
getVarFlag() now defaults to expanding by default, thus remove the True option from getVarFlag() calls with a regex search and replace. Search made with the following regex: getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-29bitbake-worker: Handle cooker/worker IO deadlockingRichard Purdie
I noiced builds where tasks seemed to be taking a surprisingly long time. When I looked at the output of top/pstree, these tasks were no longer running despite being listed in knotty. Some were in D/Z state waiting for their exit code to be collected, others were simply not present at all. strace showed communication problems between the worker and cooker, each was trying to write to the other and nearly deadlocking. Eventually, timeouts would allow them to echange 64kb of data but this was only happening every few seconds. Whilst this particularly affected builds on machines with large numbers of cores (and hence highly parallal task execution) and in cases where I had a lot of debug enabled, this situation is clearly bad in general. This patch introduces a thread to the worker which is used to write data back to cooker. This means that the deadlock can't occur and data flows much more freely and effectively. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04bitbake-worker: print full traceback instead of message onlyMarkus Lehtonen
Print full traceback instead of just the exception message in the child() function inside fork_off_task(). This makes debugging a lot easier as the function catches a generic "Exception" and the exception message alone might not give much information. [YOCTO #10393] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-11bitbake: Update version to 1.32.0Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05bitbake: Update version to 1.31.2Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28toaster: stop modifying OEROOT in toaster scriptEd Bartosh
Setting OEROOT in toaster script makes oe-init-build-env to break with error: bash: ../bitbake/bin/../../scripts/oe-buildenv-internal: No such file or directory This happens because OEROOT contains path relative to build directory. Renamed OEROOT to OE_ROOT and unset it after it's used. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06toaster: don't kill all runserver processesEd Bartosh
Toaster script kills runserver process 2 ways: - sending signal to pid from .toastermain.pid. - sending signal to pids found by grepping ps output: ps fux | grep "python.*manage.py runserver" Second approach is redundant and harmfull as it kills all django development server running on the machine. [YOCTO #7973] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06toaster: remove handling of .toasterui.pidEd Bartosh
This file is not created anywhere, but handled in toaster script code. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06toaster: don't kill toaster on startEd Bartosh
There is no point of trying to kill django development server when toaster starts because 'manage.py checksocket' command is already used in the script code to check if development server port is occupied. Even if Toaster is listening on another port, killing previous instance looks quite implicit and doesn't solve anything as there are other processes that might be still running. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>