aboutsummaryrefslogtreecommitdiffstats
path: root/lib/toaster/bldcontrol
AgeCommit message (Collapse)Author
2018-08-20Toaster: Implement the project-specific feature and releated enhancements ↵David Reyna
and defects. Here is the primary driving enhancement: * Bug 12785 - Support Project Specific configuration for external tools (e.g. ISS, Eclipse) - Isolated project-specific configuration page (full Toaster context hidden) - Support for new project, reconfigure existing project, and import existing command line project - Ability to define variables (e.g. image recipe) and pass them back to external GUI - Ability to execute the cloning phase, so that external GUI receive a buildable project - Ability to call back to the external GUI when updates are completed and ready - Compatibility of above projects with the normal full Toaster interface - Ability to pass to a 'complete' or 'cancel' web page so that the external GUI can immediately stop that Toaster instance, and not leave dangling servers nor edit sessions open Here are the supporting enhancements, where at least the back end is implemented: * Bug 12821 - Make Toaster conf changes compatible with command line usage * Bug 12822 - Support importing user changes to conf files into Toaster * Bug 12823 - Support importing user build directories into Toaster * Bug 12824 - Scan imported layers for content so that they are immediately available * Bug 12825 - show layer clone item in progress bar Here are defects fixed: * Bug 12817 - builddelete.py requires explicit 'add_arguments' * Bug 12818 - Remove orphaned imported layers when project is deleted * Bug 12826 - fix imported layer management * Bug 12819 - build using selected bitbake env, not Toaster's env * Bug 12820 - Toaster randomizes the layer order in toaster_bblayers.conf [YOCTO #12785] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-28toaster: do not fail on optional 'custom.xml' fileDavid Reyna
Explicitly capture and ignore errors when trying to load the optional 'custom.xml' fixture file. [YOCTO #12554] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-09bitbake: toaster: don't use git fetch --allRobert Yang
The '--all'' fetches all remotes, including the ones in $HOME/.gitconfig, which causes the fetching very slow, so don't use "--all", git fetch should be enough. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18toaster: enable custom env support for shell callsDavid Reyna
Allow for custom environment additions for git cloning, for example for anspass support. [YOCTO #12193] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18toaster: unset BBPATH before starting bbserver for backwards compatibilityDavid Reyna
The latest build enviroment script is now setting BBPATH. This breaks building previous releases from a master-based Toaster, because with this newly inherited BBPATH value the 'bitbake.lock' file ends up in the Toaster build directory instead of the project's build directory. Toaster should always clear BBPATH so that the bbserver's environment is clean (enough). [YOCTO #12363] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-10-06toaster: build missing toaster.conf settingsDavid Reyna
The bitbake server changed such that the Toaster custom settings from 'toaster.conf' and 'toaster-bblayers.conf' that were set when the '--server-only' mode was started were lost when the subsequent build request happened, resulting in builds missing all custom changes. This patch asserts those environment settings in both server calls. [YOCTO #12194] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-11toaster: handle early exceptionsDavid Reyna
Stop the pending build and report to the user if there is an internal exception due to a git error, a bitbake server error, or if the server-only mode halts due to for example a user syntax error in a layer or recipe. These exceptions were not caught because they occure before the normal toastergui processing was started. [YOCTO #12056] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-04toaster: set default pokydirname if no external layersOlaf Mandel
If no external layers are defined, pokydirname is not set. Rectify this by taking the 'be.sourcedir' as the pokydirname. [YOCTO #12015] Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-04toaster: debug message for lists layers missing separatorsOlaf Mandel
One of the debug messages during build contains a list of all layers but without spaces or other separators between them. Use pformat instead. [YOCTO #12014] Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com> Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-22toaster: support custom Layer Index URL and fixture overrideDavid Reyna
Toaster needs to allow the custom fixture file to specific an alternate Layer Index URL for users that host their own Layer Index Server via the 'CUSTOM_LAYERINDEX_SERVER' in 'custom.xml'. Toaster also needs to allow the ability to completely override the default fixture files, otherwise that content can leak into the custom environment (by default the custom fixture is an overlay that cannot remove existing values from the default fixture) via the 'CUSTOM_XML_ONLY' value in 'setting.xml'. [YOCTO #11938] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30toaster: move to new bitbake xmlrpc defaultDavid Reyna
The bitbake option "-t SERVERTYPE" was deprecated and can be removed since the desired XMLRPC listener now the default. The bitbake server port cannot be "-1" anymore and must be explicitly passed. There is a race condition for the bblock file to not only be created but to actually be populated, so a delay test loop is required (usually only one pass). The 'xmlrpcclient' is now the class that allows Toaster to for example kill builds. The events for populating the recipe parsing now either show only the final result or are skipped entiredly, so the progress calculator needs to be changed to not block on the parsing phase. [YOCTO #11851] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-27toaster: git clone progress barDavid Reyna
If a project has a lot of additional layers, the build may appear to hang while those layers are checked out. This patch adds a clone progress bar that is visible before the parsing progress appears. [YOCTO #9916] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-27toaster: address Django-1.10 API deprecationsDavid Reyna
There are four main API deprecations in Django-1.10: (a) String view arguments to url() must be replaced by the explicit class reference (b) New TEMPLATES stucture in settings.py consolidates TEMPLATE_DIRS, TEMPLATE_CONTEXT_PROCESSORS, TEMPLATE_LOADERS, TEMPLATE_STRING_IF_INVALID, and TEMPLATE_DEBUG (c) patterns() wrapper in url() is removed, with urlpatterns now a simple list (d) NoArgsCommand in commands() must be replace by BaseCommand, and handle_noargs() changed to handle() Also, the Django version checker must be updated to accept two digit sub-version numbers (e.g. "1.8" < "1.10") [YOCTO #11684] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: bldcontrol Move CustomImageRecipe file creation into own functionMichael Wood
Move the custom image file creation (i.e. create the layer file structure, conf and recipe file) into it's own function and remove the creation of the BRLayer as this is done at schedule_build just like all the other layers. Fix a bug where the toaster-custom-images layer was always being appened to the layer list if the directory exists. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: buildinfohelper toaster-custom-images layerMichael Wood
This fixes the unidentified layers issue by making the toaster-custom-images layer a local layer. By doing this we also fix the git assumptions made for the local layers which stop recipes and other meta data being associated with them. This also removed some of the special casing previously needed when we didn't have the concept of a local (non git) layer. Also rename created flag var to a have a different var for each returned value so that the same value isn't used multiple times. [YOCTO #10220] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: localhostbecontroller write toaster layers for project to ↵Sujith H
toaster-bblayers.conf Instead of updating conf/bblayers, here we update toaster-bblayers.conf file. So extra effort to update bblayers.conf can be removed safely. Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: localhostbecontroller accept custom init script for buildSujith H
When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster setting, it would be nice to use it. Else toaster can use oe-init script. This gives an oppurtunity to use customized build init scritps. Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: runbuilds Write the pidfile in python rather than shell scriptMichael Wood
Write the pid file out in the start up of this management command. This ensures this has happened instead of relying on the shell command having been run which may or may not be the case. This also makes it simpler for testing. Couple of clean ups of runbuilds as identified by pyflake Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04toaster: Delete useless bldcontrol/testMichael Wood
It doesn't work nor does it test anything useful Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07toaster: fix cloning of git+ssh repositoriesEd Bartosh
Replaced '+' -> '_' to avoid having '+' in folder name. Thanks Stephan Dünner for this fix. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06toaster: checksettings Remove confusing startup messagesMichael Wood
These "validation" messages are shown regardless as to whether the settings are being correctly set or not. For the time being remove them. [YOCTO #9097] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20runbuilds: code cleanup - remove unused importsEd Bartosh
Fixed pylint warning: W0611(unused-import): Unused import Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20runbuilds: code cleanup - whitespaces, long linesEd Bartosh
Fixed following pylint warnings: C0330(bad-continuation): Wrong hanging indentation before block. C0326(bad-whitespace): No space allowed around keyword argument assignment C0326(bad-whitespace): Exactly one space required before assignment C0301(line-too-long): Line too long Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20runbuilds: process builds on startEd Bartosh
If Toaster is stopped incorrectly there could be some build requests and builds in incorrect state left from the previous run. Running main processing function on start should take care of those. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20runbuilds: process builds on SIGUSR1Ed Bartosh
Run main processing function 'runbuild' only if SIGUSR1 is received. This signal is sent by Toaster when build status is changed (either started, cancelled or finished). This should stop continuous database polling as run_builds function will be called only when needed, i.e. after build status is changed. [YOCTO #8918] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20toaster: fix checking of repository urlEd Bartosh
Toaster checks gir repository url is the same as locally cloned before checking it out to existing local clone. This check can be skipped if commit is 'HEAD' as in this case repository is not hard reset to commit, so the local clone won't be changed. [YOCTO #10163] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16Add missing file from 'toaster: Allow git information to be null for BRLayer'Richard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-02toaster: localhostbecontroller Remove git assumptionMichael Wood
We don't need to force everyone to use git for the method in which openembedded-core is downloaded. For instance it could have been downloaded and extracted as a tarball. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-02toaster: Allow git information to be null for BRLayerMichael Wood
We no longer only deal with layers that have their source in a gir repository, we also allow for local directories too so update the BRLayer model to reflect this. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-02toaster: Fix oe-core fixtureMichael Wood
Due to a copy paste error we managed to get some of the wrong information in the oe fixture that provides a suggested default settings for Toaster. This meant it tested correctly when it shouldn't have. Fix: - The use of local bitbake - An incorrect call to realpath which didn't include its parent module. - The field used for the local_dir of an existing openembedded-core Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11toaster: bldcontrol model BRLayer Add corresponding local_source_dirMichael Wood
Sync the BRLayer object with the new field added to the Layer object. The BRLayer (BuildRequest Layers) are snapshots of the layers in the project at build time and therefore need to mirror the required fields of the layer object. Signed-off-by: Michael Wood <michael.g.wood@intel.com>
2016-08-11toaster: collect details for local dir imported from uiSujith H
Collect the dir path imported from UI and make sure that its getting added to bblayers.conf. This patch exactly does the same job. Any layer which is imported locally need not be cloned again to _toaster_clones dir. [YOCO #9911] Signed-off-by: Sujith H <sujith.h@gmail.com>
2016-08-11toaster: move most recent builds templating to clientElliot Smith
The most recent builds area of the all builds and project builds table needs to update as a build progresses. It also needs additional functionality to show other states (e.g. recipe parsing, queued) which again needs to update on the client side. Rather than add to the existing mix of server-side templating with client-side DOM updating, translate all of the server-side templates to client-side ones (jsrender), and add logic which updates the most recent builds area as the state of a build changes. Add a JSON API for mostrecentbuilds, which returns the state of all "recent" builds. Fetch this via Ajax from the build dashboard (rather than fetching the ad hoc API as in the previous version). Then, as new states for builds are fetched via Ajax, determine whether the build state has changed completely, or whether the progress has just updated. If the state completely changed, re-render the template on the client side for that build. If only the progress changed, just update the progress bar. (NB this fixes the task progress bar so it works for the project builds and all builds pages.) In cases where the builds table needs to update as the result of a build finishing, reload the whole page. This work highlighted a variety of other issues, such as build requests not being able to change state as necessary. This was one part of the cause of the "cancelling build..." state being fragile and disappearing entirely when the page refreshed. The cancelling state now persists between page reloads, as the logic for determining whether a build is cancelling is now on the Build object itself. Note that jsrender is redistributed as part of Toaster, so a note was added to LICENSE to that effect. [YOCTO #9631] Signed-off-by: Elliot Smith <elliot.smith@intel.com>
2016-08-11toaster: Remove old toaster config loading mechanismMichael Wood
This has been replaced using django's inbuilt loaddata. Django command documented at: https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata Signed-off-by: Michael Wood <michael.g.wood@intel.com>
2016-08-11toaster: checksettings call django's loaddata instead of custom commandMichael Wood
Call django's inbuilt loaddata command to load the appropriate fixtures. We also attempt to load a fixture called "custom" and fail silently if we don't have one. This is where initial customisations can be done to load particular settings or data into Toaster (for example layers or default values for variables) Make sure the value for TEMPLATECONF is available to checksettings so that we can have a go a working out which default data to load. Signed-off-by: Michael Wood <michael.g.wood@intel.com>
2016-07-25toaster: loadconf Partially add back some of the layerSource parsingMichael Wood
Partially add back a revised version of the layersource handling so that we can continue to support the old toasterconf.json and it's setup of the local project. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-25toaster: orm Remove the layerindex specific up_branch fieldsMichael Wood
We don't need to keep track of layerindex data in our database. And using branch==release is very confusing in the schema. Instead use the existing Release definition to keep track of which release a layer_version is for. Remove the Branch model and all references to it. Create a migration path to convert from up_branches to their corresponding releases. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-25toaster: tests: Remove references to LayerSource modelMichael Wood
Remove and replace layersource model references in the tests and test data. Remove the orm/test as this only tested LayerSource interactions which have now been removed. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-25toaster: Replace references to LayerSource modelsMichael Wood
Replace references to the now deprecated layersource models across Toaster with the new enums for layer source types. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-25toaster: loadconf remove Loading LayerSourcesMichael Wood
We don't need to configure layer sources in the initial configuration as this information is provided by the models. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-14toaster: fix wrong usage of print_exc and format_excEd Bartosh
First parameter of traceback.print_exc and traceback.format_exc APIs is a 'limit' - a number of stracktraces to print. Passing exception object to print_exc or format_exc is incorrect, but it works in Python 2 and causes printing only one line of traceback. In Python 3 comparison of integer and exception object throws exception: TypeError: unorderable types: int() < <Exception type>() As these APIs are usually used in except block of handling another exception this can cause hard to find and debug bugs. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com>
2016-06-14toaster: fix migrationsEd Bartosh
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
2016-06-01toaster: use knotty when working with bitbake serverEd Bartosh
Using empty BITBAKE_UI environment variable causes bitbake server to fail with the error: FATAL: Unable to import extension module "" from bb.ui Valid extension modules: knotty or toasterui\n' Used BITBAKE_UI="knotty" when starting and stoping bitbake server to solve above issue. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01toaster: moved import bb.server.xmlrpcEd Bartosh
Moved import xmlrpc module to the place where it's used to avoid toaster crashes when importing bitbake code. NOTE: This patch is made to be able to partly test toaster with bitbake from master. It can be removed as soon as bb.server.xmlrpc is ported to python 3. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01toaster: use decode('utf-8') for binary dataEd Bartosh
Decoded binary data to utf-8 to fix the following error when runnign on python 3: TypeError: Type str doesn't support the buffer API [YOCTO #9584] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01toaster: fix local importsEd Bartosh
Replaced local imports with absolute imports. Used .module for local imports. This should make the code to work on python 2 and python 3. [YOCTO #9584] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01toaster: use force_text instead of force_bytesEd Bartosh
Usage of force_bytes in BuildRequest.__str__ method caused python 3 to throw "__str__ returned non-string (type bytes)" error. Replaced force_bytes with force_text to make the code working on both python 2 and python 3. [YOCTO #9584] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-20toaster: runbuilds move the execution sequence out of the poll loopMichael Wood
Move the execution sequence for a build out of the polling loop and into it's own fuction. This means that we can call the function on it's own if we just want to trigger one build rather than infinite polling. This is something needed for the build tests. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-09bin, toaster: Fix print and exception syntaxRichard Purdie
This updates the print "" syntax to print() and fixes some exception handling syntax such that its compatible with python v2 and v3. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-09bitbake: Update logger.warn() -> logger.warning()Richard Purdie
python deprecated logger.warn() in favour of logger.warning(). This is only used in bitbake code so we may as well just translate everything to avoid warnings under python 3. Its safe for python 2.7. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-18 05:42:30 +0000