Commit message (Collapse)AuthorAgeFilesLines
* cooker: Ensure parseConfiguration clears parsecache_validRichard Purdie2017-09-011-0/+2
| | | | | | | | | | BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers was failing and highlighted that since parseConfiguation clears data structures, it needs to also clear parsecache_valid as it no longer contains correct data. Signed-off-by: Richard Purdie <>
* cookerdata/taskdata/runqueue: Drop remaining tryaltconfigs code and ↵Richard Purdie2017-08-314-14/+3
| | | | | | | | | | | commandline option I can't actually see how this was working, nothing connected the commandline option to the data in TaskData(). Drop the remaining pieces of this option, it was a relic from a decade ago and we want deterministic builds, not random tries until something might work. Signed-off-by: Richard Purdie <>
* siggen: Add reset to SignatureGeneratorCaio Marcelo de Oliveira Filho2017-08-311-0/+3
| | | | | | | | | Fix failure after commit "cooker/siggen: Reset siggen when reparsing" (e4c6ca9440f63761560b49bbe12654441f54687e) when executing without specifying a BB_SIGNATURE_HANDLER. Signed-off-by: Caio Marcelo de Oliveira Filho <> Signed-off-by: Richard Purdie <>
* cooker: Ensure buildFile doesn't have lasting side effectsRichard Purdie2017-08-311-0/+5
| | | | | | | | | | | | | | | | | | | | | | | BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_build_image fails, the reason is that internally, the limited_deps flag is set in the bitbake server and then never cleared. This causes the sysroots to be setup incorrectly (as per the limited dependency case) and builds break. There is also potential for corruption of recipecaches. Add shutdown/cleanup code to ensure these effects don't 'stick'. This bug is particularly nasty as you can destroy TMPDIR with large sysroots in build work directories which are prone to break. Also ensure mtime cache is cleared (to match buildTargets) and that no lasting changes are made to siggen either which ensures: BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_upgrade_git devtool.DevtoolTests.test_devtool_virtual_kernel_modify works. Signed-off-by: Richard Purdie <>
* fetch2/npm: add noverify parameter to skip lockdown/shrinkwrapPaul Eggleton2017-08-311-20/+21
| | | | | | | | | | | | | | When fetching source for the first time within scripts such as OpenEmbedded's recipetool, we don't want to be showing warnings about NPM_SHRINKWRAP or NPM_LOCKDOWN not being set since there's no way we could have set them in advance. Previously we were using ud.ignore_checksums to suppress these but since we are now using a more standard task-based path to fetch the source, we need to disable these through the metadata. Look for a "noverify" parameter set on the npm URL and skip the checks if it is set to "1". Signed-off-by: Paul Eggleton <> Signed-off-by: Richard Purdie <>
* cooker: ensure we can run buildFileInternal() after cache is populatedPaul Eggleton2017-08-311-4/+4
| | | | | | | | | | | | | | | | | | | | | | | If you run some other operations that result in the cache being populated, and then call buildFileInternal(), then you can end up in a situation where the cache already contains information about the recipe. For example in OE this can now happen when you use devtool upgrade. Normally this doesn't cause any problems, unless you have a non-absolute path in BBLAYERS - in buildFileInternal() we are calling matchfile() which will convert the filename to absolute, but later when taskdata goes to find the providers of the recipe it finds the non-absolute path, sets up the task information using this and then the runqueue can't find any tasks matching the absolute path. To fix this, back out the optimisation I did earlier in bitbake rev ba53e067a2d448dd63b4ca252557ce98aa8e6321 to avoid calling parseConfiguration() again, which is unfortunate but does result in the cached information being that causes the problem being cleared out. This fixes "Task do_unpack does not exist for target ..." running devtool upgrade within intel-iot-refkit. Signed-off-by: Paul Eggleton <> Signed-off-by: Richard Purdie <>
* tinfoil: ensure variable history tracking works when parsing a recipePaul Eggleton2017-08-311-11/+18
| | | | | | | | | | | | | | | | If you set tracking=True when creating the tinfoil object, that ensures history is collected for the main datastore, but at the end of parsing the configuration, history tracking gets turned off to save time with the result that we don't collect history for any recipes we parse. Enable tracking when we parse a recipe (and disable it afterwards if we enabled it) in order to fix this. This fixes functionality in OE's devtool that relies upon variable history (such as devtool upgrade updating PV when it's set within a recipe). Signed-off-by: Paul Eggleton <> Signed-off-by: Richard Purdie <>
* tinfoil: ensure log lines get printed when tasks failPaul Eggleton2017-08-312-23/+34
| | | | | | | | | | | | If a task fails during build_targets(), we need to print out the log lines as knotty does or the user will be missing information about the failure. (This should get some deeper refactoring, but now isn't the time for that.) Signed-off-by: Paul Eggleton <> Signed-off-by: Richard Purdie <>
* tinfoil: fix log message doubling when config_only=FalsePaul Eggleton2017-08-311-0/+11
| | | | | | | | | | | | With config_only=False we launch the UI and it sets up a logger, whereas when config_only=True we don't, with the result that with True we are seeing log messages from both our logger and the one set up by the UI. Suppress our loggers with config_only=True to avoid this. Fixes [YOCTO #11275] (again). Signed-off-by: Paul Eggleton <> Signed-off-by: Richard Purdie <>
* prserv/serv: Gracefully handle the PR server exiting quicklyRichard Purdie2017-08-311-1/+5
| | | | | | | If the server exits quickly its PID may no longer exist. Handle this gracefully. Signed-off-by: Richard Purdie <>
* prserv/serv: Rename self.quit -> self.quitflagRichard Purdie2017-08-311-6/+6
| | | | | | | self has a quit function and a variable. Separate this to two different things as the current setup is prone to breakage. Signed-off-by: Richard Purdie <>
* prserv/serv: Send sentinel to stop handler threadRichard Purdie2017-08-311-1/+4
| | | | | | | Shutdown from SIGTERM currently has to wait for the handler thread to timeout. Add a sentinel value which triggers it to loop and allows for a quick exit. Signed-off-by: Richard Purdie <>
* ui/knotty: Send updateConfig earlyRichard Purdie2017-08-311-1/+3
| | | | | | | | | | | | | | | | If for example you run: bitbake -r rm bitbake -e bitbake will crash with an error about not being able to find This is because it tries to reparse the base config for the early getVariable requests before it sees the updated missing -r option. Send the updateConfig command earlier to avoid this. Signed-off-by: Richard Purdie <>
* cookerdata: Avoid tracebacks from early reset() callsRichard Purdie2017-08-311-0/+3
| | | | | | | cooker.reset() can be called before we've actually setup the datastore. Gracefully handle this case instead of the current traceback+exit. Signed-off-by: Richard Purdie <>
* cooker: Allow changes in PRSERV_HOST to be responded toRichard Purdie2017-08-311-8/+11
| | | | | | | | | | | | When we reload the configuration, PRSERV_HOST can change. Therefore restart the PR Server depending on the new configuration at reparse. Note that the server has to be started from the right process, it can't be in the UI which shuts down as that shutdown triggers a shutdown of its children and the PR Server shuts down too. This is why we need pre_serve() which ensures its executed in the right context. Signed-off-by: Richard Purdie <>
* prserv/serv: Shut down any existing server before restartingRichard Purdie2017-08-311-0/+3
| | | | | | | This allows for cleaner code in cooker as any existing server is dealt with before a new one is started. Signed-off-by: Richard Purdie <>
* cooker: Change to consistent prefile/postfile handlingRichard Purdie2017-08-313-21/+10
| | | | | | | | | | | | Currently the original prefile and postfile passed when starting bitbake server are 'sticky'. With the new memory resident model this doesn't make sense as the server the system is started with isn't special. This patch changes the code so the prefile/postfile are used if specified on the commandline and not used otherwise. This makes the behaviour much more predictable and expected and as an added bonus simplifies the code. Signed-off-by: Richard Purdie <>
* cooker: Handle datastore tracking for showEnvironment server side onlyRichard Purdie2017-08-312-7/+11
| | | | | | | | The current passing of "tracking" backwards and forwards, client to server is ugly and complex and error prone. Instead, set this during showEnvironment commands triggering a reset there if/as required. Signed-off-by: Richard Purdie <>
* prserv/cooker: Drop unused paramRichard Purdie2017-08-312-2/+2
| | | | | | Drop pointless unused function parameter. Signed-off-by: Richard Purdie <>
* command: Fix some bugs identified by BB_SERVER_TIMEOUTRichard Purdie2017-08-301-1/+2
| | | | | | | | | | | | | | | | | | The 'needconfig' flag was meant to be the default as most commands do need it and the "False" cases were the exception. The code was written backwards with a default False value. Invert this to match the intent, resulting in the config being reparsed if metadata has changed. Also ensure the second level configuration is parsed for the getLayerPriorities command as otherwise it can return stale info. With these changes: BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_add_remove passes instead of fails. Signed-off-by: Richard Purdie <>
* main: Attempt to gain bitbake.lock rather than just waitingRichard Purdie2017-08-241-1/+9
| | | | | | | Rather than just waiting for 5s, try and get the lockfile. If we gain the lock, we know we're ready to retry and can skip any remaining timeout. Signed-off-by: Richard Purdie <>
* process: Clean up connection retry logicRichard Purdie2017-08-242-7/+14
| | | | | | | | | | | | | | | | Its possible for a connection to connect to the server as its shutting down but before its removed the socket file. This patch: a) Removes the socket file earlier to avoid connections. b) Handles EOFError in initial connections gracefully. These occur if the socket is closed during the server shutdown. c) Ensure duplicate events aren't shown on the console. This makes debugging these issues very very confusing. With these changes the backtrace that was concerning users is hidden and the server works as expected with a reconnect when it catches it in a bad state. Signed-off-by: Richard Purdie <>
* fetch2: don't mandate path element in encodeurl()Paul Eggleton2017-08-231-6/+5
| | | | | | | | | | | URLs do not have to have a path; currently our npm URLs don't, so encodeurl() needs to handle if the path element isn't specified. This fixes errors using OpenEmbedded's devtool add / recipetool create on an npm URL after OE-Core revision ecca596b75cfda2f798a0bdde75f4f774e23a95b that uses decodeurl() and encodeurl() to change URL parameter values. Signed-off-by: Paul Eggleton <> Signed-off-by: Richard Purdie <>
* bitbake: ui: Sort 'Dependent tasks' in taskexpJussi Kukkonen2017-08-231-1/+3
| | | | | | | | The underlying model is already sorted for use in the other view, add a sorting model for the 'Dependent Tasks' view. Signed-off-by: Jussi Kukkonen <> Signed-off-by: Richard Purdie <>
* toaster: support custom Layer Index URL and fixture overrideDavid Reyna2017-08-222-2/+9
| | | | | | | | | | | | | | | | | | Toaster needs to allow the custom fixture file to specific an alternate Layer Index URL for users that host their own Layer Index Server via the 'CUSTOM_LAYERINDEX_SERVER' in 'custom.xml'. Toaster also needs to allow the ability to completely override the default fixture files, otherwise that content can leak into the custom environment (by default the custom fixture is an overlay that cannot remove existing values from the default fixture) via the 'CUSTOM_XML_ONLY' value in 'setting.xml'. [YOCTO #11938] Signed-off-by: David Reyna <> Signed-off-by: Richard Purdie <>
* Toaster: custom start and stop actionsDavid Reyna2017-08-222-2/+69
| | | | | | | | | | | | Toaster needs the ability to allow custom extensions to execute when Toaster is started and stopped. Toaster will look for a custom extension script in the fixtures directory and execute any applicable hooks. [YOCTO #11938] Signed-off-by: David Reyna <> Signed-off-by: Richard Purdie <>
* toaster: custom image updates and original creationDavid Reyna2017-08-222-38/+62
| | | | | | | | | | | | | | | | | | | When Toaster removes a package from a custom image, it must also always remove the advised reverse-dependent recipes. Similarly, when adding a package it must always add its advised the packages it depends on. This code must be un-indented so that it applies to all respective added or removed packages. Toaster normally waits until a new custom image is built before creating the custom layer and the recipe. However, an intermediate different build can fail because the recipe has already been added to the project, so the image's default recipe must be created when the image is created. [YOCTO #11915] Signed-off-by: David Reyna <> Signed-off-by: Richard Purdie <>
* process: Ensure we call select() to know which fds to readRichard Purdie2017-08-211-1/+1
| | | | | | | | | | | | | | | | | There is an interesting bug in the current code where a sync command is not seen until the current async command completes, by which time the UI may have shut down. The reason is that if there are idle commands, we may not end up sleeping in the select call at all, partiularly under heavy load like parsing. Fix this by calling select with a zero timeout so that we see active fds and know to read from them. This fixes various problems toaster was having with the recent server changes. [YOCTO #11898] Signed-off-by: Richard Purdie <>
* process: Increase server startup timeoutRichard Purdie2017-08-151-1/+1
| | | | | | | | We're seeing the server fail to start within 8s on heavily loaded autobuilders so increase this timeout to 30s which should be more than enough time. Signed-off-by: Richard Purdie <>
* process: Improve client disconnectsRichard Purdie2017-08-141-15/+23
| | | | | | | | There have been cases where the server could loop indefinitely and incorrectly handle client disconnects. In the EOFError case, ensure a full disconnect happens in the alternative disconnect path to avoid this. Signed-off-by: Richard Purdie <>
* cooker: Ignore common bitbake files for the parse cache invalidationRichard Purdie2017-08-091-0/+3
| | | | | | | | | | | | Writes to the cookerdaemon log and/or the lockfile were meaning the parser cache was always being invalidated and reparsed. This is unnecessary so spot accesses to these two common cases and ignore the files from a reparse perspective. This doesn't remove many sources of reparse but does improve several common cases. Signed-off-by: Richard Purdie <>
* cooker: Improve inotify overflow handlingRichard Purdie2017-08-092-1/+8
| | | | | | | | Add a proper function for clearing the mtime cache. Clean up the inotify event overflow case to err on the side of caution and clear any potentially now out of sync caches. Signed-off-by: Richard Purdie <>
* cooker: Ensure we handle inotify before running the next commandRichard Purdie2017-08-092-7/+12
| | | | | | | | | | | | The inotify watch events are handled "at idle" which could in theory mean a command could run before any preceeding inotify events have been processed. This leads to a theoretical race window where those events may have a signficicant effect on the command. Add a mechanism to allow us to ensure all pending events are processed before running commands. Signed-off-by: Richard Purdie <>
* cooker: Track directories searched for bbappend/bb filesRichard Purdie2017-08-091-3/+21
| | | | | | | | | | | | | | | Some of the directories searched by BBFILES are not currently being added to the inotify watch list. This can mean that added append files are not noticed leading to misleading metadata results when using BB_SERVER_TIMEOUT != 0. We use glob to expand the BBFILES references and without writing our own implentation, figuring out the directories it searches is hard. We use some horrible hacks here to intecept the listdir calls, I'm open to better ways to handle this but this does ensure we have the right watches set. Signed-off-by: Richard Purdie <>
* server/process: Ensure we don't loop on client EOFErrorRichard Purdie2017-08-091-2/+2
| | | | | | | | | | | | The server currently crashes if we hit an EOFError due to controllersock still being in ready and the continue meaning ready isn't re-evaluated. Setting the value to False can mean the shutdown code doesn't handle the situation cleanly. Clear ready to avoid the crash/loop instead and handle any OSError whilst we're in here. Signed-off-by: Richard Purdie <>
* cooker/siggen: Reset siggen when reparsingRichard Purdie2017-08-092-0/+4
| | | | | | | | | | If we don't do this, we get basehash mismatch errors occurring from the reparse which would then set bitbake's error exit code. This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order to fail with a non-zero BB_SERVER_TIMEOUT. Signed-off-by: Richard Purdie <>
* main: Handle BB_SERVER_TIMEOUT = -1 for no server timeoutRobert Yang2017-08-082-3/+6
| | | | | | | Make BB_SERVER_TIMEOUT = -1 mean no unload forever. Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* process: Fix disconnect when BB_SERVER_TIMEOUTRobert Yang2017-08-081-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: $ export BB_SERVER_TIMEOUT=10000 $ bitbake --server-only $ bitbake --status-only [snip] File "/buildarea/lyang1/poky/bitbake/lib/bb/server/", line 472, in recvfds msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size)) OSError: [Errno 9] Bad file descriptor And: $ export BB_SERVER_TIMEOUT=10000 $ bitbake --server-only -B localhost:-1 $ bitbake --status-only # Everything is fine in first run $ bitbake --status-only [snip] File "/buildarea/lyang1/poky/bitbake/lib/bb/server/", line 472, in recvfds msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size)) OSError: [Errno 9] Bad file descriptor This was because self.controllersock was not set to False, so it still ran sock.recvmsg() when sock was closed. And also need set command_channel to Flase, otherwise the self.command_channel.get() will always run when EOF, and cause infinite loop. Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* main: Remove unneeded float()Robert Yang2017-08-081-1/+1
| | | | | | | | | | | | | | | | | | | | | There is already a type=float, so the float() is not needed, which also makes the error clearer: $ export BB_SERVER_TIMEOUT=10000AA With float(): $ bitbake quilt-native [snip] ValueError: could not convert string to float: '10000AA' Without float(): $ bitbake quilt-native [snip] optparse.OptionValueError: option --idle-timeout: invalid floating-point value: '10000AA' The second one tells clearly where is wrong. Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* Revert "tinfoil: fix duplication of log messages"Paul Eggleton2017-08-082-4/+3
| | | | | | | | | | | | | | | In combination with the recent server reworking, this change actually prevents messages sent from tasks from being logged properly. This will of course give us the duplicated messages back, and I really hate to do that effectively a second time, but that's better than seeing no error at all in the case of a failure - we'll have to find the proper way of avoiding the duplication that doesn't result in some messages going missing. This reverts commit 8a5bae76f91f2411187c638a42fa3c762052cf11. Signed-off-by: Paul Eggleton <> Signed-off-by: Richard Purdie <>
* daemonize: Always print any remaning UI events at exitRichard Purdie2017-08-031-1/+1
| | | | | | | | | | | If there are events in the UI queue we want to print them regardless of whether we're handling an exception or something like SystemExit. This improves error messages for some other failure cases where currently no logging would get printed and leave the user confused as to what went wrong. Signed-off-by: Richard Purdie <>
* cooker: Ensure all tasks are shown in task dependency treeRichard Purdie2017-08-011-3/+3
| | | | | | | | | | "bitbake -g m4-native -c do_unpack" doesn't list any dependencies in This is because no header describing the task was added unless a task has dependencies. Tweak the code to fix this. [YOCTO #10893] Signed-off-by: Richard Purdie <>
* daemonize: clean up codeRichard Purdie2017-07-311-134/+14
| | | | | | | | This was originally based on some other code but its mostly misleading comments now. Massively clean this up and accept its now a total fork. There are no funciton changes here, just cleanup. Signed-off-by: Richard Purdie <>
* daemonize: Ensure child process exits safelyRichard Purdie2017-07-311-3/+8
| | | | | | | | | | | When we create the child, if an exception occurred it was transfering back into the parent context. We don't want to do that us use a try/finally to ensure we exit. We need to ensure a traceback is printed and any queued UI messages which may not have made it to the client UI at this point. Signed-off-by: Richard Purdie <>
* process: Add some extra server startup logsRichard Purdie2017-07-311-1/+4
| | | | | | | | We have cases where the server is being started but we're not seeing any messages from it. Add some earlier logging so we can try and better understand where issues may be occurring. Signed-off-by: Richard Purdie <>
* process: Reorder server command processing and handle EOFErrorRichard Purdie2017-07-311-13/+18
| | | | | | | | | | | If the connection control socket and the command channel close together, we can race and hit EOFError exceptions before we close the channel. Reorder the code to handle this in the correct order and ignore the EOFError exceptions as they mean the client is disconnecting and shouldn't terminate the server. Signed-off-by: Richard Purdie <>
* wget: fix FusionForge workaroundRoss Burton2017-07-301-6/+3
| | | | | | | | | | | | | | | My previous assertion about FusionForge appears to have been wrong, or FusionForge has changed behaviour, or both. FusionForge now mandates that downloads have the Accept header set, despite that header being optional, and returns a 406 Not Acceptable error if it isn't set. As we were pretending that 406 was actually 405 (Moved) and tried to handle it as a redirect this results in an infinite loop until Python kills the recursion. Delete the handling of 406 as 405, and pass Accept: */* in the headers. Signed-off-by: Ross Burton <> Signed-off-by: Richard Purdie <>
* toaster: move to new bitbake xmlrpc defaultDavid Reyna2017-07-303-7/+18
| | | | | | | | | | | | | | | | | | | | | | | | The bitbake option "-t SERVERTYPE" was deprecated and can be removed since the desired XMLRPC listener now the default. The bitbake server port cannot be "-1" anymore and must be explicitly passed. There is a race condition for the bblock file to not only be created but to actually be populated, so a delay test loop is required (usually only one pass). The 'xmlrpcclient' is now the class that allows Toaster to for example kill builds. The events for populating the recipe parsing now either show only the final result or are skipped entiredly, so the progress calculator needs to be changed to not block on the parsing phase. [YOCTO #11851] Signed-off-by: David Reyna <> Signed-off-by: Richard Purdie <>
* toaster: enable remote HTTP API for status aggregationDavid Reyna2017-07-304-4/+100
| | | | | | | | | | | Add support for Toaster aggregators with a set of api links that return JSON data for (a) builds in progress, (b) builds completed, (c) specific build data, and (d) an is-alive health ping link. [YOCTO #11794] Signed-off-by: David Reyna <> Signed-off-by: Richard Purdie <>
* runqueue: Tweak debug message to make it more readable/diffableRichard Purdie2017-07-301-1/+1
| | | | | | | Having this as one huge long line isn't easy to manipulate, split it into multiple lines for ease of debugging issues. Signed-off-by: Richard Purdie <>