summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
* fetch2: fix checkstatus fallback to MIRRORSRoss Burton2017-07-301-3/+2
| | | | | | | | The checkstatus() code was expecting checkstatus to throw exceptions if it failed, but in general it should return False. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tests/fetch: fix GitShallowTest.test_bitbakeChristopher Larson2017-07-301-1/+1
| | | | | | | | | | | | | `git fetch --tags` seems to interact badly with `mirror=fetch`, resulting in the regular branches not being fetched, so drop the unnecessary `--tags`. This fixes this unit test failure: `bb.fetch2.FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /tmp/tmp4ag_mgmn/gitsource` [YOCTO #11698] Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* main: Always return 0 for bitbake --kill-serverRichard Purdie2017-07-281-2/+4
| | | | | | | | If the server isn't running return 0 as to do otherwise complicates scripts which do cleanup of bitbake servers which would potentially be memory resident. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process: Clean up server communication timeout errorsRichard Purdie2017-07-281-12/+3
| | | | | | | | | | | | | | | | | | | This timeout path was commonly hit due to errors starting the server. Now we have a better way to handle that, the retry logic can be improved and cleaned up. This patch: * Makes the timeout 5s rather than intervals of 1s with a message. Paul noted some commands can take around 1s to run on a server which has just been started on a loaded system. * Allows a broke connection to exit immediately rather than retrying something which will never work. * Drops the Ctrl+C masking, we shouldn't need that anymore and any issues would be better handled in other ways. This should make things clearer and less confusing for users and is much cleaner code too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process: Don't leak open pipes upon reconnectionRichard Purdie2017-07-281-1/+18
| | | | | | | If we reconnect to the server, stop leaking pipes and clean up after ourselves. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process/cooker: Allow UI process to know if the cooker was started successfullyRichard Purdie2017-07-282-3/+23
| | | | | | | | | | | | | | | Currently if the server fails to start, the user sees no error message and the server will be repeatedly attempted to be started until some longer timeouts expire. There are error messages in the cookerdeamon log but nobody thinks to look there. Add in a pipe which can be used to tell the starting process whether the cooker did actually start or not. If it fails to start, no further attempts can be made and if present, the log file can be shown to the user. [YOCTO #11834] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process: Move socket keep alive into BitBakeProcessServerConnectionRichard Purdie2017-07-281-4/+6
| | | | | | | This cleans up the socket keep alive into better class structured code and adds cleanup of the open file descriptors upon shutdown. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process: Allow BBUIEventQueue to exit cleanlyRichard Purdie2017-07-281-3/+8
| | | | | | | Currently the monitoring thread exits with some error code or runs indefinitely. Allow closure of the pipe its monitoring to have the thread exit cleanly/silently. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process: Ensure ConnectionReader/Writer have fileno() and close() methodsRichard Purdie2017-07-281-0/+9
| | | | | | | Expose the underlying close() and fileno() methods which allow connection monitoring and cleanup. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* event: Don't write duplicate logs to stdout and stderr in no UI caseRichard Purdie2017-07-281-10/+17
| | | | | | | | | | This code would duplicate messages to stdout and stderr when no UI connected and there were error level messages. Rework the code so it either uses stderr (for errors and above) or stdout for warnings/debug but not both for the same messages. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake-diffsigs: fix regression after recent server changesPaul Eggleton2017-07-273-7/+47
| | | | | | | | | | | | | | | | We were bridging the gap between the server and UI here by calling a bb.siggen.find_siginfo, a function defined and set on that module from the metadata. This worked from the UI side before but since the recent server changes is no longer accessible. Create a new command so this can execute on the server side and return the result by way of a new event. (We're still running compare_sigfiles() on the signature generator but that isn't quite the same thing and does still work.) Fixes [YOCTO #11844]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/fetch2/__init__.py: Fix unpack commentMark Hatle2017-07-271-1/+1
| | | | | Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/bb/utils.py: Add missing debug levelMark Hatle2017-07-271-2/+2
| | | | | Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2/__init__.py: replace stray logger.warn() with logger.warning()Andre McCurdy2017-07-251-1/+1
| | | | | | | | | | | | Update stray usage of deprecated logger.warn(), which was introduced to fetch2/__init__.py after all other instances had been replaced by logger.warning(): http://git.openembedded.org/bitbake/commit/?id=5a53e7d7b017769a6eb0f0a6335735a1fe51a5ec http://git.openembedded.org/bitbake/commit/?id=676a5f592e8507e81b8f748d58acfea7572f8796 Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch/wget: mitigate a wget race condition when listing FTP directoriesRoss Burton2017-07-251-12/+11
| | | | | | | | | | | | | | | | | | | | When wget is fetching a listing for a directory over FTP it writes to a temporary file called .listing in the current directory. If there are many such operations happening in parallel - for example during 'bitbake world -c checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to write to, read, and delete the same file. This results in various failures such as the file disappearing before wget has processed it or the file changing contents, which causes checkpkg to randomly fail. Mitigate the race condition by creating a temporary directory to run wget in when doing directory listings. [ YOCTO #11828 ] Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* process: Change timeout warning to a noteRichard Purdie2017-07-241-1/+1
| | | | | | | | | The warning message currently shown can occur more frequently than previously if a previous bitbake server is shutting down and we're reconnecting to a new server. Change it to a note message to match the higher level connection logging retry messages and so as not to interfer with selftests. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker/process: Drop server_main functionRichard Purdie2017-07-242-23/+21
| | | | | | | | | | | | Now that there is only one server, this abstraction is no longer needed and causes indrection/confusion. The server shutdown is also broken with the cooker post_server calls happening too late, leading to "lock held" warnings in the logs if PRServ is enabled. Remove the abstraction and put the shutdown calls in the right order with respect to the locking. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* prserv/serv: Improve process exit handlingRichard Purdie2017-07-241-2/+17
| | | | | | | | | | | | | | | | | The server shutdown is currenlty laggy and race prone. This patch: * adds a waitpid so that no zombie server is left around if its not running in daemon mode. * adds a quit "sentinal" using a pipe so that we're not sitting in a socket poll() until timeout in order just to quit. * use a select() call to poll the socket and the pipe for a quit signal. The net result of this change is that the prserv exits with the cooker server and it does so immediately and doesn't wait for the select/poll calls to timeout. This makes bitbake a lot more responsive for startup/shutdown and doesn't cause UI timeout errors as often when prserv is used. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2/wget.py: improve error handling during sstate checkPatrick Ohly2017-07-211-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the sstate is accessed via HTTP, the existence check can fail due to network issues, in which case bitbake silently continues without sstate. One such network issue is an HTTP server like Python's own SimpleHTTP which closes the TCP connection despite an explicit "Keep-Alive" in the HTTP request header. The server does that without a "close" in the HTTP response header, so the socket remains in the connection cache, leading to "urlopen failed: <urlopen error [Errno 9] Bad file descriptor>" (only visible in "bitbake -D -D" output) when trying to use the cached connection again. The connection might also get closed for other reasons (proxy, timeouts, etc.), so this is something that the client should be able to handle. This is achieved by checking for the error, removing the bad connection, and letting the check_status() method try again with a new connection. It is necessary to let the second attempt fail permanently, because bad proxy setups have been observed to also lead to such broken connections. In that case, we need to abort for real after trying twice, otherwise a build would just hang forever. [YOCTO #11782] Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: trim build target inputDavid Reyna2017-07-211-2/+2
| | | | | | | | | | Trim the entered built target value so that Toaster is not confused with no real targets nor a ghost second target. [YOCTO #11727] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: set clone progress default to offDavid Reyna2017-07-212-4/+4
| | | | | | | | | | | | Set the clone progress to be off by default for the benefit of command line projects and 'Local Yocto' builds. For Toaster managed projects that do use the clone feature the clone progress status is already explicitly set by the existing code and thus displayed. [YOCTO #11744] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster: include setscene in task progressDavid Reyna2017-07-211-3/+4
| | | | | | | | | | | Change the task progress counting from the task order field which excludes setscene to the task completion field which counts all completed tasks regardless of type. [YOCTO #9971] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Update to version 1.35.0 (development version with server rework changes)Richard Purdie2017-07-212-2/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2: fire an event when there are missing checksumsPaul Eggleton2017-07-211-0/+10
| | | | | | | | | If BB_STRICT_CHECKSUMS is set to anything other than "1" i.e. we're not going to raise an error, then fire an event so that scripts can listen for it and get the checksums. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2: allow hiding checksum warningPaul Eggleton2017-07-211-0/+6
| | | | | | | | | | | If BB_STRICT_CHECKSUMS is set to "ignore" then don't display a warning if no checksums are specified in the recipe. This is not intended to be used from recipes - it is needed when we move to using more standard code paths to fetch new files from scripts i.e. where we don't know what the checksums are in advance. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: add more doc commentsPaul Eggleton2017-07-211-0/+65
| | | | | | | | We want this API to be easier to use, so add missing function documentation to help with that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: add simple API for getting cached recipe informationPaul Eggleton2017-07-211-0/+142
| | | | | | | | | | | | | | | | | | | A common task for tinfoil-using scripts is to iterate over all recipes. This isn't too difficult with the current API, but the pkg_* variables are a little awkward and are really designed for bitbake's internal usage - and it gets a bit more difficult when you want to access some of the other information such as packages and rprovides. To resolve this, create a new recipe info class and add an all_recipes() function to generate this for all recipes. Also add a get_recipe_info() function to get the information for a specific recipe (by PN). (It might perhaps be suggested that we already have a structure similar to this in the cache, however the one we add here is designed for external use and allows the internal structures to change if needed without affecting the API). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: enable access to additional cached itemsPaul Eggleton2017-07-212-0/+40
| | | | | | | | | Add access to fn_provides, packages, packages_dynamic and rproviders on the recipecache object. This requires an additional corresponding command plumbing to be added. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: add functionality for running full buildsPaul Eggleton2017-07-211-0/+154
| | | | | | | | | | | | | | Up to this point, if you wanted to run build tasks in the normal way they get run from a python script, there was no other way than to shell out to bitbake. Worse than that, you couldn't have tinfoil active during that because only one bitbake instance could be running at once. As long as we're prepared to handle the events produced, we can create a wrapper around calling the buildTargets command. Borrow code from knotty to do this in such a way that we get the expected running task display (courtesy of TermFilter) and Ctrl+C handling. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* knotty: make it possible to use termfilter without either consolePaul Eggleton2017-07-211-2/+4
| | | | | | | | | This isn't useful for knotty itself, but for use from tinfoil in case we can't get access to either the console or errconsole, allow either to be unspecified (None). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/bb/ui/uihelper: indicate to caller of eventHandler() if events handledPaul Eggleton2017-07-211-0/+3
| | | | | | | | It is useful for the caller to know whether the uihelper has handled the event passed so that it can skip other event handling code if so. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/bb/event: refactor printing eventsPaul Eggleton2017-07-217-71/+53
| | | | | | | | | | We really ought to have just one place where the string representation of these events is produced. This doesn't take any real control away from the UI - if an alternative representation is desired, that can still be made. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: set a flag when recipes have been parsedPaul Eggleton2017-07-211-0/+3
| | | | | | | | Make it easy to determine if recipes are parsed (and thus information about available recipes is in memory). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: add a parse_recipes() functionPaul Eggleton2017-07-212-2/+8
| | | | | | | | | | Python style recommends underscore based naming rather than camelCase, and thus the former has been used for most of tinfoil's functions. Add an underscored version of parseRecipes() for consistency and change the one place we call it to use the new version. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tinfoil: add internal mode to build_file() functionPaul Eggleton2017-07-213-15/+37
| | | | | | | | | | | | | | | In OE's devtool we want to repeatedly run build_file() without showing unnecessary messages and triggering buildhistory for each call. build_file() is just a wrapper around the buildFile command. Change the final "hidewarning" parameter of the buildFile command to "internal" and have this call a new buildFileInternal() function without triggering any of the normal build events, silencing the normal info messages from the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and avoiding calling parseConfiguration() which we've already done at this point. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bb/main.py: fix logic for --observe-onlyRobert Yang2017-07-211-1/+1
| | | | | | | | | | | | | | | Fixed: $ bitbake --observe-only FATAL: '--observe-only' can only be used by UI clients connecting to a server. And even: $bitbake --observe-only -B localhost:-1 FATAL: '--observe-only' can only be used by UI clients connecting to a server. This was beucase the brackets were in a wrong position. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bb/main.py: avoid starting server when not neededRobert Yang2017-07-211-2/+8
| | | | | | | | | | | | | | | | | | | | | Fixed 1: $ . ../poky/oe-init-build-env-memres . $ bitbake -m $ bitbake -m # shutdown it again $ bitbake -m Starting bitbake server... Terminated bitbake server. It starts the server and kill it which is redundant. $ bitbake -m $ bitbake --status-only Starting bitbake server... It starts the server which is not what we need. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bb/main.py: fix infinite loop for --server-onlyRobert Yang2017-07-211-1/+1
| | | | | | | | | | | | | Fixed: $ bitbake --server-only -B localhost:-1 Infinite loop Reconnecting to bitbake server... [snip] Break the loop when server-only can fix the problem. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server/process.py: fix self.bitbake_lock.write()Robert Yang2017-07-211-2/+2
| | | | | | | | | | | | | | There is no global var "configuration", so the old code hang at self.bitbake_lock.write(), and nothing wrote to bitbake.lock. I didn't figure out why it hang (but not print errors). Reproducer: $ bitbake -B localhost:-1 world -k Check bitbake.log, there was nothing, now fixed. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cookerdata/cooker: Restore original datastore upon client disconnectRichard Purdie2017-07-212-0/+11
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server: Rework the server API so process and xmlrpc servers coexistRichard Purdie2017-07-2110-916/+789
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes the way bitbake server works quite radically. Now, the server is always a process based server with the option of starting an XMLRPC listener on a specific inferface/port. Behind the scenes this is done with a "bitbake.sock" file alongside the bitbake.lock file. If we can obtain the lock, we know we need to start a server. The server always listens on the socket and UIs can then connect to this. UIs connect by sending a set of three file descriptors over the domain socket, one for sending commands, one for receiving command results and the other for receiving events. These changes meant we can throw away all the horrid server abstraction code, the plugable transport option to bitbake and the code becomes much more readable and debuggable. It also likely removes a ton of ways you could hang the UI/cooker in weird ways due to all the race conditions that existed with previous processes. Changes: * The foreground option for bitbake-server was dropped. Just tail the log if you really want this, the codepaths were complicated enough without adding one for this. * BBSERVER="autodetect" was dropped. The server will autostart and autoconnect in process mode. You have to specify an xmlrpc server address since that can't be autodetected. I can't see a use case for autodetect now. * The transport/servetype option to bitbake was dropped. * A BB_SERVER_TIMEOUT variable is added which allows the server to stay resident for a period of time after the last client disconnects before unloading. This is used if the -T/--idle-timeout option is not passed to bitbake. This change is invasive and may well introduce new issues however I believe the codebase is in a much better position for further development and debugging. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* daemonize: Various fixesRichard Purdie2017-07-181-8/+12
| | | | | | | | | | | | | | Currently if this code is used with something like oeqa's xml logging it fails as sys.stdout is an io stream. Add in try/except to handle this case. Add a waitpid() call to remove a zombie whilst forking. Also, append to the logfile, don't overwrite it (otherwise debugging can be a real pain when the server is restarting for unknown reasons). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cookerdata: Add a function to find TOPDIRRichard Purdie2017-07-181-0/+21
| | | | | | | | | | Finding the top level build directory is currently hard and relies on having a complete cooker being setup. Add a helper function which does the same thing without all the extra overhead. This is needed to be able to locate the bitbake lockfile and hence the socket for connecting clients in the new server model. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* server: Remove base classes and inline codeRichard Purdie2017-07-183-86/+80
| | | | | | | In preparation for rewriting this code, expand the relatively useless base classes into the code itself. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* event/command: Allow UI to request the UI eventhander IDRichard Purdie2017-07-183-4/+19
| | | | | | | | | | The UI may want to change its event mask however to do this, it needs the event handler's ID. Tweak the code to allow this to be stored and add a command to query it. Use the new command in the process server backend. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fetch2: fix indentationPaul Eggleton2017-07-181-47/+47
| | | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cooker: fix whitespacePaul Eggleton2017-07-181-4/+4
| | | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache.py: fix whitespacePaul Eggleton2017-07-181-14/+13
| | | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cookerdata: fix indentation and trailing spacesPaul Eggleton2017-07-181-5/+5
| | | | | | | Fix one incorrect indentation and several lines with trailing spaces. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* knotty: fix indentationPaul Eggleton2017-07-181-1/+1
| | | | | Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>