Age | Commit message (Collapse) | Author |
|
When the UI disconnects, we can throw away any server-side remote
datastores we created in response to calls from the UI, and we *must*
drop everything in extraconfigdata or it will taint any future
operations.
Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors
when running devtool.DevtoolTests.test_devtool_update_recipe_local_files
within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you have a layer with a blank BBFILE_PATTERN the layer was ignored
when processing the list of layers with priorities. This list is not
only used for processing recipes, but also by additional programs such
as bitbake-layers show-layers.
Without this change, a layer that provides configuration or classes
only does now show up in show-layers, which is used by the
yocto-compat-layer.py script. This causes a failures in the compatibility
check.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers
was failing and highlighted that since parseConfiguation clears data
structures, it needs to also clear parsecache_valid as it no longer
contains correct data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_build_image
fails, the reason is that internally, the limited_deps flag is set in
the bitbake server and then never cleared. This causes the sysroots to
be setup incorrectly (as per the limited dependency case) and builds
break. There is also potential for corruption of recipecaches.
Add shutdown/cleanup code to ensure these effects don't 'stick'.
This bug is particularly nasty as you can destroy TMPDIR with large
sysroots in build work directories which are prone to break.
Also ensure mtime cache is cleared (to match buildTargets) and
that no lasting changes are made to siggen either which ensures:
BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_upgrade_git devtool.DevtoolTests.test_devtool_virtual_kernel_modify
works.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you run some other operations that result in the cache being
populated, and then call buildFileInternal(), then you can end up in a
situation where the cache already contains information about the recipe.
For example in OE this can now happen when you use devtool upgrade.
Normally this doesn't cause any problems, unless you have a non-absolute
path in BBLAYERS - in buildFileInternal() we are calling matchfile() which
will convert the filename to absolute, but later when taskdata goes to find
the providers of the recipe it finds the non-absolute path, sets up the
task information using this and then the runqueue can't find any tasks
matching the absolute path. To fix this, back out the optimisation I did
earlier in bitbake rev ba53e067a2d448dd63b4ca252557ce98aa8e6321 to avoid
calling parseConfiguration() again, which is unfortunate but does result
in the cached information being that causes the problem being cleared
out.
This fixes "Task do_unpack does not exist for target ..." running
devtool upgrade within intel-iot-refkit.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we reload the configuration, PRSERV_HOST can change. Therefore
restart the PR Server depending on the new configuration at reparse.
Note that the server has to be started from the right process, it
can't be in the UI which shuts down as that shutdown triggers a shutdown
of its children and the PR Server shuts down too. This is why we need
pre_serve() which ensures its executed in the right context.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the original prefile and postfile passed when starting bitbake
server are 'sticky'. With the new memory resident model this doesn't make
sense as the server the system is started with isn't special.
This patch changes the code so the prefile/postfile are used if specified
on the commandline and not used otherwise. This makes the behaviour much
more predictable and expected and as an added bonus simplifies the code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current passing of "tracking" backwards and forwards, client to server
is ugly and complex and error prone. Instead, set this during showEnvironment
commands triggering a reset there if/as required.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Drop pointless unused function parameter.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Writes to the cookerdaemon log and/or the lockfile were meaning the parser
cache was always being invalidated and reparsed. This is unnecessary so
spot accesses to these two common cases and ignore the files from a reparse
perspective.
This doesn't remove many sources of reparse but does improve several
common cases.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a proper function for clearing the mtime cache. Clean up the inotify
event overflow case to err on the side of caution and clear any potentially
now out of sync caches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The inotify watch events are handled "at idle" which could in theory mean
a command could run before any preceeding inotify events have been processed.
This leads to a theoretical race window where those events may have a
signficicant effect on the command.
Add a mechanism to allow us to ensure all pending events are processed before
running commands.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some of the directories searched by BBFILES are not currently being added
to the inotify watch list. This can mean that added append files are not
noticed leading to misleading metadata results when using
BB_SERVER_TIMEOUT != 0.
We use glob to expand the BBFILES references and without writing our own
implentation, figuring out the directories it searches is hard. We use
some horrible hacks here to intecept the listdir calls, I'm open
to better ways to handle this but this does ensure we have the right
watches set.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we don't do this, we get basehash mismatch errors occurring from the reparse
which would then set bitbake's error exit code.
This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order
to fail with a non-zero BB_SERVER_TIMEOUT.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
"bitbake -g m4-native -c do_unpack" doesn't list any dependencies in
task-depends.dot. This is because no header describing the task was
added unless a task has dependencies. Tweak the code to fix this.
[YOCTO #10893]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if the server fails to start, the user sees no error message and
the server will be repeatedly attempted to be started until some longer
timeouts expire. There are error messages in the cookerdeamon log but
nobody thinks to look there.
Add in a pipe which can be used to tell the starting process whether the cooker
did actually start or not. If it fails to start, no further attempts can be
made and if present, the log file can be shown to the user.
[YOCTO #11834]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that there is only one server, this abstraction is no longer needed
and causes indrection/confusion. The server shutdown is also broken with
the cooker post_server calls happening too late, leading to "lock held"
warnings in the logs if PRServ is enabled.
Remove the abstraction and put the shutdown calls in the right order
with respect to the locking.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.
Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.
These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.
Changes:
* The foreground option for bitbake-server was dropped. Just tail
the log if you really want this, the codepaths were complicated enough
without adding one for this.
* BBSERVER="autodetect" was dropped. The server will autostart and
autoconnect in process mode. You have to specify an xmlrpc server
address since that can't be autodetected. I can't see a use case
for autodetect now.
* The transport/servetype option to bitbake was dropped.
* A BB_SERVER_TIMEOUT variable is added which allows the server
to stay resident for a period of time after the last client
disconnects before unloading. This is used if the -T/--idle-timeout
option is not passed to bitbake.
This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently builds in multiple TMPDIRs with multiconfig can break
since the BuildStarted event is used to create directory strutures in several
cases (e.g. buildstats.bbclass) and there is only on BuildStarted event
generated in a multiconfig build.
We have two options, a) to add a new MultiConfigBuildStarted event which is
generated once per multiconfig, or b) allow multiple BuildStarted events.
Having reviewed the code and current users of BuildStarted, sending one event
per multiconfig seems like its the best way forward and the existing code looks
able to cope with the duplication of events. I did also check toaster and I think
that can handle this issue too (multiconfig builds may have other issues there).
I'm therefore proposing we send multiple BuildStarted events for multiconfig
and for consistency, send multiple BuildCompleted events too.
We need to ensure that BUILDNAME, BUILDSTART and DATE/TIME are set consistently
in all the different multiconfig datastores. These events can write to the
datastore so copies are not used. buildFile was also cleaned up to ensure it
uses the right datastore in various places.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If an event handler for bb.event.BuildCompleted fails, we still need to
call finishAsyncCommand() or else BitBake will just exit immediately
without showing any error summary, or worse in the case of memory
resident mode BitBake will hang and if you Ctrl+C to break out, the
command won't be marked as finished which means that no further commands
will be able to be executed until the server is manually restarted.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The main point of memory resident bitbake is to avoid loading data
unnecessarily on every bitbake invocation. Unfortunately the code that
updated options from the UI was simply treating the fact that either
of the "prefile" or "postfile" options were in the list of options
passed in as an indication that the configuration was invalid, which was
bad because these are always passed in. We only need to mark the
configuration as invalid and thus reload it (and thus reload the cache)
if the option value has actually changed.
At the same time, the recently handled "tracking" option needs to be
treated in a similar manner since the configuration needs to be reparsed
if that has changed. Also, add a few extra debug messages to aid
debugging this code in future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Drop a number of the commands and support functions that were created
solely to support functionality in the now-removed Hob UI. In most cases
we now have this functionality elsewhere e.g. to modify config files and
recipes we have bb.utils.edit_metadata_file() and friends, and in OE we
have oe.recipeutils.patch_recipe_file() and friends which build on top of
the former. Additionally, some of it represented pretty egregious
incursion of OE metadata-specific references into BitBake code.
For now I have left in the find*File functions and commands as they are
relatively generic and possibly still useful, but they might be removed
in future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In OpenEmbedded's buildhistory class we want access to the bitbake
command line that launched the build, and up to now we were simply using
sys.argv from within the event handler to get that. Unfortunately that
doesn't work in memory resident mode, since the event handler is
naturally executing within the server and thus will give you the command
that launched the bitbake server which is much less interesting. Add a
dynamic variable BB_CMDLINE to provide access to this, set from sys.argv
within the UI process in updateToServer().
(Note that BB_CMDLINE isn't currently passed through to the worker, so
this is only really readable from event handlers plus any explicit
getVariable calls - in theory an observe-only UI could read it for
example.)
Part of the fix for [YOCTO #11634].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we set these flags here then we can not only tidy up some of the code
that calls initConfigurationData() (and reset() which simply calls the
former), we also avoid calling initConfigurationData() an extra time
unnecessarily during startup (since setting baseconfig_valid = False in
updateConfigOpts() also resulted in initConfigurationData() being
called from updateCacheSync() at the end of the command.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you build a recipe, modify one of the files referred to in SRC_URI,
then build it again, it should re-run do_fetch and the tasks that depend
upon it to incorporate the modified file. With memory resident mode this
was not working unless you restarted the server, because the mtime of
the file was cached and thus we never checked the actual file. Clear out
the mtime caches in buildTargets() to ensure this isn't an issue. I did
a minor refactoring to break this out to its own function since we were
already clearing out the build mtime cache here.
(The test I used for this was . ./oe-init-build-env-memres,
bitbake -c fetch mdadm, modify the comments at the top of one of the
patches referred to in the recipe, and then bitbake -c fetch mdadm again
and it should re-execute the fetch task ("... 0 didn't need to be
re-run".)
Fixes [YOCTO #10732].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The loop for populating the depends of the tree data should use dep
instead of item.
Signed-off-by: Andy Voltz <andy.voltz@timesys.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The dependency graph in the bb.event.DepTreeGenerated and the
corresponding pn-buildlist and task-depends.dot from "bitbake -g"
contained entries like multiconfig:qemuarm.gcc (dot as last separator)
instead of the correct multiconfig:qemuarm:gcc (colon as separator).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Displaying the environment data in a sorted manner makes it easier
to compare data between different setups and means you can know
where to find specific entries.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When constructing an older style single regex, it's possible for BBMASK
to end up beginning with '|', which matches and masks _everything_.
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The package-depends.dot and pn-depends.dot files are inaccurate, missing
out key dependencies such those made via the [depends] flags. As such
they can be misleading to the user.
They mainly exist for historical reasons, coming from a time before
we had task based execution.
This commit removes the two dated file formats and replaces them with
a recipe-depends.dot which is a flattened version of task-depends.dot.
The old format files are removed if present so that the user can't get
confused about why data might not match between files.
The code is also rewritten to use 'with f: f.write()' syntax as is more
commonly used now.
Also update the docs to match the change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Also run sanity check for all additional multiconfig configurations
as listed in BBMULTICONFIG, not just the "default" configuration.
[YOCTO#10810]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited.
We really need a way to know this from the metadata and this new variable
provides this in worker context. This means existing code can stop having
to guess.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently "bitbake universe -c fetch" will error if every task in the system
doesn't have a fetch task. This seems unreasonable so filter the list based
upon whether the task exists. This implementation does assume if a task exits
for a given provider name, it can run, even if the preferred provider that
ends up getting built might be the namespace without the task. This is unlikely
to be a real world issue at this point.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently you cannot build a target for all the configured multiconfigs without
specifying a list. The list can be quite long, requiring to type several lines
of text.
This enhancement is to support globbing so that you can do this,
e.g. instead of:
$ bitbake multiconfig:A:bash multiconfig:B:bash bash
you can do:
$ bitbake multiconfig:*:bash
There are real world use cases where it is desirable to use multiconfig with
two different tasks. For example: SDKs with multiple toolchains but also
containing set of additional host tools, or multiconfig builds requiring one image for
the main CPU(s) and a different co-image for a companion CPU.
For this reason, two variations of the new syntax are supported.
For example, the following:
$ bitbake multiconfig:*:meta-toolhchain
would expand to:
$ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain meta-toolchain
However the following:
$ bitbake multiconfig:*:meta-toolhchain hosttools
would expand to:
$ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain hosttools
In other words, if the user specified the "default" task explicitly, it replaces the implicit
"default" task.
[YOCTO#10680]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
At end of BBCooker needs to release fd's associated with pyinotify
watchers to avoid: Too many open files (EMFILE) error in different
scenarios like several instances of tinfoil.
[YOCTO #10873]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some of our metadata assumes that BuildStarted and BuildCompleted events
see the same data store. This is the case for buildTarget but not for
buildFile and recent changes mean this is now a problem.
The update_data() call is now an empty operation and there is no difference
between the expanded_data and data so we can simply remove the expanded_data
and its references and use data everywhere. This has been inteded for a while
but the above issue makes this more pressing to finally clean up.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we want to use this function/command internally, we don't want this
warning shown.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:
* It now works when bitbake is memory-resident instead of failing with
"ERROR: Only one copy of bitbake should be run against a build
directory".
* We can now connect an actual UI, thus you get things like the recipe
parsing / cache loading progress bar and parse error handling for free
* We can now handle events generated by the server if we wish to do so
* We can potentially extend this to do more stuff, e.g. actually running
build operations - this needs to be made more practical before we can
use it though (since you effectively have to become the UI yourself
for this at the moment.)
The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.
Part of the fix for [YOCTO #5470].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In memory resident mode we don't really want to actually shut down since
it's only the client going away.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If many files change and the inotify queue overflows, rather than print
a traceback, invalidate the caches and warn the user.
[YOCTO #10676]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The processing of the "do_" prefix to tasks is currently inconsistent
and has resulted in "bitbake world -g" being broken as task prefixes
don't get handled correctly.
Make the "do_" task prefix handling consistent through various codepaths.
[YOCTO #10651]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We assume that the value taken by variable v can be string,
integer or any type which can be marshalled by xmlrpc. This
change would help us to convert the non marshallable types
to string. So that we don't get exception from xmlrpc.
[YOCTO #10740]
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There would be error when BBFILE_PATTERN = None:
BBFILE_PATTERN_foo not defined
This is the correct behaviour, but when the layer sets BBFILE_PATTERN = "",
it would match all the remaining recipes, and cause "No bb files matched BBFILE_PATTERN"
warnings for all the layers which behind it.
When a layer sets BBFILE_PATTERN = "" (for example, a layer only
provides git repos and source tarballs), now it means has no recipes.
This is different from BBFILE_PATTERN_IGNORE_EMPTY, the later one means
that it *may* not have any recipes.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some task id manipulations were suboptimal:
* taskfn_fromtid and fn_from_tid were effectively the same function
* many calls to split_tid(), then taskfn_fromtid()
* taskfn_fromtid() called split_tid() internally
This patch adds split_tid_mcfn() to replace split_tid() and returns the
"taskfn" variant being used in many places. We update all core calls
to the new function and ignore the return values we don't need since the
function call overhead of the split_tid wrapper is higher than ignoring
a return value.
The one remaining standalone use of taskfn_fromtid is replaced with
fn_from_tid. I couldn't see any external usage so it was dropped.
There is external usage of split_tid so a wrapper remains for it.
Combined together these changes should improve some of the runqueue task
manipulation performance.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|