| Age | Commit message (Collapse) | Author |
|---|
|
Since 'build/siggen: Rework stamps functions'[1] the stamps directory
for each recipe is only created if there are stamps to be written. Some
recipes - such as package-index - only have nostamp tasks so the stamps
directory is never created, resulting in the rm_work task trying to cd
into a directory that doesn't exist.
Resolve this by simply checking the stamps directory exists before
trying to delete files in it.
[1] bitbake c79ecec580e4c2a141ae483ec0f6448f70593dcf
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The do_rm_work() task is using the first available 'rm' binary
available in PATH to remove files and folders.
However, depending on the PATH setup and RECIPE_SYSROOT_NATIVE
contents, the function can be using the 'rm' binary available
in RECIPE_SYSROOT_NATIVE, a folder that will get removed.
This causes a sporadic race-condition when trying to access the
'rm' binary of a folder already deleted.
Solve this by exclusively using the HOSTTOOLS 'rm' binary, as
this folder will not get removed.
Signed-off-by: Luis Martins <luis.pinto.martins@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For now, if we use rm_work and `bitbake core-image-minimal', some
recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.
Adjust the dependency to make do_rm_work_all depend on do_rm_work
to solve this problem.
Below are the detailed explanation of why this would work.
Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]
|
[do_rm_work_all] -------------+
With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]
Such dependency chain adjustment fixes the issue because do_rm_work_all
now depends on [other deps] and thus the [depends] of these [other deps].
Take core-image-minimal as an example. Before this adjustment,
do_rm_work_all does not have any relationship with do_rootfs, and we have
do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..."
This essentially prevents 'recrdeptask' setting of do_rm_work_all extend
to makedevs-native. With this patch, the do_rm_work_all now depends
on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's
recrdeptask could have effect on makedevs-native.
With this patch, all built recipes WORKDIR will be cleaned up with
a few expected exceptions such as kernel and qemu-helper-native.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
We can exclude the SSTATETASKS from the rm_work task signature
to avoid running the task when we remove some setscene tasks
from the dependencie chain.
The inject_rm_work handler on the rm_work bbclass triggers the
rm_work task running for any signature change in the dependencie
chain of the task do_build of each recipe.
i.e INHERIT:remove = "create-spdx" will trigger the do_rm_work
when we collect the sstate cache with INHERIT = "create-spdx"
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
As stated in our top level license files, the license is MIT unless
otherwise stated. Add SPDX identifers accordingly. Replace older
license statementa with the standardised syntax. Also drop "All
Rights Reserved" expression as it isn't used now, doesn't mean anything
and is confusing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Where there isn't a copyright statement, add one to make it explicit.
Also drop editor config lines where they were present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This makes it possible to name files starting with a hyphen in the work
directory. Without this change rm will fail due to an unexpected option
being passed.
Signed-off-by: William A. Kennington III <wak@google.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Instead of overwriting the stamp name with 'dummy', handle
setscene promotion in the default case block
* Merge '*do_image_complete_setscene*' and '*do_image_qa_setscene*'
case handling
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[ YOCTO #13212 ]
Suggested-by: Romuald Jeanne <romuald.jeanne@st.com>
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Introduced in commit b3de5d5795767a4b8c331fa5040166e7e410eeec.
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fixes some 'basehash changed' errors when rm_work is being
inherited.
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current looping structure is confusing, simplify it a bit
to improve readability. Should be no functionality changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This function is a little obtuse, add more comments about what its
doing and why. Also combine some of the statements where possible
to improve clarity.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Once, there were do_setscene tasks but this hasn't been the case for years,
drop the old code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is a reworked version of the issue which 5479654eeaaa0f81bfff54ca49369c87f1658705
attempted to fix.
"""
Each time I build my image after the first, I end up with a
do_image_complete_setscene stamp file with an extra _setscene appended to
the name. Eventually, the filenames end up being so long that mv complains
and the build fails.
It looks like this behaviour was introduced when the special handling was
added for do_image_complete in 2ff9d40dc88d43567472218cf3d3faf414398c71.
"""
Instead of the original approach which broke do_package_setscene, add
an entry to explictly stop the stacking _setscene pieces on do_image_complete.
It's not straightforward to just move *do_image_complete* after the
*_setscene* pattern because do_image_complete stamps would then match
do_image*.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Whilst this fixes the do_image_complete_setscene append problem, it creates a
new problem since the code can no longer reach the *do_package_setscene.*
code block below it. This breaks builds as per [YOCTO #12765]. Revert this
change in search of a better fix.
This reverts commit 5479654eeaaa0f81bfff54ca49369c87f1658705.
|
|
Each time I build my image after the first, I end up with a
do_image_complete_setscene stamp file with an extra _setscene appended to
the name. Eventually, the filenames end up being so long that mv complains
and the build fails.
It looks like this behaviour was introduced when the special handling was
added for do_image_complete in 2ff9d40dc88d43567472218cf3d3faf414398c71.
So, let's ensure that the *_setscene* pattern is matched before anything
else so that any do_image_complete_setscene stamp file is always ignored
and the do_image_complete non-setscene stamp file is moved only once.
It's not straightforward to just move *do_image_complete* after the
*_setscene* pattern because do_image_complete stamps would then match
do_image*.
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is a race with rm_work when tasks run deltask do_build since
do_rm_work becomes a floating task. Handle this by injecting
the populate_sysroot/lic dependencies manually if the tasks returned
are empty. This depends on a change to fix bb.build.preceedtask() in
bitbake too.
[YOCTO #12365]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* since following commit:
commit 2ff9d40dc88d43567472218cf3d3faf414398c71
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Sun Jul 30 16:06:57 2017 +0100
image/rm_work: Promote do_image_complete to be more sstate like
all image rm_work tasks are failing with:
mv: 1.0-r0.do_image_complete.228730130ba309f85b73b72222ba11d7.raspberrypi3 and
1.0-r0.do_image_complete.228730130ba309f85b73b72222ba11d7.raspberrypi3 are the same file
it's because for $i 1.0-r0.do_image_complete.228730130ba309f85b73b72222ba11d7.raspberrypi3
there will be first SSTATETASK in $j do_deploy, so the sed call doesn't replace anything
It might be different order of SSTATETASKS in my builds (it might work only when
do_image_complete is the first one in the list), but here:
SSTATETASKS="do_deploy do_image_complete do_image_qa do_package do_package_qa do_package_write_ipk do_packagedata do_populate_lic do_populate_sdk do_populate_sdk_ext do_populate_sysroot"
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
do_bootdirectdisk and do_vmimg had been dropped by commit 929ba563:
[ image: Convert vmdk/vdi/qcow2 to strict CONVERSION_CMD types ]
Also drop the references to them and image-vm.
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We relied on the missing do_image_complete_setscene task to ensure the dummy
sstate tarball that was created would never be used. This lead to its own
issues and a better fix for SSTATE_SKIP_CREATION has now been merged.
We can therefore make do_image_complete look like a more standard sstate
task which means image generation doesn't keep rerunning when using rm_work.
We do need to turn do_image_complete's stamp into an sstate version to
handle this (it otherwise matches the do_image_* glob).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than requiring each user to handle this individually, handle
addto_recipe_sysroot in the core class. As well as preserving the
sysroot directory, this also ensures the stamp is preserved rather
than rerunning the task every time as currently happens.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some classes, for example populate_sdk_ext.bbclass, must be able to
trigger a full build of a recipe without also triggering
do_rm_work. They cannot depend on do_build anymore, because that would
trigger do_rm_work.
Instead, do_build_without_rm_work can be used. It has the exact same
dependencies as do_build, minus do_rm_work and do_rm_work_all.
This may also be useful in a test build of a recipe where one wants
to preserve the work directory without having to modify configuration
settings:
bitbake foobar:do_build_without_rm_work
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When rewriting the do_rm_work injection, do_rm_work_all had been
removed because it seemed obsolete, as do_build now always triggers
do_rm_work.
However, do_build does not get triggered for all recipes and thus
do_rm_work was not called for recipes that got built only
partially. For example, zlib depends indirectly on
zlib-native:do_populate_sysroot. Because of that dependency,
zlib-native got compiled, but do_rm_work was never called for it.
Re-introducing do_rm_work_all fixes that by making do_build depend on
do_rm_work_all, which then recursively depends on do_rm_work of all
dependencies. This has the unintended side-effect that do_rm_work then
also triggers additional work (like do_populate_lic) that normally
doesn't need to be done for a build. This seems like the lesser evil,
compared to an incomplete cleanup because it mostly enables the
lighter tasks after do_populate_sysroot.
The real solution would be to have two kinds of relationships: a weak
ordering relationship ("if A and B are enabled, A must run before B,
but B can also run without A") and hard dependencies ("B cannot run
unless A has run before").
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Having do_rm_work depend on do_build had one major disadvantage:
do_build depends on the do_build of other recipes, to ensure that
runtime dependencies also get built. The effect is that when work on a
recipe is complete and it could get cleaned up, do_rm_work still
doesn't run because it waits for those other recipes, thus leading to
more temporary disk space usage than really needed.
The right solution is to inject do_rm_work before do_build and after
all tasks of the recipe. Achieving that depends on the new bitbake
bb.event.RecipeTaskPreProcess and bb.build.preceedtask().
It can't just run in an anonymous function, because other anonymous
functions that run later may add more tasks. There's still such a
potential conflict when some future RecipeTaskPreProcess event handler
also wants to change task dependencies, but that's not a problem
now. Should it ever occur, the two handlers will have to know about
each other and cooperate to resolve the conflict.
Benchmarking (see "rm_work + pybootchart enhancements" on the OE-core
mailing list) showed that builds with the modified rm_work.bbclass
were both faster (albeit not by much) and required considerably less
disk space (14230MiB instead of 18740MiB for core-image-sato).
Interestingly enough, builds with rm_work.bbclass were also faster
than those without.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
By default, do_rm_work either skips recipes entirely (when listed in
RM_WORK_EXCLUDE) or removes everything except for temp.
In meta-swupd, virtual image recipes collaborate on producing update
data for the base recipe. Tasks running in the base recipe need some
information from the virtual images.
Those files could be passed via a new shared work directory, but that
scatters data in even more places. It's simpler to use the normal
WORKDIR and teach rm_work.bbclass to not remove the special output
with the new RM_WORK_EXCLUDE_ITEMS.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
We need to add do_write_qemuboot_conf to the list of tasks to be wiped as
otherwise the second time an image is built it will fail.
[ YOCTO #10758 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
We don't remove sigdata files, we also shouldn't remove sigbasedata files
as this hinders debugging.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Excluded removal of do_bootimg, do_bootdirectdisk and do_vmimg
timestamps to prevent unneeded rootfs rebuilds.
[YOCTO #10159]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
As rm_work is just cleanup it shouldn't starve more important tasks such as
do_compile of I/O, so use BB_TASK_IONICE_LEVEL to run the task in the idle
scheduler class.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
* otherwise kernel is rebuilt every single time and often it fails when
building external modules
[YOCTO #9352]
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After the separation of do_rootfs, some rootfs references need changing
to image_complete.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This prevents unneeded rootfs rebuilds if the metadata has not changed
while using rm_work.
Signed-off-by: Jate Sujjavanich <jatedev@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The do_shared_workdir task does leave behind the necessary information in
shared-work after it completes. We don't make this a "full" sstate task
however since that means tarring up and copying what is usually a large
amount of data which would be better extracted straight from the original
SCM.
The issue with rm_work occurs since it removes the do_shared_workdir stamp
meaning subsequent builds will add it back if they need to touch any kernel
modules for example. This ends up triggering a near enough complete kernerl
rebuild since if configure reruns, populate_sysroot has to rerun.
This change promotes the task to have a "setscene" variant but it doesn't use
any of the sstate class lifting to generate the sstate file. The sstate function
will therefore never get called since the sstate object will never exist.
We can add the task to the list of tasks rm_work promotes to a setscene variant
and unwanted rebuilds of the kernel should be avoided.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A previous change meant image/sdk recipes were removed unconditionally
by the class and did not respect RM_WORK_EXCLUDE. This fixes that
problem.
[YOCTO #7114]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Commands like bitbake X -c rootfs or bitbake X -c populate_sdk do not
trigger rm_work to clean up the directories afterwards since it
traditionally hooks onto do_build. This change means those two tasks now
clean up after themselves. We use the cleandirs function attribute to
handle this.
[YOCTO #6413]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When building over NFS2/3 server, removal of pseudo folders will fail in
some cases for there are files in it still used by pseudo daemon, thus
cause ".nfsXXXXX" files generated which can't be removed by clients. This
will lead rm_work task fo fail.
These failures could be safely ignored because ".nfsXXXXX" files would be
automatically cleared by NFS server when no clients keep opening them.
[YOCTO #4531]
[ CQID: WIND00412051 ]
Signed-off-by: Ming Liu <ming.liu@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
This really functions as a blacklist, not a whitelist, since we are
listing recipes to exclude. To avoid any possibility of confusion, since
this was a recent addition, rename the variable.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use RM_WORK_WHITELIST to inhibit rm_work per recipe. In this way,
one can use rm_work for the most of the recipes but still keep the
work area for the recipe(s) one is working on.
As an example, the following settings in local.conf will inhibit
rm_work for icu-native, icu and busybox.
INHERIT += "rm_work"
RM_WORK_WHITELIST += "icu-native icu busybox"
If we comment out the RM_WORK_WHITELIST line and do a rebuild, the
working area of these recipes will be cleaned up.
[YOCTO #3675]
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
* as described in
http://git.openembedded.org/openembedded-core/commit/?id=6107ee294afde395e39d084c33e8e94013c625a9
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
Typoes fixed: "enviroment", "editted", "spliting", "scheulder".
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* this makes fix for [YOCTO #1074] in bitbake 139b8a625818225c358a1b8363518d7ed6913188
much more usefull for people using rm_work
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
sstate code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to manipulate the stamps when removing WORKDIR to indicte that
tasks like compile or install can't just rerun. The most effective method
to do this is to convert the layout to match that which would have been the
case had the system been build from sstate packages.
For example, we'd task stamps like:
xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpack
and after rm_work, we'd have stamps of:
xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow
We also need to handle stamps in the form xxx-1.4.7-r3.do_package.MACHINE.TASKHASH
as used by some signature generators.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
Ross Burton
Luis
Chen Qi
Jose Quaresma
Richard Purdie
William A. Kennington III
Jacob Kroon
Ming Liu
Mike Crowe
Ross Burton
Martin Jansa
Ming Liu
Patrick Ohly
Joshua Lock
Ed Bartosh
Jate Sujjavanich
Ming Liu
Paul Eggleton
Robert P. J. Day
Richard Purdie