| Age | Commit message (Collapse) | Author |
|---|
|
Native recipes are not able to use ccache currently, due to a circular
dependency between ccache-native and cmake-native.
This affects also other native recipes, but not all, imposing a
limitation in build time optimizations.
The introduction of CCACHE_NATIVE_RECIPES_ALLOWED allows the user to specify
which native recipes should use ccache, keeping it disable by default for all
native recipes.
Signed-off-by: Angelo Ribeiro <Angelo.Ribeiro@criticaltechworks.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As stated in our top level license files, the license is MIT unless
otherwise stated. Add SPDX identifers accordingly. Replace older
license statementa with the standardised syntax. Also drop "All
Rights Reserved" expression as it isn't used now, doesn't mean anything
and is confusing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Where there isn't a copyright statement, add one to make it explicit.
Also drop editor config lines where they were present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since host's ccache is not reliable, so disable ccache for native recipes and
use ccache-native for other types of recipes. We need disable ccache for native
recipes is because ccache-native now depends on cmake-native which causes
circular dependencies, and it's not easy to break the circular.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit f5b29367af4d8e5daea5771264774aa49519f9a8.
Will use ccache-native which is more reliable.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed when build with buildtools-tarball:
$ bitbake linux-libc-headers
HOSTCC arch/x86/tools/relocs_common.o
ccache: error: Failed to create directory /run/user/0/ccache-tmp: Permission denied
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
ccache 4.x has hard dependencies on cmake-native (used as
build system) and zstd, which means inserting ccache-native
as DEPENDS into everything creates circular dependencies which
are impossible to break.
ccache 3.x did not have this problem as it used plain makefiles
and an in-tree copy of zlib.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed race issues when parallel build:
ccache: error: /path/to/ccache/i586-poky-linux/mmc-utils/ccache.conf: No such file or directory
ccache: error: /path/to/ccache/i586-poky-linux/mmc-utils/ccache.conf: No such file or directory
This is because we set CCACHE_DIR for earch recipe, and ccache will create a
ccache.conf for each CCACHE_DIR when CCACHE_CONFIGPATH is not set, but there
might be a race issue in parallel build:
ccache gcc file1.c
ccache gcc file2.c
If the two ccache processes use fopen(path, "w") to create ccache.conf at the
same time, the error would happen. Set CCACHE_CONFIGPATH to
meta/conf/ccache.conf can fix the problem, and we can add other configs to the
file when needed.
And also set cache_dir_levels to 1 (default is 2) since each recipe has a cache
dir, thus we don't have too many files in one dir.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
CCACHE_BASEDIR: ccache removes this from file path, so that hashes will be
the same in different build dirs.
CCACHE_TOP_DIR: Set it to a shared location for different builds.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some
hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
build failures (m4-native failed at do_configure, and others will also be
failed if I disable CCACHE for m4-native)
- native/nativesdk/cross/crosssdk recipes use host's ccache, but target uses
ccache-native, this may confuse user.
- The target recipes may use both host's ccache and ccache-native, this may
cause unexpected problems and be hard to debug. This is because ccache-native is
in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
rebuild target recipes, and then it would use hosttools/ccache, but the
previous ccache files were generated by ccache-native.
- Target recipes can't use ccache when no ccache is installed on the host:
CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
After refactored:
All types recipes (native, target and others) will use ccache-native except
ccache-native itself, host's cache won't be used any more. It is more
reliable now, which will work everywhere when ccache-native can be built.
And now we need use "CCACHE_DISABLE = '1'" to disable ccache for the recipe
rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
can't check whether CCACHE is set or not in anonymous function since it is
always set. Use CCACHE_DISABLE to disable it would be more clear.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
CCACHE_DISABLE was added to bitbake.conf in oe-core dd2bab (June 2012) because
autogen-native exports HOME=/dev/null during the build, which is then used by a
host ccache to construct the path to it's cache (/dev/null/.ccache) and this
fails.
However we now always export CCACHE_DIR to solve the same problem in a more
efficient way so CCACHE_DISABLE can be deleted.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
I suspect this was a typo and that TARGET_SYS makes more sense here. Its
also the only remaining user of MULTIMACH_HOST_SYS in OE-Core. Change it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As of ccache-3.3, ccache tries to ensure that the paths in the debug
information are always correct. It does this by including the current
directory in the hash if debug output is enabled. It includes support for
detecting remapping via a single -fdebug-prefix-map argument uses the
remapped directory in the hash instead.
The DEBUG_PREFIX_MAP in bitbake.conf remaps the source directory, target
sysroot and native sysroot separately which results in multiple
-fdebug-prefix-map arguments. Although ccache passes all these arguments
through to the compiler, it only enables the special behaviour described
above if the last one matches the current directory. (See
https://github.com/ccache/ccache/issues/163 )
Even if ccache did correctly honour each of the remapping arguments, the
hashes would still be different every time ${PV} or ${PR} change because
the default DEBUG_PREFIX_MAP contains maps to paths including them.
So it seems that for ccache to be of any use with this configuration,
CCACHE_NOHASHDIR needs to be set.
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Helped-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This patch is comparatively large and invasive. It does only do one thing, switching the
system to build using recipe specific sysroots and where changes could be isolated from it,
that has been done.
With the current single sysroot approach, its possible for software to find things which
aren't in their dependencies. This leads to a determinism problem and is a growing issue in
several of the market segments where OE makes sense. The way to solve this problem for OE is
to have seperate sysroots for each recipe and these will only contain the dependencies for
that recipe.
Its worth noting that this is not task specific sysroots and that OE's dependencies do vary
enormously by task. This did result in some implementation challenges. There is nothing stopping
the implementation of task specific sysroots at some later point based on this work but
that as deemed a bridge too far right now.
Implementation details:
* Rather than installing the sysroot artefacts into a combined sysroots, they are now placed in
TMPDIR/sysroot-components/PACKAGE_ARCH/PN.
* WORKDIR/recipe-sysroot and WORKDIR/recipe-sysroot-native are built by hardlinking in files
from the sysroot-component trees. These new directories are known as RECIPE_SYSROOT and
RECIPE_SYSROOT_NATIVE.
* This construction is primarily done by a new do_prepare_recipe_sysroot task which runs
before do_configure and consists of a call to the extend_recipe_sysroot function.
* Other tasks need things in the sysroot before/after this, e.g. do_patch needs quilt-native
and do_package_write_deb needs dpkg-native. The code therefore inspects the dependencies
for each task and adds extend_recipe_sysroot as a prefunc if it has populate_sysroot
dependencies.
* We have to do a search/replace 'fixme' operation on the files installed into the sysroot to
change hardcoded paths into the correct ones. We create a fixmepath file in the component
directory which lists the files which need this operation.
* Some files have "postinstall" commands which need to run against them, e.g. gdk-pixbuf each
time a new loader is added. These are handled by adding files in bindir with the name
prefixed by "postinst-" and are run in each sysroot as its created if they're present.
This did mean most sstate postinstalls have to be rewritten but there shouldn't be many of them.
* Since a recipe can have multiple tasks and these tasks can run against each other at the same
time we have to have a lock when we perform write operations against the sysroot. We also have
to maintain manifests of what we install against a task checksum of the dependency. If the
checksum changes, we remove its files and then add the new ones.
* The autotools logic for filtering the view of m4 files is no longer needed (and was the model
for the way extend_recipe_sysroot works).
* For autotools, we used to build a combined m4 macros directory which had both the native and
target m4 files. We can no longer do this so we use the target sysroot as the default and add
the native sysroot as an extra backup include path. If we don't do this, we'd have to build
target pkg-config before we could built anything using pkg-config for example (ditto gettext).
Such dependencies would be painful so we haven't required that.
* PKDDATA_DIR was moved out the sysroot and works as before using sstate to build a hybrid copy
for each machine. The paths therefore changed, the behaviour did not.
* The ccache class had to be reworked to function with rss.
* The TCBOOTSTRAP sysroot for compiler bootstrap is no longer needed but the -initial data
does have to be filtered out from the main recipe sysroots. Putting "-initial" in a normal
recipe name therefore remains a bad idea.
* The logic in insane needed tweaks to deal with the new path layout, as did the debug source
file extraction code in package.bbclass.
* The logic in sstate.bbclass had to be rewritten since it previously only performed search and
replace on extracted sstate and we now need this to happen even if the compiled path was
"correct". This in theory could cause a mild performance issue but since the sysroot data
was the main data that needed this and we'd have to do it there regardless with rss, I've opted
just to change the way the class for everything. The built output used to build the sstate output
is now retained and installed rather than deleted.
* The search and replace logic used in sstate objects also seemed weak/incorrect and didn't hold
up against testing. This has been rewritten too. There are some assumptions made about paths, we
save the 'proper' search and replace operations to fixmepath.cmd but then ignore this. What is
here works but is a little hardcoded and an area for future improvement.
* In order to work with eSDK we need a way to build something that looks like the old style sysroot.
"bitbake build-sysroots" will construct such a sysroot based on everything in the components
directory that matches the current MACHINE. It will allow transition of external tools and can
built target or native variants or both. It also supports a clean task. I'd suggest not relying on
this for anything other than transitional purposes though. To see XXX in that sysroot, you'd have
to have built that in a previous bitbake invocation.
* pseudo is run out of its components directory. This is fine as its statically linked.
* The hacks for wayland to see allarch dependencies in the multilib case are no longer needed
and can be dropped.
* wic needed more extensive changes to work with rss and the fixes are in a separate commit series
* Various oe-selftest tweaks were needed since tests did assume the location to binaries and the
combined sysroot in several cases.
* Most missing dependencies this work found have been sent out as separate patches as they were found
but a few tweaks are still included here.
* A late addition is that extend_recipe_sysroot became multilib aware and able to populate multilib
sysroots. I had hoped not to have to add that complexity but the meta-environment recipe forced my
hand. That implementation can probably be neater but this is on the list of things to cleanup later
at this point.
In summary, the impact people will likely see after this change:
* Recipes may fail with missing dependencies, particularly native tools like gettext-native,
glib-2.0-native and libxml2.0-native. Some hosts have these installed and will mask these errors
* Any recipe/class using SSTATEPOSTINSTFUNCS will need that code rewriting into a postinst
* There was a separate patch series dealing with roots postinst native dependency issues. Any postinst
which expects native tools at rootfs time will need to mark that dependency with PACKAGE_WRITE_DEPS.
There could well be other issues. This has been tested repeatedly against our autobuilders and oe-selftest
and issues found have been fixed. We believe at least OE-Core is in good shape but that doesn't mean
we've found all the issues.
Also, the logging is a bit chatty at the moment. It does help if something goes wrong and goes to the
task logfiles, not the console so I've intentionally left this like that for now. We can turn it down
easily enough in due course.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Removing the ccache directory as part of do_clean is unnecessarily
conservative and defeats many of the benefits of ccache.
The original justification for this behaviour was to avoid confusion
in the corner case that the ccache directory becomes corrupted.
However the standard approach for dealing with such highly unlikely
corner cases (ie manually removing tmp) would also recover from
corruption of the ccache directories, without the negative impact of
defeating ccache during normal development.
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
These have been deprecated for a long time, convert the remaining
references to the correct modules and prepare for removal of the
compatibility support from bitbake.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use weaker assignment for CCACHE_DIR to allow users to override it
if desired.
Signed-off-by: Wenzong Fan <wenzong.fan@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
This ensures a rebuild does really build from scratch when ccache is in use.
[YOCTO #2696]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The autogen-native built error on FC17:
ccache: failed to create /dev/null/.ccache
This is because the default gcc command of FC17 is a symlink to ccache,
so the ccache will always be used regardless to the setting of CCACHE,
ccache uses $HOME/.ccache as the CACHE_DIR by default, but autogen set
HOME=/dev/null, so the error happens.
Disable ccache explicitly if it is not enabled would fix the problem,
otherwise it would always use ccache regardless to the setting of CCACHE
on Fedora 17.
The ccache 3.1.7 has a bug, it would always create $CCCHE_DIR/.ccache
even CCACHE_DISABLE=1.
Unset CCACHE_DISABLE in ccache.bbclass, since ccache only checks whether
there is a CCACHE_DISABLE in the environment or not, it doesn't care about
its value, so we need unset it explicitly when enable ccache.
[YOCTO #2554]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Currently, ccache is used if it is present. When building from scratch it gives
no performance improvement and creates a ton of empty directories even when its
not in use.
This change moves ccache support to a bbclass file which the user can choose to
enable. This should make builds more determinstic and make it easier/clearer
to the end user when its being used and when it is not.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Angelo Ribeiro
Richard Purdie
Robert Yang
Alexander Kanavin
Ross Burton
Mike Crowe
Joshua Lock
Andre McCurdy
Wenzong Fan