| Age | Commit message (Collapse) | Author |
|---|
|
Add ${IMAGE_ROOTFS} and ${IMGDEPLOYDIR} to do_rootfs[dirs] and
do_rootfs[cleandirs], this ensures do_rootfs run from a clean
workspace, with this change, we can now remove two bb.utils.mkdirhier
lines from meta/lib/oe/rootfs.py.
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update internal variable names to improve the terms used.
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
```
File: '/home/andrew/src/openbmc/openbmc/meta/lib/oe/package_manager/ipk/manifest.py', lineno: 69, function: create_full
0065: output = pm.dummy_install(pkgs_to_install)
0066:
0067: with open(self.full_manifest, 'w+') as manifest:
0068: pkg_re = re.compile('^Installing ([^ ]+) [^ ].*')
*** 0069: for line in set(output.split('\n')):
0070: m = pkg_re.match(line)
0071: if m:
0072: manifest.write(m.group(1) + '\n')
0073:
Exception: TypeError: a bytes-like object is required, not 'str'
```
Change-Id: Ifefb13bfa22c766d20ab9f73f7abe5163b3df86f
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
```
File: '/home/andrew/src/openbmc/openbmc/meta/lib/oe/package_manager/ipk/manifest.py', lineno: 67, function: create_full
0063:
0064: output = pm.dummy_install(pkgs_to_install)
0065:
0066: with open(self.full_manifest, 'w+') as manifest:
*** 0067: pkg_re = re.compile('^Installing ([^ ]+) [^ ].*')
0068: for line in set(output.split('\n')):
0069: m = pkg_re.match(line)
0070: if m:
0071: manifest.write(m.group(1) + '\n')
Exception: NameError: name 're' is not defined
```
Change-Id: I769a2ab5e57c7b60598ea0390b576d707356db9d
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Layers could be located anywhere. The eSDK should work with them even if
they are not located in TOPDIR or in the same parent directory as
COREBASE.
For layers located in the same parent directory as COREBASE this preserves
the intent from the previous
copy_buildsystem: include layer tree during build structure creation
commit.
Related OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
On modern Power systems `uname -m` yields 'ppc64le' while the toolchain
knows the architecture as 'powerpc64le'. Provide a mapping from one to
the other to integrate with the existing architecture configuration
flags.
arch_to_rust_arch() only exists to map the OE *_ARCH variables before
any further processing, unlike arch_to_rust_target_arch() which is
specific to the internal triple handling of rust.
On Linux ppc64le systems the changes give the following config:
```
$ cat ./tmp/work/ppc64le-linux/rust-native/1.58.0-r0/targets/ppc64le-linux.json
{
"llvm-target": "powerpc64le-unknown-linux-gnu",
"data-layout": "e-m:e-i64:64-n32:64-v256:256:256-v512:512:512",
"max-atomic-width": 64,
"target-pointer-width": "64",
"target-c-int-width": "64",
"target-endian": "little",
"arch": "powerpc64",
"os": "linux",
"env": "gnu",
"vendor": "unknown",
"target-family": "unix",
"linker": "gcc",
"cpu": "generic",
"dynamic-linking": true,
"executables": true,
"linker-is-gnu": true,
"linker-flavor": "gcc",
"has-rpath": true,
"has-elf-tls": true,
"position-independent-executables": true,
"panic-strategy": "unwind"
}
```
Change-Id: Ief0c01189185d7d4da31d307270bec4e1de674ca
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want to use SPDX identifiers in LICENSE variables. There is now a
conversion script to make most of the translations. Add a list of
strings which have been replaced so we can show warnings to users
if they're still used anywhere.
Add checks to the package as insane check. This is currently a warning
by default but can be turned off or made an error as per the other standard
checks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This re-writes the INCOMPATIBLE_LICENSE checking code to replace
the WHITELIST_<lic> with
INCOMPATIBLE_LICENSE_EXCEPTIONS = '<pkg>:<lic> <pkg>:<lic> ...'
This initial change leaves most of the code structure in place,
but the code in base.bbclass needs to be re-written to make
the check more consistent around packages (PKGS) and not recipe
names (PN). This also is taking into account the changes for SPDX
licenses.
The aim is to provide a mode consistent variable where the variable
name is known and can easily be queried.
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is redundant (target arch is already in PN), and breaks
compiling a cross-canadian toolchain, as that needs populating the
sysroot with two different native-hosted toolchains built from
cross recipes. Inserting TARGET_ARCH allows only one or the other.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The previous commit introduced a small logic error. Fix the renaming issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update the comment to reflect new variable names
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There were few bugs in the _isInitialized() function which might trigger
git repo to be reinitialized and patches failing to apply.
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, SPDX SBOMs are only created for images. Add support for
SDKs.
Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Updates within bitbake now require this call to be triggered somewhere with
floating git source revs. Add the missing call. Issue can be reproduced with:
devtool check-upgrade-status dbus-wait
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is also covered in documentation since:
https://git.yoctoproject.org/yocto-docs/commit/?id=3f3e5574ac9801ad92940168b61b532e0bd53a80
[YOCTO 14605]
Signed-off-by: Daiane Angolini <daiane.angolini@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This was prompted by ltp's unreasonably long package_qa times; it has
a massive amount of executables and insane runs objdump for all of
them, serially.
This reduces the time from 4 minutes to 1m20s on my machine.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds the set helper for list properties to allow list members to be
replaced with a new list. This is necessary as it is (currently) the
only way to replace the default value for a list member.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Prelink is being dropped by glibc in 2.36. It already causes issues with
binary corruption, has a number of open bugs and is of questionable benefit
without disabling load address randomization and PIE executables.
We disabled it by default a while back but left people able to use it.
We would be unable to maintain it alone without glibc support so remove
the remaining pieces.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Native task outputs are directly run on the build system during the build
after being built. Even if the output of a native recipe doesn't change, a
change in one of its dependencies may cause a change in the output it
generates (e.g. rpm output depends on the output of its dependent zstd
library).
This can cause poor interactions with hash equivalence, since this
recipe's output-changing dependency is "hidden" and downstream tasks only
see that this recipe has the same outhash and therefore is equivalent.
This can result in different output in different cases and issues with
reproducible builds in parcular (e.g. rpm compression changes for the same
content).
To resolve this, unhide the output-changing dependency by adding it's
unihash to this tasks outhash calculation. Unfortunately, we don't know
specifically know which dependencies are output-changing, so we have to
add all of them.
[YOCTO #14685]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Extend runstrip() to accept additional argument to enable
sharing it with the kernel do_strip() so that
KERNEL_IMAGE_STRIP_EXTRA_SECTIONS can be passed.
Since is_elf() understands kernel modules there is no need to keep a
seperate list for kernmodules or hardcode the values to runstrip.
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
for git repos
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since honister host manifest stopped to generate, i.e.
manifest file is empty but all ipks/files into sdk is
ok.
Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If PATCHTOOL="git" has been specified but workdir is not git repo
bitbake fails to apply the patches with error message:
Command Error: 'git rev-parse --show-toplevel' exited with 0 Output:
fatal: not a git repository (or any of the parent directories): .git
Fix this by initializing the repo before patching.
This allows binary git patches to be applied.
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
distutils is going away and we have functionality in bitbake which can
handle these comparisions so switch to the bb.utils function.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Larger systems may have large numbers of cores, but beyond a certain
point they can't all be used for compiling: whilst purely
compute-intensive jobs can be parallelised to hundreds of cores,
operations such as compressing (needs lots of RAM) or compiling (lots of
I/O) don't scale linearly.
For example, the Marvel ThunderX2 has 32 cores, each capable of
executing four threads, and can be configured with two sockets, making
256 CPUs according to Linux. Zstd using 256 threads has been seen to
fail to allocate memory during even small recipes such as iso-codes.
Add a default cap of 64 CPUs to the cpu_count() method so that extreme
parallisation is limited. 64 is high enough that meaningful gains
beyond it are unlikely, but high enough that most systems won't be
effected.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When looking at logs involving thread pools it is useful if the threads
can be named.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Application can depend on several overlayfs mount points. Provide a
systemd unit application can depend on to make sure all overlays are
mounted before it is started to avoid any race conditions
Signed-off-by: Bruno Knittel <Bruno.Knittel@bruker.com>
Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Having annotations on relationship can provide additional information
about the relationship such as how it was derived.
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
If the function a ThreadedWorker is executing raises an exception, don't
use print() as that mostly disappears. Instead, output it to the logger.
This is done using bb.mainlogger.debug directly instead of bb.debug() as
this allows us to pass the exception instance directly, which is then
incorporated into the log stream.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Extract package_qa_write_error, package_qa_handle_error and
package_qa_add_message functions from insane.bbclass to lib/oe/qa.py and
drop the package_qa_ prefixes.
Update various bbclasses to use the new functions. No import is required
since base.bbclass puts oe.qa in OE_IMPORTS.
Stop requiring callers to manually track whether a fatal error has been
encountered via a "sane" flag. Instead replace the QA_SANE variable with
QA_ERRORS_FOUND and call oe.qa.exit_if_errors or
oe.qa.exit_with_message_if_errors at the end of each task.
Inspired by discussion resulting from
https://lists.openembedded.org/g/openembedded-core/message/156793 and
https://lists.openembedded.org/g/openembedded-core/message/156900
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want things to be reproduicble and the variable doesn't really change
much any more. Drop the remaining uses and make those code paths always
active.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reproducibility is here to stay and needs to be part of our default workflow.
Move the remaining code to base.bbclass so it is always a first class citizen
and it is clear people need to be mindful of it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To try and avoid parse/memory overhead of functions within bitbake,
move the bulk of the reproducibility functions to the function library.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow a user to validate sstate objects against a list of keys, instead
of just any known key in the user's keychain.
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since Python 3.8 visit_Num(), visit_Str() and so on are all deprecated
and replaced with visit_Constant. We can't yet remove the deprecated
functions until we require 3.8, but we can implement visit_Constant to
silence the deprecation warnings.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, installed packages are listed for images in image-info.txt, but
not for SDKs in sdk-info.txt. Add TOOLCHAIN_HOST_TASK and
TOOLCHAIN_TARGET_TASK to the output variables in sdk-info.txt.
Moreover, package output files for the SDK host are empty because
PKGDATA_DIR defaults to the target directory. Fix this bug and create a new
variable called PKGDATA_DIR_SDK which stores the correct path for the SDK
host package data.
Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
|
|
Use regex strings (rāā) to silence below deprecation warning [1]:
$ cat tmp/work/intel_x86_64-wrs-linux/linux-yocto/5.10.x+gitAUTOINC+917c420111_373c02c3ca-r0/temp/log.do_deploy
[snip]
/build/layers/oe-core/meta/lib/oe/packagedata.py:22: DeprecationWarning: invalid escape sequence \s
r = re.compile("(^.+?):\s+(.*)")
[snip]
[1] https://docs.python.org/3/library/re.html
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We hardlink some files into the build, such as licence files in
do_populate_lic tasks. Depending on the umask that the source tree
was checked out with, the group permissions would vary. This
results in inconsistent task outhashes.
Avoid this by ignoring the group/other bits unless we're under
pseudo context.
Bump the ABI numbers to ensure we don't see cache corruption from
earlier builds.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some files in the populate_sysroot tasks have hardcoded paths in them,
particularly if they are postinst-useradd- files or crossscripts.
Add some filtering logic to remove these paths.
This means that the hashequiv "outhash" matches correcting in more
cases allowing for better build artefact reuse.
To make this work a new variable is added SSTATE_HASHEQUIV_FILEMAP
which maps file globbing to replacement patterns (paths or regex)
on a per sstate task basis. It is hoped this shouldn't be needed
in many cases. We are in the process to developing QA tests which
will better detect issues in this area to allow optimal sstate
reuse.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit 2a76082363d189880613765ad339718e3614049d.
We have an issue where x86 host builds are not matching hashes with
aarch64 host builds. We'd expect that for a given target, the target
artefacts should work regardless of the host architecture, compiler
version etc. but this isn't happening and the hashes are differing.
This is due to issues from hash equivalence.
I believe the commit being reverted was added as a test and there were
other fixes at the time which resolved these issues.
As illustration of that, different gcc versions are not cauing issues
with hash equivalence. That should be similar to the aarch64 case vs.
x86-64 and hence if we're not seeing gcc verison issues, we also don't
need this special case. As such, revert it as we don't need it and it
is in fact breaking sstate reuse cross platform.
[YOCTO #14578]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds several comments to this file to explain the intention and how it
should be used
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is added to allow the create-spdx code to create annotations
that store values properly according to the SPDX Specification.
Initialy they will be used to track if a recipe is a native type.
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
collections.abc is available since 3.3 and doesn't need special handling.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The package task references WORKDIR at it's top level and we can't
easily make the timestamp for that determnistic due to writes to files
there and in other subdirs. We could try and force it to a specific value
but it is easier to just remove it from the package task, we don't need
it there or care about it in this case.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Repo-wide replacement to use newer variable to represent systemd
system unitdir directory.
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Licenses reported in the SPDX documents should be either:
A) A valid SPDX identifier cross referenced from the SPDX license
database
B) A "LicenseRef" to a license described in the SPDX document
The licensing code will now add a placeholder extracted license with
corresponding "LicenseRef" for any licenses that are not matched to the
SPDX database
Parenthesis in the license expression are now handled correctly
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds a class as a first attempt to create SPDX SBoM documents during the
build. This initial work was influenced by [meta-doubleopen][1],
although almost completely rewritten.
[1]: https://github.com/doubleopen-project/meta-doubleopen
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Ming Liu
Saul Wold
Andrew Jeffery
Daniel Wagenknecht
Richard Purdie
Alexander Kanavin
Peter Kjellerstedt
Pavel Zhukov
Andres Beltran
Daiane Angolini
Joshua Watt
Oleksiy Obitotskyy
Pavel Zhukov
Ross Burton
Vyacheslav Yurkov
Mike Crowe
Daniel McGregor
Mingli Yu
Robert P. J. Day