summaryrefslogtreecommitdiffstats
path: root/scripts
AgeCommit message (Collapse)Author
2022-02-20scripts: Add a conversion script to use SPDX license namesRichard Purdie
We're standardising on SPDX license names so it is overdue to change the ones in the LICENSE fileds to the SPDX values. Add a conversion script which makes this straightforward on the most part. Ultimately this allows the core code to be more efficient and not need to consult the mappings. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-16wic: rawcopy: Add support for packed imagesStefan Herbrechtsmeier
Add support for packed images to wic rawcopy handler do minimize disk usage in deploy directory and reuse of packed images between wic and swupdate. Add `unpack` to sourceparams to unpack an bz2, gz and xz archives. Example: part / --source rawcopy --sourceparams="file=core-image-minimal-qemu.ext4.gz,unpack" Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-16wic: partition: Support valueless keys in sourceparamsStefan Herbrechtsmeier
Accept valueless keys in sourceparams without equals sign (=) to match the comment and support Boolean entries. Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-12scriptutils: Fix handling of srcuri urlsRichard Purdie
Bitbake is now requiring there be a call to get_srcrev() when urls contain floating revisions. This breaks some recipetool workflows since it is running these queries to determine those revisions. Add a PV entry which triggers that call to avoid the errors. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-07runqemu: preload uninative libraries when host gl drivers are in useAlexander Kanavin
Some of the host distributions build the drivers in a way (RPATH/RUNPATH) that tricks uninative loader into loading pieces of the host libc, if the same pieces haven't been previously loaded by native binaries. Mixing the two libc versions leads to failures. This change ensures that the correct (uninative) versions are always in use. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-07qemu: replace a gtk wrapper with directly setting environment from runqemuAlexander Kanavin
The wrapper is executed by host bash, and host bash refuses to run when pseudo libc is preloaded via LD_PRELOAD (which is the case when gl is enabled). Only the fontconfig setting is carried over as local testing showed that only that is necessary for the gui to look ok nowadays; adjust further if necessary. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-07devtool: deploy-target: Remove stripped binaries in pseudo contextFlorian Amstutz
deploy-target may fail the second time with "pseudo abort" because devtool-deploy-target-stripped is deleted outside of pseudo's fakeroot context. Signed-off-by: Florian Amstutz <florian.amstutz@scs.ch> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05yocto-check-layer: check for duplicate layers when finding layersRoss Burton
detect_layers() is very greedy and if it recurses into poky or bitbake it will find the test suite layers, such as bitbake/lib/layerindexlib/tests/testdata/layer4. This is a dummy layer which claims to be openembedded-layer, so if the real openembedded-layer is a dependency then layer4 may be used instead, which will cause errors: initially because it's only compatible with Sumo, but later because it doesn't contain any recipes. Add a check that the set of layers we've found doesn't contain any duplicate collection names with different patterns, and abort if that is the case as the test will be non-deterministic. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05scripts/runqemu-ifdown: Don't treat the last iptables command as specialDaniel Müller
The runqemu-ifup script performs a bunch of setup steps that runqemu-ifdown attempts to undo later on. While a bunch of said setup operations are considered fatal should they fail, the iptables based NAT setup notably is not. The tear down procedure in runqemu-ifdown, however, has the iptables based tear down as the last operation, with the status of it determining the overall status of the script. Hence, if this step fails, the script is considered a failure overall. That is arguably inconsistent: If the NAT setup did not succeed, the tear down cannot succeed either. To ensure similarity of the two paths, let's not treat the last iptables tear down operation any special and allow it to fail the runqemu-ifdown script, but just ignore failures. Background: we have seen a NAT related setup problem on the ifup path (which didn't cause script failure), but then saw an issue bubbled up when this operation was meant to be undone on the ifdown path. Signed-off-by: Daniel Müller <muellerd@fb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05recipetool: Fix circular reference in SRC_URISaul Wold
When creating a new recipe.bb file for a binary, don't use BP which includes the version information, instead use BPN which is just the name base Package Name. Since PB is not specified, it takes the default: PV = "1.0+git${SRCPV}" But SRCPV is defined in terms of the SRC_URI, which leads to infinite recursion (traceback below). Here are the pertinent variables which cause the recursion: SRC_URI = "git://github.com/lvc/abi-dumper;protocol=https;subdir=${BP}" BP = "${BPN}-${PV}" PV = "1.0+git${SRCPV}" SRCPV = "${@bb.fetch2.get_srcrev(d)}" def get_srcrev(d, method_name='sortable_revision'): # ... trimmed scms = [] fetcher = Fetch(d.getVar('SRC_URI').split(), d) # ... trimmed [YOCTO #14040] Signed-off-by: Saul Wold <saul.wold@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05prelink: Drop support for itRichard Purdie
Prelink is being dropped by glibc in 2.36. It already causes issues with binary corruption, has a number of open bugs and is of questionable benefit without disabling load address randomization and PIE executables. We disabled it by default a while back but left people able to use it. We would be unable to maintain it alone without glibc support so remove the remaining pieces. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-03devtool: sdk-update: Remove an unnecessary \n from SSTATE_MIRRORSPeter Kjellerstedt
Since commit 044fb04d in bitbake (fetch2: Allow whitespace only mirror entries) there is no need to separate the entries in SSTATE_MIRRORS with "\n". Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-01yocto-check-layer: add debug output for the layers that were foundRoss Burton
When debugging weird yocto-check-layer output it is useful to know what the tool found when looking for layers. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-12recipetool/create_buildsys_python.py: less distutilsTim Orling
'distutils' is deprecated in Python 3.10 with removal in Python 3.12 (~October 2023). Replace 'distutils.command.build_py' with 'setuptools.command.build_py'. To avoid an AttributeError, we call super().__init__() which provides the missing 'distribution' attribute. However, for some reason, __init__() in 'setuptools.command.build_py.build_py' class requires a 'dist' positional argument which must be a 'Distribution' instance. It is not clear why 'distutils.command.build_py.build_py' class does not require this. There is still a check which decides to inherit setuptools3 vs distutils3 that will need to be refactored when we add pyproject.toml and setup.cfg support for more modern PEP 517 packaging. Once distutils3.bbclass is dropped, any recipe inheriting distutils3 will throw a parsing error. The plan is to move distutils*.bbclasses to meta-python. However if meta-python is not in bblayers, the parsing error would still occur. [YOCTO #14610] Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-12buildtools-installer: Update to use 3.4Richard Purdie
This updates buildtools to use the tarball from the 3.4 release which contains some bug fixes and is what the autobuilder currently uses for testing on older distros. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-11qemuboot/runqemu: fully form the ip= kernel parameterAlexander Kanavin
New systemd is actually parsing this in systemd-network-generator and fails if it is not fully formed. 'off' means 'static ip, do nothing': https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-05scripts: Update to use exec_module() instead of load_module()Richard Purdie
This is deprecated in python 3.12 and Fedora 35 is throwing warnings so move to the new functions. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-04scripts/buildhistory-diff: drop use of distutilsTim Orling
The use of distutils.version.LooseVersion to check for GitPython > 0.3.1 is not really needed anymore since any supported distribution has at least 1.0.0 (centos-7 via epel7, debian-9, ubuntu-16.04) If we want to reinstate this check, alternatives would be to require python3-packaging on all hosts and use packaging.version.Version or use an imported LooseVersion in bb.version. [YOCTO #14610] Signed-off-by: Tim Orling <timothy.t.orling@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-20recipetool: Separate licenses with & operatorStefan Herbrechtsmeier
Separate licenses with & operator since it should be satisfied most use cases and it is a reasonable assumption that all the licenses apply. Furthermore flat, split and sort the licenses to minimize license string changes. Separate package licenses with & operator: -LICENSE:${PN} = "MIT ISC" +LICENSE:${PN} = "ISC & MIT" Respect | and brackets in LICENSE: -LICENSE = "BSD-3-Clause & (ISC & | & MIT)" +LICENSE = "BSD-3-Clause & (ISC | MIT)" Sort licenses: -LICENSE = "MIT & BSD-3-Clause & ISC" +LICENSE = "BSD-3-Clause & ISC & MIT" Remove duplicates: -LICENSE = "MIT & ISC & MIT" +LICENSE = "ISC & MIT" Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-20recipetool: Sort output of guess_license function to be deterministicStefan Herbrechtsmeier
Sort the output of guess_license function by license file to be deterministic and to support self-testing. Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-17runqemu: additional setting to force software rendering with sdl 2.0.18Alexander Kanavin
Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-09recipetool: Set master branch only as fallbackStefan Herbrechtsmeier
The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)' sets the branch= parameter too early to master and thereby breaks the -B/--srcbranch option. ERROR: branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-09runqemu: check the qemu PID has been set before kill()ing itRoss Burton
If runqemu is killed, check that we have a valid PID for the qemu before sending a kill() to it. [ YOCTO #14651 ] Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-04yocto-check-layer: Relax README case checksJoshua Watt
Relaxes the case requirements for checks in the README file so that word like "Maintainer" and "Patch" are allowed Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-04yocto-check-layer: disregard checks if referencing another README fileJon Mason
It is permissible to have a layer's README reference another README. If this is the case, avoid other checks. Do this by checking for the word README (case insensitive) in the README file. This might be too permissive, but follows the pattern of looking for exact words (like "patch" or "maintainer") in READMEs. Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-03yocto-check-layer: Add additional README checksDhruva Gole
This adds: - Test case to review if the README file specifies the maintainer. - Test case to review if the README file specifies the word patch. - Test case to review if the README file contains an email address. [YOCTO #11131] Signed-off-by: Dhruva Gole <goledhruva@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-03recipetool: extend curl detection when creating recipesRoss Burton
If a configure.ac uses LIBCURL_CHECK_CONFIG it wants curl. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-03recipetool: handle GitLab URLs like we do GitHubRoss Burton
GitHub URLs are automatically transformed to git: fetches, so handle GitLab URLs too. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-01scripts/checklayer/common.py: Fixed a minor grammatical errorDhruva Gole
Signed-off-by: Dhruva Gole <goledhruva@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-26native/cross: Add ar wrapper for determinismJacob Kroon
Add a wrapper around ar calls for native/cross recipes. This wrapper adds the -D option so that deterministic archives are built for native/cross output. This improves the changes of hash equivalence matches and hence build artefact reuse. We don't need this in the target case since we compile binutils-cross with an option making this the default. We need a wrapper since we need to remove the "u" option and replace it with "D" but also allow things like "--version" to continue to work too. Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-26nativesdk: Handle chown/chgrp calls in nativesdk do_install tasksRichard Purdie
We disable the useradd code for nativesdk targets since we don't support postinstalls or multiple users in those cases. This means any usage of chown/chgrp inside do_install tasks won't work and would have to be conditional. Rather than require all recipes to do that, add intercepts of the calls and map those to root/root user/groups. We can't just ignore them as some calls are used to remove host contamination from the host user ID so they need to be made, just as root. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-24wic: support rootdev identified by partition labelSchmidt, Adriaan
We already support specifying the rootfs by PARTUUID. This adds general support for letting the kernel find the rootfs by PARTLABEL. Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-21wic: use shutil.whichMingli Yu
Use shutil.which to find the executable instead to silence the below warning: $ cat tmp/work/intel_x86_64-poky-linux/core-image-base/1.0-r5/temp/log.do_image_wic [snip] DEBUG: Executing shell function do_image_wic /path/layers/oe-core/scripts/wic:27: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives from distutils import spawn INFO: Creating image(s)... [snip] [RP: Added conversion for missed function reference] Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-21scripts/lnr: removeRoss Burton
lnr is a script that implements the same behaviour as 'ln --relative --symlink', as at the time of creation --relative was only available in coreutils 8.16 onwards which was too new for the older supported distros. Now, however, everyone has a new enough coreutils, so we can remove this script. All users of lnr should be replaced with ln --relative --symbolic. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-13runqemu: support rootfs mounted roAdrian Freihofer
Optionally allow to set QB_KERNEL_ROOT to e.g. "/dev/vda ro" to mount the rootfs reay-only in Qemu. Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-11scripts/lib/wic/help.py: Update Fedora Kickstart URLsJon Mason
The URLs describing Kickstart are no longer valid and do not redirect to the correct location. Update them with the correct location. Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-11scripts/oe-package-browser: Handle no packages being builtRichard Purdie
Give the user a proper error message if there aren't packages built, rather than a less friendly traceback. [YOCTO #14619] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-11scripts/oe-package-browser: Fix after overrides changeRichard Purdie
After the overrides change, the format of pkgdata changed and this usage of configparser no longer works. This change is a bandaid to make things work but the pkgdata format isn't very similar to ini files so this may need to be reimplmented in a better way in the long run. [YOCTO #14619] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-11convert-srcuri.py: use regex to check space in SRC_URIKai Kang
There may be none, one or more spaces including tab before backslash in SRC_URI. Use regex to check and update. It helps to avoid malformed uri such as recipe open-iscsi-user in meta-openstack: SRC_URI = "git://github.com/open-iscsi/open-iscsi.git;protocol=https ;branch=master \ And help to check more recipes such as concurrent-ruby in the same layer: SRC_URI = "git://github.com/ruby-concurrency/concurrent-ruby.git;protocol=https;tag=v1.1.6\ Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-10wic: Use os.rename instead of bb.utils.renameKhem Raj
This is not invoked with bitbake context as a result bb.utils is not visible when this function is called during image creation and builds fail e.g. NameError: name 'bb' is not defined Signed-off-by: Khem Raj <raj.khem@gmail.com> Cc: Joshua Watt <JPEWhacker@gmail.com> Cc: Devendra Tewari <devendra.tewari@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03runqemu: work without SERIAL_CONSOLES being definedJon Mason
Not all machine definitions need to have SERIAL_CONSOLES defined, but runqemu currently will fail with the following script error if not present: Traceback (most recent call last): File "/builder/meta-arm/poky/scripts/runqemu", line 1604, in main config.setup_final() File "/builder/meta-arm/poky/scripts/runqemu", line 1446, in setup_final self.setup_serial() File "/builder/meta-arm/poky/scripts/runqemu", line 1381, in setup_serial self.kernel_cmdline_script += ' console=%s' %entry.split(';')[1] IndexError: list index out of range To get around this issue, add a sanity check to runqemu to avoid the parsing of SERIAL_CONSOLES if empty. Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03scripts/convert-srcuri: Update SRC_URI conversion script to handle github ↵Richard Purdie
url changes Github are dropping support for git:// protocol fetching. Update the script to learn about corner cases found in the previous conversion and support remapping the github urls as needed too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30meta/scripts: Manual git url branch additionsRichard Purdie
Following the scripted conversion adding branches to git:// SRC_URI entries, add the remaining references, mainly in the selftests and recipetool. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30meta: Add explict branch to git SRC_URIsRichard Purdie
There is uncertainty about the default branch name in git going forward. To try and cover the different possible outcomes, add branch names to all git:// and gitsm:// SRC_URI entries. This update was made with the script added to contrib in this patch which aims to help others convert other layers. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30runqemu: unbreak non-gl displaysAlexander Kanavin
Correct two issues: 1. Looking for dri.pc is only needed when gl is enabled. 2. virtio-vga-gl works only when gl is enabled via -display, otherwise virtio-vga needs to be selected. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
2021-10-23recipetool: Simplify common source files skip in guess_licenseStefan Herbrechtsmeier
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-15sstate: Fixes for eSDK generation after zstd switchRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14scripts/autobuilder-worker-prereq-tests: jinja2 checkMichael Halstead
Ensure the jinja2 module is available during bringup. Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14sstate: Switch to ZStandard compressor supportHenry Kleynhans
This patch switches the compressor from Gzip to ZStandard for ssate cache files. Zstandard compression provides a significant improvement in decompression speed as well as improvement in compression speed and disk usage over the 'tgz' format in use. Furthermore, its configurable compression level offers a trade-off between time spent compressing sstate cache files and disk space used by those files. The reduced disk usage also contributes to saving network traffic for those sharing their sstate cache with others. Zstandard should therefore be a good choice when: * disk space is at a premium * network speed / resources are limited * the CI server can sstate packages can be created at high compression * less CPU on the build server should be used for sstate decompression Signed-off-by: Henry Kleynhans <hkleynhans@fb.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14runqemu: add DEPLOY_DIR_IMAGE replacement in QB_OPT_APPENDJon Mason
Add the ability to replace DEPLOY_DIR_IMAGE with that path in QB_OPT_APPEND. This allows for anything present in DEPLOY_DIR_IMAGE to be passed into the qemu parameters. This is especially useful if you want to run multiple flash images (as -bios only allows for one). Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>