| Age | Commit message (Collapse) | Author |
|---|
|
The dependencies for do_build became a little too minimal after the
removal of recrdeptask since "bitbake go" would not package go-runtime
despite it being in DEPENDS and the resulting package having a dependency
on it.
A reasonable compromise is probably rdeptask instead of recrdeptask
which is a lot lower overhead but makes the build target more useful
and importantly, lets world builds do what you'd expect them to.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reproducibility is here to stay and needs to be part of our default workflow.
Move the remaining code to base.bbclass so it is always a first class citizen
and it is clear people need to be mindful of it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When signing a package, we were resolving the `PKGV` version using the
original data store (`d`). However, since that store does not have the
package name in its `OVERRIDES` list the wrong version can be returned in
recipes that produce multiple packages. One such example would be
`external-arm-toolchain.bb`.
The above issue is fixed by using `localdata` instead of `d` when
resolving the needed variables.
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@verkada.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is a controversial change which removes the recursive dependencies
from the do_build target of packaging tasks of recipes.
Currently this means when you "bitbake <image>" or "bitbake <recipe>",
the packaging tasks run for all packaging backends enabled for all recipes
in the dependency chain. The same therefore then applies to images.
We don't actually need that, it is a convinience thing. Removing it
massively simplifies the task graph and causes much fewer tasks to execute
in many common scenarios. It also means less sstate is fetched for
example when building an image.
This means when building a recipe all package formats would be built
but when building an image, only the format used by the image would be
generated. This should be an improvement in most cases but some CI systems
may need to be explict about what they're building.
'bitbake XXX --runall build' would give the previous behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that 022 is the default in BB_DEFAULT_UMASK in bitbake.conf, we
don't need any of these task flags, clean up.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Switch do_populate_sdk for the ipk package manager to use a separate target
opkg config file and separate the lockfiles restricting do_rootfs and
do_populate_sdk from running in parallel.
This way if an image recipe includes a dependency to do_populate_sdk by
default then it will run in parallel to do_rootfs saving time compared to the
sequential execution.
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add PACKAGE_ADD_METADATA_IPK to IPKEXTRAVARS so it affects the sstate hash
of do_package_write_ipk.
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Several variables used by the task are not included in the hash, as highlighted
by recent changes to MAINTAINERS not causing rebuilds. Fix this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The recent opkg-utils change didn't update all the packages since its
marked as ABISAFE (and has to be due to update-alternatives).
Fix a minor import issue to avoid multiple imports of glob which
also causes packages to re-generate.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The opkg-utils change didn't trigger a full repackage due to its mention
in layer.conf for update-alternatives hash exclusion. Do it manually
to invalidate broken ipks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some distributions for various reasons (like for example mounting a
tmpfs over /var at runtime) can't use /var/lib to store the opkg
metadata, so a different path is required to have a functioning
package manager.
${localstatedir} can't be modified to something other than the
hardcoded value in bitbake.conf because other recipes depending on it
will fail to install.
So the only recourse, which is also the least invasive, is to allow
distros to overwrite the OPKGLIBDIR variable just like they are also
allowed to overwrite OPKGBUILDCMD.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
when building with opkg backend and huge packages e.g. chromium/llvm all
going in parallel, memory pressure causes xz to catapult with
do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory
since there are many tasks going on in parallel, xz adds to memory pressure
and it wants it all, put an upper limit for memory xz can use
We add a variable XZ_MAXRAM with 30% of RAM limit and can be customized
if builders have more memory one can set it like
XZ_DEFAULTS = "-M 0 -T 0"
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Before, threading was enabled via a patch to opkg-build. Now that
opkg-build provides a hook for extra arguments, use that.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current code had broken exception handling due to the use of a
"traceback" variable as well as an import. Use the new library code
for this instead which reduces code duplication and has fixed/improved
exception handling.
The chdir code can be dropped since any directory changes are in other
processes now so there is no need for it here and the code no longer
changes directory.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a dependency on xz-native and pass '-Z xz' to opkg-build to use xz instead
of gzip when building packages.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If PACKAGE_EXCLUDE is constructed using _append then it's possible
that the final value will contain only a space. Currently that
results in build failures due to an invalid opkg command line.
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Current implementation does not handle possible exceptions coming from child
processes, the latter responsible for creating packages. With the aim to have more
control, use pipes to communicate exceptions and stop package creation in case
of failure.
Helps to debug [YOCTO #12012].
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Redirecting stderr to stdout helps debugging issues, i.e instead of just
getting the return code, get also the error log from the pkg manger
This commit is in the way to figure out the root cause of [YOCTO #12012],
where dpkg-deb fails with a 2 return code and according to the man page,
there are multiple issues leading to the same code.
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Allow the creation of ipks to happen in parallel, making best use of resources
on multiprocessor systems.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is the potential for sensitive information to leak through the urls
there and removing it brings this into the behavior of the other package
backends since filtering it is likely error prone.
Since ipks don't appear to be generated at all if we don't set this, set
the field to the recipe name used (basename only, no paths). This avoids
information leaking. We may want to drop the field if opkg can allow that
at a future point but the recipe name is a suitable identifier for now.
Reported-by: Andrej Valek <andrej.valek@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Empty descriptions lines are set with a space following by a dot and
the multiline ones require a leading space. Also, for non-empty lines,
there is no need for formating with textwrap, so remove the logic for
the latter. The documentation for multiline description was taken from [1].
[1] https://web.archive.org/web/20100727133051/http://handhelds.org:80/moin/moin.cgi/BuildingIpkgs
[YOCTO #10677]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This uses more modern formatting to handle the lockfiles and control
file cleanup with try/finally, taking advantage of the previous
extra indentation.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This prepares the way to parallelise ipk generation and splits the iteration
over packages and the package generation into separate functions. Whitespace
indentation is unchanged deliberately and is fixed in a followup patch. There
should be no functional change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
opkg uses empty lines as separator for next package and if an ipk file was
packaged with empty lines in DESCRIPTION opkg won't be able to handle such ipk
file, this happens at execution time.
This commit will replace empty lines in DESCRIPTION with a '.' when generating
an ipk package to avoid this issue.
[YOCTO #10677]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The exception handling in this function seemed mildly crazy. Python will
given perfectly good or in several cases better information if we let its
standard traceback/exception handling happen. Remove the pointless code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if the dpkg-deb command fails you see an error message like this:
ERROR: nss-3.27.1-r0 do_package_write_ipk: opkg-build execution failed
ERROR: nss-3.27.1-r0 do_package_write_ipk: Function failed: do_package_ipk
which is pretty much useless. If we use subprocess.check_output, we see a
traceback and then:
Exception: subprocess.CalledProcessError: Command '<cmd>' returned non-zero exit status 1
Subprocess output:
<output>
which is much easier to debug from.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a new variable to allow markup of postinstall (and preinst)
script dependnecies on native/cross tools.
If your postinstall can execute at rootfs creation time rather than on
target but depends on a native tool in order to execute, you need to
list that tool in PACKAGE_WRITE_DEPENDS.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This sets a good example and avoids unnecessarily contributing to
perceived complexity and cargo culting.
Motivating quote below:
< kergoth> the *original* intent was for the function/task to error via
whatever appropriate means, bb.fatal, whatever, and
funcfailed was what you'd catch if you were calling
exec_func/exec_task. that is, it's what those functions
raise, not what metadata functions should be raising
< kergoth> it didn't end up being used that way
< kergoth> but there's really never a reason to raise it yourself
FuncFailed.__init__ takes a 'name' argument rather than a 'msg'
argument, which also shows that the original purpose got lost.
Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The dict.fromkeys() creates a dict without order, there might be a
problem when build the same recipe again, for example:
- First build of make:
Provides: es-translation, make-locale
- Second build of acl:
Provides: make-locale, es-translation
They are exactly the same Provides, but tools like "diff" doesn't think
so. Sort RPROVIDES will fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
opkg-build needs to be executed in the root of the package, so save and restore
the current directory so this task doesn't modify the state.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The syntax for octal values changed in python3, adapt to it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Implement ipk signing inside the sign_ipk bbclass using the gpg_sign
module and configure signing similar to how rpm does it. sign_ipk uses
gpg_sign's detach_sign because its functionality is identical to package
feed signing.
IPK signing process is a bit different from rpm:
- Signatures are stored outside ipk files; opkg connects to a feed
server and downloads them to verify a package.
- Signatures are of two types (both supported by opkg): binary or
ascii armoured. By default we sign using ascii armoured.
- Public keys are stored on targets to verify ipks using the
opkg-keyrings recipe.
Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If user specific parameters to opkg are set in local.conf, they are
rewritten in package_ipk.bbclass and ignored, instead append
package_ipk specific arguments to the user defined ones.
The change is needed, if a user has to pass an alternative path to a
temporary directory for opkg, e.g.
OPKG_ARGS = "--tmp-dir=${TOPDIR}/tmp-opkg"
The default /tmp directory may be unusable for do_rootfs task, for
example if there is no enough space or /tmp is mounted with noexec
mount option, then an alternative path allows to complete do_rootfs
and fix the problems like this:
ERROR: Unable to install packages.
...
sh: /tmp/opkg-5jPLag/run-postinsts-UsUtaI/preinst: /bin/sh: bad interpreter: Permission denied
sh: /tmp/opkg-5jPLag/base-files-4hFwQS/preinst: /bin/sh: bad interpreter: Permission denied
sh: /tmp/opkg-5jPLag/run-postinsts-UsUtaI/preinst: /bin/sh: bad interpreter: Permission denied
sh: /tmp/opkg-5jPLag/busybox-syslog-sJmfbw/preinst: /bin/sh: bad interpreter: Permission denied
...
Signed-off-by: Vladimir Zapolskiy <vz@mleia.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Changes to OVERRIDES (e.g. from changing MACHINE) should not change
the sstate signatures of do_package_write_*. Exclude the variable
explicitly in the package classes to avoid this.
(From OE-Core rev: 5826a9260138c437f87ba1a9f84d5c08442b997d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In some scenarios rpm needs version information from RPROVIDES. We can
add this to the metadata where needed however we need to stop it entering
the ipk/deb packages. This data is not needed due to the way opkg/dpkg
handle the data.
This patch ensures that data isn't used.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow globs in CONFFILES.
This patch changes the way of CONFFILES handling. After this change,
the CONFFILES can take the same form as FILES. That means, we don't
have to list a bunch of files for CONFFILES. It will just be expanded
like the FILES variable.
We don't assume default value for CONFFILES in OE. But distro vendors could
provide a default value for CONFFILES in their distro configuration file
like below.
CONFFILES = "${sysconfdir}"
In this way, files under /etc are treated as configuration files by
default. Of course, setting CONFFILES in recipes take precedence over
the CONFFILES. For example, if the recipe author decides that package A
should only treat files under ${sysconfdir}/default/ as config files,
he/she can write like this.
CONFFILES_A = "${sysconfdir}/default"
[YOCTO #5200]
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The SRC_URI may contain whitespace, but be otherwise empty. This can happen
in the case:
MYSRC = ""
MYSRC_arm = "file://myarm.patch"
SRC_URI += "${MYSRC}"
Unless we strip blank spaces, to determine if it is empty, we can end up
generating a "Source: " line which in invalid. This leads to the error:
invalid Source: field is speified in the generated CONTROL file
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch allows for an optional new layout for ipk feed directories which I've
called a 'hierarchical feed' and is based on how Debian pools package files. It
is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".
In the traditional feed layout, package files are placed in <outdir>/<arch>/.
This can lead to several thousand files existing in a single directory which is
often a problem if developers want to upload a package feed to a shared web
hosting provider. For example, in my case, listing files via FTP only shows the
first 2000 files, breaking my scripts which attempt to upload only new and
changed files via FTP.
In the hierarchical feed, package files are written to
<outdir>/<arch>/<pkg_prefix>/<pkg_subdir>, where pkg_prefix is the first letter
of the package file name for non-lib packages or "lib" plus the 4th letter of
the package file name for lib packages (eg, 'l' for less, 'libc' for libc6).
pkg_subdir is the root of the package file name, discarding the version and
architecture parts and the common suffixes '-dbg', '-dev', '-doc', '-staticdev',
'-locale' and '-locale-*' which are listed in meta/conf/bitbake.conf.
This change relies on recent patches to opkg-utils which support hierarchical
package feeds.
Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The package_write task was previously removed. Remove a remaining superfluous
reference to it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The reasons this task was introduced are lost in the mists of time. It
allowed for the a single "package_write" task instead of spelling out
the explicit package backends, however in all but one case we do that
anyway.
As such as might as well give in and delete the task, converting that
single reference into explicit dependencies.
This gives bitbake a bit less work to to when processing the runqueue
since there are less tasks (but more dependencies in some cases).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Package indexing is done in python and package-index.bb uses the new
routines.
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The default value for HOMEPAGE of "unknown" has been in place since the
early OE-Classic days, but it doesn't really make sense - "unknown" is
not a valid URL and it just means we have to explicitly check for this
hardcoded string if we're displaying the value in some form of UI, such
as Toaster.
This has required some changes to the packaging classes as they
previously did not expect the value to be blank.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This commit cleans up the functions that were ported to python.
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
|
Richard Purdie
Kenfe-Mickael Laventure
Michael Ho
Adrian Ratiu
Khem Raj
Alejandro del Castillo
Ross Burton
Andre McCurdy
Leonardo Sandoval
Mariano Lopez
Joshua Lock
Ulf Magnusson
Robert Yang
Ioan-Adrian Ratiu
Vladimir Zapolskiy
Chen Qi
Mark Hatle
Paul Barker
Laurentiu Palcu
Paul Eggleton