path: root/meta/lib/oe/copy_buildsystem.py
Commit message (Collapse)AuthorAgeFilesLines
* oe/copy_buildsystem.py: dereference symlinkRobert Yang2016-11-061-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | When there is a relative symlink in the layer, for example: symA -> ../out/of/layer/file symA will be invalid fater copied, it would be invalid from build time if it points to a relative path, and would be invalid after extracted the sdk if it points to a absolute py. Dereference symlink when copy will fix the problem. Use tar rather than shutil.copytree() to copy is because: 1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks: https://bugs.python.org/issue21697 And Ubunutu 1404 doesn't upgrade python3 to fix the problem. 2) shutil.copytree(symlinks=False) raises errors when there is a invalid symlink, and tar just prints a warning, tar is preferred here since the real world is unpredicatable 3) tar is faster than shutil.copytree() as said by oe.path.copytree() So use tar to copy. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* lib/oe/copy_buildsystem: fix building eSDK with indirect paths in BBLAYERSPaul Eggleton2016-09-141-2/+2
| | | | | | | | | | | | | | | | Indirect paths (e.g. ${TOPDIR}/../meta-something) do generally work if used in BBLAYERS in bblayers.conf. However, if you built an extensible SDK with this configuration then the creation of the workspace within the SDK using devtool in do_populate_sdk_ext failed. This is because the copy_buildsystem code was no longer correctly recognising that the core layer ("meta") was part of a repository (e.g. openembedded-core / poky) that should be shipped together - because of the indirection - and thus it was splitting out the meta directory, and a number of places in the code assume that the meta directory is next to the scripts directory. Use os.path.abspath() to flatten out any indirections. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/copy_buildsystem: fix merging sstate directories for eSDKPaul Eggleton2016-08-171-8/+12
| | | | | | | | | | | | | | | | | | | | | | | | When we don't have uninative enabled there's more merging to be done in the default configuration (SDK_EXT_TYPE = "full" which by default means SDK_INCLUDE_TOOLCHAIN = "1") and there are likely files that already exist in the sstate feed we're assembling, so we need to take care to merge the directory contents rather than just moving the directories over. Additionally we now only run this if uninative genuinely isn't enabled (i.e. NATIVELSBSTRING is different to the fixed value of "universal".) In the process of fixing this I discovered an unusual behaviour in os.rename() - when we're merging these feeds we're dealing with hard-linked sstate artifacts, and whilst os.rename() is supposed to silently overwrite an existing destination (permissions allowing), if you have the source and destination as hardlinks to the same file then the os.rename() call will just silently fail. As a result the code now just checks if the destination exists and deletes the source if so (since we know it will be the same file, we don't need to check in this case.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* classes/populate_sdk_ext: filter sstate within the extensible SDKPaul Eggleton2016-07-251-2/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use the new oe-check-sstate to filter the sstate artifacts shipped with the extensible SDK by effectively running bitbake within the produced eSDK and and getting it to tell us which tasks it will restore from sstate. This has several benefits: 1) We drop the *-initial artifacts from the minimal + toolchain eSDK. This still leaves us with a reasonably large SDK for this configuration, however it does pave the way for future reductions since we are actually filtering by what will be expected to be there on install rather than hoping that whatever cuts we make will match. 2) We verify bitbake's basic operation within the eSDK, i.e. that we haven't messed up the configuration 3) We verify that the sstate artifacts we expect to be present are present (at least in the sstate cache for the build producing the eSDK). Outside deletion of sstate artifacts has been a problem up to now, and this should at least catch that earlier i.e. during the build rather than when someone tries to install the eSDK. This does add a couple of minutes to the do_populate_sdk_ext time, but it seems like the most appropriate way to handle this. Should mostly address [YOCTO #9083] and [YOCTO #9626]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/populate_sdk_ext: allow including toolchain in eSDK on installPaul Eggleton2016-07-251-2/+3
| | | | | | | | | | | | | | | | | | | If we're to completely replace the standard SDK with the extensible SDK, we need to be able to provide the standard toolchain on install without doing anything other than installing it, so that you can install the SDK and then point your IDE at it. This is particularly applicable to the minimal SDK which normally installs nothing by default. NOTE: enabling this option currently adds ~280MB to the size of the minimal eSDK installer. If we need to reduce this further we would have to look at adjusting the dependencies and/or the sstate_depvalid() function in sstate.bbclass which eliminates dependencies, or look at reducing the size of the artifacts themselves. Implements [YOCTO #9751]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib: Update to explictly create lists where neededRichard Purdie2016-06-021-1/+1
| | | | | | | Iterators now return views, not lists in python3. Where we need lists, handle this explicitly. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib: Update to match python3 iter requirementsRichard Purdie2016-06-021-1/+1
| | | | | | | python3 standardises its use of iteration operations. Update the code to match the for python3 requires. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: don't copy .git when building the eSDKStephano Cetola2016-04-141-1/+1
| | | | | | | | | When creating an eSDK ensure that any .git directories are not included. [ YOCTO #9426 ] Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: add build-sdk subcommandPaul Eggleton2016-03-041-6/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a build-sdk command which is only available within the extensible SDK that builds a derivative extensible SDK. The idea is recipes in the workspace become a part of the new SDK - for example, this allows taking a vendor provided SDK, adding a few libs and then producing a new SDK with those included. When normally building the extensible SDK, the workspace is excluded; here we need to copy into the new SDK (renaming it in the process); the recipes' task signatures become locked and thus the sources are no longer needed, so they are removed along with the workspace bbappends which would interfere with the locked signatures. Additionally we need to just copy the configuration files (i.e. local.conf and auto.conf) rather than filtering and appending to them since that work has already been done when constructing the original SDK. The extra sstate artifacts from workspace recipes are also determined and copied into the new SDK in minimal mode (on the assumption that you won't set up a new sstate mirror). This reuses some code from build-image, so that needed to be generalised to allow that. Implements [YOCTO #8892]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* copy_buildsystem.py: Pass the nativelsb argument to gen-lockedsig-cacheRandy Witt2016-02-061-1/+8
| | | | | | | | | | | | If the nativelsb argument is not used, then create_locked_sstate_cache() can get collisions when moving the files from the input_sstate_cache to the output_sstate_cache. The specific case where this was encountered was when a "universal" nativelsb directory already existed in the input_sstate_cache. Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* copy_buildsystem: add ability to exclude layersChen Qi2016-02-041-0/+6
| | | | | | | | | | | | | | | In some cases, we may have some kind of download layers in BBLAYERS, so that we can set BB_NO_NETWORK to "1". This results in extremely large extensible SDK. And we actually don't need these download layers in the SDK. Add a new variable, SDK_LAYERS_EXCLUDE, to enable users to explicitly exclude some layers when generating the extensible SDK. [YOCTO #8878] Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* gen-lockedsig-cache: copy correct native sstate into ext SDKPaul Eggleton2016-01-221-2/+3
| | | | | | | | | | | | | | | | When constructing the sstate-cache directory for the extensible SDK, we were copying in any matching native sstate packages, and as the signature doesn't actually change when the distro changes (since NATIVELSBSTRING is just a path separator for the artifacts and is not part of the signature) we ended up copying duplicated packages when the distro changed e.g. upon host distro upgrade. Only search in the NATIVELSBSTRING-named subdirectory for native packages and the issue goes away. Fixes [YOCTO #8885]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/populate_sdk_ext: add option to bring in pkgdata for worldPaul Eggleton2016-01-221-2/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a variable SDK_INCLUDE_PKGDATA which you can set to "1" to include pkgdata for all recipes in the world target. There are a couple of uses for this: 1) If you use "devtool add" to add a recipe that builds something which depends on anything in world, the dependency can then be correctly mapped to the recipe providing it and that recipe can be added to DEPENDS, since we have the pkg-config and shared library dependency data within pkgdata. 2) You'll be able to search for these recipes and any files they package for the target with "devtool search" since that also uses pkgdata This of course assumes you've tailored world through EXCLUDE_FROM_WORLD to only include recipes you'd want built in your distro, but I think that's a reasonable assumption; failing that there is a WORLD_PKGDATA_EXCLUDE variable that you can set to exclude any recipes you don't want. Note that this patch relies on functionality implemented in a recent BitBake patch and will not work without it. Implements [YOCTO #8600]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* populate_sdk_ext: Change to include siginfo and non sstate task sigsRichard Purdie2016-01-111-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | Right now, the locked task hashes list for the extensible SDK locks down only the sstate tasks. Whilst asthetically pleasing, this gives two problems: * Half the task are left floating meaning checksum mismatches are a pain to debug * The later code which copies relavent data files out the sstate cache can't use any of this data. This patch modifies things so all the checksums are listed in the locked file. An exclusion of tasks probably makes more sense for the library function rather than an allowed list. The only sstate task being deliberaly excluded here was do_package so add in a function to explictly exclude those sstate object files. The net result of this that siginfo files for all tasks are included in the SDK, which means commands like "bitbake -S printdiff" now function. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/populate_sdk_ext: tweak reporting of workspace exclusionPaul Eggleton2015-12-011-2/+3
| | | | | | | | | | | | If you have a local workspace layer enabled when building the extensible SDK, we explicitly exclude that from the SDK (mostly because the SDK has its own for the user to use). Adjust the message we print notifying the user of this so it's clear that we're excluding it from the SDK, and scale it back from a warning to a note printed with bb.plain(). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
* lib/oe/copy_buildsystem: Don't expand BB_TASKDEPDATARichard Purdie2015-11-241-1/+1
| | | | | | | The value isn't a string so don't try and expand it. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Ross Burton <ross.burton@intel.com>
* copy_buildsystem: make sure bitbake directory is copiedQi.Chen@windriver.com2015-09-071-4/+3
| | | | | | | | | | The previous code assumes that bitbake/ directory is under the core layer. This is the case for Yocto project. But users might clone oe-core and bitbake separately. So we use bb.__file__ to locate the bitbake directory to make sure it's copied into the extensible SDK. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* copy_buildsystem.py: Add methods to copy shared state.Randy Witt2015-02-231-0/+32
| | | | | | | | | Added the helper functions necessary to copy the sstate from the current build, and generate the file to "lock" it. Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* copy_buildsystem.py: Add a way to copy buildsystem to a directory.Randy Witt2015-02-231-0/+70
This file provides a way to take bitbake and the layers in the current build and copy them to a target specified. Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>