summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/selftest/cases/reproducible.py
AgeCommit message (Collapse)Author
2023-06-15selftest/reproducible: Allow native/cross reuse in testRichard Purdie
We don't compare reproducibility of the native/cross components, only the target ones. With the long build times of rust-native, the test now takes crazy lengths of time so this tweak should allow us to reuse native/cross artefacts from sstate whilst still testing the target output is reproducible. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b494d83c639a877cefeb7cbab6d37195e492f059) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-03-06apr-util: Fix CFLAGS used in buildRichard Purdie
We need to use CFLAGS with the correct WORKDIR in them, replace those in the sysroot file with the ones appropriate to the current recipe. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 45edf189961aff1858be9bb7b63116073c0a0c10) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-11-10selftest/reproducible: add webkitgtk back to exclusion list for dunfellSteve Sakoman
We are still getting occassional failures to reproduce Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-11-04oeqa: reproducible: Fix test not producing diffsJoshua Watt
Diffoscope changed the --exclude-directory-metadata option to require an argument. Add a test to validate that diffoscope is functioning as expected to ensure that future upgrades do not unintentionally break the reproducibility tests. [YOCTO #14025] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ea8fbcb7978ce48d7a9a83143d09402329535f86) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-10-25selftest/reproducible: adjust exclusion list for dunfellSteve Sakoman
Signed-off-be: Steve Sakoman <steve@sakoman.com>
2021-05-13reproducible.py: add quilt-ptest and valgrind-ptestSteve Sakoman
Both don't seem to be reproducible with fedora 33 Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-04-19selftest/reproducible: Sort the unused exclusion listRichard Purdie
This makes comparisions between lists easier. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d2c52125d1cdc06c7e08d507ca68f3e4612a4314) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-04-19selftest/reproducible: track unusued entries in the exclusion listAlexander Kanavin
This helps with trimming down the list, and towards 100% reproducibility :) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit da7a173d7a01524229c8515326465968a845e96f) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-04-19selftest/reproducible: adjust exclusion list for dunfellSteve Sakoman
Signed-off-be: Steve Sakoman <steve@sakoman.com>
2021-04-19selftest/reproducible: add an exclusion list for items that are not yet ↵Alexander Kanavin
reproducible Hopefully over time this list will be reduced to an empty one. Non-reproducible excluded packages are not given to diffoscope and do not cause a failure, but still saved side-by-side with non-reproducible failing ones to make investigation easier. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 406bd0d48d8f90e2c836f7d3e204f21d5f13c833) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-04-19selftest/reproducible: enable world reproducibility testAlexander Kanavin
Add systemd and pam distro features, and commercial license flag to include more recipes into the world set. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ed8f94942c07784f9f6eef5c00b75aa9b398ea5e) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-03-11selftest/reproducible: Add ability to pull some objects from sstateRichard Purdie
When debugging why a single recipe doesn't reproduce, its a pain to wait for the world to rebuild from scratch. Update the selftest to allow this to be configured, for example you could set targets as ['perf'] and sstate_targets as ['virtual/kernel'] and then it should only be rebuilding perf in the test rather than things like the toolchain (parts of the kernel may be unavoiable as they're not in sstate). Can be run as: OEQA_DEBUGGING_SAVED_OUTPUT=/tmp/perf-diffoscope oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds to save diffoscope output. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 132a17d02f29711572e14a2f38a841323fbb6df6) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-03-04selftest/reproducible: Don't call sync between each file compareRichard Purdie
Calling sync between each file compare is horrible performance wise as we compare thousands of files. We don't care about IO latency here so disable. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 25f78abd8bbeb201fd9452e7983e015027954948) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-03-04oeqa: reproducible: Add more loggingJoshua Watt
The reproducible build tests can take a long time, so having more logging messages at various points in the build can help debug where the build is taking a long time. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6b792afe8759d62af8e713b86dad8f6721961a05) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2021-03-04oeqa: reproducible: Fix SSTATE_MIRRORS variableJoshua Watt
The SSTATE_MIRRORS variable was misspelled, which allowed the "clean" test build to pull from the mirror. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e42497bd84d0bb370a9f7b0448bff29f01fd1b0c) Signed-off-by: Steve Sakoman <steve@sakoman.com>
2020-02-19oeqa: reproducible: Include jquery in resultsJoshua Watt
Includes jquery in the output directory to make the diffoscope output easier to navigate Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-14oeqa: reproducible: Run diffoscope on saved outputJoshua Watt
If there are differing packages and they are being saved for review, automatically run diffoscope on them and include the output in the saved output. The output is currently done in HTML format since these are typically published on a webpage by the autobuilder. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-06oeqa/reproducible: Improve test output and ensure deb+ipk comparedRichard Purdie
Adding newline characters between the packages in the failure output massively improves readability. Also ensure to output ipk failures when there are deb failures by calling self.fail() at the end, else sometimes only partial differences are returned. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-03oeqa: reproducible: Do not strip packagesJoshua Watt
Do not strip packages when testing reproducible builds. In some cases, stripped data differs between builds, but then gets removed. However, the contents affect the generation of the GCC build-id, which then differs in the resulting ELF files, even though the data that caused this is no longer there. Inhibit stripping so that their causes can be evaluated. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-02reproducible: Allow configuration of saved debug outputRichard Purdie
If OEQA_DEBUGGING_SAVED_OUTPUT is set in the environment, use this location to store reproducibile build failure output. This aids debugging on the YP autobuilder in particular. Use a date in the directory name to make it easier to find failure output. Also clean up empty directories as they're unnecessary distracting noise. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-22oeqa/reproducible: Fix unset save_dir variableRichard Purdie
Previous refactoring broke the case where save_dir was set. Fix this. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-22oeqa/reproducible: Fix extra data reportingJoshua Watt
A typo was preventing the extra data about the reproducible build from being reported in the test results Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-22oeqa/reproducible: Add flag for building from sstateJoshua Watt
Adds a flag to control if the reproducible QA test should allow building from sstate or not. Building from sstate may not be reproducible depending on how the sstate is populated. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-18oeqa: reproducible: Test core-image-sato and core-image-full-cmdlineJoshua Watt
Adds core-image-sato and core-image-full-cmdline to the list of images that the reproducible build test builds. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-07oeqa: reproducible: Add option to capture bad packagesJoshua Watt
Adds an option that can be used to copy the offending packages to a temp directory for later evaluation. This is useful on the Autobuilder to investigate failures. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19oeqa/selftest/reproducible: test ipkgs tooRoss Burton
Now that opkg-build can build reproducible ipkgs, we can also add those to the test case. [ YOCTO #13513 ] Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07oeqa/selftest: Use extraresults on self instead of self.tcNathan Rossi
In order to take advantage of multiprocess execution of tests the extraresults must be passed through the TestResult. With changes to how oeqa/core handles test cases the extraresults attribute of the testcase is passed to the TestResult, with passing across process boundaries handled automatically. Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06oeqa: reproducible: Preserve sstate mirror for first buildJoshua Watt
Preserves the SSTATE_MIRROR variable for the first build in the reproducible test. This patch is intended to test the theory that using the Yocto autobuilder sstate mirror will still be reproducible. The autobuilder always does clean builds and never rebuilds recipes that were built previously, thus building with the mirror but not sharing tmpdir and sstate with previous builds should be reproducible. There is no guarantee that all sstate caches are populated from clean builds so this patch cannot be submitted to master, and I'm not bothering sending it to the mailing list. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30oeqa: reproducible: Do two clean buildsJoshua Watt
Perform two clean builds without sstate instead of one partial rebuild with sstate and one clean build without. There are some classes of reproducibility issues that this solves, and while we would like to resolve them in the long term the direction to do so is not currently clear. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30oeqa: Enable reproducible build testJoshua Watt
[YOCTO #13323] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30oeqa: reproducible: Record packages in test resultsJoshua Watt
Records the results of the reproducibility analysis in the JSON test results file. This makes it easier to do post-test analysis on the packages. [YOCTO #13324] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30oeqa: reproducible: Use subTest for packagesJoshua Watt
Runs each package class reproducibility test in a separate sub-test. This allows the other sub tests to still run in the event that one fails. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30oeqa: reproducible: Cleanup reproducible buildJoshua Watt
Cleans up the output from the reproducible build before building to ensure consistent results. Note that the output put is purposely left after around after the build so that non-reproducible packages can be diffed. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10oeqa: Add reproducible build selftestJoshua Watt
Adds an initial test for reproducible builds to the OE selftest. This initial test builds core-image-minimal using sstate, then does a clean build without sstate in another build directory, and finally does a binary comparison of the resulting package files between the two builds. The test is currently always skipped since it doesn't pass yet, but it can easily be enabled locally Signed-off-by: Joshua Watt <JPEWHacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-14 08:16:25 +0000