summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/runtime
AgeCommit message (Collapse)Author
2022-03-13oeqa/runtime/ping: Improve failure message to include more detailRichard Purdie
When the ping test fails due to a timeout we only get limited debug information. Tweak the code to improve that in case it sheds any light on intermittent failures. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20core-image-testmaster: Rename to core-image-testcontrollerRichard Purdie
Rename the image, the test controller class/code/module and the underlying image sentinel file to all match the controller terminology. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05weston: upgrade 9.0.0 -> 10.0.0Alexander Kanavin
Release announcement with changes: https://lists.freedesktop.org/archives/wayland-devel/2022-February/042103.html Upstream has deprecated both weston-launch and fbdev backend, so let's not delay the inevitable and find replacements. Fbdev can be replaced by passing --use-pixman to drm backend; this will bypass the opengl paths and use CPU for rendering. Apply where GL is too slow or unavailable. weston-launch can be replaced by starting weston directly, with a seat management daemon for support. This is provided either by systemd, or on systemd-less systems, by seatd. The sysvinit startup scripts and tests have been rewritten accordingly. Bonus fix: under sysvinit weston now starts under weston user as it should, and not under root. Upstream discussion: https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/725 License-Update: copyright years Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05oqea/runtime/oe_syslog: Improve testRichard Purdie
We're seeing failures on the autobuilder where the log message isn't present in the log. This could just be a race in the compound command but it is hard to say due to the poor log output from the test. Splitting the command into two sections should add more of a delay whilst ssh reconnects and may well fix the issues but this change should also improve the log output in the failure case too so if the failure isn't solved, we should be better able to debug it. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-01oeqa/runtime/stap: improve systemtap testRoss Burton
Split the test up into compile and execute phases, as the stap binary is known to be quite memory-hungry and this can result in the probe being unable to allocate enough memory for the buffers it needs. If the test fails, dump the dmesg as any useful messages will be there. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-20logrotate: update 3.18.1 -> 3.19.0, drop rotate-across-filesystems patchesAlexander Kanavin
Upstream review and request to address it got no reaction from the author, and the patches are an ongoing rebase burden, so if someone needs this feature, please complete the upstreaming work first. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-19oeqa/runtime/stap: increase buffer sizeRoss Burton
We're seeing random failures in the SystemTap tests in qemuarm on kernel 5.10. This might be related to the buffer between user and kernel space being too small, so explicitly set the size. If this cures the problem it should be considered a workaround and not the solution. [ YOCTO #14673 ] Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-15oeqa/runtime/stap: rewrite testRoss Burton
There's no need to copy files to the target when stap can take the short probe directly. Add more has-package decorators for the kernel source and GCC symlinks. Remove the test dependencies as it's not a hard dependency. Change the probe to print a message with some minimal logic, and verify that the message was correctly constructed in the output. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-11parselogs: add a couple systemd false positivesAlexander Kanavin
New systemd has changed the phrasing when skipping things, with unfortunate use of 'failed': [ 1.623667] systemd[1]: Journal Audit Socket was skipped because of a failed condition check (ConditionSecurity=audit). [ 1.688258] systemd[1]: Load Kernel Modules was skipped because all trigger condition checks failed. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-17rust-hello-world: test at runtimeAlexander Kanavin
This adds a smoke check for whether the rust toolchain actually produces working executables across a range of architectures. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-29go-helloworld: test at runtimeAlexander Kanavin
This adds a smoke check for whether the Go toolchain actually produces working executables across a range of architectures. Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-29oeqa/parselogs: Fix quotingRichard Purdie
Fix deprecation warnings about invalid escape sequences. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-23oeqa: fix warnings for append operators combined with +=Yi Zhao
Signed-off-by: Yi Zhao <yi.zhao@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-28oeqa/runtime/parselogs: modified drm error in common errors listTeoh Jay Shen
Changed the following line from: [drm] Cannot find any crtc or sizes - going 1024x768 > [drm] Cannot find any crtc or sizes This will expand the coverage of the failure to also cover the case when fallback size is not set. Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-23oeqa/runtime: search sys.path explicitly for modulesRoss Burton
The controller module loading code needs to be told what directories to search for modules via the target_modules_path keyword argument, which is set to BBPATH. However, as the actual module loading is done via importlib this relies on the paths being on sys.path, which it is as base.bbclass puts each layer's lib/ in sys.path. Simplify the code by removing this indirection, and simply search sys.path directly. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-23oeqa/runtime: load modules using importlibRoss Burton
Instead of using __import__() which is low-level and discouraged, use importlib. Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-11rpm: update 4.16.1.3 -> 4.17.0Alexander Kanavin
The flagship features are migration from bdb to sqlite and zstd support, both are enabled and taken into use. The relationship and upstream preference between sqlite and ndb formats isn't quite clear. Lua is now a hard dependency. Added packageconfig option for r/o support for bdb (that doesn't need bdb itself), but not enabled it as upstream marks it EXPERIMENTAL in capital letters. Drop sed adjustment for a file that is not anymore installed. Adjust oeqa test to check for sqlite database instead of bdb. Drop 0001-Fix-build-with-musl-C-library.patch (nss support removed upstream) 0001-rpm-rpmio.c-restrict-virtual-memory-usage-if-limit-s.patch (difficult to undersand and rebase; obsolete with the move to zstd) 0011-Do-not-require-that-ELF-binaries-are-executable-to-b.patch (upstream made the same change) Portions of 0001-tools-Add-error.h-for-non-glibc-case.patch dropped (upstream moved the files to a separate component). Added 0001-docs-do-not-build-manpages-requires-pandoc.patch to avoid pandoc dependency. Added 0001-build-pack.c-do-not-insert-payloadflags-into-.rpm-me.patch to restore reproducibility when compression thread amount varies between hosts. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-03oeqa/runtime/parselogs: Make DVD ata error apply to all qemux86 machinesRichard Purdie
This log checking fix is needed for both qemux86 and qemux86-64 so move to the common section. [YOCTO #14528] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-27parselogs.py: ignore intermittent CD/DVDROM identification failureBruce Ashfield
We don't use the CD/DVD ROM drive in any of our tests, but it periodically fails discovery and that leads to a QA error: [ 6.403477] ata3.00: failed to IDENTIFY (I/O error, err_mask=0x4) The only way to disable the optical ROM drive in qemu is to use the '-nodefaults' option, which disables the CDROM (among other things). We can't be sure that none of our tests, or extended users are relying on default devices, so using that option is more of a risk than adding the message to our ignore list. To date, no one has sent a patch to just disable the optical drive (either in qemu or the BIOS), but that is something we could consider in the future. [YOCTO #14528] Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-13oeqa/runtime/cases: make date.DateTest.test_date more reliableAlexandre Belloni
The test uses the broken out time and can only handle about 59s of delay, use a UNIX timestamp to allow for up to a 300s delay. [YOCTO #14463] Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-06oeqa/runtime/cases/ptest: Increase test timeout from 300s to 450sRichard Purdie
Some tests such as lttng-tools are marginal and timing out on the autobuilder with the current 300s default. Increase to avoid this noise in the ptest failures list. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30Convert to new override syntaxRichard Purdie
This is the result of automated script conversion: scripts/contrib/convert-overrides.py <oe-core directory> converting the metadata to use ":" as the override character instead of "_". Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-29parselogs.py: qemuarm should be qemuarmv5Jon Mason
All of the errors being masked off for qemuarm are legacy from before the migration of qemuarm to qemuarmv5. Rename the machine to that to allow for qemuarmv5 to pass parselog test. Light testing shows no errors in dmesg for qemuarm. Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-20parselogs.py: ignore rndr initialization failureMingli Yu
After rng-tools upgraded to 6.13, the RNDR instruction added as an entropy source [1]. But for some cpu which doesn't support rndr, there comes below warning message: # systemctl status rngd [snip] Jul 20 06:48:07 qemuarm64 rngd[163]: [rndr ]: No HW SUPPORT Jul 20 06:48:07 qemuarm64 rngd[163]: [rndr ]: Initialization Failed [snip] Actually the failed message doesn't matter as it only indicates one entropy source rndr fails to initialize and won't affect rngd function, so ignore the failure message to fix below error during do_testimage. NOTE: ====================================================================== NOTE: FAIL: test_parselogs (parselogs.ParseLogsTest) NOTE: ---------------------------------------------------------------------- NOTE: Traceback (most recent call last): File "/build/layers/oe-core/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/build/layers/oe-core/meta/lib/oeqa/runtime/cases/parselogs.py", line 382, in test_parselogs self.assertEqual(errcount, 0, msg=self.msg) AssertionError: 1 != 0 : Log: /build/tmp/work/qemuarm64-wrs-linux/wrlinux-image-std/1.0-r5/target_logs/daemon.log ----------------------- Central error: 2021-06-23T20:44:18.374494+00:00 qemuarm64 rngd[162]: [rndr ]: Initialization Failed Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-28oeqa/runtime/rpm: Drop log message counting test componentRichard Purdie
This test is flawed since multiple parts of the system can write to the log and we obtain different numbers of log messages depending on factors we can't control. Drop the log testing component of the test. [YOCTO #12465] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18oeqa/runtime/cases: Only disable/enable for current bootWes Lindauer
Previously doing a stop/start worked, but using a disable/enable does not work on a read-only rootfs. Add a --runtime flag to systemctl so that systemd only modifies the current boot and does not attempt to write to the filesystem. This also keeps the test from making a permanent (one could argue policy) change to the running system being tested. i.e. What if the image being tested had intentionally disabled the timesyncd service in preference to using chrony or ntpd? The test shouldn't assume that the user wants the timesyncd service enabled. Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18parselogs: ignore floppy error on qemu-system-x86 at boot stageYanfei Xu
We can disable floppy drive by BIOS on a hardware, but an empty floppy drive is connected by default on qemu-system-x86. Linux usually detect the device and modprode the matched floppy.ko at the boot stage. Due to we don't specify a floppy deivce in qemu boot arguments, then the errors about floppy reading comes out. It is harmless and normal, so we could ignore this error message on qemux86. Seen if kernel-modules is included in the image which pulls in the relavent kernel module. https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg01402.html Signed-off-by: Yanfei Xu <yanfei.xu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18oeqa/runtime: space neededJon Mason
Messages are currently being printed as: Test requires dropbear, oropenssh-sshd to be installed but should be Test requires dropbear, or openssh-sshd to be installed Adding the space after the 'or' corrects this. Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11parselogs: Allow expected kernel messages for qemuppc64Khem Raj
pseries machine emulation ends up with these three messages in kernel logs which are harmless, so ignore them for now https://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg5600561.html https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1827162 Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-06oeqa/weston: correctly run wayland-info when systemd is in useAlexander Kanavin
Under systemd weston's socket is in /run, not in /run/user/0. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-27oeqa/pam: Need shadow installed for the testsKhem Raj
Distros may not use shadow for login manager secondly login --help returns 1 only with shadow provided login utility, with busybox it returns normal 0, the testcase clearly assumes shadow being part of image, therefore add needed checks to ensure the pre-requisites are met Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-19df.py: Add feature check for read-only-rootfsWes Lindauer
If the target is using a read-only rootfs, the available space on '/' will be zero. This will cause the test to incorrectly fail and skipping seems appropriate in this case. Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-19oeqa/runlevel : add test for runlevelsTeoh Jay Shen
This test mimic the boot_from_runlevel_3 and boot_from_runlevel_5 test cases from oeqa/manual/bsp-hw.json. The boot_from_runlevel_3 and boot_from_runlevel_5 manual test cases should be remove from oeqa/manual/bsp-hw.json if this patch get merged. Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-03oeqa/usb_hid.py : add test to check the usb/human interface device status ↵Teoh Jay Shen
after suspend state This test mimic the Test_if_usb_hid_device_works_well_after_resume_from_suspend_state manual test case from oeqa/manual/bsp-hw.json. Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-28oeqa/ethernet_ip_connman : add test for network connectionsTeoh Jay Shen
This test mimic the ethernet_static_ip_set_in_connman and ethernet_get_IP_in_connman_via_DHCP test case from oeqa/manual/bsp-hw.json. The ethernet_static_ip_set_in_connman and ethernet_get_IP_in_connman_via_DHCP manual test case should be remove from oeqa/manual/bsp-hw.json if this patch get merged. Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-17oeqa/ptest: print a warning if ptests failedAlexander Kanavin
This allows spotting ptest regressions without having hard ptest failures (for that full ptest stability should be achieved). Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-31oeqa/suspend : add test for suspend stateTeoh Jay Shen
This test case is checking the command and LAN device behaviour before and after suspend state. The Test_if_LAN_device_works_well_after_resume_from_suspend_state and standby manual test cases from oeqa/manual/bsp-hw can be replace by this runtime test. Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-21oeqa/terminal : improve the test caseTeoh Jay Shen
-Improve this test case to fulfill the requirements of replacing the click_terminal_icon_on_X_desktop manual test case from oeqa/manual/bsp-hw : 1) verify that the terminal window is working without problem 2) verify that there's only 1 terminal window is launched Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-11wayland-utils: introduce a recipeAlexander Kanavin
wayland-utils contains wayland-info utility which deprecates and replaces weston-info from weston. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-11weston-init: correctly start under systemdAlexander Kanavin
Several issues are addressed: 1. weston requires pam when starting under systemd 2. systemd was attempting to launch weston twice (from sysvinit script and from systemd unit file) which caused confusion and errors. 3. runtime test should stop/start weston via systemd only if systemd actually controls system startup, not merely when systemd is present. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-28oeqa/runtime : add test for RTC(Real Time Clock)Teoh Jay Shen
This test is checking the functionality of the RTC(Real Time Clock). The Check_if_RTC_(Real_Time_Clock)_can_work_correctly manual test case from oeqa/manual/bsp-hw can be replace by this runtime test. Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-14oeqa/runtime: add test for matchbox-terminalTeohJayShen
This test is checking that the terminal application is able to run. The click_terminal_icon_on_X_desktop manual test case from oeqa/manual/bsp-hw can be replace by this runtime test. Signed-off-by: TeohJayShen <jay.shen.teoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-14oeqa/weston: Fix tests to run with systemdKhem Raj
Currently, weston tests fail when using systemd, fix it by providing transient unit file and inject it via systemd-run, which generates a service file automatically and launches another weston instance to test if it can launch a nested instance. Use systemctl stop to end the service and cleanup, instead of brutal kill Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-28oeqa/multilib: un-skip the connman testAlexander Kanavin
It was previously skipped as it was expecting plain connman to not be in the image, and for core-image-sato connman is always there. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-28oeqa/multilib: rework to use readelf from the hostAlexander Kanavin
It is difficult and error-prone to ensure binutils gets installed into target images where this test may run; on the other hand readelf is always present on the host, as it is a part of HOSTTOOLS. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-28oeqa/x32lib: rework to use readelf from the hostAlexander Kanavin
It is difficult and error-prone to ensure binutils gets installed into target images where this test may run; on the other hand readelf is always present on the host, as it is a part of HOSTTOOLS. Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-18buildcpio.py: Apply patch to fix build with -fno-commonKhem Raj
This sed expression implements the needed patch to fix compilation with -fno-common, hopefully this patch will get included in 2.14 release and we can remove this operation Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-06ltp: make copyFrom scp command non-fatalMatthew
[YOCTO #13802] Make the scp failure non-fatal so the ltp tests continue to run and the rest of the logs will be available to see afterwards. Signed-off-by: Mingde (Matthew) Zeng <matthew.zeng@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-03logrotate.py: fix testimage occasionally failureChangqing Li
testcase test_systemd_failed occasionally failed with below error: Failed to start Rotate log files. logrotate.service: Failed with result 'exit-code'. logrotate.service: Main process exited, code=exited, status=1/FAILURE error: stat of /var/log/logrotate_test failed: No such file or directory error: logrotate_test:1 lines must begin with a keyword or a filename (possibly in double quotes) above failure caused since testcase test_logrotate_wtmp add /etc/logrotate.d/logrotate_test, which need /var/log/logrotate_test, but there is no such file. so when logrotate.service is triggerd by logrotate.timer after testcase test_logrotate_wtmp is runned, the testcase test_systemd_failed will fail. these 3 lines are useless, so remove them to fix above problem. Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-23oeqa/runtime/cases/ptest: Make output content path absoluteAndrej Valek
The output content is created in current directory, because json content has no defined absolute path to WORKDIR as in bitbake. Signed-off-by: Andrej Valek <andrej.valek@siemens.com> Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>