Execution time varies widely with the existing tests. Provides a way to
produce a summary of the time taken for each test, along with a
histogram.
This is enabled with the --timing flag.
Enable it for sandbox in CI.
Example:
Duration : Number of tests
======== : ========================================
<1ms : 1
<8ms : 1
<20ms : # 20
<30ms : ######## 127
<50ms : ######################################## 582
<75ms : ####### 102
<100ms : ## 39
<200ms : ##### 86
<300ms : # 29
<500ms : ## 42
<750ms : # 16
<1.0s : # 15
<2.0s : # 23
<3.0s : 13
<5.0s : 9
<7.5s : 1
<10.0s : 6
<20.0s : 12
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
This script can fail if there is no toolchain available for the board.
At present this is not handled very nicely, in that only the error
output is reported. It is much more useful to see everything, so
combine stdout and stderr and report them both.
Signed-off-by: Simon Glass <sjg@chromium.org>
The u-boot-test-getrole script runs before the normal environment
variables have been set up. This is unavoidable since the script is
providing necessary information to test.py
This means that U_BOOT_SOURCE_DIR is not set in the environment.
As a result, Labgrid uses its default source path, configured in its
environment variable. While this may happen to work, it is not correct.
Also, it causes problems when running from Gitlab, where the runner may
not have access to that source path.
Provide the required source path in U_BOOT_SOURCE_DIR so that Labgrid
does the right thing.
Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: bf89a8f1fc ("test: Introduce the concept of a role")
[1] https://patchwork.ozlabs.org/project/uboot/patch/20241211131858.520639-1-sjg@chromium.org/
Simon Glass <sjg@chromium.org> says:
The current method of running unit tests relies on subcommands of the
ut command. Only the code in each subcommand knows how to find the tests
related to that subcomand.
This is not ideal and we now have quite a few subcommands which do
nothing but locate the relevant tests in a linker list, then call a
common function to run them.
This series adds a list of test suites, so that these subcommands can be
removed.
An issue with 'ut all' is that it doesn't record how many tests failed
overall, so it is necessary to examine copious amounts of output to look
for failures. This series adds a new 'total' feature allow recording the
total number of failed tests.
To help with 'ut all' a new pytest is created which runs it (as well as
'ut info') and makes sure that all is well. Due to the 'ut all' failures
this does not pass, so the test is disabled for now. It is here because
it provides security against misnaming a test suite and causing it not
to run.
Future work may:
- get 'ut all' passing
- enable test_suite() in CL, to ensure that 'ut all' keeps passing
- record duration of each suite
- allow running the tests in random order to tease out dependencies
- tweak the output to remove common prefixes
- getting rid of bootstd, optee and seame 'ut' subcommands
Link: https://lore.kernel.org/r/20250120212613.516664-1-sjg@chromium.org
Add a function to show the stats, so we can decide when to print it.
This slightly adjusts the output, so that any 'test not found' message
appears on its own line after all other output.
The 'failures' message now appears in lower case so update pytest
accordingly.
Signed-off-by: Simon Glass <sjg@chromium.org>
This is not needed anymore. If a test suite is not built, then it will
have no linker-list entries. So we can just check for that and know that
the suite is not present.
This allows removal of the #ifdefs and the need to keep them in sync
with the associated Makefile rules, which has actually failed, since the
help does not match what commands are actually present.
Signed-off-by: Simon Glass <sjg@chromium.org>
The 'info' test is not a real test. With the new suite array we can drop
this and the associated special-case code.
Signed-off-by: Simon Glass <sjg@chromium.org>
Most test suites have a _test suffix. This is not necessary as there is
also a ut_ prefix.
Drop the suffix so that (with future work) the suite name can be used as
the linker-list name.
Remove the suffix from the pytest regex as well, moving it to the top of
the file, as it is a constant.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a Python test which runs 'ut all' and then checks that the expected
suites are present and all tests in each suite are run.
This can help to check that nothing is missing.
Update 'ut info' to ignore the 'all' suite when counting the number of
suites, since that is really just a combination of all the other suites.
Adjust the message for skipped tests so that appears even if no
particular test was selected. This helps the new 'test_suite' test see
what is going on.
Signed-off-by: Simon Glass <sjg@chromium.org>
Simon Glass <sjg@chromium.org> says:
This series provides a way to keep track of the images used in bootstd,
including the type of each image.
At present this is sort-of handled by struct bootflow but in quite an
ad-hoc way. The structure has become quite large and is hard to query.
Future work will be able to reduce its size.
Ultimately the 'bootflow info' command may change to also show images as
a list, but that is left for later, as this series is already fairly
long. So for now, just introduce the concept and adjust bootstd to use
it, with a simple command to list the images.
This series includes various alist enhancements, to make use of this new
data structure a little easier.
[trini: Drop patch 18 and 19 for now due to size considerations]
Link: https://lore.kernel.org/r/20241115231926.211999-1-sjg@chromium.org
Add a new 'bootstd images' command, which lists the images which have
been loaded.
Update some existing tests to use it. Provide some documentation about
images in general and this command in particular.
Use a more realistic kernel command-line to make the test easier to
follow.
Signed-off-by: Simon Glass <sjg@chromium.org>
Raymond Mao <raymond.mao@linaro.org> says:
Motivations for changes:
Current SMBIOS library and command-line tool is not fully matching with
the requirements:
1. Missing support for other mandatory types (#7, #9, #16, #17, #19).
2. Only a few platforms support SMBIOS node from the device tree.
3. Values of some fields are hardcoded in the library other than fetching
from the device hardware.
4. Embedded data with dynamic length is not supported (E.g. Contained
Object Handles in Type #2 and Contained Elements in Type #3)
Changes:
1. Refactor the SMBIOS library and command-line tool to better align with
the SMBIOS spec.
2. Create an arch-specific driver for all aarch64-based platforms to fetch
SMBIOS private data from the device hardware (processor and cache).
3. Create a sysinfo driver to poppulate platform SMBIOS private data.
4. Add generic SMBIOS DTS file for arm64 platforms for those common strings
and values which cannot be retrieved from the system registers.
Vendors can create their own SMBIOS node using this as an example.
For those boards without SMBIOS nodes, this DTS file can be included to
have a generic SMBIOS information of the system.
5. Add support for Type #7 (Cache Information) and link its handles to
Type #4.
6. To minimize size-growth for those platforms which have not sufficient
ROM spaces or the platforms which don't need detailed SMBIOS
information, new added fields are only being built when kconfig
GENERATE_SMBIOS_TABLE_VERBOSE is selected.
Once this patch is acceptted, subsequent patch sets will add other missing
types (#9, #16, #17, #19).
Tests:
To test this with QEMU arm64, please follow the guide on dt_qemu.rst to
get a merged DT to run with.
```
qemu-system-aarch64 -machine virt -machine dumpdtb=qemu.dtb
cat <(dtc -I dtb qemu.dtb) <(dtc -I dtb ./dts/dt.dtb | grep -v /dts-v1/) \
| dtc - -o merged.dtb
qemu-system-aarch64 -machine virt -nographic -bios u-boot.bin \
-dtb merged.dtb
```
Link: https://lore.kernel.org/r/20241206225438.13866-1-raymond.mao@linaro.org
Update the cmd according to the changes of the smbios library:
1. Refactor smbios cmd print functions to match the content defined
by the specification.
2. Add new print functions for Type 3, 4 and 7.
3. Remove the fallback string "Not specified" from smbios_get_string,
as the spec requires a NULL output for those undefined strings.
4. Update the test_cmd_smbios_sandbox pytest expected result to
align with the smbios library changes and add new pytest
test_cmd_smbios_sysinfo_verbose to test the verbose smbios
output.
Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
test_extension.py assumes that no extension is known at test start.
This assumption is wrong because we do not come out of reboot.
A prior test may have already hunted for the extension bootdev.
Remove the invalid assert.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
The logic in get_details() retrieves the default source directory from
the Labgrid settings. This is convenient for interactive use, since it
allows pytests to be run from any directory and still find the source
tree.
However, it is not actually correct.
Gitlab sets the current directory to the source tree and expects that to
be used. At present it is ignored. The result is that Gitlab builds
whatever happens to be in the default source directory, ignoring the
tree it is supposed to be building.
Fix this by using the directory of the source tree, always. This is
obtained by looking at the grandparent of the conftest.py file.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
Fixes: bf89a8f1fc ("test: Introduce the concept of a role")
Tested-by: Tom Rini <trini@konsulko.com>
binman:
- Separate binman description from main DT
zynqmp:
- Enable binman for ZynqMP platforms
- DT sync with Linux v6.12
- Update usb5744 hub for SOMs
common:
- Drop SPL_FIT_GENERATOR support
versal2
- Enable OPTEE layers
ospi:
- Refactor the flash reset functionality
pytest:
- Fix tcminit mode handling
-----BEGIN PGP SIGNATURE-----
iF0EABECAB0WIQQbPNTMvXmYlBPRwx7KSWXLKUoMIQUCZ0m53AAKCRDKSWXLKUoM
Ie4PAJ4r7EHkSngpW3QdBEb15aLxN1pC4ACcC9DDGGn04kCTChpv0CEi/kRLUBU=
=Veyg
-----END PGP SIGNATURE-----
Merge tag 'xilinx-for-v2025.04-rc1' of https://gitlab.denx.de/u-boot/custodians/u-boot-microblaze into next
AMD/Xilinx changes for v2025.04-rc1
binman:
- Separate binman description from main DT
zynqmp:
- Enable binman for ZynqMP platforms
- DT sync with Linux v6.12
- Update usb5744 hub for SOMs
common:
- Drop SPL_FIT_GENERATOR support
versal2
- Enable OPTEE layers
ospi:
- Refactor the flash reset functionality
pytest:
- Fix tcminit mode handling
Tom Rini <trini@konsulko.com> says:
Hey all,
This is picking up Simon's v5 of the above-named series and making a few
more changes so that the follow-up series I have leads to arm64 being
supported for almost all jobs. To quote Simon's cover letter:
All gitlab runners are currently amd64 machines. This series attempts to
create a docker image which can also support arm64 so that sandbox tests
can be run on it.
The TARGET_... environment variables for grub could perhaps be adjusted,
using the new variables, but I have not done that for now.
Adding to what Simon said, we now build grub for all architectures as
the reason to install it was to be able to use the binaries in QEMU.
That won't provide us with amd64 binaries on arm64 hosts so we can't use
that shortcut anymore.
Link: https://lore.kernel.org/r/20241127172247.1488685-1-trini@konsulko.com
When a timeout occurs while executing a command a 'Boot fail' message is
written and testing is stopped. The user is left in the dark about the
failure cause.
! _pytest.outcomes.Exit: Boot fail: Marking connection bad - no other tests will run !
Add the executed command to the message.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
When a timeout occurs while executing a command a 'Lab failure' message is
written and testing is stopped. The user is left in the dark about the
failure cause.
! _pytest.outcomes.Exit: Lab failure: Marking connection bad - no other tests will run !
Add the word 'Timeout' and the executed command to the message.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
We don't need the fallback anymore. Remove the code which uses these
files.
Signed-off-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Richard Weinberger <richard@nod.at>
We don't need the fallback anymore. As a first step to removing it,
drop the try...except clauses and unindent the code.
This produces a large diff but there are no other code changes.
Signed-off-by: Simon Glass <sjg@chromium.org>
Like for test_fs, no need to mess with loop mounts.
Tweaks to reduce diff (keep mnt variable):
Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a parameter to indicate the size of the image to build.
Signed-off-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Richard Weinberger <richard@nod.at>
Since no mounting happens anymore, rename the "mnt"
directory to "scratch" and the related variables.
Signed-off-by: Richard Weinberger <richard@nod.at>
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
There is no need to mount the filesystem on the host side.
All filesystem tools offer some way to fill the fs without mounting.
So, create the content on the host side, create and fill the fs
without mounting.
No more sudo or guestmount needed.
This new approach works because the tests don't care about user IDs
and no device files are needed.
If user IDs start to matter it's still possible to use wrapper
tools like fakeroot in future while filling the fs.
Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
We don't need the fallback anymore. Remove the code which uses these
files.
Signed-off-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Richard Weinberger <richard@nod.at>
We don't need the fallback anymore. As a first step to removing it,
drop the try...except clauses and unindent the code.
This produces a large diff but there are no other code changes.
Signed-off-by: Simon Glass <sjg@chromium.org>
Like for test_fs, no need to mess with loop mounts.
Signed-off-by: Richard Weinberger <richard@nod.at>
Tweaks to reduce diff (keep mnt variable):
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a parameter to indicate the size of the image to build.
Signed-off-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Richard Weinberger <richard@nod.at>
Since no mounting happens anymore, rename the "mnt"
directory to "scratch" and the related variables.
Signed-off-by: Richard Weinberger <richard@nod.at>
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
There is no need to mount the filesystem on the host side.
All filesystem tools offer some way to fill the fs without mounting.
So, create the content on the host side, create and fill the fs
without mounting.
No more sudo or guestmount needed.
This new approach works because the tests don't care about user IDs
and no device files are needed.
If user IDs start to matter it's still possible to use wrapper
tools like fakeroot in future while filling the fs.
Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
Rename actual android bootmethod test to specify it's for boot image
version 4.
Add a unit test for testing the Android bootmethod with boot image
version 2.
This requires another mmc image (mmc8) to contain the following
partitions:
- misc: contains the Bootloader Control Block (BCB)
- boot_a: contains a fake generic kernel image
we can test this with:
$ ./test/py/test.py --bd sandbox --build -k test_ut # build the mmc8.img
$ ./test/py/test.py --bd sandbox --build -k bootflow_android
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
Signed-off-by: Guillaume La Roque <glaroque@baylibre.com>
Link: https://lore.kernel.org/r/20241126-adnroidv2-v4-5-11636106dc69@baylibre.com
Signed-off-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
Currently, MMC test runs on default mmc modes, adding a provision to
support multiple mmc modes through user defined parameters.
Signed-off-by: Love Kumar <love.kumar@amd.com>
'usb part' command shows the partition maps and shows the partition type
by displaying number such as 0c, 83 etc. Observed that ext2 and ext4
partitions shows the same number, i.e, 83, so, using the fstype command
to distiniguish between ext2 and ext4 partitions.
Signed-off-by: Love Kumar <love.kumar@amd.com>
'mmc part' command shows the partition maps and shows the partition type
by displaying number such as 0c, 83 etc. Observed that ext2 and ext4
partitions shows the same number, i.e, 83, so, using the fstype command
to distiniguish between ext2 and ext4 partitions.
Signed-off-by: Love Kumar <love.kumar@amd.com>
Use an 'r' string to avoid a warning:
test/py/tests/test_spi.py:698: DeprecationWarning: invalid escape
sequence '\s'
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Love Kumar <love.kumar@amd.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
