Execution time varies widely with the existing tests. Provides a way to
produce a summary of the time taken for each test, along with a
histogram.
This is enabled with the --timing flag.
Enable it for sandbox in CI.
Example:
Duration : Number of tests
======== : ========================================
<1ms : 1
<8ms : 1
<20ms : # 20
<30ms : ######## 127
<50ms : ######################################## 582
<75ms : ####### 102
<100ms : ## 39
<200ms : ##### 86
<300ms : # 29
<500ms : ## 42
<750ms : # 16
<1.0s : # 15
<2.0s : # 23
<3.0s : 13
<5.0s : 9
<7.5s : 1
<10.0s : 6
<20.0s : 12
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
bloblist_find function only returns the pointer of blob data,
which is fine for those self-describing data like FDT.
But as a common scenario, an interface is needed to retrieve both
the pointer and the size of the blob data.
Add a few ut test cases for the new api.
Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
The $loadaddr variable is a hexadecimal value, not a string, it must be
assigned using env_set_hex(). This may break follow up tests, like the
dm_test_cmd_hash_md5 in CI. To avoid any interference with other tests,
set $wgetaddr variable which is specific to this test and use it in the
test.
Fixes: 20f641987f ("test/cmd/wget.c: move net_test_wget() to the cmd test suite")
Signed-off-by: Marek Vasut <marek.vasut+renesas@mailbox.org>
Acked-by: Jerome Forissier <jerome.forissier@linaro.org>
Reviewed-by: Fabio Estevam <festevam@gmail.com>
Up to now we could only use log level, category, and file for filtering.
Allow filtering on a list of functions.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
This script can fail if there is no toolchain available for the board.
At present this is not handled very nicely, in that only the error
output is reported. It is much more useful to see everything, so
combine stdout and stderr and report them both.
Signed-off-by: Simon Glass <sjg@chromium.org>
The u-boot-test-getrole script runs before the normal environment
variables have been set up. This is unavoidable since the script is
providing necessary information to test.py
This means that U_BOOT_SOURCE_DIR is not set in the environment.
As a result, Labgrid uses its default source path, configured in its
environment variable. While this may happen to work, it is not correct.
Also, it causes problems when running from Gitlab, where the runner may
not have access to that source path.
Provide the required source path in U_BOOT_SOURCE_DIR so that Labgrid
does the right thing.
Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: bf89a8f1fc ("test: Introduce the concept of a role")
[1] https://patchwork.ozlabs.org/project/uboot/patch/20241211131858.520639-1-sjg@chromium.org/
Simon Glass <sjg@chromium.org> says:
The current method of running unit tests relies on subcommands of the
ut command. Only the code in each subcommand knows how to find the tests
related to that subcomand.
This is not ideal and we now have quite a few subcommands which do
nothing but locate the relevant tests in a linker list, then call a
common function to run them.
This series adds a list of test suites, so that these subcommands can be
removed.
An issue with 'ut all' is that it doesn't record how many tests failed
overall, so it is necessary to examine copious amounts of output to look
for failures. This series adds a new 'total' feature allow recording the
total number of failed tests.
To help with 'ut all' a new pytest is created which runs it (as well as
'ut info') and makes sure that all is well. Due to the 'ut all' failures
this does not pass, so the test is disabled for now. It is here because
it provides security against misnaming a test suite and causing it not
to run.
Future work may:
- get 'ut all' passing
- enable test_suite() in CL, to ensure that 'ut all' keeps passing
- record duration of each suite
- allow running the tests in random order to tease out dependencies
- tweak the output to remove common prefixes
- getting rid of bootstd, optee and seame 'ut' subcommands
Link: https://lore.kernel.org/r/20250120212613.516664-1-sjg@chromium.org
Rather than having the help in the longhelp, put it in the suite info
so 'ut info -s' can show it. This is tidier, particular due to the
removal of #ifdefs
This means that the help text is present in the image (although not
displayed with 'ut info -s') so the image-size increases. But with
UNIT_TEST enabled, we expect large images so this doesn't seem
important.
Signed-off-by: Simon Glass <sjg@chromium.org>
Put the suites in order by name, for easier code-maintenance. This also
helps find test results for a particular swuit in the 'ut all' output.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a function to show the stats, so we can decide when to print it.
This slightly adjusts the output, so that any 'test not found' message
appears on its own line after all other output.
The 'failures' message now appears in lower case so update pytest
accordingly.
Signed-off-by: Simon Glass <sjg@chromium.org>
This is useful information and is not always the same as the 'count' arg
to ut_run_list() so add it as a separate stat.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use a struct to hold the stats, since we also want to have the same
stats for all runs as we have for each suite.
Signed-off-by: Simon Glass <sjg@chromium.org>
Update this function to access a unit-test state, so that the caller can
collect results from running multiple suites.
Signed-off-by: Simon Glass <sjg@chromium.org>
This is not needed anymore. If a test suite is not built, then it will
have no linker-list entries. So we can just check for that and know that
the suite is not present.
This allows removal of the #ifdefs and the need to keep them in sync
with the associated Makefile rules, which has actually failed, since the
help does not match what commands are actually present.
Signed-off-by: Simon Glass <sjg@chromium.org>
The 'info' test is not a real test. With the new suite array we can drop
this and the associated special-case code.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new suite-runner to run these tests instead.
It is not clear that these actually work, since they are not enabled on
sandbox for some reason.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new suite-runner to run these tests instead.
It is not clear that these actually work, since they are not enabled on
sandbox for some reason.
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
The current cmd_ut_sub[] array was fine when there were only a few test
suites. But is quite unwieldy now:
- it requires a separate do_ut_xxx for each suite, even though the code
for most is almost identical
- running more than one suite requires running multiple commands, and
there is no record of which suites passed or failed
- 'ut all' runs all suites but reports their results individually
- we need lots of #ifdefs in the array, mirroring those in the makefile
but maintained in a separate place
In fact the tests are all in the same linker list. The suites are
grouped, so it is possible to access the information without a command.
Introduce a 'suite' array, which holds the cmd_ut_...() function to
call, but can also support running a suite without that function. This
means that the array of struct cmd_tbl is transformed into an array of
'struct suite'.
This will allow removal of many of the functions, particularly those
without test-specific init.
Signed-off-by: Simon Glass <sjg@chromium.org>
Most test suites have a _test suffix. This is not necessary as there is
also a ut_ prefix.
Drop the suffix so that (with future work) the suite name can be used as
the linker-list name.
Remove the suffix from the pytest regex as well, moving it to the top of
the file, as it is a constant.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a Python test which runs 'ut all' and then checks that the expected
suites are present and all tests in each suite are run.
This can help to check that nothing is missing.
Update 'ut info' to ignore the 'all' suite when counting the number of
suites, since that is really just a combination of all the other suites.
Adjust the message for skipped tests so that appears even if no
particular test was selected. This helps the new 'test_suite' test see
what is going on.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is helpful to see a list of available suites. At present this is
handled by the longhelp for the 'ut' command, but this is not in a
format which can be easily parsed by python tests.
Add a -s option to show this. At present it is not possible to show the
number of tests in each suite, but future work will address this. For
now, show a ?
Signed-off-by: Simon Glass <sjg@chromium.org>
Some suites have a different name from that used in the linker list.
That makes it hard to programmatically match the name printed when the
suite runs to the linker-list name it has.
Update the names so they are the same.
Signed-off-by: Simon Glass <sjg@chromium.org>
A few messages lack a newline so the test output shows the next
test-name on the same line. For example:
Beware: this test sets local variable dollar_bar and dollar_quux
and they cannot be unset!Test: hush_test_env_dollar: dollar.c
This is confusing, so fix it.
Signed-off-by: Simon Glass <sjg@chromium.org>
Move these operations into separate functions so that it is clearer what
is needed. These functions can also be called from somewhere other than
ut_run_list().
Signed-off-by: Simon Glass <sjg@chromium.org>
The string conversion functions are implemented in lib/strto.c which is
only compiled if CONFIG_STRTO=y.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
