At present the 'ut' command handles its flags in a strange way, in that
they must come after the subcommand.
So, we must use 'ut bloblist -r2' to run the bloblist tests twice. This
is an artefact of the way tests were run, through subcommands.
It is now possible to correct this, by doing flag-processing before
running the suite.
Update the code to handle this, so that 'ut -r2 bloblist' works. Update
the 'test_suite' test to check the new arguments.
Add a sanity-check for -I while we are here.
Signed-off-by: Simon Glass <sjg@chromium.org>
Move the logic from this function into run_suite(), on the way to having
flag parsing in the top-level 'ut' command instead of its children.
Signed-off-by: Simon Glass <sjg@chromium.org>
Enhance the ut command to accept a comma-separated list of test suites
to run. Report the summary information for these at the end.
Signed-off-by: Simon Glass <sjg@chromium.org>
Now that everything is using the new test-suite features, drop support
for running commands.
Fix a missing closing-bracket while we are here.
Signed-off-by: Simon Glass <sjg@chromium.org>
When tests are all in the same suite it is annoying to have to read all
the common text after each name. Skip this to help the user.
Signed-off-by: Simon Glass <sjg@chromium.org>
Show the average duration of a test, so we can keep track of how it is
trending. Report the suite with the longest average test to encourage
people to improve it.
Add a function to update the stats based on the results from a single
suite and another to show the summary information.
Make this optional, since sandbox's SPL tests do not have a timer driver
and people may want to print results without times.
Signed-off-by: Simon Glass <sjg@chromium.org>
All tests should belong to a suite, but if there is a suite we don't
know about (e.g. not added to cmd_ut.c) then the totals will not add up.
Add a check for this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Rather than having the help in the longhelp, put it in the suite info
so 'ut info -s' can show it. This is tidier, particular due to the
removal of #ifdefs
This means that the help text is present in the image (although not
displayed with 'ut info -s') so the image-size increases. But with
UNIT_TEST enabled, we expect large images so this doesn't seem
important.
Signed-off-by: Simon Glass <sjg@chromium.org>
Put the suites in order by name, for easier code-maintenance. This also
helps find test results for a particular swuit in the 'ut all' output.
Signed-off-by: Simon Glass <sjg@chromium.org>
Update this function to access a unit-test state, so that the caller can
collect results from running multiple suites.
Signed-off-by: Simon Glass <sjg@chromium.org>
This is not needed anymore. If a test suite is not built, then it will
have no linker-list entries. So we can just check for that and know that
the suite is not present.
This allows removal of the #ifdefs and the need to keep them in sync
with the associated Makefile rules, which has actually failed, since the
help does not match what commands are actually present.
Signed-off-by: Simon Glass <sjg@chromium.org>
The 'info' test is not a real test. With the new suite array we can drop
this and the associated special-case code.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new suite-runner to run these tests instead.
It is not clear that these actually work, since they are not enabled on
sandbox for some reason.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new suite-runner to run these tests instead.
It is not clear that these actually work, since they are not enabled on
sandbox for some reason.
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
The current cmd_ut_sub[] array was fine when there were only a few test
suites. But is quite unwieldy now:
- it requires a separate do_ut_xxx for each suite, even though the code
for most is almost identical
- running more than one suite requires running multiple commands, and
there is no record of which suites passed or failed
- 'ut all' runs all suites but reports their results individually
- we need lots of #ifdefs in the array, mirroring those in the makefile
but maintained in a separate place
In fact the tests are all in the same linker list. The suites are
grouped, so it is possible to access the information without a command.
Introduce a 'suite' array, which holds the cmd_ut_...() function to
call, but can also support running a suite without that function. This
means that the array of struct cmd_tbl is transformed into an array of
'struct suite'.
This will allow removal of many of the functions, particularly those
without test-specific init.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a Python test which runs 'ut all' and then checks that the expected
suites are present and all tests in each suite are run.
This can help to check that nothing is missing.
Update 'ut info' to ignore the 'all' suite when counting the number of
suites, since that is really just a combination of all the other suites.
Adjust the message for skipped tests so that appears even if no
particular test was selected. This helps the new 'test_suite' test see
what is going on.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is helpful to see a list of available suites. At present this is
handled by the longhelp for the 'ut' command, but this is not in a
format which can be easily parsed by python tests.
Add a -s option to show this. At present it is not possible to show the
number of tests in each suite, but future work will address this. For
now, show a ?
Signed-off-by: Simon Glass <sjg@chromium.org>
Having CONFIG_OF_EMBED=y && CONFIG_BLOBLIST=n leads to the link
error:
```
ld: /tmp/ccwtRVty.ltrans28.ltrans.o:(.data.rel+0x4b0): undefined \
reference to `do_ut_bloblist'
```
Fixes: 6ea5df39e8 ("test: Only enable bloblist test when supported")
Signed-off-by: Evgeny Bachinin <EABachinin@salutedevices.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
There is no particular need for the time tests to have their own test
command. Move them into the lib suite instead.
Update the test functions to match the normal unit-test signature.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Tom Rini <trini@konsulko.com> # rpi_3, rpi_4, rpi_arm64, am64x_evm_a53, am64-sk
There is no particular need for the unicode tests to have their own test
suite. Move them into the lib suite instead.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Tested-by: Tom Rini <trini@konsulko.com> # rpi_3, rpi_4, rpi_arm64, am64x_evm_a53, am64-sk
