Some test can only be run by a particular board in a lab, e.g. because
they are loaded with an OS image used by the test. Add a way to specify
this in tests.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is sometimes tricky to figure out what modules test.py is loading
when it starts up. The result can be a silent failure with no clue as to
what when wrong.
Add a section which lists the modules loaded as well as those not
found.
Signed-off-by: Simon Glass <sjg@chromium.org>
We know this is U-Boot so the prefix serves no purpose other than to
make things longer and harder to read. Drop it and rename the files.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com> # test_android / test_dfu
This fixture name is quite long and results in lots of verbose code.
We know this is U-Boot so the 'u_boot_' part is not necessary.
But it is also a bit of a misnomer, since it provides access to all the
information available to tests. It is not just the console.
It would be too confusing to use con as it would be confused with
config and it is probably too short.
So shorten it to 'ubman'.
Signed-off-by: Simon Glass <sjg@chromium.org>
Link: https://lore.kernel.org/u-boot/CAFLszTgPa4aT_J9h9pqeTtLCVn4x2JvLWRcWRD8NaN3uoSAtyA@mail.gmail.com/
Execution time varies widely with the existing tests. Provides a way to
produce a summary of the time taken for each test, along with a
histogram.
This is enabled with the --timing flag.
Enable it for sandbox in CI.
Example:
Duration : Number of tests
======== : ========================================
<1ms : 1
<8ms : 1
<20ms : # 20
<30ms : ######## 127
<50ms : ######################################## 582
<75ms : ####### 102
<100ms : ## 39
<200ms : ##### 86
<300ms : # 29
<500ms : ## 42
<750ms : # 16
<1.0s : # 15
<2.0s : # 23
<3.0s : 13
<5.0s : 9
<7.5s : 1
<10.0s : 6
<20.0s : 12
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
This script can fail if there is no toolchain available for the board.
At present this is not handled very nicely, in that only the error
output is reported. It is much more useful to see everything, so
combine stdout and stderr and report them both.
Signed-off-by: Simon Glass <sjg@chromium.org>
The u-boot-test-getrole script runs before the normal environment
variables have been set up. This is unavoidable since the script is
providing necessary information to test.py
This means that U_BOOT_SOURCE_DIR is not set in the environment.
As a result, Labgrid uses its default source path, configured in its
environment variable. While this may happen to work, it is not correct.
Also, it causes problems when running from Gitlab, where the runner may
not have access to that source path.
Provide the required source path in U_BOOT_SOURCE_DIR so that Labgrid
does the right thing.
Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: bf89a8f1fc ("test: Introduce the concept of a role")
[1] https://patchwork.ozlabs.org/project/uboot/patch/20241211131858.520639-1-sjg@chromium.org/
Most test suites have a _test suffix. This is not necessary as there is
also a ut_ prefix.
Drop the suffix so that (with future work) the suite name can be used as
the linker-list name.
Remove the suffix from the pytest regex as well, moving it to the top of
the file, as it is a constant.
Signed-off-by: Simon Glass <sjg@chromium.org>
The logic in get_details() retrieves the default source directory from
the Labgrid settings. This is convenient for interactive use, since it
allows pytests to be run from any directory and still find the source
tree.
However, it is not actually correct.
Gitlab sets the current directory to the source tree and expects that to
be used. At present it is ignored. The result is that Gitlab builds
whatever happens to be in the default source directory, ignoring the
tree it is supposed to be building.
Fix this by using the directory of the source tree, always. This is
obtained by looking at the grandparent of the conftest.py file.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
Fixes: bf89a8f1fc ("test: Introduce the concept of a role")
Tested-by: Tom Rini <trini@konsulko.com>
The Beagleplay board uses an SoC from the TI K3 family. This has both a
Cortex-R core and a Cortex-A core and the R core needs to come up before
the A core. In both cases we have U-Boot SPL then U-Boot proper being
used.
In practice this means we need two entirely separate builds to produce
an image.
Handle this in test.py by adding more parameters.
Signed-off-by: Simon Glass <sjg@chromium.org>
In Labgrid there is the concept of a 'role', which is similar to the
U-Boot board ID in U-Boot's pytest subsystem.
The role indicates both the target and information about the U-Boot
build to use. It can also provide any amount of other configuration.
The information is obtained using the 'labgrid-client query' operation.
Using this role, all required configuration for the board is stored
within the Labgrid environment, with pytest simply querying it. This
allows connecting to boards using an interactive console, something that
isn't possible without some kind of mapping. It also means that we don't
need to replicate the pytest functionality in tbot, since Labgrid can
handle the console and kick off builds as needed.
Make use of this in tests, so that only the role is required in gitlab
and other situations. The board type and other things can be queried
as needed.
Use a new 'u-boot-test-getrole' script to obtain the requested
information.
With this it is possible to run lab tests in gitlab with just a single
'ROLE' variable for each board.
Note that, without this feature:
- interactive use of boards with Labgrid-sjg would require repeating the
id/board in a separate configuration file
- Gitlab yaml file would need to specify both the id and board
This feature is entirely optional, however, with the code gracefully
falling back to using a separate ID and board.
Link: https://tbot.tools
Signed-off-by: Simon Glass <sjg@chromium.org>
Sometimes we know that the board is already running the right software,
so provide an option to allow running of tests directly, without first
resetting the board.
This saves time when re-running a test where only the Python code is
changing.
Note that this feature is open to errors, since the user must know that
the board is in a fit state to execute tests. It is useful for repeated
iteration on a particular test, where it can save quite a bit of time.
Signed-off-by: Simon Glass <sjg@chromium.org>
When the connection to a board dies, assume it is dead forever until
some user action is taken. Skip all remaining tests. This avoids CI
runs taking an hour, with hundreds of 30-second timeouts all to no
avail.
Signed-off-by: Simon Glass <sjg@chromium.org>
The settings are decoded in two places. Combine them into a new
function, before (in a future patch) expanding the number of items.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we normally write tests either in Python or in C. But most
Python tests end up doing a lot of checks which would be better done in C.
Checks done in C are orders of magnitude faster and it is possible to get
full access to U-Boot's internal workings, rather than just relying on
the command line.
The model is to have a Python test set up some things and then use C code
(in a unit test) to check that they were done correctly. But we don't want
those checks to happen as part of normal test running, since each C unit
tests is dependent on the associate Python tests, so cannot run without
it.
To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check'
tests, so that they can be skipped by default when the 'ut' command is
used. Require that tests have a name ending with '_norun', so that pytest
knows to skip them.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present when -n is used, all workers try to build U-Boot at once.
Add a lock to ensure that only one of them builds, with the others using
the build that is produced.
The lock file is removed on startup.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present the collection function used by pytest is quite strict on the
naming of the functions it detects. In particular it requires the name of
the test to be repeated in the function name.
This is not enforced anywhere else, but instead the tests are silently
omitted from the pytest run. This affects a few dozen tests.
The rule does not seem to have any particular purpose. Relax it, so that
all tests that use the UNIT_TEST() macro will run, regardless of the name
of the test function.
Signed-off-by: Simon Glass <sjg@chromium.org>
Improve the regular expression that matches unittest symbols in
u-boot.sym.
Currently we do not enforce no prefix in symbol string, but with the
soon to come change in linker lists declaring lists and entries with the
__ADDRESSABLE macro (because of LTO), the symbol file will contain for
every symbol of the form
_u_boot_list_2_ut_X_2_Y
also symbol
__UNIQUE_ID___addressable__u_boot_list_2_ut_X_2_YN,
(where N at the end is some number).
In order to avoid matching these additional symbols, ensure that the
character before "_u_boot_list_2_ut" is not a symbol name character.
Signed-off-by: Marek Behún <marek.behun@nic.cz>
Reviewed-by: Simon Glass <sjg@chromium.org>
At present each test suite has its own portion of the linker_list section
of the image, but other lists are interspersed. This makes it hard to
enumerate all the available tests without knowing the suites that each one
is in.
Place all tests together in a single contiguous list by giving them
common prefix not used elsewhere in U-Boot. This makes it possible to find
the start and end of all tests.
Signed-off-by: Simon Glass <sjg@chromium.org>
The implementation of pytest_runtest_protocol() must call
pytest_runtest_logstart() and pytest_runtest_logfinish(). This appears to
be necessary even in pytest 5.2.1 judging by the default version of
pytest_runtest_protocol(), but evidently some form of code reorganization
in pytest only made this have a practical effect in the newer version. I'd
previously been under the impression that 100% of the required work of
pytest_runtest_protocol() was handled by the fact it called
runtestprotocol() as its implementation. However, it appears that custom
implementations do need to do a little more than this.
Reported-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
When building U-Boot we should not blindly use make -j8 but consider the
actual core count given by os.cpu_count().
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
Acked-by: Stephen Warren <swarren@nvidia.com>
It is a pain to have to set the CROSS_COMPILE environment variable when
using test.py's --build option. It is possible to get this using the -A
option from buildman. But it seems better to just use buildman to do the
build when it is available.
However using buildman adds a new dependency to the test system which we
want to avoid. So leave the default as is and add a flag to make it use
buildman.
Note that most of these changes relate to test.py and the parts of the
travis/gitlab/azure scripts which relate to running test and building a
suitable U-Boot to run the tests on.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
Consider the following test sample:
@pytest.mark.buildconfigspec('fit')
@pytest.mark.notbuildconfigspec('generate_acpi_table')
def test_sample(u_boot_console):
Whatever the argument of the 'notbuildconfigspec' is,
the test ends up being skipped with the message:
('/uboot/test/py/conftest.py', 463,
'Skipped: .config feature "fit" enabled')
Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Buildman doesn't store this file in the same directory as a normal build.
Update the conftest code to handle both cases.
Change-Id: I1fd0e56054d7dc77394a7589336aa0991bd0133d
Signed-off-by: Simon Glass <sjg@chromium.org>
- Modern pytest is more visible in telling us about parameters that we
had not described, so describe a few more.
- ConfigParser.readfp(...) is now configparser.read_file(...)
- As part of the "strings vs bytes" conversions in Python 3, we use the
default encoding/decoding of utf-8 but in some places tell Python to
replace problematic conversions rather than throw a fatal error.
- Fix a typo noticed while doing the above ("tot he" -> "to the").
- As suggested by Stephen, re-alphabetize the import list
- Per Heinrich, replace how we write contents in test_fit.py
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
Use the 2to3 tool to perform numerous automatic conversions from Python
2 syntax to Python 3. Also fix whitespace problems that Python 3
catches that Python 2 did not.
Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
Fix the following spit from pytest:
u-boot/test/py/conftest.py:438: RemovedInPytest4Warning: MarkInfo objects are deprecated as they contain merged marks which are hard to deal with correctly.
Please use node.get_closest_marker(name) or node.iter_markers(name).
Docs: https://docs.pytest.org/en/latest/mark.html#updating-code
for board in mark.args:
In both cases, the later suggestion is applicable.
Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Marek Vasut <marek.vasut+renesas@gmail.com>
Cc: Igor Opaniuk <igor.opaniuk@gmail.com>
[trini: Update for current file with a few more cases, un-pin pytest in CI]
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Tested-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
We already can let a Python test depend on a build option being set via
@pytest.mark.buildconfigspec(). It may be necessary to let a test depend on
a build option *not* being set. So let's introduce
@pytest.mark.notbuildconfigspec
for this purpose.
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Enable usage of gdbserver for all sandbox targets (sandbox,
sandbox_flattree etc.).
Signed-off-by: Igor Opaniuk <igor.opaniuk@linaro.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
In python 3.x the configparser module is named with all lower case.
Import it as such in order to avoid errors when running on python 3.x,
and fall back to the CamelCase version in order to keep working with
python 2.x.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
In python 3.x print must be called as a function rather than used as a
statement. Update uses of print to the function call syntax in order to
be python 3.x safe.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.
In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.
This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.
Signed-off-by: Tom Rini <trini@konsulko.com>
Currently, if a test emits a warning message but otherwise passes, there's
no indication of this in the log summary, which can lead to warnings being
missed. Enhance the test logic to explicitly mention warnings in otherwise
passing tests, and not to collapse the log sections for tests with
warnings, so that they're more easily seen when scanning the log.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
The u-boot.sym file is scanned to find unit test suites for execution. At
present it only finds those whose names start with 'dm' or 'env'. This
code is buried in the bowels of the test code so when adding a new suite
it is not easy to discover why it is ignored by the test framework.
There seems to be no need to make this restriction. Drop it.
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Stephen Warren <swarren@nvidia.com>
Some tests use external tools (executables) during their operation. Add
a test.py mark to indicate this. This allows those tests to be skipped if
the required tool is not present.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
When skipping tests, explicitly mention the board type or config option
that caused the skip. This will help people understand/fix any issues.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
If a test uses a fixture which is expensive to setup, the fixture can
possibly created with session or module scope. As u_boot_console has
function scope, it can not be used in this case.
Signed-off-by: Stefan Brüns <stefan.bruens@rwth-aachen.de>
Acked-by: Stephen Warren <swarren@wwwdotorg.org>
In pytest 3, runtestprotocol() may not call pytest_runtest_setup() if
the test is skipped. That call is required to create a section for the
test in the log file. If this is skipped, the call to log.end_section()
at the tail of pytest_runtest_protocol() will throw an exception. This
patch ensures that a log section always exists, both to avoid the
exception and to ensure that a consistently structured log file is
always created.
Cc: Stefan Brüns <stefan.bruens@rwth-aachen.de>
Reported-by: Stefan Brüns <stefan.bruens@rwth-aachen.de>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Tom Rini <trini@konsulko.com>
This board can sometimes be used for tests. Handle it the same way as
sandbox.
Note: I plan to drop the sandbox_spl board at some point and merge its
features into sandbox. So this commit may not be necessary.
Signed-off-by: Simon Glass <sjg@chromium.org>
Normally tests will run with the test.dtb file designed for this purpose.
However, the verified boot tests need to run with their own device-tree
file, containing a public key.
Make the device-tree file a config option so that it can be adjusted by
tests. The default is to keep the current behaviour.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Teddy Reed <teddy.reed@gmail.com>
Use lists rather than sets to record the status of tests. This causes
the test summary in the HTML file to be generated in the same order as
the tests are (or would have been) run. This makes it easier to locate
the first failed test. The log for this test might have interesting
first clues re: interaction with the environment (e.g. hardware flashing,
serial console, ...) and may help tracking down external issues.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
The code replaced pexpect with custom code long ago. Don't import the
unused module so it doesn't need to be installed.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Invoke each "ut"-based unit test as a separate pytest.
Now that the DM unit test runs under test/py, remove the manual shell
script that invokes it.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org> # v2, on sandbox
Implement three improvements to the HTML log file:
- Ability to expand/contract sections. All passing sections are contracted
at file load time so the user can concentrate on issues requiring
action.
- The overall status report is copied to the top of the log for easy
access.
- Add links from the status report to the test logs, for easy navigation.
This all relies on Javascript and the jquery library. If the user doesn't
have Javascript enabled, or jquery can't be downloaded, the log should
look and behave identically to how it did before this patch.
A few notes on the diff:
- A few more 'with log.section("xxx")' were added, so that all stream
blocks are kept within a section block for consistent HTML entity
nesting structure. This changed indentation in a few places, making
the diff look slightly larger.
- HTML entity IDs are cleaned up. We assign simple incrementing integer
IDs now, rather than using mangled test names which were possibly
invalid.
- Sections and streams now use common CSS class names (in addition to the
current separate class names) to more easily share the new behaviour.
This also reduces the CSS file size since rules don't need to be
duplicated.
- An "OK" status is logged after some external command executions so that
make and flash steps are auto-contracted at log file load time, assuming
they passed.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Implement command--line option --gdbserver COMM, which does two things:
a) Run the sandbox process under gdbserver, using COMM as gdbserver's
communication channel.
b) Disables all timeouts, so that if U-Boot is halted under the debugger,
tests don't fail. If the user gives up in the middle of a debugging
session, they can simply CTRL-C the test script to abort it.
This allows easy debugging of test failures without having to manually
re-create the failure conditions. Usage is:
Window 1:
./test/py/test.py --bd sandbox --gdbserver localhost:1234
Window 2:
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
When using this option, it likely makes sense to use pytest's -k option
to limit the set of tests that are executed.
Simply running U-Boot directly under gdb (rather than gdbserver) was
also considered. However, this was rejected because:
a) gdb's output would then be processed by the test script, and likely
confuse it causing false failures.
b) pytest by default hides stdout from tests, which would prevent the
user from interacting with gdb.
While gdb can be told to redirect the debugee's stdio to a separate
PTY, this would appear to leave gdb's stdio directed at the test
scripts and the debugee's stdio directed elsewhere, which is the
opposite of the desired effect. Perhaps some complicated PTY muxing
and process hierarchy could invert this. However, the current scheme
is simple to implement and use, so it doesn't seem worth complicating
matters.
c) Using gdbserver allows arbitrary debuggers to be used, even those with
a GUI. If the test scripts invoked the debugger themselves, they'd have
to know how to execute arbitary applications. While the user could hide
this all in a wrapper script, this feels like extra complication.
An interesting future idea might be a --gdb-screen option, which could
spawn both U-Boot and gdb separately, and spawn the screen into a newly
created window under screen. Similar options could be envisaged for
creating a new xterm/... too.
--gdbserver currently only supports sandbox, and not real hardware.
That's primarily because the test hooks are responsible for all aspects of
hardware control, so there's nothing for the test scripts themselves can
do to enable gdbserver on real hardware. We might consider introducing a
separate --disable-timeouts option to support use of debuggers on real
hardware, and having --gdbserver imply that option.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
