Add tests for the exfat filesystem. These tests are largely an
extension of the FS_GENERIC tests with the following notable
exceptions.
The filesystem image for exfat tests is generated using combination
of exfatprogs mkfs.exfat and python fattools. The fattols are capable
of generating exfat filesystem images too, but this is not used, the
fattools are only used as a replacement for dosfstools 'mcopy' and
'mdir', which are used to insert files and directories into existing
fatfs images and list existing fatfs images respectively, without the
need for superuser access to mount such images.
The exfat filesystem has no filesystem specific command, there is only
the generic filesystem command interface, therefore check_ubconfig()
has to special case exfat and skip check for CONFIG_CMD_EXFAT and
instead check for CONFIG_FS_EXFAT.
Signed-off-by: Marek Vasut <marex@denx.de>
The generic filesystem interface was so far untested. The interface
is similar to the FS specific interfaces with FS specific prefixes,
like ext4ls, fatmkdir, ... but it does not have any prefixes, i.e.
it provides plain ls, mkdir, ... commands.
Extend the test parameters to include 'fs_cmd_prefix' and optionally
'fs_cmd_write' parameters. The 'fs_cmd_prefix' allow specifying the
filesystem specific command prefix, like 'ext4' in 'ext4ls'. The
'fs_cmd_write' allows selecting between 'write'/'save' command name
for storing files into the filesystem, see last paragraph.
Introduce new 'fs_generic' fs_type which is used to parametrize existing
tests and run them without any prefixes if detected, thus testing the
generic filesystem interface. Use the fatfs as the backing store for the
generic FS tests.
The check_ubconfig needs to be slightly adjusted to avoid test for
CMD_FS_GENERIC_WRITE which does not exist separately from CMD_FS_GENERIC.
The CMD_FS_GENERIC does not provide generic 'write' command, instead
the generic equivalent command is called 'save' . Add simple ternary
oeprator to use 'save' command for CMD_FS_GENERIC tests and '..write'
commands for filesystem specific tests.
Enable generic filesystem tests for basic/extended/mkdir/unlink tests.
Signed-off-by: Marek Vasut <marex@denx.de>
Now that we have a shorter name, we don't need this sort of thing.
Drop it.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com> # test_android
We know this is U-Boot so the prefix serves no purpose other than to
make things longer and harder to read. Drop it and rename the files.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com> # test_android / test_dfu
This fixture name is quite long and results in lots of verbose code.
We know this is U-Boot so the 'u_boot_' part is not necessary.
But it is also a bit of a misnomer, since it provides access to all the
information available to tests. It is not just the console.
It would be too confusing to use con as it would be confused with
config and it is probably too short.
So shorten it to 'ubman'.
Signed-off-by: Simon Glass <sjg@chromium.org>
Link: https://lore.kernel.org/u-boot/CAFLszTgPa4aT_J9h9pqeTtLCVn4x2JvLWRcWRD8NaN3uoSAtyA@mail.gmail.com/
The implementation roughly follows the POSIX specification for
rename() [1]. The ordering of operations attempting to minimize the chance
for data loss in unexpected circumstances.
The 'mv' command was implemented as a front end for the rename operation
as that is what most users are likely familiar with in terms of behavior.
The 'FAT_RENAME' Kconfig option was added to prevent code size increase on
size-oriented builds like SPL.
[1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/rename.html
Signed-off-by: Gabriel Dalimonte <gabriel.dalimonte@gmail.com>
Since no mounting happens anymore, rename the "mnt"
directory to "scratch" and the related variables.
Signed-off-by: Richard Weinberger <richard@nod.at>
Reviewed-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
There is no need to mount the filesystem on the host side.
All filesystem tools offer some way to fill the fs without mounting.
So, create the content on the host side, create and fill the fs
without mounting.
No more sudo or guestmount needed.
This new approach works because the tests don't care about user IDs
and no device files are needed.
If user IDs start to matter it's still possible to use wrapper
tools like fakeroot in future while filling the fs.
Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Mattijs Korpershoek <mkorpershoek@baylibre.com>
Now that opendir, readir, closedir are implemented for ext4 we can use
fs_ls_generic() for implementing the ls command.
Adjust the unit tests:
* fs_ls_generic() produces more spaces between file size and name.
* The ext4 specific message "** Can not find directory. **\n" is not
written anymore.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Recently pylint has started to complain about:
No name 'fs_helper' in module 'tests' (no-name-in-module)
Due to:
from tests import fs_helper
However, we have:
test/py/tests/fs_helper.py
And since we do not want to add a dummy test/py/tests/__init__.py to
silence this warning we instead just disable it as needed.
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
To quote the author:
This series fixes an issue where the FAT type (FAT12, FAT16) is not
correctly detected, e.g. when the BPB field BS_FilSysType contains the
valid value "FAT ".
This issue occures, for example, if a partition is formatted by
swupdate using its diskformat handler. swupdate uses the FAT library
from http://elm-chan.org/fsw/ff/ internally.
See https://groups.google.com/g/swupdate/c/7Yc3NupjXx8 for a
discussion in the swupdate mailing list.
Please refer to the commit messages for more details.
1. Added bootsector checks
Most tests from https://www.win.tue.nl/~aeb/linux/fs/fat/fat-2.html
are added in the commit 'fs: fat: add bootsector validity check'.
Only the tests VIII, IX and X are not implemented.
I also checked the Linux kernel code (v6.6) and did not find any
checks on 'vistart->fs_type'. This is the reason why is skipped them
here.
See section '2. Size comparisons' for the impact on the binary size.
2. Size comparisons
I executed bloat-o-meter from the Linux kernel for an arm64
target (config xilinx_zynqmp_mini_emmc0_defconfig):
Comparison of the binary spl/u-boot-spl between master (rev
e17d174773) and this patch
series (including the added validity checks of the boot sector):
add/remove: 0/0 grow/shrink: 1/1 up/down: 100/-12 (88)
Function old new delta
read_bootsectandvi 308 408 +100
fat_itr_root 444 432 -12
Total: Before=67977, After=68065, chg +0.13%
When compare the size of the binary spl/u-boot-spl between master this
series without the the validity checks of the boot sector:
add/remove: 0/0 grow/shrink: 0/2 up/down: 0/-24 (-24)
Function old new delta
read_bootsectandvi 308 296 -12
fat_itr_root 444 432 -12
Total: Before=67977, After=67953, chg -0.04%
So the size of the spl on this arm64 target increases by 88 bytes for
this series. When i remove the validity check the size decreases by 24 bytes.
Ensure that a large FAT12 filesystem and a small FAT16 filesystem are
detected correctly.
Signed-off-by: Christian Taedcke <christian.taedcke@weidmueller.com>
The tests fs_ext, fs_mkdir and fs_unlink support fat12 without
modifications.
The fs_basic test uses a partition that is too large for fat12, so it
is omitted here.
Signed-off-by: Christian Taedcke <christian.taedcke@weidmueller.com>
Changes for complying to EFI spec §3.5.1.1
'Removable Media Boot Behavior'.
Boot variables can be automatically generated during a removable
media is probed. At the same time, unused boot variables will be
detected and removed.
Please note that currently the function 'efi_disk_remove' has no
ability to distinguish below two scenarios
a) Unplugging of a removable media under U-Boot
b) U-Boot exiting and booting an OS
Thus currently the boot variables management is not added into
'efi_disk_remove' to avoid boot options being added/erased
repeatedly under scenario b) during power cycles
See TODO comments under function 'efi_disk_remove' for more details
The original efi_secboot tests expect that BootOrder EFI variable
is not defined. With this commit, the BootOrder EFI variable is
automatically added when the disk is detected. The original
efi_secboot tests end up with unexpected failure.
The efi_secboot tests need to be modified to explicitly set
the BootOrder EFI variable.
squashfs and erofs ls tests are also affected by this modification,
need to clear the previous state before squashfs ls test starts.
Co-developed-by: Masahisa Kojima <masahisa.kojima@linaro.org>
Signed-off-by: Masahisa Kojima <masahisa.kojima@linaro.org>
Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
Reviewed-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Reviewed-by: Joao Marcos Costa <jmcosta944@gmail.com>
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Ensure that a freshly written fat file with a lower case filename which
fits into the upper case 8.3 short filename is not mangeled with a tilde
and number.
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
This test seems to interfere with the other test in this file. Mark it
single-threaded to avoid any problems.
Signed-off-by: Simon Glass <sjg@chromium.org>
Testing with mksquasshfs 4.5.1 results in an error
ValueError: could not convert string to float: '4.5.1'
Version 4.10 would be considered to be lower than 4.4.
Fixes: 04c9813e95 ("test/py: rewrite common tools for SquashFS tests")
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Some builds of squashfs-tools append version string with "-git" or
similar. The float() conversion will fail in this case.
Improve the code to only convert to float() the string before the '-'
character.
Signed-off-by: Marek Behún <marek.behun@nic.cz>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joao Marcos Costa <jmcosta944@gmail.com>
Add more details to test cases by comparing each expected line with the
command's output. Add new test cases:
- sqfsls at an empty directory
- sqfsls at a sub-directory
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Joao Marcos Costa <jmcosta944@gmail.com>
The previous strategy to know if a file was correctly loaded was to
check for how many bytes were read and compare it against the file's
original size. Since this is not a good solution, replace it by
comparing the checksum of the loaded bytes against the original file's
checksum. Add more test cases: files at a sub-directory and non-existent
file.
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Joao Marcos Costa <jmcosta944@gmail.com>
Remove the previous OOP approach, which was confusing and incomplete.
Add more test cases by making SquashFS images with various options,
concerning file fragmentation and its compression. Add comments to
properly document the code.
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Joao Marcos Costa <jmcosta944@gmail.com>
Some filesystem tests are failing when their image is prepared with
guestmount, but succeeding if loop mounts are used instead. The reason
seems to be a race condition the guestmount(1) manual page explains:
When guestunmount(1)/fusermount(1) exits, guestmount may still be
running and cleaning up the mountpoint. The disk image will not be
fully finalized.
This means that scripts like the following have a nasty race condition:
guestmount -a disk.img -i /mnt
# copy things into /mnt
guestunmount /mnt
# immediately try to use 'disk.img' ** UNSAFE **
The solution is to use the --pid-file option to write the guestmount
PID to a file, then after guestunmount spin waiting for this PID to
exit.
The Python standard library has an os.waitpid() function for waiting a
child to terminate, but it cannot wait on non-child processes. Implement
a utility function that can do this by polling the process repeatedly
for a given duration, optionally killing the process if it won't
terminate on its own. Apply the suggested solution with this utility
function, which makes the failing tests succeed again.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
If guestmount isn't available on the system, filesystem test setup falls
back to using loop mounts to prepare its disk images. If guestmount is
available but fails to work, the tests are immediately skipped. Instead
of giving up on a guestmount failure, try using loop mounts as an
attempt to keep tests running.
Also stop checking if guestmount is in PATH, as trying to run a missing
guestmount can now follow the same failure codepath and fall back to
loop mounts anyway.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
On some distributions the mkfs is under /sbin and /sbin is not set
for mere users. Include /sbin to the PATH when creating file system,
so that users won't get a scary traceback from Python.
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Commit 1ba21bb06b ("test: Don't unmount not (yet) mounted system")
fixes an issue in the filesystem tests where the test setup may fail
to mount an image and still attempt to unmount it. However, the commit
unintentionally breaks the test setups in two ways.
The newly created unmounted filesystem images are being immediately
deleted due to some cleanup steps being misplaced into finally blocks,
which makes them always run instead of only on failures. The mount calls
always fail since the images never exist, causing the tests to be always
skipped. This patch moves these cleanup calls into the except blocks to
fix this and makes the tests run again.
There are also unmount calls misplaced into finally blocks, making them
run after the tests instead of before the tests. These unmount calls
make the filesystem image file consistent with the changes made to it as
part of the test setup, and this misplacement is making a number of
tests fail unexpectedly.
The unmount calls must be run before the tests use the image, meaning
before the yield call and not in the finally block. They must also be
run as a cleanup step when the filesystem setup fails, so they can't be
placed as the final call in the try blocks since they would be skipped
on such failures. For these reasons, this patch places the unmount calls
both in the except blocks and the else blocks of the final setup step.
This makes the unexpectedly failing tests to succeed again.
Furthermore, this isolates the mount calls to their own try-except
statement to avoid reintroducing the original issue of unmounting a
not-mounted image while fixing the unmount misplacement.
After these fixes, running "make tests" with guestmount available results
in two test failures not related to the mentioned commit. If the
guestmount executables are unavailable, the mounts fallback to using
sudo and result in no failures.
Fixes: 1ba21bb06b ("test: Don't unmount not (yet) mounted system")
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
When test suite tries to create a file for a new filesystem test case and fails,
the clean up of the exception tries to unmount the image, that has not yet been
mounted. When it happens, the fuse_mounted global variable is set to False and
inconveniently the test case tries to use sudo, so without this change the
admin of the machine gets an (annoying) email:
Subject: *** SECURITY information for example.com ***
example.com : Feb 5 19:43:47 : ... COMMAND=/bin/umount .../build-sandbox/persistent-data/mnt
and second run of the test cases on uncleaned build folder will ask for sudo
which is not what expected.
Besides that there is a double unmount calls during successfully run test case.
All of these due to over engineered Python try-except clause and people didn't
get it properly at all. The rule of thumb is that don't use more keywords than
try-except in the exception handling code. Nevertheless, here we adjust code
to be less intrusive to the initial logic behind that complex and unclear
constructions in the test case, although it adds a lot of lines of the code,
i.e. splits one exception handler to three, so on each step we know what
cleanup shall perform.
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
As noted in comments, yield_fixture has been deprecated for longer than
our minimum required version of pytest. Newer versions of pytest cause
this to be a louder warning, and as the migration is trivial, perform it
now.
Signed-off-by: Tom Rini <trini@konsulko.com>
do_save() function defined in fs.c also supports FAT file system
re-use the same for fatwrite command.
Also fix the FAT test script to match the expected output.
Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
Reviewed-by: Biju Das <biju.das.jz@bp.renesas.com>
Improve SquashFS tests architecture. Add 'Compression' class. LZO
algorithm may crash if the file is fragmented, so the fragments are
disabled when testing LZO.
Signed-off-by: Joao Marcos Costa <joaomarcos.costa@bootlin.com>
Use "cons.config.build_dir" instead of writing to the source directory
(read-only). This will fix the test failures in Azure.
Signed-off-by: Joao Marcos Costa <joaomarcos.costa@bootlin.com>
Modify various test/py filesystem creation routines to support systems
that don't implement the metadata_csum ext4 feature.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Add Python scripts to test 'ls' and 'load' commands. The scripts
generate a SquashFS image and clean the directory after the assertions,
or if an exception is raised.
Signed-off-by: Joao Marcos Costa <joaomarcos.costa@bootlin.com>
Added command "fstypes" to list supported/included filesystems.
Signed-off-by: Niel Fourie <lusus@denx.de>
Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
[trini: Limit to sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
For non-root users mkfs.vfat is not in the search path at least on Debian.
Hence when running 'make tests' a message indicates that file system tests
have been skipped:
SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16
This message is not really helpful as the executed program is not
indicated. Provide a more complete message like
SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for
filesystem: fat16.
Command 'mkfs.vfat -F 16 build-sandbox/persistent-data/3GB.fat16.img'
returned non-zero exit status 127.
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
# This is actually a resent patch of
# [1] https://lists.denx.de/pipermail/u-boot/2019-May/369170.html
Two test cases are added under test_fs_ext:
test case 10: for root directory
test case 11: for non-root directory
Those will verify a behavior fixed by the commits related to
root directory
("fs: fat: allocate a new cluster for root directory of fat32" and
"fs: fat: flush a directory cluster properly"), and focus on
handling long-file-name directory entries under a directory.
Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
The check_output function from the subprocess Python module by default
returns data as encoded bytes and leaves decoding to the application.
Given our uses of the call, it makes the most sense to immediately
decode the results.
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
Use the 2to3 tool to perform numerous automatic conversions from Python
2 syntax to Python 3. Also fix whitespace problems that Python 3
catches that Python 2 did not.
Reviewed-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Simon Glass <sjg@chromium.org> [on sandbox]
Signed-off-by: Tom Rini <trini@konsulko.com>
Currently enabling fsck on FAT16/FAT32 exposes that we have problems
with:
TestFsBasic.test_fs13[fat16]
TestFsBasic.test_fs11[fat32]
TestFsBasic.test_fs12[fat32]
TestFsBasic.test_fs13[fat32]
TestFsExt.test_fs_ext1[fat32]
TestFsExt.test_fs_ext2[fat32]
TestFsExt.test_fs_ext3[fat32]
TestFsExt.test_fs_ext4[fat32]
TestFsExt.test_fs_ext5[fat32]
TestFsExt.test_fs_ext6[fat32]
TestFsExt.test_fs_ext7[fat32]
TestFsExt.test_fs_ext8[fat32]
TestFsExt.test_fs_ext9[fat32]
TestMkdir.test_mkdir6[fat16]
TestMkdir.test_mkdir1[fat32]
TestMkdir.test_mkdir2[fat32]
TestMkdir.test_mkdir3[fat32]
TestMkdir.test_mkdir4[fat32]
TestMkdir.test_mkdir5[fat32]
TestMkdir.test_mkdir6[fat32]
TestUnlink.test_unlink1[fat16]
TestUnlink.test_unlink2[fat16]
TestUnlink.test_unlink3[fat16]
TestUnlink.test_unlink4[fat16]
TestUnlink.test_unlink5[fat16]
TestUnlink.test_unlink6[fat16]
TestUnlink.test_unlink7[fat16]
TestUnlink.test_unlink1[fat32]
TestUnlink.test_unlink2[fat32]
TestUnlink.test_unlink3[fat32]
TestUnlink.test_unlink4[fat32]
TestUnlink.test_unlink5[fat32]
TestUnlink.test_unlink6[fat32]
TestUnlink.test_unlink7[fat32]
This is because we don't update the "information sector" on FAT32.
While in the future we should resolve this problem and include that
feature, we should enable fsck for ext4 to ensure that things remain in
good shape there.
Signed-off-by: Tom Rini <trini@konsulko.com>
Test cases are:
1) basic link creation, verify it can be followed
2) chained links, verify it can be followed
3) replace exiting file a with a link, and a link with a link. verify it
can be followed
4) create a broken link, verify it can't be followed
Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
We need to make sure that file writes,file creation, etc. are properly
performed and do not corrupt the filesystem.
To help with this, introduce the assert_fs_integrity() function that
executes the appropriate fsck tool. It should be called at the end of any
test that modify the content/organization of the filesystem.
Currently only supports FATs and EXT4.
Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
If the metadata checksums are enabled, all write operations will fail.
Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Some tests have ended up using double quotes where single quotes could be
used. Adjust this for consistency with the rest of U-Boot's Python code.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
At present tests are quite slow to run, over a minute on my machine. This
presents a considerable barrier to bisecting for failures.
The slowest tests are the filesystem ones and the buildman --fetch-arch
test. Add a new 'qcheck' target that skips these tests. This reduces test
time down to about 40 second, still too long, but bearable.
Signed-off-by: Simon Glass <sjg@chromium.org>
After Siomon's comment, add a descriptive comment (docstring) to each of
helper functions in conftest.py. No functionality changed.
Signed-off-by: Akashi Takahiro <takahiro.akashi@linaro.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Since there is no use of fs_type in umount_fs(), just remove it.
Signed-off-by: Akashi Takahiro <takahiro.akashi@linaro.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
In this commit, test cases for unlink interfaces are added as part of
"test_fs" test suite.
Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
Signed-off-by: Alexander Graf <agraf@suse.de>
