Errata application is painful for performance. For a start, it's done
when the core has just come out of reset, which means branch predictors
and caches will be empty so a branch to a workaround function must be
fetched from memory and that round trip is very slow. Then it also runs
with the I-cache off, which means that the loop to iterate over the
workarounds must also be fetched from memory on each iteration.
We can remove both branches. First, we can simply apply every erratum
directly instead of defining a workaround function and jumping to it.
Currently, no errata that need to be applied at both reset and runtime,
with the same workaround function, exist. If the need arose in future,
this should be achievable with a reset + runtime wrapper combo.
Then, we can construct a function that applies each erratum linearly
instead of looping over the list. If this function is part of the reset
function, then the only "far" branches at reset will be for the checker
functions. Importantly, this mitigates the slowdown even when an erratum
is disabled.
The result is ~50% speedup on N1SDP and ~20% on AArch64 Juno on wakeup
from PSCI calls that end in powerdown. This is roughly back to the
baseline of v2.9, before the errata framework regressed on performance
(or a little better). It is important to note that there are other
slowdowns since then that remain unknown.
Change-Id: Ie4d5288a331b11fd648e5c4a0b652b74160b07b9
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Similar to the cpu_rev_var and cpu_ger_rev_var functions, inline the
call_reset_handler handler. This way we skip the costly branch at no
extra cost as this is the only place where this is called.
While we're at it, drop the options for CPU_NO_RESET_FUNC. The only cpus
that need that are virtual cpus which can spare the tiny bit of
performance lost. The rest are real cores which can save on the check
for zero.
Now is a good time to put the assert for a missing cpu in the
get_cpu_ops_ptr function so that it's a bit better encapsulated.
Change-Id: Ia7c3dcd13b75e5d7c8bafad4698994ea65f42406
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
The simplistic view of a core's powerdown sequence is that power is
atomically cut upon calling `wfi`. However, it turns out that it has
lots to do - it has to talk to the interconnect to exit coherency, clean
caches, check for RAS errors, etc. These take significant amounts of
time and are certainly not atomic. As such there is a significant window
of opportunity for external events to happen. Many of these steps are
not destructive to context, so theoretically, the core can just "give
up" half way (or roll certain actions back) and carry on running. The
point in this sequence after which roll back is not possible is called
the point of no return.
One of these actions is the checking for RAS errors. It is possible for
one to happen during this lengthy sequence, or at least remain
undiscovered until that point. If the core were to continue powerdown
when that happens, there would be no (easy) way to inform anyone about
it. Rejecting the powerdown and letting software handle the error is the
best way to implement this.
Arm cores since at least the a510 have included this exact feature. So
far it hasn't been deemed necessary to account for it in firmware due to
the low likelihood of this happening. However, events like GIC wakeup
requests are much more probable. Older cores will powerdown and
immediately power back up when this happens. Travis and Gelas include a
feature similar to the RAS case above, called powerdown abandon. The
idea is that this will improve the latency to service the interrupt by
saving on work which the core and software need to do.
So far firmware has relied on the `wfi` being the point of no return and
if it doesn't explicitly detect a pending interrupt quite early on, it
will embark onto a sequence that it expects to end with shutdown. To
accommodate for it not being a point of no return, we must undo all of
the system management we did, just like in the warm boot entrypoint.
To achieve that, the pwr_domain_pwr_down_wfi hook must not be terminal.
Most recent platforms do some platform management and finish on the
standard `wfi`, followed by a panic or an endless loop as this is
expected to not return. To make this generic, any platform that wishes
to support wakeups must instead let common code call
`psci_power_down_wfi()` right after. Besides wakeups, this lets common
code handle powerdown errata better as well.
Then, the CPU_OFF case is simple - PSCI does not allow it to return. So
the best that can be done is to attempt the `wfi` a few times (the
choice of 32 is arbitrary) in the hope that the wakeup is transient. If
it isn't, the only choice is to panic, as the system is likely to be in
a bad state, eg. interrupts weren't routed away. The same applies for
SYSTEM_OFF, SYSTEM_RESET, and SYSTEM_RESET2. There the panic won't
matter as the system is going offline one way or another. The RAS case
will be considered in a separate patch.
Now, the CPU_SUSPEND case is more involved. First, to powerdown it must
wipe its context as it is not written on warm boot. But it cannot be
overwritten in case of a wakeup. To avoid the catch 22, save a copy that
will only be used if powerdown fails. That is about 500 bytes on the
stack so it hopefully doesn't tip anyone over any limits. In future that
can be avoided by having a core manage its own context.
Second, when the core wakes up, it must undo anything it did to prepare
for poweroff, which for the cores we care about, is writing
CPUPWRCTLR_EL1.CORE_PWRDN_EN. The least intrusive for the cpu library
way of doing this is to simply call the power off hook again and have
the hook toggle the bit. If in the future there need to be more complex
sequences, their direction can be advised on the value of this bit.
Third, do the actual "resume". Most of the logic is already there for
the retention suspend, so that only needs a small touch up to apply to
the powerdown case as well. The missing bit is the powerdown specific
state management. Luckily, the warmboot entrypoint does exactly that
already too, so steal that and we're done.
All of this is hidden behind a FEAT_PABANDON flag since it has a large
memory and runtime cost that we don't want to burden non pabandon cores
with.
Finally, do some function renaming to better reflect their purpose and
make names a little bit more consistent.
Change-Id: I2405b59300c2e24ce02e266f91b7c51474c1145f
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Currently both FEAT_MTE and FEAT_MTE_PERM aren't used for enabling
of any feature bits in EL3. So remove both FEAT handling.
All mte regs that are currently context saved/restored are needed
only when FEAT_MTE2 is enabled, so move to usage of FEAT_MTE2 and
remove FEAT_MTE usage.
BREAKING CHANGE: Any platform or downstream code trying to use
SCR_EL3.ATA bit(26) will see failures as this is now moved to be
used only with FEAT_MTE2 with
commit@ef0d0e5478a3f19cbe70a378b9b184036db38fe2
Change-Id: Id01e154156571f7792135639e17dc5c8d0e17cf8
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>
Add support for feat mte2. tfsr_el2 is available only with mte2,
however currently its context_save/restore is done with mte rather than
mte2, so introduce 'is_feat_mte2_supported' to check mte2.
Change-Id: I108d9989a8f5b4d1d2f3b9865a914056fa566cf2
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>
The former link pointed to a page which displayed the following warning
message:
We could not find that page in the latest version, so we have taken
you to the first page instead
Signed-off-by: Sandrine Bailleux <sandrine.bailleux@arm.com>
Change-Id: Icf9277770e38bc5e602b75052c2386301984238d
Currently CTX_INCLUDE_MTE_REGS is used for dual purpose,
to enable allocation tags register and to context save and restore
them and also to check if mte feature is available.
To make it more meaningful, remove CTX_INCLUDE_MTE_REGS
and introduce FEAT_MTE. This would enable allocation tags register
when FEAT_MTE is enabled and also supported from platform.
Also arch features can be conditionally enabled disabled based on
arch version from `make_helpers/arch_features.mk`
Change-Id: Ibdd2d43874634ad7ddff93c7edad6044ae1631ed
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>
There is no platform function to retrieve the info in the generic code.
Populate the BL32 image base, size and max limit in arg2, arg3 and arg4.
Signed-off-by: Nishant Sharma <nishant.sharma@arm.com>
Change-Id: I35527fb41829102083b488a5150c0c707c5ede15
commit@7794d6c8f8c44acc14fbdc5ada5965310056be1e added a march utility
but the details were not updated in docs.
Update docs to provide a glimpse of march utility added.
Change-Id: I696cb9a701a30d7bf36a1ecd38a80d07df1fd551
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>
Both march32-directive and march64-directive eventually generate the
same march option that will passed to compiler.
Merge this two separate directives to a common one as march-directive.
Change-Id: I220d2b782eb3b54e13ffd5b6a581d0e6da68756a
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>
Also add a recommended Procedure Call Standard (PCS) to use inside CPU
files and split everything into sections to make it easier to follow.
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: Idd876d8e598b5dfe1193aa3e7375c52f6edf5671
PSCI specification reference in the documentation is updated
to point to latest specification and duplicate PSCI references are
removed.
Change-Id: I35ee365f08c557f3017af4d51f6d063a7501b27e
Signed-off-by: Manish V Badarkhe <Manish.Badarkhe@arm.com>
Add symbols to mark end of TEXT/RODATA before page alignment.
Similar change was done by commit 8d69a03f6a ("Various
improvements/cleanups on the linker scripts") for
RO_END/COHERENT_RAM. These symbols help to know how much free
space is in the final binary because of page alignment.
Also show all *UNALIGNED__ symbols via poetry.
For example:
poetry run memory -p zynqmp -b debug
Change-Id: I322beba37dad76be9f4e88ca7e5b3eff2df7d96e
Signed-off-by: Michal Simek <michal.simek@amd.com>
Updated the documentation to include missing details about the
TZC secured DRAM mapping for the FVP and Juno platforms.
Change-Id: I10e59b9f9686fa2fef97f89864ebc272b10e5c0b
Signed-off-by: Manish V Badarkhe <Manish.Badarkhe@arm.com>
It hasn't been updated since 2017 and the documentation around that bit
of code is fairly good so it is redundant to be there.
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: Idee4523e97cb6039fae1efae35eda2b45e8f7345
BL2_AT_EL3 is an overloaded macro which has two uses:
1. When BL2 is entry point into TF-A(no BL1)
2. When BL2 is running at EL3 exception level
These two scenarios are not exactly same even though first implicitly
means second to be true. To distinguish between these two use cases we
introduce new macros.
BL2_AT_EL3 is renamed to RESET_TO_BL2 to better convey both 1. and 2.
Additional macro BL2_RUNS_AT_EL3 is added to cover all scenarious where
BL2 runs at EL3 (including four world systems).
BREAKING CHANGE: BL2_AT_EL3 renamed to RESET_TO_BL2 across the
repository.
Change-Id: I477e1d0f843b44b799c216670e028fcb3509fb72
Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Signed-off-by: Maksims Svecovs <maksims.svecovs@arm.com>
Some of our specialized sections are not prefixed with the conventional
period. The compiler uses input section names to derive certain other
section names (e.g. `.rela.text`, `.relacpu_ops`), and these can be
difficult to select in linker scripts when there is a lack of a
delimiter.
This change introduces the period prefix to all specialized section
names.
BREAKING-CHANGE: All input and output linker section names have been
prefixed with the period character, e.g. `cpu_ops` -> `.cpu_ops`.
Change-Id: I51c13c5266d5975fbd944ef4961328e72f82fc1c
Signed-off-by: Chris Kay <chris.kay@arm.com>
As the max ESPI can be 5119, so enlarge the intr_num range of
structure interrupt_prop_t. After the patch the ESPI can be add to
this macro:
define PLATFORM_G1S_PROPS(grp) \
INTR_PROP_DESC(197 - 32 + 4576, GIC_HIGHEST_SEC_PRIORITY, (grp), GIC_INTR_CFG_EDGE), \
INTR_PROP_DESC(199 - 32 + 4576, GIC_HIGHEST_SEC_PRIORITY, (grp), GIC_INTR_CFG_EDGE)
The firmware-design.rst will be updated accordingly.
Signed-off-by: Ming Huang <huangming@linux.alibaba.com>
Change-Id: Ic923868bb1b00c017410dc2aeabfda58ee54782f
SMCCCv1.3 introduces the SVE hint bit added to the SMC FID (bit 16)
denoting that the world issuing an SMC doesn't expect the callee to
preserve the SVE state (FFR, predicates, Zn vector bits greater than
127). Update the generic SMC handler to copy the SVE hint bit state
to SMC flags and mask out the bit by default for the services called
by the standard dispatcher. It is permitted by the SMCCC standard to
ignore the bit as long as the SVE state is preserved. In any case a
callee must preserve the NEON state (FPCR/FPSR, Vn 128b vectors)
whichever the SVE hint bit state.
Signed-off-by: Olivier Deprez <olivier.deprez@arm.com>
Change-Id: I2b163ed83dc311b8f81f96b23c942829ae9fa1b5
As per change [1], now HW_CONFIG gets loaded in secure and
non-secure memory. Hence updated the documentation to show
secure and non-secure load region of HW_CONFIG in FVP Arm
platform.
Additionally, added a note on how FW_CONFIG address gets
passed from BL2 to BL31/SP_MIN.
[1]: https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/14620
Change-Id: I37e02ff4f433c87bccbe67c7df5ecde3017668b9
Signed-off-by: Manish V Badarkhe <Manish.Badarkhe@arm.com>
This patch expands the RME documentation with description of TF-A
changes for RME. It also modifies some other parts of TF-A documentation
to account for RME changes.
Signed-off-by: Zelalem Aweke <zelalem.aweke@arm.com>
Change-Id: I9e6feeee235f0ba4b767d239f15840f1e0c540bb
there are features which are marked as experimental even though they
are stable and used for quite some time.
Following features are no longer marked as experimental
- SPMD
- MEASURED_BOOT
- FCONF and associated build flags
- DECRYPTION_SUPPORT and associated build flags
- ENABLE_PAUTH
- ENABLE_BTI
- USE_SPINLOCK_CAS
- GICv3 Multichip support
Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: I4bb653d9c413c66095ec31f0b8aefeb13ea04ee9
To avoid trapping from EL0/1, FPEN bits need to be set 0x3, not
clearing.
Signed-off-by: Peng Fan <peng.fan@nxp.com>
Change-Id: Ic34e9aeb876872883c5f040618ed6d50f21dacd0
These broken links were found with the help of this command:
$> sphinx-build -M linkcheck . build
A sample broken link is reported as follows:
(line 80) -local- firmware-design.rst#secure-el1-payloads-and-dispatchers
Change-Id: I5dcefdd4b8040908658115647e957f6c2c5da7c2
Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
The link to the exception handling framework page on the System Design /
Firmware Design / Section 4.3 just links to itself, so I changed it to
link to the exception handling framework component document.
Signed-off-by: John Powell <john.powell@arm.com>
Change-Id: I6711b423a789b2b3d1921671e8497fffa8ba33d1
Captured the increase in firmware configuration area from
4KB to 8kB in memory layout document. Updated the documentation
to provide details about fw_config separately.
Signed-off-by: Manish V Badarkhe <Manish.Badarkhe@arm.com>
Change-Id: Ifbec443ced479301be65827b49ff4fe447e9109f
This is a fixup for patch 3ba55a3c5f
("docs: Update SMCCC doc, other changes for release"), where some
links names got changed but their references didn't.
Change-Id: I980d04dde338f3539a2ec1ae2e807440587b1cf5
Signed-off-by: Sandrine Bailleux <sandrine.bailleux@arm.com>
To accommodate the increasing size of the SCP_BL2 binary, the base
address of the memory region allocated to SCP_BL2 has been moved
downwards from its current (mostly) arbitrary address to the beginning
of the non-shared trusted SRAM.
Change-Id: I086a3765bf3ea88f45525223d765dc0dbad6b434
Signed-off-by: Chris Kay <chris.kay@arm.com>
This patch provides a fix for incorrect crash dump data for
lower EL when TF-A is built with HANDLE_EA_EL3_FIRST=1 option
which enables routing of External Aborts and SErrors to EL3.
Change-Id: I9d5e6775e6aad21db5b78362da6c3a3d897df977
Signed-off-by: Alexei Fedorov <Alexei.Fedorov@arm.com>
This patch adds information on the PMU configuration registers
and security considerations related to the PMU.
Signed-off-by: Petre-Ionut Tudor <petre-ionut.tudor@arm.com>
Change-Id: I36b15060b9830a77d3f47f293c0a6dafa3c581fb
Some platforms are extremely memory constrained and must split BL31
between multiple non-contiguous areas in SRAM. Allow the NOBITS
sections (.bss, stacks, page tables, and coherent memory) to be placed
in a separate region of RAM from the loaded firmware image.
Because the NOBITS region may be at a lower address than the rest of
BL31, __RW_{START,END}__ and __BL31_{START,END}__ cannot include this
region, or el3_entrypoint_common would attempt to invalidate the dcache
for the entire address space. New symbols __NOBITS_{START,END}__ are
added when SEPARATE_NOBITS_REGION is enabled, and the dcached for the
NOBITS region is invalidated separately.
Signed-off-by: Samuel Holland <samuel@sholland.org>
Change-Id: Idedfec5e4dbee77e94f2fdd356e6ae6f4dc79d37
The User Guide document has grown organically over time and
now covers a wide range of topics, making it difficult to
skim read and extract information from. Currently, it covers
these topics and maybe a couple more:
- Requirements (hardware, tools, libs)
- Checking out the repo
- Basic build instructions
- A comprehensive list of build flags
- FIP packaging
- Building specifically for Juno
- Firmware update images
- EL3 payloads
- Preloaded BL33 boot flow
- Running on FVPs
- Running on Juno
I have separated these out into a few groups that become new
documents. Broadly speaking, build instructions for the tools,
for TF-A generally, and for specific scenarios are separated.
Content relating to specific platforms (Juno and the FVPs are
Arm-specific platforms, essentially) has been moved into the
documentation that is specific to those platforms, under
docs/plat/arm.
Change-Id: Ica87c52d8cd4f577332be0b0738998ea3ba3bbec
Signed-off-by: Paul Beesley <paul.beesley@arm.com>
From AArch64 state, arguments are passed in registers W0-W7(X0-X7)
and results are returned in W0-W7(X0-X7) for SMC32(SMC64) calls.
From AArch32 state, arguments are passed in registers R0-R7 and
results are returned in registers R0-R7 for SMC32 calls.
Most of the functions and macros already existed to support using
upto 8 registers for passing/returning parameters/results. Added
few helper macros for SMC calls from AArch32 state.
Link to the specification:
https://developer.arm.com/docs/den0028/c
Change-Id: I87976b42454dc3fc45c8343e9640aa78210e9741
Signed-off-by: Madhukar Pappireddy <madhukar.pappireddy@arm.com>
Tidying up a few Sphinx warnings that had built-up over time.
None of these are critical but it cleans up the Sphinx output.
At the same time, fixing some spelling errors that were detected.
Change-Id: I38209e235481eed287f8008c6de9dedd6b12ab2e
Signed-off-by: Paul Beesley <paul.beesley@arm.com>
Currently links between documents are using the format:
<path/to/><filename>.rst
This was required for services like GitHub because they render each
document in isolation - linking to another document is like linking
to any other file, just provide the full path.
However, with the new approach, the .rst files are only the raw
source for the documents. Once the documents have been rendered
the output is now in another format (HTML in our case) and so,
when linking to another document, the link must point to the
rendered version and not the .rst file.
The RST spec provides a few methods for linking between content.
The parent of this patch enabled the automatic creation of anchors
for document titles - we will use these anchors as the targets for
our links. Additional anchors can be added by hand if needed, on
section and sub-section titles, for example.
An example of this new format, for a document with the title
"Firmware Design" is :ref:`Firmware Design`.
One big advantage of this is that anchors are not dependent on
paths. We can then move documents around, even between directories,
without breaking any links between documents. Links will need to be
updated only if the title of a document changes.
Change-Id: I9e2340a61dd424cbd8fd1ecc2dc166f460d81703
Signed-off-by: Paul Beesley <paul.beesley@arm.com>
Make the spinlock implementation use ARMv8.1-LSE CAS instruction based
on a platform build option. The CAS-based implementation used to be
unconditionally selected for all ARM8.1+ platforms.
The previous CAS spinlock implementation had a bug wherein the spin_unlock()
implementation had an `sev` after `stlr` which is not sufficient. A dsb is
needed to ensure that the stlr completes prior to the sev. Having a dsb is
heavyweight and a better solution would be to use load exclusive semantics
to monitor the lock and wake up from wfe when a store happens to the lock.
The patch implements the same.
Change-Id: I5283ce4a889376e4cc01d1b9d09afa8229a2e522
Signed-off-by: Soby Mathew <soby.mathew@arm.com>
Signed-off-by: Olivier Deprez <olivier.deprez@arm.com>
A new build flag, CTX_INCLUDE_MTE_REGS, has been added; this patch adds
documentation for it in the User Guide along with instructions of what
different values mean.
Change-Id: I430a9c6ced06b1b6be317edbeff4f5530e30f63a
Signed-off-by: Justin Chadwell <justin.chadwell@arm.com>
This patch adds the functionality needed for platforms to provide
Branch Target Identification (BTI) extension, introduced to AArch64
in Armv8.5-A by adding BTI instruction used to mark valid targets
for indirect branches. The patch sets new GP bit [50] to the stage 1
Translation Table Block and Page entries to denote guarded EL3 code
pages which will cause processor to trap instructions in protected
pages trying to perform an indirect branch to any instruction other
than BTI.
BTI feature is selected by BRANCH_PROTECTION option which supersedes
the previous ENABLE_PAUTH used for Armv8.3-A Pointer Authentication
and is disabled by default. Enabling BTI requires compiler support
and was tested with GCC versions 9.0.0, 9.0.1 and 10.0.0.
The assembly macros and helpers are modified to accommodate the BTI
instruction.
This is an experimental feature.
Note. The previous ENABLE_PAUTH build option to enable PAuth in EL3
is now made as an internal flag and BRANCH_PROTECTION flag should be
used instead to enable Pointer Authentication.
Note. USE_LIBROM=1 option is currently not supported.
Change-Id: Ifaf4438609b16647dc79468b70cd1f47a623362e
Signed-off-by: Alexei Fedorov <Alexei.Fedorov@arm.com>
