Errata application is painful for performance. For a start, it's done
when the core has just come out of reset, which means branch predictors
and caches will be empty so a branch to a workaround function must be
fetched from memory and that round trip is very slow. Then it also runs
with the I-cache off, which means that the loop to iterate over the
workarounds must also be fetched from memory on each iteration.
We can remove both branches. First, we can simply apply every erratum
directly instead of defining a workaround function and jumping to it.
Currently, no errata that need to be applied at both reset and runtime,
with the same workaround function, exist. If the need arose in future,
this should be achievable with a reset + runtime wrapper combo.
Then, we can construct a function that applies each erratum linearly
instead of looping over the list. If this function is part of the reset
function, then the only "far" branches at reset will be for the checker
functions. Importantly, this mitigates the slowdown even when an erratum
is disabled.
The result is ~50% speedup on N1SDP and ~20% on AArch64 Juno on wakeup
from PSCI calls that end in powerdown. This is roughly back to the
baseline of v2.9, before the errata framework regressed on performance
(or a little better). It is important to note that there are other
slowdowns since then that remain unknown.
Change-Id: Ie4d5288a331b11fd648e5c4a0b652b74160b07b9
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Errata printing is done directly via generic_errata_report.
This commit removes the unused \_cpu\()_errata_report
functions for all cores, and removes errata_func from cpu_ops.
Change-Id: I04fefbde5f0ff63b1f1cd17c864557a14070d68c
Signed-off-by: Ryan Everett <ryan.everett@arm.com>
Also, convert checker functions of errata which are enabled for all cpu
revisions to report correctly in preparation of the errata ABI.
Although the script from commit 250919 was used to check that errata
code did not change, this CPU only loosely adhered to convention and its
output was not particularly useful. Nevertheless, the discrepancies were
manually verified. All errata have been checked that they get invoked.
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: I988db6e7b6d1732f1d2258dbdf945cb475781894
This involves replacing:
* the reset_func with the standard cpu_reset_func_{start,end} to apply
errata automatically
* the <cpu>_errata_report with the errata_report_shim to report errata
automatically
...and for each erratum:
* the prologue with the workaround_<type>_start to do the checks and
framework registration automatically
* the epilogue with the workaround_<type>_end
* the checker function with the check_erratum_<type> to make it more
descriptive
It is important to note that the errata workaround and checking
sequences remain unchanged and preserve their git blame.
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: I30556f438859d17f054cb6bc96f3069b40474b58
Errata report order is enforced to be in ascending order. To achieve
this with the errata framework this has to be done at the definition
level.
Also rename the disable_non_temporal_hint to its erratum number to
conform to convention.
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: Id474872afebf361ab3d21c454ab3624db8354045
The ERRATA_XXX macros, used in cpu_helpers.S, are necessary for the
check_errata_xxx family of functions. The CPU_REV should be used in the
cpu files but for whatever reason the values have been hard-coded so far
(at the cost of readability). It's evident this file is not strictly for
status reporting.
The new purpose of this file is to make it a one-stop-shop for all
things errata.
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Change-Id: I1ce22dd36df5aa0bcfc5f2772251f91af8703dfb
Reported the status (applies, missing) of AT speculative workaround
which is applicable for below CPUs.
+---------+--------------+
| Errata | CPU |
+=========+==============+
| 1165522 | Cortex-A76 |
+---------+--------------+
| 1319367 | Cortex-A72 |
+---------+--------------+
| 1319537 | Cortex-A57 |
+---------+--------------+
| 1530923 | Cortex-A55 |
+---------+--------------+
| 1530924 | Cortex-A53 |
+---------+--------------+
Also, changes are done to enable common macro 'ERRATA_SPECULATIVE_AT'
if AT speculative errata workaround is enabled for any of the above
CPUs using 'ERRATA_*' CPU specific build macro.
Signed-off-by: Manish V Badarkhe <Manish.Badarkhe@arm.com>
Change-Id: I3e6a5316a2564071f3920c3ce9ae9a29adbe435b
We previously left our caches on during power-down to prevent any
non-caching accesses to memory that is cached by other cores. Now with
the last accessed areas all being marked as non-cached by
USE_COHERENT_MEM we can rely on that to workaround our interconnect
issues. Remove the old workaround.
Change-Id: Idadb7696d1449499d1edff4f6f62ab3b99d1efb7
Signed-off-by: Andrew F. Davis <afd@ti.com>
Errata 819472, 824069, and 827319 are currently reported in a warning as
missing during boot for platforms that do not need them. Only warn when
the errata is needed for a given revision but not compiled in like other
errata workarounds.
Fixes: bd393704d2 ("Cortex-A53: Workarounds for 819472, 824069 and 827319")
Signed-off-by: Andrew F. Davis <afd@ti.com>
Change-Id: Ifd757b3d0e73a9bd465b98dc20648b6c13397d8d
The workarounds for these errata are so closely related that it is
better to only have one patch to make it easier to understand.
Change-Id: I0287fa69aefa8b72f884833f6ed0e7775ca834e9
Signed-off-by: Ambroise Vincent <ambroise.vincent@arm.com>
Enforce full include path for includes. Deprecate old paths.
The following folders inside include/lib have been left unchanged:
- include/lib/cpus/${ARCH}
- include/lib/el3_runtime/${ARCH}
The reason for this change is that having a global namespace for
includes isn't a good idea. It defeats one of the advantages of having
folders and it introduces problems that are sometimes subtle (because
you may not know the header you are actually including if there are two
of them).
For example, this patch had to be created because two headers were
called the same way: e0ea0928d5 ("Fix gpio includes of mt8173 platform
to avoid collision."). More recently, this patch has had similar
problems: 46f9b2c3a2 ("drivers: add tzc380 support").
This problem was introduced in commit 4ecca33988 ("Move include and
source files to logical locations"). At that time, there weren't too
many headers so it wasn't a real issue. However, time has shown that
this creates problems.
Platforms that want to preserve the way they include headers may add the
removed paths to PLAT_INCLUDES, but this is discouraged.
Change-Id: I39dc53ed98f9e297a5966e723d1936d6ccf2fc8f
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
Leave the caches on and explicitly flush any data that
may be stale when the core is powered down. This prevents
non-coherent interconnect access which has negative side-
effects on AM65x.
Signed-off-by: Andrew F. Davis <afd@ti.com>
A fix for errata 835769 may be available in revisions r0p2, r0p3 or r0p4
of the Cortex-A53 processor. The presence of the fix is determined by
checking bit 7 in the REVIDR register.
If the fix is present we report ERRATA_NOT_APPLIES which silences the
erroneous 'missing workaround' warning.
Change-Id: Ib75b008e755e9ac648554ca9398024fdbea4a91a
Signed-off-by: Jonathan Wright <jonathan.wright@arm.com>
A fix for errata 843419 may be available in revision r0p4 of the
Cortex-A53 processor. The presence of the fix is determined by checking
bit 8 in the REVIDR register.
If the fix is present we report ERRATA_NOT_APPLIES which silences the
erroneous 'missing workaround' warning.
Change-Id: Ibd2a478df3e2a6325442a6a48a0bb0259dcfc1d7
Signed-off-by: Jonathan Wright <jonathan.wright@arm.com>
At present, various CPU register macros that refer to CPUACTLR are named
ACTLR. This patch fixes that.
The previous register names are retained, but guarded by the
ERROR_DEPRECATED macro, so as not to break platforms that continue using
the old names.
Change-Id: Ia872196d81803f8f390b887d149e0fd054df519b
Signed-off-by: Eleanor Bonnici <Eleanor.bonnici@arm.com>
These errata are only applicable to AArch64 state. See the errata notice
for more details:
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.epm048406/index.html
Introduce the build options ERRATA_A53_835769 and ERRATA_A53_843419.
Enable both of them for Juno.
Apply the 835769 workaround as following:
* Compile with -mfix-cortex-a53-835769
* Link with --fix-cortex-a53-835769
Apply the 843419 workaround as following:
* Link with --fix-cortex-a53-843419
The erratum 843419 workaround can lead the linker to create new sections
suffixed with "*.stub*" and 4KB aligned. The erratum 835769 can lead the
linker to create new "*.stub" sections with no particular alignment.
Also add support for LDFLAGS_aarch32 and LDFLAGS_aarch64 in Makefile for
architecture-specific linker options.
Change-Id: Iab3337e338b7a0a16b0d102404d9db98c154f8f8
Signed-off-by: Douglas Raillard <douglas.raillard@arm.com>
This patch makes all the defines in the CPU libraries unique,
by prefixing them with the CPU name.
NOTE: PLATFORMS USING THESE MACROS WILL HAVE TO UPDATE THEIR CODE
TO START USING THE UPDATED NAMES
Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>
To make software license auditing simpler, use SPDX[0] license
identifiers instead of duplicating the license text in every file.
NOTE: Files that have been imported by FreeBSD have not been modified.
[0]: https://spdx.org/
Change-Id: I80a00e1f641b8cc075ca5a95b10607ed9ed8761a
Signed-off-by: dp-arm <dimitris.papastamos@arm.com>
ARM erratum 855873 applies to all Cortex-A53 CPUs.
The recommended workaround is to promote "data cache clean"
instructions to "data cache clean and invalidate" instructions.
For core revisions of r0p3 and later this can be done by setting a bit
in the CPUACTLR_EL1 register, so that hardware takes care of the promotion.
As CPUACTLR_EL1 is both IMPLEMENTATION DEFINED and can be trapped to EL3,
we set the bit in firmware.
Also we dump this register upon crashing to provide more debug
information.
Enable the workaround for the Juno boards.
Change-Id: I3840114291958a406574ab6c49b01a9d9847fec8
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
The errata reporting policy is as follows:
- If an errata workaround is enabled:
- If it applies (i.e. the CPU is affected by the errata), an INFO
message is printed, confirming that the errata workaround has been
applied.
- If it does not apply, a VERBOSE message is printed, confirming
that the errata workaround has been skipped.
- If an errata workaround is not enabled, but would have applied had
it been, a WARN message is printed, alerting that errata workaround
is missing.
The CPU errata messages are printed by both BL1 (primary CPU only) and
runtime firmware on debug builds, once for each CPU/errata combination.
Relevant output from Juno r1 console when ARM Trusted Firmware is built
with PLAT=juno LOG_LEVEL=50 DEBUG=1:
VERBOSE: BL1: cortex_a57: errata workaround for 806969 was not applied
VERBOSE: BL1: cortex_a57: errata workaround for 813420 was not applied
INFO: BL1: cortex_a57: errata workaround for disable_ldnp_overread was applied
WARNING: BL1: cortex_a57: errata workaround for 826974 was missing!
WARNING: BL1: cortex_a57: errata workaround for 826977 was missing!
WARNING: BL1: cortex_a57: errata workaround for 828024 was missing!
WARNING: BL1: cortex_a57: errata workaround for 829520 was missing!
WARNING: BL1: cortex_a57: errata workaround for 833471 was missing!
...
VERBOSE: BL31: cortex_a57: errata workaround for 806969 was not applied
VERBOSE: BL31: cortex_a57: errata workaround for 813420 was not applied
INFO: BL31: cortex_a57: errata workaround for disable_ldnp_overread was applied
WARNING: BL31: cortex_a57: errata workaround for 826974 was missing!
WARNING: BL31: cortex_a57: errata workaround for 826977 was missing!
WARNING: BL31: cortex_a57: errata workaround for 828024 was missing!
WARNING: BL31: cortex_a57: errata workaround for 829520 was missing!
WARNING: BL31: cortex_a57: errata workaround for 833471 was missing!
...
VERBOSE: BL31: cortex_a53: errata workaround for 826319 was not applied
INFO: BL31: cortex_a53: errata workaround for disable_non_temporal_hint was applied
Also update documentation.
Change-Id: Iccf059d3348adb876ca121cdf5207bdbbacf2aba
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
Various CPU drivers in ARM Trusted Firmware register functions to handle
power-down operations. At present, separate functions are registered to
power down individual cores and clusters.
This scheme operates on the basis of core and cluster, and doesn't cater
for extending the hierarchy for power-down operations. For example,
future CPUs might support multiple threads which might need powering
down individually.
This patch therefore reworks the CPU operations framework to allow for
registering power down handlers on specific level basis. Henceforth:
- Generic code invokes CPU power down operations by the level
required.
- CPU drivers explicitly mention CPU_NO_RESET_FUNC when the CPU has no
reset function.
- CPU drivers register power down handlers as a list: a mandatory
handler for level 0, and optional handlers for higher levels.
All existing CPU drivers are adapted to the new CPU operations framework
without needing any functional changes within.
Also update firmware design guide.
Change-Id: I1826842d37a9e60a9e85fdcee7b4b8f6bc1ad043
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
This patch adds cpumerrsr_el1 and l2merrsr_el1 to the register dump on
error for applicable CPUs.
These registers hold the ECC errors on L1 and L2 caches.
This patch updates the A53, A57, A72, A73 (l2merrsr_el1 only) CPU libraries.
Signed-off-by: Naga Sureshkumar Relli <nagasure@xilinx.com>
The assembler helper function `print_revision_warning` is used when a
CPU specific operation is enabled in the debug build (e.g. an errata
workaround) but doesn't apply to the executing CPU's revision/part number.
However, in some cases the system integrator may want a single binary to
support multiple platforms with different IP versions, only some of which
contain a specific erratum. In this case, the warning can be emitted very
frequently when CPUs are being powered on/off.
This patch modifies this warning print behaviour so that it is emitted only
when LOG_LEVEL >= LOG_LEVEL_VERBOSE. The `debug.h` header file now contains
guard macros so that it can be included in assembly code.
Change-Id: Ic6e7a07f128dcdb8498a5bfdae920a8feeea1345
In the Cortex-A35/A53/A57 CPUs library code, some of the CPU specific
reset operations are skipped if they have already been applied in a
previous invocation of the reset handler. This precaution is not
required, as all these operations can be reapplied safely.
This patch removes the unneeded test-before-set instructions in
the reset handler for these CPUs.
Change-Id: Ib175952c814dc51f1b5125f76ed6c06a22b95167
The LDNP/STNP instructions as implemented on Cortex-A53 and
Cortex-A57 do not behave in a way most programmers expect, and will
most probably result in a significant speed degradation to any code
that employs them. The ARMv8-A architecture (see Document ARM DDI
0487A.h, section D3.4.3) allows cores to ignore the non-temporal hint
and treat LDNP/STNP as LDP/STP instead.
This patch introduces 2 new build flags:
A53_DISABLE_NON_TEMPORAL_HINT and A57_DISABLE_NON_TEMPORAL_HINT
to enforce this behaviour on Cortex-A53 and Cortex-A57. They are
enabled by default.
The string printed in debug builds when a specific CPU errata
workaround is compiled in but skipped at runtime has been
generalised, so that it can be reused for the non-temporal hint use
case as well.
Change-Id: I3e354f4797fd5d3959872a678e160322b13867a1
In order for the symbol table in the ELF file to contain the size of
functions written in assembly, it is necessary to report it to the
assembler using the .size directive.
To fulfil the above requirements, this patch introduces an 'endfunc'
macro which contains the .endfunc and .size directives. It also adds
a .func directive to the 'func' assembler macro.
The .func/.endfunc have been used so the assembler can fail if
endfunc is omitted.
FixesARM-Software/tf-issues#295
Change-Id: If8cb331b03d7f38fe7e3694d4de26f1075b278fc
Signed-off-by: Kévin Petit <kevin.petit@arm.com>
The CPU specific reset handlers no longer have the freedom
of using any general purpose register because it is being invoked
by the BL3-1 entry point in addition to BL1. The Cortex-A57 CPU
specific reset handler was overwriting x20 register which was being
used by the BL3-1 entry point to save the entry point information.
This patch fixes this bug by reworking the register allocation in the
Cortex-A57 reset handler to avoid using x20. The patch also
explicitly mentions the register clobber list for each of the
callee functions invoked by the reset handler
Change-Id: I28fcff8e742aeed883eaec8f6c4ee2bd3fce30df
This patch adds support to call the reset_handler() function in BL3-1 in the
cold and warm boot paths when another Boot ROM reset_handler() has already run.
This means the BL1 and BL3-1 versions of the CPU and platform specific reset
handlers may execute different code to each other. This enables a developer to
perform additional actions or undo actions already performed during the first
call of the reset handlers e.g. apply additional errata workarounds.
Typically, the reset handler will be first called from the BL1 Boot ROM. Any
additional functionality can be added to the reset handler when it is called
from BL3-1 resident in RW memory. The constant FIRST_RESET_HANDLER_CALL is used
to identify whether this is the first version of the reset handler code to be
executed or an overridden version of the code.
The Cortex-A57 errata workarounds are applied only if they have not already been
applied.
FixesARM-software/tf-issue#275
Change-Id: Id295f106e4fda23d6736debdade2ac7f2a9a9053
This patch adds level specific cache maintenance functions
to cache_helpers.S. The new functions 'dcsw_op_levelx',
where '1 <= x <= 3', allow to perform cache maintenance by
set/way for that particular level of cache. With this patch,
functions to support cache maintenance upto level 3 have
been implemented since it is the highest cache level for
most ARM SoCs.
These functions are now utilized in CPU specific power down
sequences to implement them as mandated by processor specific
technical reference manual.
Change-Id: Icd90ce6b51cff5a12863bcda01b93601417fd45c
This patch adds handlers for dumping Cortex-A57 and Cortex-A53 specific register
state to the CPU specific operations framework. The contents of CPUECTLR_EL1 are
dumped currently.
Change-Id: I63d3dbfc4ac52fef5e25a8cf6b937c6f0975c8ab
This patch adds CPU core and cluster power down sequences to the CPU specific
operations framework introduced in a earlier patch. Cortex-A53, Cortex-A57 and
generic AEM sequences have been added. The latter is suitable for the
Foundation and Base AEM FVPs. A pointer to each CPU's operations structure is
saved in the per-cpu data so that it can be easily accessed during power down
seqeunces.
An optional platform API has been introduced to allow a platform to disable the
Accelerator Coherency Port (ACP) during a cluster power down sequence. The weak
definition of this function (plat_disable_acp()) does not take any action. It
should be overriden with a strong definition if the ACP is present on a
platform.
Change-Id: I8d09bd40d2f528a28d2d3f19b77101178778685d
This patch introduces a framework which will allow CPUs to perform
implementation defined actions after a CPU reset, during a CPU or cluster power
down, and when a crash occurs. CPU specific reset handlers have been implemented
in this patch. Other handlers will be implemented in subsequent patches.
Also moved cpu_helpers.S to the new directory lib/cpus/aarch64/.
Change-Id: I1ca1bade4d101d11a898fb30fea2669f9b37b956
