Certain erratum workarounds like Neoverse N1 1542419, need a part
of their mitigation done in EL3 and the rest in lower EL. But currently
such workarounds return HIGHER_EL_MITIGATION which indicates that the
erratum has already been mitigated by a higher EL(EL3 in this case)
which causes the lower EL to not apply it's part of the mitigation.
This patch fixes this issue by adding support for split workarounds
so that on certain errata we return AFFECTED even though EL3 has
applied it's workaround. This is done by reusing the chosen field of
erratum_entry structure into a bitfield that has two bitfields -
Bit 0 indicates that the erratum has been enabled in build,
Bit 1 indicates that the erratum is a split workaround and should
return AFFECTED instead of HIGHER_EL_MITIGATION.
SDEN documentation:
https://developer.arm.com/documentation/SDEN885747/latest
Signed-off-by: Arvind Ram Prakash <arvind.ramprakash@arm.com>
Change-Id: Iec94d665b5f55609507a219a7d1771eb75e7f4a7
Errata application is painful for performance. For a start, it's done
when the core has just come out of reset, which means branch predictors
and caches will be empty so a branch to a workaround function must be
fetched from memory and that round trip is very slow. Then it also runs
with the I-cache off, which means that the loop to iterate over the
workarounds must also be fetched from memory on each iteration.
We can remove both branches. First, we can simply apply every erratum
directly instead of defining a workaround function and jumping to it.
Currently, no errata that need to be applied at both reset and runtime,
with the same workaround function, exist. If the need arose in future,
this should be achievable with a reset + runtime wrapper combo.
Then, we can construct a function that applies each erratum linearly
instead of looping over the list. If this function is part of the reset
function, then the only "far" branches at reset will be for the checker
functions. Importantly, this mitigates the slowdown even when an erratum
is disabled.
The result is ~50% speedup on N1SDP and ~20% on AArch64 Juno on wakeup
from PSCI calls that end in powerdown. This is roughly back to the
baseline of v2.9, before the errata framework regressed on performance
(or a little better). It is important to note that there are other
slowdowns since then that remain unknown.
Change-Id: Ie4d5288a331b11fd648e5c4a0b652b74160b07b9
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
The existing DSU errata workarounds hijack the errata framework's inner
workings to register with it. However, that is undesirable as any change
to the framework may end up missing these workarounds. So convert the
checks and workarounds to macros and have them included with the
standard wrappers.
The only problem with this is the is_scu_present_in_dsu weak function.
Fortunately, it is only needed for 2 of the errata and only on 3 cores.
So drop it, assuming the default behaviour and have the callers handle
the exception.
Change-Id: Iefa36325804ea093e938f867b9a6f49a6984b8ae
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
The errata framework has a helper to invoke workarounds, complete with a
cpu rev_var check. We can use that directly instead of the
apply_cpu_pwr_dwn_errata to save on some code, as well as an extra
branch. It's also more readable.
Also, apply_erratum invocation in cpu files don't need to check the
rev_var as that was already done by the cpu_ops dispatcher for us to end
up in the file.
Finally, X2 erratum 2768515 only applies in the powerdown sequence, i.e.
at runtime. It doesn't achieve anything at reset, so we can label it
accordingly.
Change-Id: I02f9dd7d0619feb54c870938ea186be5e3a6ca7b
Signed-off-by: Boyan Karatotev <boyan.karatotev@arm.com>
Errata printing is done directly via generic_errata_report.
This commit removes the unused \_cpu\()_errata_report
functions for all cores, and removes errata_func from cpu_ops.
Change-Id: I04fefbde5f0ff63b1f1cd17c864557a14070d68c
Signed-off-by: Ryan Everett <ryan.everett@arm.com>
Conversion to use CPU helpers for Neoverse-N1 testing done with
framework adaptation patch.
Change-Id: I2103f6e64daf0ee4c7b756083e5bf485f15c0e21
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
For N1, this involves replacing:
- The reset_func with the standard cpu_reset_func_{start,end}
to apply errata automatically
- The <cpu>_errata_report with the errata_report_shim to
report errata automatically
And for each erratum:
- The prologue with the workaround_<type>_start to do the checks and
framework registration automatically at reset or runtime
- The epilogue with the workaround_<type>_end
- The checker function with the check_erratum_<type> to check whether
the erratum applies on the revision of the CPU.
Testing was conducted by:
- Manual comparison of disassembly of converted functions with non-
converted functions:
aarch64-none-elf-objdump -D
<trusted-firmware-a with conversion>/build/fvp/release/bl31/bl31.elf
vs
aarch64-none-elf-objdump -D
<trusted-firmware-a clean repo>/build/fvp/release/bl31/bl31.elf
- Build for debug with all errata enabled and step through ArmDS
to ensure all functions are entered and the path remains the same
as before conversion to the new framework.
Change-Id: I2ea612d4c197dd73835fadda81f59732c19534f8
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Moving neoverse_n1_disable_speculative_loads function before reset
function to maintain git blame with refactor to new framework.
Change-Id: I79a4de9955a6f37e289456a743b946c0c4c8c27f
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
If RAS Extension is not implemented esb instruction executes as a NOP.
No need to have a check for RAS presence in the code.
Also, The handler is related to a synchronous exceptions which
implicitly is part of BL31 image only, so remove that check too.
Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: If4264504cba9f0642b7b9c581ae66cd4deace32b
So far we have the ENABLE_AMU build option to include AMU register
handling code for enabling and context switch. There is also an
ENABLE_FEAT_AMUv1 option, solely to protect the HAFGRTR_EL2 system
register handling. The latter needs some alignment with the new feature
scheme, but it conceptually overlaps with the ENABLE_AMU option.
Since there is no real need for two separate options, unify both into a
new ENABLE_FEAT_AMU name in a first step. This is mostly just renaming at
this point, a subsequent patch will make use of the new feature handling
scheme.
Change-Id: I97d8a55bdee2ed1e1509fa9f2b09fd0bdd82736e
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Neoverse N1 erratum 2743102 is a Cat B erratum that applies to
all revisions <=r4p1 and is still open. The workaround is to
insert a dsb before the isb in the power down sequence.
SDEN documentation:
https://developer.arm.com/documentation/SDEN885747/latest
Signed-off-by: Bipin Ravi <bipin.ravi@arm.com>
Change-Id: I81a8793c1a118764df3ac97b67f5e088f56f6a20
Cortex N1 erratum 1946160 is a Cat B erratum present in r0p0, r1p0,
r2p0, r3p0, r3p1, r4p0, and r4p1. The workaround is to insert a DMB ST
before acquire atomic instructions without release semantics. This
issue is present starting from r0p0 but this workaround applies to
revisions r3p0, r3p1, r4p0, and r4p1, for previous revisions there is no
workaround.
SDEN can be found here:
https://documentation-service.arm.com/static/5fa9304cd8dacc30eded464f
Signed-off-by: John Powell <john.powell@arm.com>
Change-Id: I36e4d6728c275f1c2477dcee9b351077cf7c53e4
Enable basic support for Neoverse-N2 CPUs.
Signed-off-by: Javier Almansa Sobrino <javier.almansasobrino@arm.com>
Change-Id: I498adc2d9fc61ac6e1af8ece131039410872e8ad
Neoverse N1 erratum 1868343 is a Cat B erratum, present in older
revisions of the Neoverse N1 processor core. The workaround is to
set a bit in the CPUACTLR_EL1 system register, which delays instruction
fetch after branch misprediction. This workaround will have a small
impact on performance.
SDEN can be found here:
https://documentation-service.arm.com/static/5f2c130260a93e65927bc92f
Signed-off-by: John Powell <john.powell@arm.com>
Change-Id: I37da2b3b2da697701b883bff9a1eff2772352844
The DSU contains system control registers in the SCU and L3 logic to
control the functionality of the cluster. If "DIRECT CONNECT" L3
memory system variant is used, there won't be any L3 cache,
snoop filter, and SCU logic present hence no system control register
will be present. Hence check SCU presence before accessing DSU register
for DSU_936184 errata.
Signed-off-by: Pramod Kumar <pramod.kumar@broadcom.com>
Change-Id: I1ffa8afb0447ae3bd1032c9dd678d68021fe5a63
This reverts commit 11af40b630, reversing
changes made to 2afcf1d4b8.
This errata workaround did not work as intended so we are reverting this
change. In the future, when the corrected workaround is published in an
SDEN, we will push a new workaround.
This is the patch being reverted:
https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/4750
Signed-off-by: John Powell <john.powell@arm.com>
Change-Id: I20aa064c1bac9671939e657bec269d32b9e75a97
Neoverse N1 erratum 1800710 is a Cat B erratum, present in older
revisions of the Neoverse N1 processor core. The workaround is to
set a bit in the ECTLR_EL1 system register, which disables allocation
of splintered pages in the L2 TLB.
This errata is explained in this SDEN:
https://static.docs.arm.com/sden885747/f/Arm_Neoverse_N1_MP050_Software_Developer_Errata_Notice_v21.pdf
Signed-off-by: John Powell <john.powell@arm.com>
Change-Id: Ie5b15c8bc3235e474a06a57c3ec70684361857a6
CPUECTLR_EL1.EXTLLC bit indicates the presense of internal or external
last level cache(LLC) in the system, the reset value is internal LLC.
To cater for the platforms(like N1SDP) which has external LLC present
introduce a new build option 'NEOVERSE_N1_EXTERNAL_LLC' which can be
enabled by platform port.
Signed-off-by: Manish Pandey <manish.pandey2@arm.com>
Change-Id: Ibf475fcd6fd44401897a71600f4eafe989921363
Even though ERET always causes a jump to another address, aarch64 CPUs
speculatively execute following instructions as if the ERET
instruction was not a jump instruction.
The speculative execution does not cross privilege-levels (to the jump
target as one would expect), but it continues on the kernel privilege
level as if the ERET instruction did not change the control flow -
thus execution anything that is accidentally linked after the ERET
instruction. Later, the results of this speculative execution are
always architecturally discarded, however they can leak data using
microarchitectural side channels. This speculative execution is very
reliable (seems to be unconditional) and it manages to complete even
relatively performance-heavy operations (e.g. multiple dependent
fetches from uncached memory).
This was fixed in Linux, FreeBSD, OpenBSD and Optee OS:
679db7080129fb48ace43a08873eceabfd092aa1
It is demonstrated in a SafeSide example:
https://github.com/google/safeside/blob/master/demos/eret_hvc_smc_wrapper.cchttps://github.com/google/safeside/blob/master/kernel_modules/kmod_eret_hvc_smc/eret_hvc_smc_module.c
Signed-off-by: Anthony Steinhauser <asteinhauser@google.com>
Change-Id: Iead39b0b9fb4b8d8b5609daaa8be81497ba63a0f
Fix a few white space errors and remove #if defined in workaround
for N1 Errata 1542419.
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Change-Id: I07ac5a2fd50cd63de53c06e3d0f8262871b62fad
Coherent I-cache is causing a prefetch violation where when the core
executes an instruction that has recently been modified, the core might
fetch a stale instruction which violates the ordering of instruction
fetches.
The workaround includes an instruction sequence to implementation
defined registers to trap all EL0 IC IVAU instructions to EL3 and a trap
handler to execute a TLB inner-shareable invalidation to an arbitrary
address followed by a DSB.
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Change-Id: Ic3b7cbb11cf2eaf9005523ef5578a372593ae4d6
Replacing ISB instructions in each Errata workaround with a single ISB
instruction before the RET in the reset handler.
Change-Id: I08afabc5b98986a6fe81664cd13822b36cab786f
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1275112 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined
CPUACTLR_EL1 system register, which delays instruction fetch after
branch misprediction.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: If7fe55fe92e656fa6aea12327ab297f2e6119833
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1262888 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined
CPUECTLR_EL1 system register, which disables the MMU hardware prefetcher.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: Ib733d748e32a7ea6a2783f3d5a9c5e13eee01105
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1262606 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined
CPUACTLR_EL1 system register, which delays instruction fetch after
branch misprediction.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: Idd980e9d5310232d38f0ce272862e1fb0f02ce9a
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1257314 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined
CPUACTLR3_EL1 system register, which prevents parallel
execution of divide and square root instructions.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: I54f0f40ff9043efee40d51e796b92ed85b394cbb
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1220197 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set two bits in the implementation defined
CPUECTLR_EL1 system register, which disables write streaming to the L2.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: I9c3373f1b6d67d21ee71b2b80aec5e96826818e8
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1207823 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined
CPUACTLR2_EL1 system register.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: Ia932337821f1ef0d644db3612480462a8d924d21
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1165347 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set two bits in the implementation defined
CPUACTLR2_EL1 system register.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: I163d0ea00578245c1323d2340314cdc3088c450d
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1130799 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined
CPUACTLR2_EL1 system register.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: I252bc45f9733443ba0503fefe62f50fdea61da6d
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1073348 is a Cat B erratum [1],
present in older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined
CPUACTLR_EL1 system register, which disables static prediction.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdoc-466751330-10325/index.html
Change-Id: I674126c0af6e068eecb379a190bcf7c75dcbca8e
Signed-off-by: Lauren Wehrmeister <lauren.wehrmeister@arm.com>
Neoverse N1 erratum 1315703 is a Cat A (rare) erratum [1], present in
older revisions of the Neoverse N1 processor core.
The workaround is to set a bit in the implementation defined CPUACTLR2_EL1
system register, which will disable the load-bypass-store feature.
[1] http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.pjdocpjdoc-466751330-1032/index.html
Change-Id: I5c708dbe0efa4daa0bcb6bd9622c5efe19c03af9
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Some cores support only AArch64 mode. In those cores, only a limited
subset of the AArch32 system registers are implemented. Hence, if TF-A
is supposed to run on AArch64-only cores, it must be compiled with
CTX_INCLUDE_AARCH32_REGS=0.
Currently, the default settings for compiling TF-A are with the AArch32
system registers included. So, if we compile TF-A the default way and
attempt to run it on an AArch64-only core, we only get a runtime panic.
Now a compile-time check has been added to ensure that this flag has the
appropriate value when AArch64-only cores are included in the build.
Change-Id: I298ec550037fafc9347baafb056926d149197d4c
Signed-off-by: John Tsichritzis <john.tsichritzis@arm.com>
Examine the ID_AA64PFR1_EL1 bits 7:4 to see if speculative
loads (SSBS) is implemented, before disabling speculative
loads.
Change-Id: I7607c45ed2889260d22a94f6fd9af804520acf67
Signed-off-by: Sami Mujawar <sami.mujawar@arm.com>
This patch fixes this issue:
https://github.com/ARM-software/tf-issues/issues/660
The introduced changes are the following:
1) Some cores implement cache coherency maintenance operation on the
hardware level. For those cores, such as - but not only - the DynamIQ
cores, it is mandatory that TF-A is compiled with the
HW_ASSISTED_COHERENCY flag. If not, the core behaviour at runtime is
unpredictable. To prevent this, compile time checks have been added and
compilation errors are generated, if needed.
2) To enable this change for FVP, a logical separation has been done for
the core libraries. A system cannot contain cores of both groups, i.e.
cores that manage coherency on hardware and cores that don't do it. As
such, depending on the HW_ASSISTED_COHERENCY flag, FVP includes the
libraries only of the relevant cores.
3) The neoverse_e1.S file has been added to the FVP sources.
Change-Id: I787d15819b2add4ec0d238249e04bf0497dc12f3
Signed-off-by: John Tsichritzis <john.tsichritzis@arm.com>
This patch forces all cacheable atomic instructions to be near, which
improves performance in highly contended parallelized use-cases.
Change-Id: I93fac62847f4af8d5eaaf3b52318c30893e947d3
Signed-off-by: Louis Mayencourt <louis.mayencourt@arm.com>
This patch applies the new MSR instruction to directly set the
PSTATE.SSBS bit which controls speculative loads. This new instruction
is available at Neoverse N1 core so it's utilised.
Change-Id: Iee18a8b042c90fdb72d2b98f364dcfbb17510728
Signed-off-by: John Tsichritzis <john.tsichritzis@arm.com>
