diff --git a/docs/about/index.rst b/docs/about/index.rst index 06973ef16..7cf2a6828 100644 --- a/docs/about/index.rst +++ b/docs/about/index.rst @@ -7,6 +7,7 @@ About features release-information + lts maintainers contact acknowledgements diff --git a/docs/about/lts.rst b/docs/about/lts.rst new file mode 100644 index 000000000..91d3d408a --- /dev/null +++ b/docs/about/lts.rst @@ -0,0 +1,329 @@ +LTS - Long-Term Support +======================= + +.. table:: Table 1: Document History + + +-------------+--------------------+-------------------------------------------------------+ + | Date | Author | Description | + +=============+====================+=======================================================+ + | 2022-07-20 | Okash Khawaja, | Initial draft. | + | | Varun Wadekar | | + +-------------+--------------------+-------------------------------------------------------+ + | 2022-07-21 | Varun Wadekar | Refine the Maintainership guidelines and planning | + | | | sections. Introduce a new section documenting a day | + | | | in the life of a LTS branch maintainer | + +-------------+--------------------+-------------------------------------------------------+ + | 2022-08-05 | Okash Khawaja, | Merge two drafts (draft 1 and 2), address comments | + | | Varun Wadekar | made by both authors, cosmetic changes to the content | + | | | all over the document | + +-------------+--------------------+-------------------------------------------------------+ + | 2022-08-05 | Okash Khawaja | Add note about testing support available from TF.org | + +-------------+--------------------+-------------------------------------------------------+ + | 2022-08-05 | Varun Wadekar | Changed the “Future plans” section to “FAQ” and | + | | | answered some of the questions with feedback from | + | | | the community. | + +-------------+--------------------+-------------------------------------------------------+ + |2025-01-07 | Govindraj Raja | Convert from pdf to rst. | + +-------------+--------------------+-------------------------------------------------------+ + +This document proposes a plan for long-term support (LTS) of the |TF-A| project. + +Why is LTS required? +-------------------- +LTS is needed for commercial reasons. More specifically, on the device side, +when a product is released, the companies have to support that in-market product +such that the amount of changes to the firmware are kept to a minimum to avoid +the risk of regression. At the same time the companies don't want to exclude +critical patches such as those for security advisories. Similarly on the server side, +companies want to minimize the churn when deploying fixes during incident +response, e.g. due to critical security bugs. + +This means that those companies have to maintain and backport critical updates to +old branches internally. As this effort is duplicated across different companies +using TF-A, it makes sense to factor out this effort into a community-wide LTS. + +What does LTS mean for TF-A? +---------------------------- +In this section we will define exactly what constitutes LTS for TF-A. +Specifically, we will define the following characteristics: + +- criteria for selecting patches which will be backported to LTS branches +- lifetime and frequency of LTS branches + +**Criteria** + +We must have an objective criterion for selecting patches to be backported to +LTS branches. This will make maintenance easy because: + +a. there will be less -- ideally no -- discussion when selecting patches to backport +b. large parts of the process can be automated + +Below is the criteria + +#. No features will be backported. +#. Security advisories: Any patch that makes it into :ref:`Security Advisories` + is automatically selected for back porting. This includes patches to external + components too, e.g. libfdt. +#. Workarounds for CPU and other ARM IP errata +#. Workarounds for non-ARM IP errata, e.g. TI UART +#. Fixes for platform bugs. These patches must not modify any code outside of + the specific platform that the fix applies to. +#. Patches can only be backported from the master branch. In other words, the + master branch will be a superset of all the changes in any LTS branch. + +**Lifetime and frequency** + +This section approaches three questions: for how long should an LTS release be +supported, how frequently should LTS releases be made and at which time(s) of +the year should the releases be made. + +1. For how long should an LTS release be supported? + +Linux kernel supports an LTS branch for 5 years. Since firmware tends to +have less churn and longer lifetime than HLOS, TF-A should support at least +5 years for its LTS. We should leave the room open for discussions about +extending it to 7 years. + +2. How frequently should LTS releases be made? + +Given that many products that have a release cycle, have a yearly release +cycle, it would make sense to have yearly TF-A releases. + +3. Which time(s) of the year should the releases be made? + +TF-A releases are cut twice a year: May and November. Basing LTS release +on the November TF-A release has a few benefits. First, it aligns with Linux +LTS releases which happen towards the end of each year. Second, it aligns +with Android releases which tend to fall in Q3 each year. Since product +releases are timed with Android release, this gives enough time to harden +the TF-A LTS release during development so that it's ready for launch in +Q3 following year. On the other hand, if the May release of TF-A is chosen as +the basis for LTS then developers will have little time -- about a month, +taking into account the test-and-debug phase before LTS is cut (see below) -- +before Android release. + +To summarize, there will be one LTS release per year. It will be supported for +5 years and we can discuss extending it to 7 years later on. The LTS release +will be based on the November release of TF-A. + +**Testing Criteria** + +Every patch merged to the LTS branch will complete the following tests before +getting approved. + +#. TFTF tests currently running in the testing farm +#. CI/CD static analysis scans +#. Coverity scans +#. Platform tests + +Platforms that are not maintained upstream will undergo testing downstream in a +pre-defined window. The platform maintainer will complete the testing and provide +a verified score on the patch once testing is completed. + +** A note about test coverage from TF.org ** + +Currently TF.org maintains a CI system to run TF-A automated tests on a +selection of HW boards donated by TF.org members (a benefit reserved to project +members, see the project charter for more details). This automated test coverage +will be extended to cover testing for LTS as well for boards that are part of +the CI system. + +**TFTF branching** + +A note about testing here. After a patch is backported to an LTS branch, that +branch will need to be regression tested. Since TFTF moves forward with latest +TF-A changes, newer TFTF tests may not apply to old LTS branches. Therefore +TFTF will also need to be branched, in-sync with TF-A LTS branches. In other +words, there will be one TFTF LTS branch corresponding to each TF-A LTS branch. +The TFTF LTS branch will be used to regression test the corresponding TF-A LTS +branch. + +As we work with the LTS branch of TFTF, we might also need fixes for TFTF +itself to be ported to LTS. However, decision-making about those patches need +not be as stringent as for TF-A. + +Release details +--------------- +This section goes into details of what the LTS release process will look like. + + +**Test-and-debug period** + +Since the LTS branch will be used in product releases, it is expected that more +testing and debugging will be done on the November release of TF-A. Therefore +it would make sense to leave at least a month after the November release and +then cut the LTS branch. We recommend two months, given that one of the months +is December which tends to be slower due to holidays. So, an end-of-November +TF-A release would result in a beginning-of-February LTS release. Note that +the LTS branch will be created at the same time as the TF-A November release, +but it will be officially released at the end of January or early February. +Going forward we should strive to make the period smaller and smaller until +ideally it coincides with TF-A November release which means that our test +and CI/CD infra is good enough to allow that to happen. + +**Example timeline** + +Below is an example timeline starting from the November 2022 release of TF-A. + +.. image:: ../resources/diagrams/lts-timeline-example.png + +- Nov 2022: TF-A 2.8 is released towards the end of Nov, 2022. Not shown in the + diagram, at the same time LTS release candidate branch is made which is based + on TF-A 2.8. This means new features going in 2.8 won’t go in the LTS branch. + We can call it `LTS 2.8-rc`. +- Feb 2023: After testing and debugging LTS 2.8-rc for a couple of months, + LTS 2.8.0 is officially released in early Feb 2023. +- May 2023: TF-A 2.9 is released but since this is not an LTS branch it doesn’t + affect LTS. +- Somewhere between May and Nov of 2023: A security advisory comes up and the + related patches go into TF-A master branch. Since these patches fall under + LTS criteria, they are backported to LTS 2.8.0 which results in LTS 2.8.1 + being released. Note that here we don’t allow the extra testing and debugging + time that we had between Nov 2022 and early Feb 2023. This is because there + isn’t as much to test and debug as an annual LTS release has. Also companies + might want to deploy critical patches soon. +- Nov 2023: TF-A 2.10 is released. Not shown in the diagram, at the same time + LTS 2.10-rc is made. It’s tested by partners for a couple of months. +- Feb 2024: LTS 2.10.1 is released in early Feb. Now there are two LTS + branches: 2.8.1 and 2.10.1. + +Note that TFTF will follow similar branching model as TF-A LTS, i.e. there will +be TFTF LTS 2.8.0 in Feb 2023, 2.8.1 (if new TFTF tests need to be added for +the security advisory) when there is TF-A LTS 2.8.1 and so on. + +Maintainership +-------------- + +**Guidelines & Responsibilities** + +#. Maintainers shall be impartial and strive to work for the benefit of + the community +#. Objective and well-defined merge criteria to avoid confusion and discussions + at random points in time when there is a "candidate" patch +#. The maintainers shall explain the lifecycle of a patch to the community, + with a detailed description of the maximum time spent in each step +#. Automate, automate, automate +#. Reviewers should not focus too much on "what" and instead focus on "how" +#. Constantly refine the merge criteria to include more partner use cases +#. Ensure that all candidate patches flow from the main branch to all LTS branches + +**Options** + +These are some options in the order of preference. + +#. Current set of :ref:`lts maintainers` from tf.org(or hired contractor) take care of the LTS +#. From the community, create a set of maintainers focused solely on the LTS branches + +A day in the life of a maintainer +********************************* +This section documents the daily tasks that a maintainer might perform to +support the LTS program. It is expected that a maintainer follows clearly laid +down steps and does not have to make policy level decisions for merge, testing, +or candidate patch selection. + +#. Monitor the main branch to identify candidate patches for the LTS branches +#. Inform the LTS maintainers mailing list of a new candidate patch for LTS and solicit feedback +#. Start the review process and CI/CD cycle for the patch +#. Review the CI/CD output to ensure that the quality bar is met +#. After reviews are complete, merge the patch and bump the minor version, if required +#. Monitor the mailing list for any LTS related issues +#. Propose or solicit patches to the main branch and tag them as candidates for LTS + +Execution Plan +************** +This section lists the steps needed to put the LTS system in place. However, +to kick start LTS in Nov ‘22, only a few steps are needed. The rest can follow +in the background. + +Initial release steps +********************* + +The following steps are necessary to kickstart the project and potentially +create the first LTS from the Nov’22 release. + +#. Create a TF-A LTS release-candidate branch and a TFTF LTS branch immediately + after the Nov’22 release +#. Request all platform-owners to test and debug the RC branch +#. Gather feedback from the test and debug cycle +#. Mark the TF-A LTS branch ready by the end of January +#. Announce the official LTS release availability on the mailing lists + +Long term release plan +********************** +Above will buy us time to then work on the rest of the execution plan which +is given below. + +#. The review criteria for LTS patches must be the same as TF-A patches +#. The maintainers shall publish the well-defined merge criteria to allow + the community to choose candidate patches +#. The maintainers shall publish a well-defined test specification for any + patch entering the LTS branch + + a. Tests required to pass in the CI/CD flow + b. Static analysis scans + c. Coverity scans + +#. The maintainers shall publish a mechanism to choose candidate patches for + the LTS branch +#. The maintainers shall publish a mechanism to report bugs `[1]`_ seen with + an LTS branch +#. The maintainers shall publish a versioning mechanism for the LTS branch + + a. Bump minor version for every “logical” `[2]`_ fix that gets merged + +#. The CI/CD infrastructure shall provide test support for all “live” LTS + branches at any given point in time +#. The CI/CD infrastructure shall provide means to + + a. notify all maintainers that a patch is ready for review + b. automatically cherry-pick a patch to a given LTS branch + c. get it through the CI/CD testing flow + d. send nag emails to maintainers at regular intervals to ensure reviews keep moving + +FAQ +*** + +In our discussions, in addition to the above points we also considered some +questions. They have been discussed on the mailing list too. + +| Q. What happens when a bug fix applies just to a LTS branch and not to the + master branch? +| A. This will be treated as a special case and the bug, and the fix will be + discussed + +| Q. When testing a backported patch, what if one of the partners needs more + time while the patch fix is time-critical and, hence slowing other + partners? +| A. The maintainers will add more detail to the review and merge process to + handle this scenario. + +| Q. How do we handle the increasing version numbers for errata fixes? +| A. Too many CPU errata workarounds resulting in too many LTS releases. + We propose bumping the version number for each logical fix as + described in the section “Long term release plan” above because + that will help accurately track what changes have been deployed in-field. + +| Q. What if LTS support duration needs to be extended to longer than 5 years? +| A. Still under discussion. + +These are uncharted waters, and we will face some unseen problems. When they +become real problems, then we will have concrete data and be better able to +address them. This means that our LTS definition as presented in this document +is not the final one. We will constantly be discussing it and deciding how to +adapt it as we see practical problems. + +.. _[1]: + +[1] The plan is to create a system where reviewers can tag a patch on mainline which +gets automatically rebased on LTS and pushed to Gerrit. On seeing this patch, +the CI/CD starts tests and provides a score. In parallel, the system also sends +an email to the maintainers announcing the arrival of a candidate patch for the +LTS branch. + +.. _[2]: + +[2] Logical will be a patch or patches implementing a certain fix. For example, if a +security mitigation is fixed with the help of three patches, then all of them are +considered as one "logical" fix. The version is incremented only after all these +patches are merged. with the maintainers. If agreed unanimously, the bug fix +will be merged to the affected LTS branches after completing the review process. diff --git a/docs/about/maintainers.rst b/docs/about/maintainers.rst index 12c0f3699..c45a757b1 100644 --- a/docs/about/maintainers.rst +++ b/docs/about/maintainers.rst @@ -58,6 +58,8 @@ Maintainers :|M|: Govindraj Raja :|G|: `govindraj-arm`_ +.. _lts maintainers: + LTS Maintainers --------------- diff --git a/docs/about/release-information.rst b/docs/about/release-information.rst index 9d6bbf6aa..dc2b2fd73 100644 --- a/docs/about/release-information.rst +++ b/docs/about/release-information.rst @@ -35,7 +35,7 @@ TF-A version is given in Makefile, through several macros: For example, TF-A v2.10 has VERSION_MAJOR=2, VERSION_MINOR=10 and VERSION_PATCH=0. -This VERSION_PATCH macro is only increased for LTS releases. +This VERSION_PATCH macro is only increased for :ref:`LTS - Long-Term Support` releases. Upcoming Releases ~~~~~~~~~~~~~~~~~ @@ -120,4 +120,4 @@ Populated table provides details about build options that were removed or deprec -------------- -*Copyright (c) 2018-2024, Arm Limited and Contributors. All rights reserved.* +*Copyright (c) 2018-2025, Arm Limited and Contributors. All rights reserved.* diff --git a/docs/glossary.rst b/docs/glossary.rst index f19897c7d..20ad21c14 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -130,6 +130,9 @@ You can find additional definitions in the `Arm Glossary`_. Memory Tagging Extension. An optional Armv8.5 extension that enables hardware-assisted memory tagging. + LTS + Long-Term Support + OEN Owning Entity Number diff --git a/docs/resources/diagrams/lts-timeline-example.png b/docs/resources/diagrams/lts-timeline-example.png new file mode 100644 index 000000000..edb1145a1 Binary files /dev/null and b/docs/resources/diagrams/lts-timeline-example.png differ