doc/xnu_build_consolidation.md

   1 # XNU Build Consolidation
   2
   3 ## Introduction and motivation
   4
   5 XNU is supported on approximately 20 different targets. Whilst in some cases the differences between two
   6 given targets are small (e.g. when they both support the same ISA), XNU has traditionally required to have
   7 separate builds in cases where the topology of the targets differ (for example, when they feature different
   8 core/cluster counts or cache sizes). Similarly, SoC-specific fix-ups are usually conditionally compiled
   9 based on the target.
  10
  11 Given the time it takes to compile all three different variants (release, debug and development) for each
  12 supported SoC, usually several times a day for various teams across Apple, the goal of this project was to
  13 reduce the number of existing builds, as well as to set up a simple framework that makes it easier to share
  14 builds across different SoCs moving forward.
  15
  16 Although this effort could be extended to KEXTs, and hence lead to shared KernelCaches across devices, the
  17 scope of this document only includes XNU. In cases where KEXTs also differ across targets, or perhaps the
  18 required KEXTs are completely different in the first place, the kernel still needs to be linked
  19 appropriately with different sets of KEXTs and hence KernelCaches cannot be shared.
  20
  21
  22 ## Changes required in XNU
  23
  24 The kernel itself is relatively SoC-agnostic, although strongly architecture-dependent; this is because most
  25 of the SoC-specific aspects of the KernelCache are abstracted by the KEXTs. Things that pertain to the
  26 kernel include:
  27
  28 * Number of cores/clusters in the system, their physical IDs and type.
  29 * Addresses of PIO registers that are to be accessed from the XNU side.
  30 * L1/L2 cache geometry parameters (e.g. size, number of set/ways).
  31 * Just like other components, the kernel has its share of responsibility when it comes to setting up HID
  32 registers and applying fix-ups at various points during boot or elsewhere at runtime.
  33 * Certain kernel-visible architectural features are optional, which means that two same-generation SoCs may
  34 still differ in their feature set.
  35
  36 All of these problems can be solved through a mix of relying more heavily on device tree information and
  37 performing runtime checks. The latter is possible because both the ARM architecture and the Apple's
  38 extensions provide r/o registers that can be checked at runtime to discover supported features as well as
  39 various CPU-specific parameters.
  40
  41 ### Obtaining cache geometry parameters at runtime
  42
  43 Although not often, the kernel may still require deriving, one way or another, parameters like cache sizes
  44 and number of set/ways. XNU needs most of this information to perform set/way clean/invalidate operations.
  45 Prior to this work, these values were hardcoded for each supported target in `proc_reg.h`, and used in
  46 `caches_asm.s`. The ARM architecture provides the `CCSIDR_EL1` register, which can be used in conjunction
  47 with `CSSELR_EL1` to select the target cache and obtain geometry information.
  48
  49
  50 ### Performing CPU/Revision-specific checks at runtime
  51
  52 CPU and revision checks may be required at various places, although the focus here has been the application
  53 of tunables at boot time.
  54
  55 Tunables are often applied:
  56
  57 * On a specific core type of a specific SoC.
  58 * On a subset of all of the CPU revisions.
  59 * On all P-cores or all E-cores.
  60
  61 This has led in the past to a number of nested, conditionally-compiled blocks of code that are not easy to
  62 understand or manage as new tunables are added or SoCs/revisions are deprecated.
  63
  64 The changes applied as part of this work focus mainly on:
  65
  66 1. Decoupling the tunable-application code from `start.s`.
  67 2. Splitting the tunable-application code across different files, one per supported architecture (e.g.
  68 `tunables_h7.h`, or `tunables_h11.h`).
  69 3. Providing "templates" for the most commonly-used combinations of tunables.
  70 4. Providing a family of assembly macros that can be used to conditionally execute code on a specific core
  71 type, CPU ID, revision(s), or a combination of these.
  72
  73 All of the macros live in the 64-bit version of `proc_reg.h`, and are SoC-agnostic; they simply check the
  74 `MIDR_EL1` register against a CPU revision that is passed as a parameter to the macro, where applicable.
  75 Similarly, where a block of code is to be executed on a core type, rather than a specific core ID, a couple
  76 of the provided macros can check this against `MPIDR_EL1`.
  77
  78
  79 ### Checking for feature compatibility at runtime
  80
  81 Some architectural features are optional, which means that, when disabled at compile-time, this may cause
  82 two same-generation SoCs to diverge.
  83
  84
  85 Rather than disabling features, and assuming this does not pose security risks or performance regressions,
  86 the preferred approach is to compile them in, but perform runtime checks to enable/disable them, possibly in
  87 early boot. The way these checks are performed varies from feature to feature (for example, VHE is an ARM
  88 feature, and the ARM ARM specifies how it can be discovered). For Apple-specific features, these are all
  89 advertised through the `AIDR_EL1` register. One of the changes is the addition of a function,
  90 ml_feature_supported(), that may be used to check for the presence of a feature at runtime.
  91
  92
  93 ### Deriving core/cluster counts from device tree
  94
  95 One of the aspects that until now has been hardcoded in XNU is the system topology: number of cores/clusters
  96 and their physical IDs. This effort piggybacks on other recent XNU changes which aimed to consolidate
  97 topology-related information into XNU, by parsing it from the device tree and exporting it to KEXTs through
  98 well-defined APIs.
  99
 100 Changes applied as part of the XNU consolidation project include:
 101
 102 * Extending the `ml_*` API to extract cluster information from the topology parser. New APIs include the following:
 103     * `ml_get_max_cluster_number()`
 104     * `ml_get_cluster_count()`
 105     * `ml_get_first_cpu_id()`
 106 * Removing hardcoded core counts (`CPU_COUNT`) and cluster counts (`ARM_CLUSTER_COUNT`) from XNU, and
 107 replacing them with `ml_*` calls.
 108 * Similarly, deriving CPU physical IDs from the topology parser.
 109
 110
 111 ### Allocating memory that is core size/cluster size/cache size aligned
 112
 113 In some cases, certain statically-allocated arrays/structures need to be cache line-aligned, or have one
 114 element per core or cluster. Whilst this information is not known precisely at compile time anymore, the
 115 following macros have been added to provide a reasonably close upper bound:
 116
 117 * `MAX_CPUS`
 118 * `MAX_CPU_CLUSTERS`
 119 * `MAX_L2_CLINE`
 120
 121 These macros are defined in `board_config.h`, and should be set to the same value for a group of targets
 122 sharing a single build. Note that these no longer reflect actual counts and sizes, and the real values need
 123 to be queried at runtime through the `ml_` API.
 124
 125 The L1 cache line size is still hardcoded, and defined as `MMU_CLINE`. Since this value is always the same
 126 and very often checked at various places across XNU and elsewhere, it made sense to keep it as a compile
 127 time macro rather than relying on runtime checks.
 128
 129 ### Restrictions on conditional compilation
 130
 131 Currently, a family of per-SoC macros are defined at build time to enable XNU to conditionally compile code
 132 for different targets. These are named `ARM[64]_BOARD_CONFIG_[TARGET_NAME]`, and have historically been used
 133 in different places across the kernel; for example, when applying tunables, various fixes, or enabling
 134 disabling features. In order not to create divergences in the future across same-generation SoCs, but also
 135 to keep the codebase consistent, the recommendation is to avoid the use of these macros whenever possible.
 136
 137 Instead, XNU itself defines yet another family of macros that are defined for all targets of a particular
 138 generation. These are named after the P-CORE introduced by each (for example, `APPLEMONSOON`, or
 139 `APPLEVORTEX`), and are preferred over the SoC-specific ones. Where a generation macro is not enough to
 140 provide correctness (which happens, for example, when the code block at hand should not be executed on a
 141 given SoC of the same family), appropriate runtime checks can be performed inside the conditionally-compiled
 142 code block. `machine_read_midr()` and `get_arm_cpu_version()` may be used for this purpose.