DTB loading for Linux is harder than it looks

And it's not getting better, unless we act now.

What do I mean by that? It's simple: the current solutions all have problems, and in the end, are all done at the wrong layer.

At best, this is causing duplicated work, at worst, using wrong or incomplete data, which could lead to the system being unusable. (i.e. Drivers not working as expected, e.g. no display output, no networking, no usb, etc.).

This started as a much longer write-up, which will be rehashed later.

Instead, this will be direct and to the point.

After all, something had to be rushed out, and the other write-up was stuck in rewrite hell. This overall message needs to get out right now, and was precipitated by systemd's announcement of .dtbauto in UKIs.

Though, this is not meant as a slight against systemd specifically, Solving this problem is, as far as I can tell from many private and public sightings, in the zeitgeist. It's not surprising they received contributions attempting to solve this.

This write-up is coming from the point of view of a distribution developer and Platform Firmware developer, with a focus on user experience. This is describing the current situation, not the intended use-case or best case scenario. The failure modes described are not theoretical.

The target audience for this write-up are power-users and developers who are already somewhat aware of what devicetrees are (i.e. DT, FDT, dtb, etc.).

Table of contents

Even though it was supposed to be short and to the point, the subject at hand is too complicated to do so.

I hope this table of contents helps.

The current situations

Situations, plural. And this is part of the problem: there are too many different parties attempting to handle the life-cycle of dtb files, all doing it with different implementation details that matter.

And even if they were all doing it in the same manner, it still means that the semantics are duplicated across different projects, and only usable for users of these projects.

For brevity I will only focus on the downsides, but I want to highlight again that doing this “correctly” is a really hard task. Every one of those attempts at handling the problem is a valiant effort, despite their issues.

Only methods that are generic are detailed here. It's not useful to describe methods specifying a specific dtb file to load (e.g. devicetree in GRUB) or methods that are specific to a “product” (e.g. Android, ChromeOS). Similarly, vendor-specific approaches are not described (e.g. Raspberry Pi's config.txt).

U-Boot's FDTDIR

What is it?

The FDTDIR label is used with the extlinux-compatible scheme from U-Boot.

The label is part of a boot entry in the boot configuration file, and will load a dtb file from the provided path, either from a hard-coded environment value, or if missing, formed from a plausible name (i.e. ${soc}-${board}.dtb).

See: pxe_utils.c~#L610

What's the problem?

U-Boot's EFI /dtbs dir

What is it?

U-Boot will attempt to load the dtb file for the current hardware from a few well-known global paths in the ESP, when it starts attempting EFI boot.

The “heuristics” for the dtb file name are similar to the previously described ones, but also differ a bit. The fallback file name, when needed, is equivalent to ${soc}-${board}${boardver}.dtb

See:

What's the problem?

dtbloader

What is it?

dtbloader is an EFI driver, which can load and install dtb files.

Since it is largely a reimplementation of DtbLoader.efi from the aarch64-laptops project, the overall design decisions, and problems, apply to the original project, too.

It is currently focused on Windows on ARM devices, and uses DMI information, building a set of CHIDs to pick a dtb file name from a built-in lookup table. The file will be loaded from a few well-known global paths in the ESP.

NOTE: It also implements the EFI_​DT_FIXUP_​PROTOCOL, adding missing functionality to the Platform Firmware. More on that later.

What's the problem?

systemd's .dtb and .dtbauto UKI sections

What is it?

This is a pair of mechanism available in systemd-stub, an older one (.dtb section) and a new mechanism from systemd v257 (.dtbauto section).

It will load dtb files as follows:

[Specifically the PE file include the following sections:] .dtbauto […]. systemd-stub will always use the first matching one. The match is performed by taking the first DeviceTree's compatible string supplied by the firmware in configuration tables and comparing it with the first compatible string from each of the .dtbauto sections. If the firmware does not provide a DeviceTree, the match is done using the .hwids section instead. After selecting a .hwids section (see the description below), the compatible string from that section will be used to perform the same matching procedure. If a match is found, that .dtbauto section will be loaded and will override .dtb if present.

— systemd@3f3b4959/man/systemd-stub.xml

What's the problem?

The failure modes

Bespoke selection logic

Every method has its own semantics about how it goes to select the dtb file to be loaded.

For example, .dtbauto matches on compatible name, dtbloader using DMI information; implementations using a globally available path are not using the same paths and order either.

I think it's also important to highlight that implementations that are using file paths are tied to file names that are part of an external project, making it an unexpected *interface* to support.

The more selection heuristics there are out there, the harder it is to reason about which dtb file is effective.

The dtb file selection should be consistent across all different combinations of boot stacks.

Dealing with selection requirements changing

Looking at the words, it looks like it would just require a Platform Firmware update, an UEFI program update, or an Operating System Loader update.

And that would be true, if and only if those were tightly coupled with the kernel's life-cycle. (i.e. for any kernel change, those components will always be changed in lockstep.)

Otherwise, the requirements may end-up changing, but the selection will not match for the kernel intended to be booted.

This failure mode can be illustrated with an example. Let's say a specific device ends-up being split into different variants, for any reason, there is now a different file that may need to be loaded depending on the exact device in use, and the exact dtb files in use.

Now, that might be workable, though not ideal, if we could rely on forcing an update for the component doing the dtb file loading, or if the lookup rules were flexible enough to add more conditions. (Though this would still needlessly duplicate the logic, see the preceding failure mode.)

I'll add a note that this is a non-trivial problem to handle outside of the kernel. Even current implementations fail to agree on the details of the rules to use, and sometimes even within the same codebase.

It should not be required to be running the latest platform firmware to run the latest kernel release.

Handled at the wrong moment

This failure mode is exercised with, for example, a generic (“USB”) installer image that would have both the LTS and Latest kernel version as a boot menu item.

Loading the “correct” dtb file is not trivial, as highlighted previously.

This is exacerbated by the fact that support for booting different kernel versions is required.

Stages preceding the Operating System Loader are unaware of the kernel that is going to be loaded, so they should be considered unaware of which rule set to apply for the selected boot option.

The Operating System Loader is the only component in the chain that could change its behaviour according to the kernel version selected to boot. Though doing so only brings us back to the bespoke selection logic problem.

And really, this ties the Operating System Loader to what would need to become a stable interface provided by the kernel. The kernel developers (and it's the right move) don't want to provide one for drivers. I believe most of the rationale applies for handling the kernel's own dtb files.

It should not be required to update to the latest kernel when using the latest platform firmware update.

Missing additional fixups

This is a very nuanced problem space, and fixups might even be considered somewhat controversial. Those “fixups” are changes that needs to be applied to the “generic” devicetree. They are a mix of runtime facts, device-specific information, board information, and platform information.

Some of what is missing might not produce an unusable system. Though missing some other fixups may cause problems.

I wrote more details about fixups, to highlight how they matter, but this can be skipped.

What even are fixups?

It depends. Not all fixups are the same, here's a few examples:

Runtime facts, by their nature, need to come from the Platform Firmware. In the absence of a protocol to handle the fixups, the properties that the kernel relies need to be forwarded.

All other fixups lives in an in-between world. Some of it cannot be detected at runtime, and for those, relying on imperative configuration is necessary (i.e. the Platform Firmware knows what is the intended configuration1). Some if it can be detected or derived at runtime, and for some the detection could be handled in the kernel (too). Though the details are specific to the implementation details of the drivers and subsystems affected.

What differs in device-specific facts, board configuration and platform configuration really is just the nature of “where” the fixups apply. Some always apply (but are not static data), some apply the same way, but conditionally, and some are unique to the specific device.

Since some of the fixups cannot be derived or detected, using the Platform Firmware fixups is necessary.

How are those handled currently?

Generally speaking, two (interconnected) categories:

The Platform Firmware specific handling could be anything, really. As a simple example, at any point U-Boot itself loads a devicetree, it ends-up calling image_setup_libfdt, which might do some general fixups, and then follow thorugh the diverse implementations of ft_board_setup.

The EFI_​DT_FIXUP_​PROTOCOL is generally implemented using the Platform Fimware specific handling. This is how U-Boot does it.

Though the EFI_​DT_FIXUP_​PROTOCOL could be implemented in another form too. Like how dtbloader installs its own.

The main takeaway here is that this is another difficult part when dealing with loading devicetrees.

Solving this

Solving this requires a paradigm shift in the handling of dtb files, and their lifecycles.

One option could be to start relying solely on platform firmware's innate FDTs, for all systems, but I know from experience that this is a non-starter.

So we're left with this option: The kernel needs to handle dtb loading for its own dtb files, by itself2.

This is not a change that is trivial, likely will be controversial, and definitely will take time.

With that said, solving this entirely within the kernel would be missing helping the situation with existing kernel releases and currently supported LTS releases. As such, even if the next suggestion wasn't needed as a proof of concept, and as a staging ground, it would still be needed to solve this with backward compatibility in mind.

Since supporting older kernels is a necessity, and adding extensive changes to the startup code to existing kernel releases is likely undesirable, an independent compatibility component needs to be authored. Serving as a proof of concept, and staging grounds for the heuristics and semantics around loading the “correct” devicetree for the kernel.

Here I'm suggesting an UEFI-centric component, since this is where the issue is the most pronounced. Other non-standard boot methods already have their own workarounds for the problem.

This component would insert itself between the Operating System Loader's life-cycle and the kernel startup.

It would rely on data to look up the appropriate dtb file to use, according to any available ambient facts. This data can hold additional knowledge to properly guide the dtb file lookup. The data would be generated, from the same kernel source tree as the kernel image to be booted.

While I use the terms dtb file, it does not imply the use of the ESP's filesystem, or any filesystem-like construct. It's meant to refer to the output from make dtbs. The exact form they will be made available as is an open question, and a future implementation detail at this point in time.

Once the implementation is proven, it can be ported into the kernel, with confidence that it is solid.

Why an independent compatibility component?

The word independent may not be the best word to use here.

The project should be independent from any of the diverse set of components making the boot stages before the kernel. The goal is (at first) to reduce the diverging semantics for UEFI boot.

But truthfully, it should be a project under the greater kernel.org umbrella, and not independent in that sense. Being independent from the other components serves as an affordance to later make that easier.

And really, the goal of that compatibility tool is to become unneeded in future kernel releases, since the kernel should be doing the work by itself. After all, this is not an UEFI problem.

And thus, the goal of the independent project is to become unneeded, the kernel organization adopting it as the recommended solution.

Why not in the Platform Firmware?

The Platform Firmware cannot know which kernel will be used, and thus which dtb files to use.

Furthermore, this would be encoding Linux-specific implementation details, duplicated in different state of support across different Platform Firmwares. Creating, in effect, a new kind of stable interface with the kernel.

Why not in a UEFI program?

Well, actually… That's not the right way to frame the issue.

Really it's: why can't it be done before knowing which kernel is in use?.

After all, a UEFI program is part of the suggested solution, but it needs to load the dtb files in-sync with the kernel that is going to boot.

Leading us to the next question…

Why is it that important that the dtb files be in sync with the kernel?

Because in practice, the kernel dtb files that are presumed to be used are the ones coming from the same tree from which the kernel drivers come from.

Often, changes in kernel drivers ends-up changing the devicetrees to match new semantics.

In turn, this means that even though a lot of care is often taken so it's not an issue, in practice there is no guarantee regarding backward compatibility and forward compatibility for dtb files between kernel versions. Breakage already has happened in the past.

In practice, devicetrees originating from the kernel source tree already are an unstable semi-internal interface made for the kernel. Though they are left to be provided by the previous stages.

NOTE: It is deemed acceptable for distributions to provide additional dtb files, or patching dtb files. It should be considered morally equivalent to shipping kernel patches.

Why not in the OS Loaders?

Simply put: we don't need more ways to maybe load maybe the appropriate dtb file. The implementations, as they are currently, are likely to differ, whether in scope, or details.

Given that there is no defined “protocol” to load the “correct” dtb file, this is likely to lead to implementation-defined behaviour.

This is undesirable. Distributions are already chasing after problems caused by how the life-cycle of dtb files is ill-defined.

There needs to be a single solution, so it can be ported into the kernel (once deemed acceptable), and kept in sync with new semantics from the kernel whenever they happen, without requiring OS Loaders to be updated.

And even if the kernel was to write-up a precise protocol to follow, making the implementations (assumedly) equivalent, it would still require the Operating System Loaders to be updated, which lives in a different life-cycle than the kernel.

And why not in systemd-stub / UKIs?

(In addition to the previous points, as systemd-stub and UKIs are straddling the line and are like OS Loaders in a lot of ways…)

Plainly: to prevent a monoculture around either systemd-stub or UKIs to form. It might even hamper work to implement this directly in the kernel, since it would be seen as solved.

Additionally, it would require coupling those to kernel release and its changing semantics.

It might sound counter-intuitive to point out concerns around a monoculture, when suggesting a single implementation, but it's not since the goal of the compatibility component is to act as a stand-in for the future in-kernel implementation.

As such, it is not forming a monoculture from the external point of view of the kernel, but acting within the existing monoculture of the Linux kernel.

And fixups?

As stated previously, this is a very nuanced problem space, and the solution is just as much.

In actuality, this is an open question, and might require different solutions depending on the nature of the fixups.

First of all, runtime information from the Platform Firmware will need a proper “protocol” to be put in place. It could be as simple as providing a list of DT paths to forward from the innate FDT to the loaded dtb file.

As for the other information? Still an open question! It may very well become something that is decided on a case-by-case basis.

It's too early to solve fixups at this moment in time. Luckily, we can rely on the EFI_​DT_FIXUP_​PROTOCOL in the compatibility tooling, and consider the options when designing the in-kernel loading.

Conclusion

This needs to be dealt with. It's long past due.

I can see only one solution (in two parts):

My opinion is that all current and new workarounds around this defect are technical debt.

I have even made my own proof-of-concept, which is why I'm confident in my suggestions. Though please don't scrutinize the proof-of-concept, it needs to be worked on, a lot, still.

Side-tangent: I'm not sure if I should, but I'm also thinking that asking for a moratorium on usage of .dtbauto from systemd is necessary. At the very least it should not be added to the UKI spec, as it would require supporting a feature that we should, instead, aim to make entirely irrelevant and unneeded.

Finally, note how I never stated which ISA this is about. It is because this is NOT an ISA-specific issue, this is not an Arm problem. Arm is not “hard to boot”, like some think. The exact same family of problems exists on RISC-V, and could on x86, or other architectures. In reality, most, if not all, of these issues end-up tied to the (lack) of ownership of the life-cycle of the Linux kernel's own devicetree files.

  1. Whether it's through a “BIOS configuration menu”, or through different Platform Firmware installations, is unimportant. The fixup would look at the same facts. ↩

  2. This does not mean it's forbidden to rely solely on any given device's innate FDT from its Platform Firmware. Only that when a dtb file exists for a platform, it must be used, and from the same source tree as the kernel. It is an exclusive or: for a given kernel version either the device is supported only using the innate FDT of any platform firmware release for the device, or only from the dtb files from the kernel. ↩ ↩2