Hello coreboot community,
Google's vboot has been with us (coreboot) for many years already, probably mainly on Chromebook devices. In 3mdeb we have been using it with coreboot for a couple years as well.
From a user and developer point of view, vboot is a great library that hardens the firmware with cryptographic verification of the executable code and provides a redundant A/B + recovery layout scheme. However, it comes with a couple of limitations we have observed throughout the years:
1. Sometimes when the coreboot-based firmware is updated on a device after a longer period of time, the vboot submodule revisions get updated too. Occasionally it may lead to backwards incompatibility between recovery and A/B parts (assuming that only A/B is updated and recovery remain write protected) . 2. Sometimes when the coreboot-based firmware is updated on a device after a longer period of time, the car.ld (or any program layout shared across multiple stages in both recovery and A/B like bootblock and romstage/postcar) gets updated too. Almost every time it leads to backwards incompatibility and a brick.
There may be other cases as well, which need special handling and testing, because it may not be safe to flash an update on a device. One solution may be to freeze the codebase on given revisions and avoid updating them, but that's not what we typically want. I came up with the following idea:
Leverage the top-swap mechanism to provide two bootblocks with verstages (assuming verstage starts in bootblock and no separate verstage):
- Top Swap RO - used for booting recovery only (built with vboot submodule from coreboot revision X) - Top Swap RW - used for booting RW A/B only (built with vboot submodule from coreboot revision Y)
Example (simplified) flashmap layout could look like this:
RW_B (optional RW_A WP_RO { FMAP GBB COREBOOT TOP_SWAP_RO } TOP_SWAP_RW
- TOP_SWAP_RW would be the default one placed right under top of 4G. - Whenever vb2api_fail is called, coreboot would have to switch the top swap to TOP_SWAP_RO. - TOP_SWAP_RW is updatable so should be unlocked, like RW_A/B. But verstage should be protected
coreboot has some code/logic for building an image with top swaps, so I think it is doable.
Unsolved problems or questions:
1. Having two distinct vboot builds does not prevent from backwards incompatibility in vboot NV storage layout. Does the layout of vbnv even change? For the past 10 years the offsets have not changed (only one new flag and one flag name modification). So we can assume it is not a problem. 2. There might be other incompatibilities I am unaware of. Different recovery reasons codes shouldn't be a problem generally. 3. Chromebooks usually write-protect WP_RO region. In the scheme above, the TOP_SWAP_RW would be unprotected. Do Chromebooks still use WP screws and SPI chip provided protections?
- In 3mdeb we use the chipset PRx registers to protect the recovery part of the firmware. It gives a possibility of 4K granular protection of the SPI firmware. Multiple PRx registers could be used here: 1 - protect WP_RO, 2 - protect TOP_SWAP_RW. On updates lift the protection on second PR only - Ideally the top swaps could be protected/verified by solutions like Boot Guard (AFAIK boot Guard can support top swap, haven't researched yet how it works in practice). Then write protection of TOP_SWAP_RW would not be a concern.
4. Changes in the flashmap layout. This is rather out of scope, as any change in flashmap layout would imply updating whole BIOS region typically. So I would exclude this for now.
The solution is Intel-only, not aware of top-swaps on AMD, except PSP/ASP 16MB flash page limitations and top-swap like behavior. Also AMD is a whole different story, because bootblock is integrated into PSP/ASP directory, which has its own A/B (and two level) mechanism + psp_verstage... So leaving that aside for now.
Looking for your insights, comments, suggestions, any problems you see with the approach (haven't started any development yet) etc. Appreciated in advance.
Best regards,
Hi Michał,
we have had a discussion about this last fall in the coreboot leadership meeting on 2024-08-21 [1]. That time Matt brought it up. And your mentioned points where named back then, too. It was a intense discussion and several approaches were discussed together with their ups and downs. Nevertheless, the TOP_SWAP solution was not discussed to my memory and may indeed be helpful in this regard (though it might not be able to solve all known issues).
My proposal would be to bring this to the coreboot meeting again, showing that others (3mdeb) has the same issues. I think there is added value for the project if we could find a solution to this issue which fits all requirements from different parties.
Regards Werner
[1]: https://docs.google.com/document/d/1spx5B45u2IymaQitE-Thbqx4e4ykTok6a_g453Av...
Hello coreboot community,
Google's vboot has been with us (coreboot) for many years already, probably mainly on Chromebook devices. In 3mdeb we have been using it with coreboot for a couple years as well.
From a user and developer point of view, vboot is a great library that hardens the firmware with cryptographic verification of the executable code and provides a redundant A/B + recovery layout scheme. However, it comes with a couple of limitations we have observed throughout the years:
- Sometimes when the coreboot-based firmware is updated on a device
after a longer period of time, the vboot submodule revisions get updated too. Occasionally it may lead to backwards incompatibility between recovery and A/B parts (assuming that only A/B is updated and recovery remain write protected) . 2. Sometimes when the coreboot-based firmware is updated on a device after a longer period of time, the car.ld (or any program layout shared across multiple stages in both recovery and A/B like bootblock and romstage/postcar) gets updated too. Almost every time it leads to backwards incompatibility and a brick.
There may be other cases as well, which need special handling and testing, because it may not be safe to flash an update on a device. One solution may be to freeze the codebase on given revisions and avoid updating them, but that's not what we typically want. I came up with the following idea:
Leverage the top-swap mechanism to provide two bootblocks with verstages (assuming verstage starts in bootblock and no separate verstage):
- Top Swap RO - used for booting recovery only (built with vboot
submodule from coreboot revision X)
- Top Swap RW - used for booting RW A/B only (built with vboot submodule
from coreboot revision Y)
Example (simplified) flashmap layout could look like this:
RW_B (optional RW_A WP_RO { FMAP GBB COREBOOT TOP_SWAP_RO } TOP_SWAP_RW
- TOP_SWAP_RW would be the default one placed right under top of 4G.
- Whenever vb2api_fail is called, coreboot would have to switch the top
swap to TOP_SWAP_RO.
- TOP_SWAP_RW is updatable so should be unlocked, like RW_A/B. But
verstage should be protected
coreboot has some code/logic for building an image with top swaps, so I think it is doable.
Unsolved problems or questions:
- Having two distinct vboot builds does not prevent from backwards
incompatibility in vboot NV storage layout. Does the layout of vbnv even change? For the past 10 years the offsets have not changed (only one new flag and one flag name modification). So we can assume it is not a problem. 2. There might be other incompatibilities I am unaware of. Different recovery reasons codes shouldn't be a problem generally. 3. Chromebooks usually write-protect WP_RO region. In the scheme above, the TOP_SWAP_RW would be unprotected. Do Chromebooks still use WP screws and SPI chip provided protections?
- In 3mdeb we use the chipset PRx registers to protect the recovery part
of the firmware. It gives a possibility of 4K granular protection of the SPI firmware. Multiple PRx registers could be used here: 1 - protect WP_RO, 2 - protect TOP_SWAP_RW. On updates lift the protection on second PR only
- Ideally the top swaps could be protected/verified by solutions like
Boot Guard (AFAIK boot Guard can support top swap, haven't researched yet how it works in practice). Then write protection of TOP_SWAP_RW would not be a concern.
- Changes in the flashmap layout. This is rather out of scope, as any
change in flashmap layout would imply updating whole BIOS region typically. So I would exclude this for now.
The solution is Intel-only, not aware of top-swaps on AMD, except PSP/ASP 16MB flash page limitations and top-swap like behavior. Also AMD is a whole different story, because bootblock is integrated into PSP/ASP directory, which has its own A/B (and two level) mechanism + psp_verstage... So leaving that aside for now.
Looking for your insights, comments, suggestions, any problems you see with the approach (haven't started any development yet) etc. Appreciated in advance.
Best regards,
Hi Werner,
On 4/13/25 13:35, Werner Zeh via coreboot wrote:
Hi Michał,
we have had a discussion about this last fall in the coreboot leadership meeting on 2024-08-21 [1]. That time Matt brought it up. And your mentioned points where named back then, too. It was a intense discussion and several approaches were discussed together with their ups and downs. Nevertheless, the TOP_SWAP solution was not discussed to my memory and may indeed be helpful in this regard (though it might not be able to solve all known issues).
Thank you for the reference. I will take a look at it.
TOP_SWAP can help with some issues, but it does not solve all of them, indeed.
My proposal would be to bring this to the coreboot meeting again, showing that others (3mdeb) has the same issues. I think there is added value for the project if we could find a solution to this issue which fits all requirements from different parties.
Allright, I will put it on the agenda of the meeting on May 14th (I won't be able to attend and present it sooner than May 14th).
Regards Werner
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Best regards,
Hi Michał,
Sorry for being more confused than helpful on this topic in the meeting. As usual, once one has 5 minutes to sit down and calmly think the problem through, things became much clearer. :D
So, at the fundamental level, vboot is a mechanism (based on PKI crypto) to verify a mutable part of firmware (RW) from an immutable part of firmware (RO), so that the mutable part can get regular updates while the immutable part remains trusted. That is its entire reason for existence. If you don't need that, then you don't need vboot and all its RSA signature stuff.
What you're proposing here is that the topswap RW bootblock+verstage remain mutable (and maybe protected by some other mechanism involving flash protection registers and capsule updates and stuff). In that case, you don't need PKI crypto, you don't need RSA keys. You can simply use hash-based verification only, which is basically what CBFS_VERIFICATION is (which can be used without vboot), because every time you update your romstage/ramstage/etc. (meaning their hashes change), you can also update your bootblock (which stores the top metadata hash that all those other file hashes are verified from). The only situations where you ever really need PKI keys and signatures are situations where you want something that cannot be updated to verify something that can be updated, and that's no longer the case for the model you want.
So let's talk about the solution you want in a vacuum first, without relating to vboot: I don't think you can have both a "recovery" slot and A/B update slots at the same time, because that requires three separate slots and you only have two. You will need to decide which of the two options (recovery+normal or A+B) you care about more. To clarify, by "recovery" slot I mean a slot that will always have exactly the code your device was first built with (ideally ensured by hard write-protection). This is mostly useful if you're worried that some bug or malicious attack might erase all the writable firmware flash and disk on your system, so that you will still have some firmware left to recover from. However, to allow this your recovery firmware actually needs to be able to do that recovery all by itself (without assuming that there's still something on the disk it can boot, because when someone erases all your firmware flash they'll probably also erase your disk). On Chromebooks, we do that by letting it boot from a USB stick, but if you don't have a mechanism like that in your bootloader payload, a recovery slot is probably not that important to you. (Also, if you use other mechanisms like flash protection registers to protect the rest of your firmware, maybe this scenario isn't as relevant to you.)
An A/B slot system, on the other hand, means that you always have one "active" slot that you boot out of and one "update" slot that you can safely write new updates to. This allows you to both protect against asynchronous resets during the update, and to fall back from the updated version to the older version if the update didn't work for some reason. In this case, both of those slots need to be mutable (from the perspective of your update system, not necessarily from your running OS). I assume(?) that this is probably more what you care about in your situation.
So if we want to design an A/B update system based on topswap, we need a flash layout that's split into two areas (let's say RW_SECTION_A and RW_SECTION_B, but no RO_SECTION) which each contain a separate CBFS. You probably also need extra sections for the bootblock outside of those CBFSes since the topswap mechanism requires those to be in a special place, let's call those BOOTBLOCK_A and BOOTBLOCK_B. You also need some mechanism to select which of the two bootblocks to boot (this must be provided by the Intel hardware already, right? How do they do that? Via CMOS?). If you want the "fall back to older version when newer version doesn't work" thing, then you also need some extra NVRAM mechanism (e.g. CMOS) to store a counter that can be decremented by your bootblock every time it attempts to boot the new slot (however, note that if the new bootblock itself gets corrupted during the update, or is so broken that it doesn't even reach the part where it updates the try counter, the system may not be so easily recoverable... unless Intel themselves integrated a try counter mechanism directly in the topswap hardware?).
From a coreboot build perspective, you need the CBFS verification from BOOTBLOCK_A to tie through to the CBFS in RW_SECTION_A and from BOOTBLOCK_B to the CBFS in RW_SECTION_B. We could try to make cbfstool aware of all that, but maybe the easier option is that cbfstool only knows about BOOTBLOCK_A and RW_SECTION_A, and you only copy the entire sections into the B slots when you are performing an update. So basically, while the flash image is in a file only the A slot is valid, but when your system wants to do an update and the A slot is currently active (meaning B is the target slot for the update), your update system would flash BOOTBLOCK_A and RW_SECTION_A from the new image file into BOOTBLOCK_B and RW_SECTION_B on your flash. (This requires that all code including the bootblock itself is position-independent, I'm not sure if that's true nowadays on x86? Or can be made true by moving the page table setup a bit earlier, maybe? Otherwise, the build and update process would become a lot more complicated.)
I think this scheme would be so different from vboot that it probably wouldn't make sense to try to integrate it into vboot as a separate option. Because vboot is a solution built around PKI crypto verification and you don't need the PKI crypto part. The only things worth reusing for what you need might be the NVRAM storage solution (but the underlying code to access CMOS NVRAM in coreboot is outside of vboot anyway) and maybe a small part of the slot selection logic. It would basically be like trying to design a new car based on an existing model, but the only thing you want to keep are the tires and the windshield wipers (and not the chassis or the engine). I think designing this as a completely separate new mechanism (based on the underlying CBFS_VERIFICATION) makes more sense.
-
Julius Werner -
Michał Żygowski -
Werner Zeh