Dear coreboot folks,
Arthur pushed the proof of concept *Use FDT as payload handoff instead lb_tables* [1] to pass information to the payload via FDT instead of coreboot tables.
Julius asked for more background, and Arthur replied. I am moving the discussion to the list for broader visibility, and better way of discussing these things than Gerrit comments.
Were there any discussions about this that I missed (if so would you mind linking to them)? Can you explain who needs this and why, what's the overall goal? (I know there have been vague discussions about various new firmware handoff formats from Arm and Intel and others, but if this is related to those I missed the parts where there was suddenly consensus on how it should look, and why that is FDT.)
Most of the discussion happened verbally at osfc, so I can't point to any written communication. There is https://www.osfc.io/2022/talks/collaborative-firmware-payload-handoff-design... if you have the time to listen to a talk. So if I have to summarize the idea's/goals:
- Intel Universal scalable firmware suggested using UEFI HOBs as a handoff structure
- coreboot won't do that as it's no improvement over coreboot tables
- USF won't make coreboot tables a standard either
- Maybe something new can be created that would fit both the needs of payload handoff and other binary information passing between programs (e.g. FSP)?
- USF people suggested using CBOR to encode data to so that serialization/deserialization of data is explicit instead of using structs.
- CBOR is an improvement for the data format, but not what coreboot/(other project) should tell the payload: i.e. what entries are needed
- FDT was suggested as an alternative as it covers both explicit serialization/deserialization + what payload tell each other since the spec already exists, with a lot of existing compatible node strings. (e.g. I reuse simple-framebuffer from Linux here to hand of the framebuffer) Also most payloads and firmware have code for working with FDT.
- (unrelated to payload handof but part of the discussions) Another idea is to also use FDT as an interface between coreboot and FSP.
- a lot of payloads already have code to work with FDT so having FDT as a handoff structure would not be a lot of work
- FDT should make compatibility between coreboot and payload easier when handoff entries are added or changed (plain structs are painful if they are updated as both payload and coreboot definitions need to match)
- debugging the handoff structure is easier with FDT as you can use DTC in the OS userspace or coreboot code can also print out the full structure in human readable format.
What are your thoughts?
- Is it a good idea to use FDT and eventually replace coreboot tables.
- If so, do we keep both for a while to make transitioning easier? I think maintaining two structures is not sustainable long term.
- How should the 'entries' be structured? I made compatible nodes that map somewhat to existing coreboot table structs.
Kind regards,
Paul
Hello,
This would fit nicely for POWER9 boards using Skiboot as a payload. I started a discussion related to that some time ago [1], since then we decided to switch to generating FDT in code. Basically, we did some parts of mentioned POC but at SoC level, to not break existing code. Having it as common code would make it much easier to maintain.
What are your thoughts?
* Is it a good idea to use FDT and eventually replace coreboot tables.
I think FDT is widely supported, from embedded platforms to servers, on many different architectures. I've got a feeling that whenever Intel talks about "universal", it usually means "x86-only", or at least "lets make it work for x86 and others can worry about their architectures later".
FDT is usually exposed by kernel so you don't need `cbmem` util for simplest cases, but I guess it would still be nice to have to decode specific tables. However:
* If so, do we keep both for a while to make transitioning easier? I think maintaining two structures is not sustainable long term.
Persuading kernel to parse FDT on x86 platforms and propagating that change to kernels included in distros can take some time. Meanwhile, `cbmem` could be used to read the data instead, but it also would have to be made aware of the transition. IIUC currently it chooses between FDT and coreboot tables based on architecture it is built for. Users would have to update `cbmem` together with firmware, which may not always be the case.
* How should the 'entries' be structured? I made compatible nodes that map somewhat to existing coreboot table structs.
As for standard coreboot tables like timestamps or console, for me existing format looks good. Just make sure to explicitly define endianness of fields :)
Based on experience with POWER9, we need to be able to both create entries specific to given SoC (e.g. XSCOM ranges, which is something between MSR and root complex "PCIe" devices) and to modify entries generated by common FDT code (Skiboot expects chip IDs on some of the entries when there are more CPUs on board). This can be easily done by having callouts to platform- and arch-specific code. Current FIT code doesn't work for the latter case, because call to `fit_update_memory()` happens after platform is allowed to modify the tree.
[1] https://mail.coreboot.org/hyperkitty/list/coreboot@coreboot.org/thread/6EZWU...
Hi Arthur,
Thanks for the detailed response. I went ahead and watched your presentation as well. FWIW, I think there were a few technical inaccuracies about the current state of coreboot tables in your presentation that might be good to get out of the way first:
1. You said that coreboot tables were GPLv2 licensed, and this could be an adoption problem for payloads... however, while src/commonlib/include/commonlib/coreboot_tables.h has a GPLv2 header, the same definitions for all structures have always been available in payloads/libpayload/include/coreboot_tables.h with the BSD license, so there's no actual issue for payloads. This is just a consequence of the historic duplication we always had between coreboot and libpayload. Now that we have the infrastructure to share commonlib/bsd code between the two, I hope sooner or later someone will find the time to delete coreboot's old (GPLv2) headers and move the libpayload (BSD) headers into commonlib/bsd to be shared by both.
2. You said that there was an ABI problem due to different alignment requirements for 64-bit types on different architectures. This has actually never been true (unless someone made a mistake and slipped a raw uint64_t into the table definitions, but we are generally pretty vigilant in reviews about this so if it did happen it must have been fixed quickly). If you look at commit 0d304c18 (so old that I can't even link to Gerrit for it), we used to use a struct solution with custom accessor functions to create 32-bit-aligned 64-bit types in the past. More recently in https://review.coreboot.org/63494 we replaced that with __attribute__((aligned, 4)) which does the same thing in a less cumbersome way.
3. You also said that the coreboot tables aren't versioned, which is of course technically true. However, the general guideline that I think we have been trying to maintain in reviews is that coreboot tables should be backwards-compatible, meaning that once a tag has been widely used its structure layout should not change anymore (at most new fields might be added at the end so that the size field can be used to differentiate between both versions). Older versions of libpayload should always be compatible with newer versions of coreboot and as far as I'm aware we have done a pretty good job of maintaining that standard. When a structure needs to be updated, a simple solution is to just create a new tag and stop using the old one, and this has been done in the past (e.g. LB_TAG_VBOOT_HANDOFF has been deprecated and LB_TAG_VBOOT_WORKBUF basically contains the equivalent stuff in a different format today). Fundamentally, this is pretty similar to how FDT is used by Linux -- there's no explicit versioning for nodes or properties, but bindings are meant to be stable, and when you really need to change stuff up you can just make a new binding with different compatible string and deprecate the old one.
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that. One of the things you mentioned was so that we might be able to use the same handoff format for FSP and the payload, and the first question that comes to my mind is... why? What does that actually give us? I think it's easy to sometimes fall into the xkcd.com/927 trap of unifying things just for unification's sake, because it feels "right", but ending up with a solution that's less suited for either target than the sum of the burden of carrying all individual existing solutions. The information that needs to be passed to the FSP is (according to my understanding, I'm admittedly not an FSP expert) completely different from the information that needs to be passed to the payload (except maybe for one or two tiny specks like console information?), so it's not like you'd actually want to reuse the actual data structure you built and pass both things the union of the stuff they each need to know, when the intersection of that is almost empty. So what other reason is there to unify the format, then? It's not like we have a ton of coreboot table support libraries or tooling that we could drop if we unified them with something else -- that's the beauty about a very simple C structure format like that, it's basically free to use.
The other aspect is the payload itself, and that whole "universal payload" idea (I assume that's what Intel is calling "USF" now, or is that something else?). I have to admit I'm just fundamentally very skeptical of that whole concept, and I just don't believe it's gonna work out in a way that's actually going to generate more benefit than problems for anybody. The way we use payloads nowadays is often very deeply tied to coreboot, with things like CBFS, verification, console, timestamps, VPD, etc. all implemented in a very coreboot-specific way (not just the handoff format but the very implementation of the thing itself). You can't unify those things without unifying those implementations which is practically impossible (i.e. if you unify every aspect of how everything works between different firmware projects then there's little point in having separate projects at all anymore, because they'd all be doing the same thing anyway). So in practice, I think the only thing a universal handoff would accomplish is that e.g. you can then have a payload that will boot from coreboot and start parsing coreboot's handoff, but then hangs because the flash access bindings it is looking for don't exist in coreboot's handoff and it doesn't know what to do with CBFS bindings. If you want a payload that can support more than one firmware, with or without universal handoff you'll have to implement specific support for each anyway (e.g. like GRUB has implemented CBFS support), and once you're doing that the extra cost of implementing a parser for a very simple, dependency-less handoff format like the coreboot tables is not really noticeable on top.
That said, I understand Intel wants to push this and there are people who want to try this idea out, and I don't want to block them just because I don't believe it'll work. So if you want to implement this as an experimental, default-off Kconfig option that can hopefully stay pretty isolated and self-contained in the code base, I'm fine with that (and I don't particularly care about CBOR vs FDT... I think both are pretty similar, both have tiny advantages and disadvantages for your goals, and both add a similarly large overhead compared to coreboot tables). But I wouldn't want this added as a replacement for the coreboot tables either right away or with some implicit agreement that we will transition there eventually. We don't even have any practical use cases for universal payloads yet, but we have plenty of big existing use cases that only combine coreboot with a single, coreboot-specific payload and don't have any plans to require any more flexibility on either of those sides in the future (ChromeOS is one of them), and I wouldn't want those use cases to suffer from being preemptively forward-ported onto a new solution that just adds a lot of extra weight without having any proven benefits as of yet.
So basically, I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone. I'm not trying to stand in the way of experimenting with new alternatives if people want to do that.
Hi Julius
So FDT has a few advantages over C structs for passing information between 2 separate programs: - reflection is easy - no problem of 'header mismatch' to have the right definition of a struct - easy to update with a text editor without needing to write code - No need to be careful about ABI, writing structs e.g. 64bit alignment
I agree with you that these, while nice, are not very compelling arguments to replace coreboot tables, which are very established by now. For instance the header mismatch problem does not really exist given how stable coreboot tables have actually been. There is also no use case of wanting to edit the payload handoff by hand. At last x86 Linux can be launched in around 200 bytes of stackless assembly code... That's not possible with FDT or at least I'm not aware of any stackless assembly written FDT parser.
For FSP it makes a lot more sense to improve the binary interface over the existing C struct: - reflection is a problem as FSP comes with default and printing out a struct in C is a pain - You never know whether the FSP binary will match a given header - FSP validation is a manual process afaict. The people doing it don't want to write code. That is why coreboot is not used in FSP validation. Intel uses UEFI so that there is a menu to change FSP values for validation.
So it's not that FSP would need the same information as the payload. The intersection is indeed almost 0. It's just that FDT is actually a nice format for 2 separate programs to talk to each other. For instance u-boot also uses it to pass information from spl to u-boot (bootloader) afaik.
I think Intel only wanted to try FDT as an interface for FSP if it's also used in the payload (read EDK2) handoff (don't ask me why). Maybe someone needs to convince Intel that using FDT for just FSP is a good idea regardless of how payload handoff is handled.
So basically, I'd like to make sure that we can agree to keep coreboot
tables as a first-class citizen now and in the future, and don't force a new format on anyone. I'm not trying to stand in the way of experimenting with new alternatives if people want to do that.
Having multiple ways to do the same thing is a maintenance problem. It was a fun experiment, but it has no place in the master branch if the end goal is just to have FDT live side by side with coreboot tables.
Arthur
On Fri, Nov 4, 2022 at 3:41 AM Julius Werner jwerner@chromium.org wrote:
Hi Arthur,
Thanks for the detailed response. I went ahead and watched your presentation as well. FWIW, I think there were a few technical inaccuracies about the current state of coreboot tables in your presentation that might be good to get out of the way first:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads... however, while src/commonlib/include/commonlib/coreboot_tables.h has a GPLv2 header, the same definitions for all structures have always been available in payloads/libpayload/include/coreboot_tables.h with the BSD license, so there's no actual issue for payloads. This is just a consequence of the historic duplication we always had between coreboot and libpayload. Now that we have the infrastructure to share commonlib/bsd code between the two, I hope sooner or later someone will find the time to delete coreboot's old (GPLv2) headers and move the libpayload (BSD) headers into commonlib/bsd to be shared by both.
- You said that there was an ABI problem due to different alignment
requirements for 64-bit types on different architectures. This has actually never been true (unless someone made a mistake and slipped a raw uint64_t into the table definitions, but we are generally pretty vigilant in reviews about this so if it did happen it must have been fixed quickly). If you look at commit 0d304c18 (so old that I can't even link to Gerrit for it), we used to use a struct solution with custom accessor functions to create 32-bit-aligned 64-bit types in the past. More recently in https://review.coreboot.org/63494 we replaced that with __attribute__((aligned, 4)) which does the same thing in a less cumbersome way.
- You also said that the coreboot tables aren't versioned, which is
of course technically true. However, the general guideline that I think we have been trying to maintain in reviews is that coreboot tables should be backwards-compatible, meaning that once a tag has been widely used its structure layout should not change anymore (at most new fields might be added at the end so that the size field can be used to differentiate between both versions). Older versions of libpayload should always be compatible with newer versions of coreboot and as far as I'm aware we have done a pretty good job of maintaining that standard. When a structure needs to be updated, a simple solution is to just create a new tag and stop using the old one, and this has been done in the past (e.g. LB_TAG_VBOOT_HANDOFF has been deprecated and LB_TAG_VBOOT_WORKBUF basically contains the equivalent stuff in a different format today). Fundamentally, this is pretty similar to how FDT is used by Linux -- there's no explicit versioning for nodes or properties, but bindings are meant to be stable, and when you really need to change stuff up you can just make a new binding with different compatible string and deprecate the old one.
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that. One of the things you mentioned was so that we might be able to use the same handoff format for FSP and the payload, and the first question that comes to my mind is... why? What does that actually give us? I think it's easy to sometimes fall into the xkcd.com/927 trap of unifying things just for unification's sake, because it feels "right", but ending up with a solution that's less suited for either target than the sum of the burden of carrying all individual existing solutions. The information that needs to be passed to the FSP is (according to my understanding, I'm admittedly not an FSP expert) completely different from the information that needs to be passed to the payload (except maybe for one or two tiny specks like console information?), so it's not like you'd actually want to reuse the actual data structure you built and pass both things the union of the stuff they each need to know, when the intersection of that is almost empty. So what other reason is there to unify the format, then? It's not like we have a ton of coreboot table support libraries or tooling that we could drop if we unified them with something else -- that's the beauty about a very simple C structure format like that, it's basically free to use.
The other aspect is the payload itself, and that whole "universal payload" idea (I assume that's what Intel is calling "USF" now, or is that something else?). I have to admit I'm just fundamentally very skeptical of that whole concept, and I just don't believe it's gonna work out in a way that's actually going to generate more benefit than problems for anybody. The way we use payloads nowadays is often very deeply tied to coreboot, with things like CBFS, verification, console, timestamps, VPD, etc. all implemented in a very coreboot-specific way (not just the handoff format but the very implementation of the thing itself). You can't unify those things without unifying those implementations which is practically impossible (i.e. if you unify every aspect of how everything works between different firmware projects then there's little point in having separate projects at all anymore, because they'd all be doing the same thing anyway). So in practice, I think the only thing a universal handoff would accomplish is that e.g. you can then have a payload that will boot from coreboot and start parsing coreboot's handoff, but then hangs because the flash access bindings it is looking for don't exist in coreboot's handoff and it doesn't know what to do with CBFS bindings. If you want a payload that can support more than one firmware, with or without universal handoff you'll have to implement specific support for each anyway (e.g. like GRUB has implemented CBFS support), and once you're doing that the extra cost of implementing a parser for a very simple, dependency-less handoff format like the coreboot tables is not really noticeable on top.
That said, I understand Intel wants to push this and there are people who want to try this idea out, and I don't want to block them just because I don't believe it'll work. So if you want to implement this as an experimental, default-off Kconfig option that can hopefully stay pretty isolated and self-contained in the code base, I'm fine with that (and I don't particularly care about CBOR vs FDT... I think both are pretty similar, both have tiny advantages and disadvantages for your goals, and both add a similarly large overhead compared to coreboot tables). But I wouldn't want this added as a replacement for the coreboot tables either right away or with some implicit agreement that we will transition there eventually. We don't even have any practical use cases for universal payloads yet, but we have plenty of big existing use cases that only combine coreboot with a single, coreboot-specific payload and don't have any plans to require any more flexibility on either of those sides in the future (ChromeOS is one of them), and I wouldn't want those use cases to suffer from being preemptively forward-ported onto a new solution that just adds a lot of extra weight without having any proven benefits as of yet.
So basically, I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone. I'm not trying to stand in the way of experimenting with new alternatives if people want to do that.
I was involved in the creation of coreboot tables. They were the best we could come up with at the time.
Over time, their limits have been more and more apparent. This is why, in oreboot, we removed the initial support for coreboot tables and moved to FDT.
FDT is better than coreboot tables. I'll repeat Arthur's points, the ones I think are important: " - reflection is easy - no problem of 'header mismatch' to have the right definition of a struct - easy to update with a text editor without needing to write code "
These are pretty critical. Further, Linux (and many if not most kernels in fact) have built-in support for FDT, where they may not for coreboot tables.
Finally, FDT exists outside coreboot; unlike coreboot tables. There are parsers in every language I've looked at (including Rust and Go) for fdt. I think that's good. I think any time we can replace a coreboot-specific standard with a standard that exists in a broader context than "just coreboot", that's a good thing.
ron
On Fri, Nov 4, 2022 at 12:58 AM Arthur Heymans arthur@aheymans.xyz wrote:
Hi Julius
So FDT has a few advantages over C structs for passing information between 2 separate programs:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
- No need to be careful about ABI, writing structs e.g. 64bit alignment
I agree with you that these, while nice, are not very compelling arguments to replace coreboot tables, which are very established by now. For instance the header mismatch problem does not really exist given how stable coreboot tables have actually been. There is also no use case of wanting to edit the payload handoff by hand. At last x86 Linux can be launched in around 200 bytes of stackless assembly code... That's not possible with FDT or at least I'm not aware of any stackless assembly written FDT parser.
For FSP it makes a lot more sense to improve the binary interface over the existing C struct:
- reflection is a problem as FSP comes with default and printing out a
struct in C is a pain
- You never know whether the FSP binary will match a given header
- FSP validation is a manual process afaict. The people doing it don't
want to write code. That is why coreboot is not used in FSP validation. Intel uses UEFI so that there is a menu to change FSP values for validation.
So it's not that FSP would need the same information as the payload. The intersection is indeed almost 0. It's just that FDT is actually a nice format for 2 separate programs to talk to each other. For instance u-boot also uses it to pass information from spl to u-boot (bootloader) afaik.
I think Intel only wanted to try FDT as an interface for FSP if it's also used in the payload (read EDK2) handoff (don't ask me why). Maybe someone needs to convince Intel that using FDT for just FSP is a good idea regardless of how payload handoff is handled.
So basically, I'd like to make sure that we can agree to keep coreboot
tables as a first-class citizen now and in the future, and don't force a new format on anyone. I'm not trying to stand in the way of experimenting with new alternatives if people want to do that.
Having multiple ways to do the same thing is a maintenance problem. It was a fun experiment, but it has no place in the master branch if the end goal is just to have FDT live side by side with coreboot tables.
Arthur
On Fri, Nov 4, 2022 at 3:41 AM Julius Werner jwerner@chromium.org wrote:
Hi Arthur,
Thanks for the detailed response. I went ahead and watched your presentation as well. FWIW, I think there were a few technical inaccuracies about the current state of coreboot tables in your presentation that might be good to get out of the way first:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads... however, while src/commonlib/include/commonlib/coreboot_tables.h has a GPLv2 header, the same definitions for all structures have always been available in payloads/libpayload/include/coreboot_tables.h with the BSD license, so there's no actual issue for payloads. This is just a consequence of the historic duplication we always had between coreboot and libpayload. Now that we have the infrastructure to share commonlib/bsd code between the two, I hope sooner or later someone will find the time to delete coreboot's old (GPLv2) headers and move the libpayload (BSD) headers into commonlib/bsd to be shared by both.
- You said that there was an ABI problem due to different alignment
requirements for 64-bit types on different architectures. This has actually never been true (unless someone made a mistake and slipped a raw uint64_t into the table definitions, but we are generally pretty vigilant in reviews about this so if it did happen it must have been fixed quickly). If you look at commit 0d304c18 (so old that I can't even link to Gerrit for it), we used to use a struct solution with custom accessor functions to create 32-bit-aligned 64-bit types in the past. More recently in https://review.coreboot.org/63494 we replaced that with __attribute__((aligned, 4)) which does the same thing in a less cumbersome way.
- You also said that the coreboot tables aren't versioned, which is
of course technically true. However, the general guideline that I think we have been trying to maintain in reviews is that coreboot tables should be backwards-compatible, meaning that once a tag has been widely used its structure layout should not change anymore (at most new fields might be added at the end so that the size field can be used to differentiate between both versions). Older versions of libpayload should always be compatible with newer versions of coreboot and as far as I'm aware we have done a pretty good job of maintaining that standard. When a structure needs to be updated, a simple solution is to just create a new tag and stop using the old one, and this has been done in the past (e.g. LB_TAG_VBOOT_HANDOFF has been deprecated and LB_TAG_VBOOT_WORKBUF basically contains the equivalent stuff in a different format today). Fundamentally, this is pretty similar to how FDT is used by Linux -- there's no explicit versioning for nodes or properties, but bindings are meant to be stable, and when you really need to change stuff up you can just make a new binding with different compatible string and deprecate the old one.
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that. One of the things you mentioned was so that we might be able to use the same handoff format for FSP and the payload, and the first question that comes to my mind is... why? What does that actually give us? I think it's easy to sometimes fall into the xkcd.com/927 trap of unifying things just for unification's sake, because it feels "right", but ending up with a solution that's less suited for either target than the sum of the burden of carrying all individual existing solutions. The information that needs to be passed to the FSP is (according to my understanding, I'm admittedly not an FSP expert) completely different from the information that needs to be passed to the payload (except maybe for one or two tiny specks like console information?), so it's not like you'd actually want to reuse the actual data structure you built and pass both things the union of the stuff they each need to know, when the intersection of that is almost empty. So what other reason is there to unify the format, then? It's not like we have a ton of coreboot table support libraries or tooling that we could drop if we unified them with something else -- that's the beauty about a very simple C structure format like that, it's basically free to use.
The other aspect is the payload itself, and that whole "universal payload" idea (I assume that's what Intel is calling "USF" now, or is that something else?). I have to admit I'm just fundamentally very skeptical of that whole concept, and I just don't believe it's gonna work out in a way that's actually going to generate more benefit than problems for anybody. The way we use payloads nowadays is often very deeply tied to coreboot, with things like CBFS, verification, console, timestamps, VPD, etc. all implemented in a very coreboot-specific way (not just the handoff format but the very implementation of the thing itself). You can't unify those things without unifying those implementations which is practically impossible (i.e. if you unify every aspect of how everything works between different firmware projects then there's little point in having separate projects at all anymore, because they'd all be doing the same thing anyway). So in practice, I think the only thing a universal handoff would accomplish is that e.g. you can then have a payload that will boot from coreboot and start parsing coreboot's handoff, but then hangs because the flash access bindings it is looking for don't exist in coreboot's handoff and it doesn't know what to do with CBFS bindings. If you want a payload that can support more than one firmware, with or without universal handoff you'll have to implement specific support for each anyway (e.g. like GRUB has implemented CBFS support), and once you're doing that the extra cost of implementing a parser for a very simple, dependency-less handoff format like the coreboot tables is not really noticeable on top.
That said, I understand Intel wants to push this and there are people who want to try this idea out, and I don't want to block them just because I don't believe it'll work. So if you want to implement this as an experimental, default-off Kconfig option that can hopefully stay pretty isolated and self-contained in the code base, I'm fine with that (and I don't particularly care about CBOR vs FDT... I think both are pretty similar, both have tiny advantages and disadvantages for your goals, and both add a similarly large overhead compared to coreboot tables). But I wouldn't want this added as a replacement for the coreboot tables either right away or with some implicit agreement that we will transition there eventually. We don't even have any practical use cases for universal payloads yet, but we have plenty of big existing use cases that only combine coreboot with a single, coreboot-specific payload and don't have any plans to require any more flexibility on either of those sides in the future (ChromeOS is one of them), and I wouldn't want those use cases to suffer from being preemptively forward-ported onto a new solution that just adds a lot of extra weight without having any proven benefits as of yet.
So basically, I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone. I'm not trying to stand in the way of experimenting with new alternatives if people want to do that.
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
FDT is better than coreboot tables. I'll repeat Arthur's points, the ones I think are important: "
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
"
These are pretty critical. Further, Linux (and many if not most kernels in fact) have built-in support for FDT, where they may not for coreboot tables.
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with. When was the last time you actually had to look at coreboot tables in a hexdump? I think I have done that maybe once or twice total in my ~10 years of working with coreboot. We just don't need reflection, or updating with a text editor. It's not a persistent structure that's stored as a file on anyone's disk, it gets written out at runtime and then consumed again immediately after. There are no environments with text editors between coreboot and the payload.
I also disagree that FDTs are somehow more robust to "mismatches" than C structures. Yes, the data format is self-describing, but what that data means still has to be agreed upon by both sides. Where's the difference between changing a member in a C structure from an integer to a pointer, and changing a property in an FDT from an integer cell to a phandle? Either way it's going to be broken if encountered by an old reader -- maybe one leads to a segfault and the other "just" to a clean parsing error, but that still doesn't mean it's not a problem and you still need conventions to prevent it. So the convention is that you just keep the structure layouts / bindings stable after you've started using them, and that's what we're doing with coreboot tables, just like projects using FDT are doing that with their bindings. (FWIW, I'm not aware of any instances where we broke coreboot table backwards compatibility in the past.)
I don't want to say things about FSP because I honestly don't know enough about how that works and what people struggle with in practice to have an informed opinion. If most of the people working with FSP platforms think that using FDT as a handoff format there is a good idea, then sure, do that. But that doesn't really make a difference for what's better for payload handoff.
Hi all,
On 04.11.22 08:57, Arthur Heymans wrote:
So FDT has a few advantages over C structs for passing information between 2 separate programs:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
- No need to be careful about ABI, writing structs e.g. 64bit alignment
this is something I've been seeing a lot during the whole universal payload discussions: The arguments often miss the point. These are actually good arguments. But they are about the implementation that handles the table data and not about the table format. One could describe the coreboot table data in some equivalent of FDT bindings and one could also write C structs for FDT bindings. I've personally often argued for explicit serialization code instead of using C structs in coreboot. It's doable. In case of the coreboot tables, there's not that much benefit because the structs are well defined and make no trouble with alignment (compared to weird `packed` structs).
(I hope I'm using the terms right as I'm not yet used to FDT things.)
I agree with you that these, while nice, are not very compelling arguments
to replace coreboot tables, which are very established by now. For instance the header mismatch problem does not really exist given how stable coreboot tables have actually been. There is also no use case of wanting to edit the payload handoff by hand. At last x86 Linux can be launched in around 200 bytes of stackless assembly code... That's not possible with FDT or at least I'm not aware of any stackless assembly written FDT parser.
If there is no such thing yet, I'd argue to write a compiler for it. One that takes the bindings and a description where to put the data, and writes the assembler for you :)
I think Intel only wanted to try FDT as an interface for FSP if it's also used in the payload (read EDK2) handoff (don't ask me why). Maybe someone needs to convince Intel that using FDT for just FSP is a good idea regardless of how payload handoff is handled.
They probably assume that FSP would talk to their payload directly with only a shim in between. Remember, the whole discussion started with somebody wanting to hand over HOBs to edk2 (which are currently produced by FSP).
And if we are going to talk about the special-purpose payload (I think that is what Intel means when they say `universal`) topic... one impor- tant point that was discussed at the OSFC was that we could have a neutral place to host the bindings (osfw.foundation was mentioned IIRC). So that Intel would have to agree to an interface that they couldn't dictate nor change whenever they like. IMO, this is the most important point of the whole endeavor.
If we can't get to some common ground there, we can still talk about FDT in coreboot. Although, IMHO, it would boil down to the simple question if we want to use FDT so we can use already written libraries and tools around it. Unless we wanted to replace ACPI with FDT¹ xD
So basically, I'd like to make sure that we can agree to keep coreboot
tables as a first-class citizen now and in the future, and don't force a new format on anyone. I'm not trying to stand in the way of experimenting with new alternatives if people want to do that.
Having multiple ways to do the same thing is a maintenance problem. It was a fun experiment, but it has no place in the master branch if the end goal is just to have FDT live side by side with coreboot tables.
Don't remember who said it. Somebody already said at the OSFC how a future could look like: We should keep the coreboot table for compati- bility in any case. We could freeze the format (structs) though and say new things should be described in FDT. But that would mean that we should keep two implementations in the tree, at least for a while. Alternatively, keep one in a shim layer payload. But that's still two things to maintain.
After all, I actually like the idea to use FDT. We just need to find a good use case for it :)
Nico
¹ I wouldn't mind that btw., replacing ACPI. Right now FDT seems to me like the better choice, it's just not what we have started with on x86. As it's less complex (not a programming language), assuming we'd gene- rate FDT info everywhere where we have ACPI now, it would be simple to write a compatibility layer, i.e. something that knows the FDT bindings we use and generates ACPI tables for them. Then one would only need ACPI in coreboot (or a shim payload?) if they want compatibility to OSs that don't understand FDT.
Hi Arthur,
- You never know whether the FSP binary will match a given header
slightly off-topic for this thread, but it's possible to check for mismatches between the expected FSP binary versions and the actual FSP binary version during runtime by implementing soc_validate_fspm_header that compares the version information from the FSP binary to some version defines in the FSP headers. See [1] for the implementation used by the AMD SoCs that use FSP.
Regards, Felix
[1] https://review.coreboot.org/plugins/gitiles/coreboot/+/aab7f0490419c50a1bcb5...
Julius Werner wrote:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads...
The only generator code I know is GPLv2.
One could argue that only coreboot should ever generate this format, but I think one goal here is to enable or even encourage new situations.
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that.
This isn't just a technical change, for me there is also the more political goal to make more of what Intel does with USF going forward (please see https://github.com/UniversalScalableFirmware/Introduction/blob/main/USF_Over... ) maximally useful for and within the coreboot context, ideally while encouraging reuse.
Since Intel has planned to use CBOR and are unlikely to use coreboot tables for anything outside of coreboot I see using FDT as a way to improve the situation for everyone while making players come together more - that's an unusually rare win-win.
I strongly prefer FDT over CBOR. FDT is simple on byte level, which I find important to not lose the simplicity of coreboot tables. FDT can easily be parsed without RAM, CBOR not so much.
use the same handoff format for FSP and the payload, What does that actually give us?
More common (data) patterns, more code re-use and potentially even some future innovation once common patterns are more broadly used.
ending up with a solution that's less suited for either target
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
information that needs to be passed to the FSP is (according to my understanding, I'm admittedly not an FSP expert) completely different from the information that needs to be passed to the payload (except maybe for one or two tiny specks like console information?), so it's not like you'd actually want to reuse the actual data structure you built and pass both things the union of the stuff they each need to know, when the intersection of that is almost empty.
1. Already the possibility to maintain only a single dataset in a single format is quite nice, both conceptually and for usability.
2. Both of these software packages change over time, given the above possibility the intersection may expand.
The other aspect is the payload itself, and that whole "universal payload" idea (I assume that's what Intel is calling "USF" now, or is that something else?).
Please do look at the diagram on the last page of the USF PDF.
I think it makes clear what Intel is working towards.
The way we use payloads nowadays is often very deeply tied to coreboot,
More so if "often" means number of shipped systems, less so if "often" means the number of individual payloads.
You can't unify those things without unifying those implementations
Yes, but some compatibility is already a win in my opinion.
the extra cost of implementing a parser for a very simple, dependency-less handoff format like the coreboot tables is not really noticeable on top.
This is similar enough for coreboot tables and FDT, that's why I consider them really comparable.
That said, I understand Intel wants to push this
Intel wants to push USF and CBOR.
"Our" people want that energy to fit well with coreboot.
If we manage to improve some interfaces within UEFI in the process that would be even better.
can hopefully stay pretty isolated and self-contained in the code base
I think there should be just one source of truth, from which other formats are translated/derived.
I don't particularly care about CBOR vs FDT... I think both are pretty similar, both have tiny advantages and disadvantages for your goals, and both add a similarly large overhead compared to coreboot tables).
On the byte level only FDT and coreboot tables are really comparable.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
Julius Werner wrote:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
(FWIW, I'm not aware of any instances where we broke coreboot table backwards compatibility in the past.)
I agree that coreboot tables have also been pretty stable.
Nico Huber wrote:
I've personally often argued for explicit serialization code instead of using C structs in coreboot. It's doable. In case of the coreboot tables, there's not that much benefit because the structs are well defined and make no trouble with alignment (compared to weird `packed` structs).
I think the most important improvement we could make to coreboot tables is to add explicit (de)serialization code, while others value reflection more. I agree however that reflection would create opportunity for new runtime UIs that are less doable or nonsensical without it.
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different compilers. "Just use the coreboot compiler" is not a good answer.
There is also no use case of wanting to edit the payload handoff by hand. At last x86 Linux can be launched in around 200 bytes of stackless assembly code... That's not possible with FDT or at least I'm not aware of any stackless assembly written FDT parser.
If there is no such thing yet, I'd argue to write a compiler for it. One that takes the bindings and a description where to put the data, and writes the assembler for you :)
Fun idea! It reminds me of romcc though.
Parsing FDT in assembler is anyway eminently doable, CBOR much less so.
Unless we wanted to replace ACPI with FDT¹ ¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
Having multiple ways to do the same thing is a maintenance problem. It was a fun experiment, but it has no place in the master branch if the end goal is just to have FDT live side by side with coreboot tables.
Don't remember who said it. Somebody already said at the OSFC how a future could look like: We should keep the coreboot table for compati- bility in any case. We could freeze the format (structs) though and say new things should be described in FDT. But that would mean that we should keep two implementations in the tree, at least for a while. Alternatively, keep one in a shim layer payload. But that's still two things to maintain.
I talked a bit about the idea to have a single origin format and translation to other formats. Generic translators would not need much maintainance.
//Peter
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads...
The only generator code I know is GPLv2.
One could argue that only coreboot should ever generate this format, but I think one goal here is to enable or even encourage new situations.
Yes, I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense. I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea and not worth the cost".
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that.
This isn't just a technical change, for me there is also the more political goal to make more of what Intel does with USF going forward (please see https://github.com/UniversalScalableFirmware/Introduction/blob/main/USF_Over... ) maximally useful for and within the coreboot context, ideally while encouraging reuse.
Since Intel has planned to use CBOR and are unlikely to use coreboot tables for anything outside of coreboot I see using FDT as a way to improve the situation for everyone while making players come together more - that's an unusually rare win-win.
I strongly prefer FDT over CBOR. FDT is simple on byte level, which I find important to not lose the simplicity of coreboot tables. FDT can easily be parsed without RAM, CBOR not so much.
I'm a bit confused by the hate for CBOR here to be honest, I think it's a very simple format and not much different from FDT in terms of complexity (maybe the CBOR tag parsing is slightly more complicated, but in return FDT has weird quirks like the memory reserve map, separate strings table and phandles you have to deal with), and I don't think parsing one of them without RAM would be much more cumbersome than the other. But I don't want to replace the coreboot tables with either so it's probably not worth going off on that tangent.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other, I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
ending up with a solution that's less suited for either target
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
The way we use payloads nowadays is often very deeply tied to coreboot,
More so if "often" means number of shipped systems, less so if "often" means the number of individual payloads.
Well, if we want to open that line of argument we also have to consider what's left in the handoff for the payloads where this isn't true. If your payload isn't even accessing CBFS or printing logs to the CBMEM console (both of which are supported directly within libpayload), then what do you still need a handoff for at all? Just to parse the mainboard vendor string?
can hopefully stay pretty isolated and self-contained in the code base
I think there should be just one source of truth, from which other formats are translated/derived.
I mean I generally agree with that sentiment, but then we are at an impasse because I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project. I think keeping these things side-by-side in the code base with a Kconfig to switch between them is the only way we're going to let everyone achieve what they want here, even if it is going to add extra maintenance burden.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch. I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture, and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
Julius Werner wrote:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case? When have you ever had a situation where you'd want to edit coreboot table structures at runtime? (FWIW, even if you wanted to do that for some reason it would still not be very hard to write such a thing for coreboot tables -- the surrounding UI and stuff of this shim payload would probably take the lion's share of the effort, compared to the code to be able to delete/add/change table entries for the known structures.)
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different compilers. "Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
Unless we wanted to replace ACPI with FDT¹ ¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it). We already have support for FDT for kernel handoff in coreboot today, after all.
For the record, I just met with Lean from 9elements about this as well and basically discussed that same position -- that I don't want to block anyone from doing this but that I would like it to remain a compile-time option and would like coreboot tables to continue being available and supported as a first-party citizen going forward (and not "soft deprecated" in favor of FDT, or have the FDT stuff seep into something that can not be compiled out). As I understood, Lean (and Arthur?) want to continue pursuing this implementation under that premise, and then we'll have to see in practice how well we can make them coexist side-by-side without stepping on each other's toes.
Hi
I appreciate all the solid input on this one!
If coreboot had to be rewritten, I'd argue FDT would be a nicer way of structuring a payload handoff. However that is not the case, and coreboot tables are well established in many different payloads, supporting very different and specialized use cases. I also share the scepticism about the universal payload concept: the reality is that hardware init and payload are tightly coupled. The increased complexity for doing it otherwise may not be worth it, since it involves a lot of runtime decisions based on the payload. Is it really worth overhauling coreboot tables, which only pass limited information like memory map, board id, framebuffer,... ? Maybe... Is it worth maintaining 2 payload handoffs in the tree? My answer is pretty solid to that: *no*. So I won't be pursuing this FDT as a payload handoff as something that lives side by side with coreboot tables, as I've had my fill with multiple codepaths doing the same thing in the tree: romcc vs. C env bootblock, CAR teardown in ramstage vs. postcar, resource allocation v3 vs. v4, legacy LAPIC_CPU_INIT vs. parallel_mp, ... It's always a big mess to maintain multiple solutions. I'd hate to introduce more of that...
Currently EDK2 as a USF payload is built with a layer (shimlayer) that translates coreboot tables and then loads the DXE stage as a separate program. That shimlayer is just very poorly implemented: it tries to guess the flash memory mapping and then loads ELF at runtime. A better way would be to use libpayload to parse coreboot tables and pass whatever format EDK2 wants and have cbfstool process the DXE stage ELF (the universal one ;-) ) so that it's easy to find and load. As Libpayload is maintained with coreboot, using it to create a new specialized payload loader is the proper way to deal with payloads requiring a different format.
And completely unrelated to the payload handoff (so it indeed should not impact that decision/discussion) - FDT as an interface to FSP would be a big improvement for reusability. Currently FSP is totally not reusable and you need code for each SOC, which is exactly the opposite of what Intel USF claims to want to achieve. - FDT to replace ACPI... For static information that might work, but ACPI has a lot of real code in there that would require actual drivers in Linux to replace it with. It's a lot of work, so it would best be set up as a collaborative effort: let's pick a platform and go for it?
Kind regards Arthur
On Tue, Nov 8, 2022 at 1:14 AM Julius Werner jwerner@chromium.org wrote:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads...
The only generator code I know is GPLv2.
One could argue that only coreboot should ever generate this format, but I think one goal here is to enable or even encourage new situations.
Yes, I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense. I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea and not worth the cost".
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that.
This isn't just a technical change, for me there is also the more political goal to make more of what Intel does with USF going forward (please see
https://github.com/UniversalScalableFirmware/Introduction/blob/main/USF_Over... )
maximally useful for and within the coreboot context, ideally while encouraging reuse.
Since Intel has planned to use CBOR and are unlikely to use coreboot tables for anything outside of coreboot I see using FDT as a way to improve the situation for everyone while making players come together more - that's an unusually rare win-win.
I strongly prefer FDT over CBOR. FDT is simple on byte level, which I find important to not lose the simplicity of coreboot tables. FDT can easily be parsed without RAM, CBOR not so much.
I'm a bit confused by the hate for CBOR here to be honest, I think it's a very simple format and not much different from FDT in terms of complexity (maybe the CBOR tag parsing is slightly more complicated, but in return FDT has weird quirks like the memory reserve map, separate strings table and phandles you have to deal with), and I don't think parsing one of them without RAM would be much more cumbersome than the other. But I don't want to replace the coreboot tables with either so it's probably not worth going off on that tangent.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other, I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
ending up with a solution that's less suited for either target
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
The way we use payloads nowadays is often very deeply tied to coreboot,
More so if "often" means number of shipped systems, less so if "often" means the number of individual payloads.
Well, if we want to open that line of argument we also have to consider what's left in the handoff for the payloads where this isn't true. If your payload isn't even accessing CBFS or printing logs to the CBMEM console (both of which are supported directly within libpayload), then what do you still need a handoff for at all? Just to parse the mainboard vendor string?
can hopefully stay pretty isolated and self-contained in the code base
I think there should be just one source of truth, from which other formats are translated/derived.
I mean I generally agree with that sentiment, but then we are at an impasse because I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project. I think keeping these things side-by-side in the code base with a Kconfig to switch between them is the only way we're going to let everyone achieve what they want here, even if it is going to add extra maintenance burden.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch. I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture, and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
Julius Werner wrote:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a
struct
- easy to update with a text editor without needing to write code
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case? When have you ever had a situation where you'd want to edit coreboot table structures at runtime? (FWIW, even if you wanted to do that for some reason it would still not be very hard to write such a thing for coreboot tables -- the surrounding UI and stuff of this shim payload would probably take the lion's share of the effort, compared to the code to be able to delete/add/change table entries for the known structures.)
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different compilers. "Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
Unless we wanted to replace ACPI with FDT¹ ¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it). We already have support for FDT for kernel handoff in coreboot today, after all.
For the record, I just met with Lean from 9elements about this as well and basically discussed that same position -- that I don't want to block anyone from doing this but that I would like it to remain a compile-time option and would like coreboot tables to continue being available and supported as a first-party citizen going forward (and not "soft deprecated" in favor of FDT, or have the FDT stuff seep into something that can not be compiled out). As I understood, Lean (and Arthur?) want to continue pursuing this implementation under that premise, and then we'll have to see in practice how well we can make them coexist side-by-side without stepping on each other's toes. _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Hi, following is my perspective:
On FDT as a coreboot payload interface: I fully agree that coreboot table is well established, and we want to maintain the stability of the coreboot payload interface. That being said, FDT offers more than coreboot table does. Ron illustrated that very well, I would like to add that: a. FDT is what Linux kernel expects on most of the embedded devices. b. FDT is one of the two methods that ARM Linux server kernel expects. The other method is UEFI/ACPI. For coreboot to boot LinuxBoot on ARM server, coreboot need to provide either FDT or UEFI interface. So my proposal is to add FDT as a parallel path, but do not retire coreboot table until FDT is very robust. I agree that 2 ways of doing same thing is generally not desired; I suppose that applies to more to coreboot internal, but not necessarily for external facing interfaces.
On FDT as an interface with FSP: such a discussion will not be effective without silicon vendor in the loop. When coreboot community and silicon vendor have trust and rapport with each other, such a discussion will go smoothly.
On FDT vs. ACPI: At least for target OS kernel on server, ACPI will be expected.
Thanks, Jonathan
On Nov 8, 2022, at 1:19 AM, Arthur Heymans arthur@aheymans.xyz wrote:
This Message Is From an External Sender Hi
I appreciate all the solid input on this one!
If coreboot had to be rewritten, I'd argue FDT would be a nicer way of structuring a payload handoff. However that is not the case, and coreboot tables are well established in many different payloads, supporting very different and specialized use cases. I also share the scepticism about the universal payload concept: the reality is that hardware init and payload are tightly coupled. The increased complexity for doing it otherwise may not be worth it, since it involves a lot of runtime decisions based on the payload. Is it really worth overhauling coreboot tables, which only pass limited information like memory map, board id, framebuffer,... ? Maybe... Is it worth maintaining 2 payload handoffs in the tree? My answer is pretty solid to that: *no*. So I won't be pursuing this FDT as a payload handoff as something that lives side by side with coreboot tables, as I've had my fill with multiple codepaths doing the same thing in the tree: romcc vs. C env bootblock, CAR teardown in ramstage vs. postcar, resource allocation v3 vs. v4, legacy LAPIC_CPU_INIT vs. parallel_mp, ... It's always a big mess to maintain multiple solutions. I'd hate to introduce more of that...
Currently EDK2 as a USF payload is built with a layer (shimlayer) that translates coreboot tables and then loads the DXE stage as a separate program. That shimlayer is just very poorly implemented: it tries to guess the flash memory mapping and then loads ELF at runtime. A better way would be to use libpayload to parse coreboot tables and pass whatever format EDK2 wants and have cbfstool process the DXE stage ELF (the universal one ;-) ) so that it's easy to find and load. As Libpayload is maintained with coreboot, using it to create a new specialized payload loader is the proper way to deal with payloads requiring a different format.
And completely unrelated to the payload handoff (so it indeed should not impact that decision/discussion)
- FDT as an interface to FSP would be a big improvement for reusability. Currently FSP is totally not reusable and you need code for each SOC, which is exactly the opposite of what Intel USF claims to want to achieve.
- FDT to replace ACPI... For static information that might work, but ACPI has a lot of real code in there that would require actual drivers in Linux to replace it with. It's a lot of work, so it would best be set up as a collaborative effort: let's pick a platform and go for it?
Kind regards Arthur
On Tue, Nov 8, 2022 at 1:14 AM Julius Werner jwerner@chromium.org wrote:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads...
The only generator code I know is GPLv2.
One could argue that only coreboot should ever generate this format, but I think one goal here is to enable or even encourage new situations.
Yes, I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense. I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea and not worth the cost".
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that.
This isn't just a technical change, for me there is also the more political goal to make more of what Intel does with USF going forward (please see https://github.com/UniversalScalableFirmware/Introduction/blob/main/USF_Over... ) maximally useful for and within the coreboot context, ideally while encouraging reuse.
Since Intel has planned to use CBOR and are unlikely to use coreboot tables for anything outside of coreboot I see using FDT as a way to improve the situation for everyone while making players come together more - that's an unusually rare win-win.
I strongly prefer FDT over CBOR. FDT is simple on byte level, which I find important to not lose the simplicity of coreboot tables. FDT can easily be parsed without RAM, CBOR not so much.
I'm a bit confused by the hate for CBOR here to be honest, I think it's a very simple format and not much different from FDT in terms of complexity (maybe the CBOR tag parsing is slightly more complicated, but in return FDT has weird quirks like the memory reserve map, separate strings table and phandles you have to deal with), and I don't think parsing one of them without RAM would be much more cumbersome than the other. But I don't want to replace the coreboot tables with either so it's probably not worth going off on that tangent.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other, I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
ending up with a solution that's less suited for either target
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
The way we use payloads nowadays is often very deeply tied to coreboot,
More so if "often" means number of shipped systems, less so if "often" means the number of individual payloads.
Well, if we want to open that line of argument we also have to consider what's left in the handoff for the payloads where this isn't true. If your payload isn't even accessing CBFS or printing logs to the CBMEM console (both of which are supported directly within libpayload), then what do you still need a handoff for at all? Just to parse the mainboard vendor string?
can hopefully stay pretty isolated and self-contained in the code base
I think there should be just one source of truth, from which other formats are translated/derived.
I mean I generally agree with that sentiment, but then we are at an impasse because I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project. I think keeping these things side-by-side in the code base with a Kconfig to switch between them is the only way we're going to let everyone achieve what they want here, even if it is going to add extra maintenance burden.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch. I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture, and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
Julius Werner wrote:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case? When have you ever had a situation where you'd want to edit coreboot table structures at runtime? (FWIW, even if you wanted to do that for some reason it would still not be very hard to write such a thing for coreboot tables -- the surrounding UI and stuff of this shim payload would probably take the lion's share of the effort, compared to the code to be able to delete/add/change table entries for the known structures.)
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different compilers. "Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
Unless we wanted to replace ACPI with FDT¹ ¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it). We already have support for FDT for kernel handoff in coreboot today, after all.
For the record, I just met with Lean from 9elements about this as well and basically discussed that same position -- that I don't want to block anyone from doing this but that I would like it to remain a compile-time option and would like coreboot tables to continue being available and supported as a first-party citizen going forward (and not "soft deprecated" in favor of FDT, or have the FDT stuff seep into something that can not be compiled out). As I understood, Lean (and Arthur?) want to continue pursuing this implementation under that premise, and then we'll have to see in practice how well we can make them coexist side-by-side without stepping on each other's toes. _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Hi all, Thanks again everyone for chipping in on this, I think that's what makes the coreboot community healthy - that we have honest discussions while allowing new ideas to grow. I share the same sentiment as Ron, Nico and Peter - the goodness of implementing FDT outweighs the cons, hence I will continue to pursue this path.
As mentioned by Julius, we had a candid discussion yesterday on this, while he still reserves his opinions on FDT, he will not block the implementation. For now we will guard it with kconfig and give it a year of trial to see how it turns out. IMHO coreboot continues to grow because it allows people to take risks and innovate new things (that's why we are much cooler), and the born of new things like resource allocator V2-V4 and parallel MP init, yes it introduced maintenance burden, but the maintenance problems did not discredit the introduction of those ideas, because the in the end the benefits of new innovations make coreboot project better, while others complained why other firmware projects always 20 years lagging behind of the technology of OS/ kernels.
There is no way to please everyone, and there are always risks with new innovation, and I believe coreboot is in a much more mature shape than 20 years ago to allow new ideas to flourish in the sense that it would benefit the overall open source firmware communities.
Best Regards, *Lean Sheng Tan*
On Tue, 8 Nov 2022 at 03:19, Arthur Heymans arthur@aheymans.xyz wrote:
Hi
I appreciate all the solid input on this one!
If coreboot had to be rewritten, I'd argue FDT would be a nicer way of structuring a payload handoff. However that is not the case, and coreboot tables are well established in many different payloads, supporting very different and specialized use cases. I also share the scepticism about the universal payload concept: the reality is that hardware init and payload are tightly coupled. The increased complexity for doing it otherwise may not be worth it, since it involves a lot of runtime decisions based on the payload. Is it really worth overhauling coreboot tables, which only pass limited information like memory map, board id, framebuffer,... ? Maybe... Is it worth maintaining 2 payload handoffs in the tree? My answer is pretty solid to that: *no*. So I won't be pursuing this FDT as a payload handoff as something that lives side by side with coreboot tables, as I've had my fill with multiple codepaths doing the same thing in the tree: romcc vs. C env bootblock, CAR teardown in ramstage vs. postcar, resource allocation v3 vs. v4, legacy LAPIC_CPU_INIT vs. parallel_mp, ... It's always a big mess to maintain multiple solutions. I'd hate to introduce more of that...
Currently EDK2 as a USF payload is built with a layer (shimlayer) that translates coreboot tables and then loads the DXE stage as a separate program. That shimlayer is just very poorly implemented: it tries to guess the flash memory mapping and then loads ELF at runtime. A better way would be to use libpayload to parse coreboot tables and pass whatever format EDK2 wants and have cbfstool process the DXE stage ELF (the universal one ;-) ) so that it's easy to find and load. As Libpayload is maintained with coreboot, using it to create a new specialized payload loader is the proper way to deal with payloads requiring a different format.
And completely unrelated to the payload handoff (so it indeed should not impact that decision/discussion)
- FDT as an interface to FSP would be a big improvement for reusability.
Currently FSP is totally not reusable and you need code for each SOC, which is exactly the opposite of what Intel USF claims to want to achieve.
- FDT to replace ACPI... For static information that might work, but ACPI
has a lot of real code in there that would require actual drivers in Linux to replace it with. It's a lot of work, so it would best be set up as a collaborative effort: let's pick a platform and go for it?
Kind regards Arthur
On Tue, Nov 8, 2022 at 1:14 AM Julius Werner jwerner@chromium.org wrote:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads...
The only generator code I know is GPLv2.
One could argue that only coreboot should ever generate this format, but I think one goal here is to enable or even encourage new situations.
Yes, I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense. I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea and not worth the cost".
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that.
This isn't just a technical change, for me there is also the more political goal to make more of what Intel does with USF going forward (please see
https://github.com/UniversalScalableFirmware/Introduction/blob/main/USF_Over... )
maximally useful for and within the coreboot context, ideally while encouraging reuse.
Since Intel has planned to use CBOR and are unlikely to use coreboot tables for anything outside of coreboot I see using FDT as a way to improve the situation for everyone while making players come together more - that's an unusually rare win-win.
I strongly prefer FDT over CBOR. FDT is simple on byte level, which I find important to not lose the simplicity of coreboot tables. FDT can easily be parsed without RAM, CBOR not so much.
I'm a bit confused by the hate for CBOR here to be honest, I think it's a very simple format and not much different from FDT in terms of complexity (maybe the CBOR tag parsing is slightly more complicated, but in return FDT has weird quirks like the memory reserve map, separate strings table and phandles you have to deal with), and I don't think parsing one of them without RAM would be much more cumbersome than the other. But I don't want to replace the coreboot tables with either so it's probably not worth going off on that tangent.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other, I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
ending up with a solution that's less suited for either target
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
The way we use payloads nowadays is often very deeply tied to coreboot,
More so if "often" means number of shipped systems, less so if "often" means the number of individual payloads.
Well, if we want to open that line of argument we also have to consider what's left in the handoff for the payloads where this isn't true. If your payload isn't even accessing CBFS or printing logs to the CBMEM console (both of which are supported directly within libpayload), then what do you still need a handoff for at all? Just to parse the mainboard vendor string?
can hopefully stay pretty isolated and self-contained in the code base
I think there should be just one source of truth, from which other formats are translated/derived.
I mean I generally agree with that sentiment, but then we are at an impasse because I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project. I think keeping these things side-by-side in the code base with a Kconfig to switch between them is the only way we're going to let everyone achieve what they want here, even if it is going to add extra maintenance burden.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch. I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture, and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
Julius Werner wrote:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a
struct
- easy to update with a text editor without needing to write code
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case? When have you ever had a situation where you'd want to edit coreboot table structures at runtime? (FWIW, even if you wanted to do that for some reason it would still not be very hard to write such a thing for coreboot tables -- the surrounding UI and stuff of this shim payload would probably take the lion's share of the effort, compared to the code to be able to delete/add/change table entries for the known structures.)
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different
compilers.
"Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
Unless we wanted to replace ACPI with FDT¹ ¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it). We already have support for FDT for kernel handoff in coreboot today, after all.
For the record, I just met with Lean from 9elements about this as well and basically discussed that same position -- that I don't want to block anyone from doing this but that I would like it to remain a compile-time option and would like coreboot tables to continue being available and supported as a first-party citizen going forward (and not "soft deprecated" in favor of FDT, or have the FDT stuff seep into something that can not be compiled out). As I understood, Lean (and Arthur?) want to continue pursuing this implementation under that premise, and then we'll have to see in practice how well we can make them coexist side-by-side without stepping on each other's toes. _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Hi,
I wasn't sure if I should also shim in here, but I just wanted to leave my opinion here. I agree with adding FDT as another method to handoff data to a payload. Despite all the advantages that Ron and others pointed out, I think something that passed data into another project e.g. payload here - whatever that is, should never be specific to one project. It should not be the status-quo that other projects, if they want to be loaded as a payload after coreboot need to implement other project specific implementations. What I mean by this is that hands-off's between different projects should always rely on an open standard that is *not* specific to the project.
In this particular case, my opinion is that other payloads implement coreboot tables should not be the defacto standard. coreboot should be encouraged to have a flexible option i.e. FDT to pass data into a payload - so that the payload would not care if that data is coming form coreboot - or any other underlying solution (oreboot?).
Of course I do understand the need for being backwards compatible, so I also would propose the approach to have these two solutions living site-by-site next to each other for some time - and then we as a community can decide how we want to move forward. However, we should never block those kind of approaches to make the open-source firmware would a bit more unified - and we should definitely not do it "just because Intel wants us to do it".
Best,
Chris
On 11/8/22 18:25, Jonathan Zhang (Infra) via coreboot wrote:
Hi, following is my perspective:
On FDT as a coreboot payload interface: I fully agree that coreboot table is well established, and we want to maintain the stability of the coreboot payload interface. That being said, FDT offers more than coreboot table does. Ron illustrated that very well, I would like to add that: a. FDT is what Linux kernel expects on most of the embedded devices. b. FDT is one of the two methods that ARM Linux server kernel expects. The other method is UEFI/ACPI. For coreboot to boot LinuxBoot on ARM server, coreboot need to provide either FDT or UEFI interface. So my proposal is to add FDT as a parallel path, but do not retire coreboot table until FDT is very robust. I agree that 2 ways of doing same thing is generally not desired; I suppose that applies to more to coreboot internal, but not necessarily for external facing interfaces.
On FDT as an interface with FSP: such a discussion will not be effective without silicon vendor in the loop. When coreboot community and silicon vendor have trust and rapport with each other, such a discussion will go smoothly.
On FDT vs. ACPI: At least for target OS kernel on server, ACPI will be expected.
Thanks, Jonathan
On Nov 8, 2022, at 1:19 AM, Arthur Heymansarthur@aheymans.xyz wrote:
This Message Is From an External Sender Hi
I appreciate all the solid input on this one!
If coreboot had to be rewritten, I'd argue FDT would be a nicer way of structuring a payload handoff. However that is not the case, and coreboot tables are well established in many different payloads, supporting very different and specialized use cases. I also share the scepticism about the universal payload concept: the reality is that hardware init and payload are tightly coupled. The increased complexity for doing it otherwise may not be worth it, since it involves a lot of runtime decisions based on the payload. Is it really worth overhauling coreboot tables, which only pass limited information like memory map, board id, framebuffer,... ? Maybe... Is it worth maintaining 2 payload handoffs in the tree? My answer is pretty solid to that: *no*. So I won't be pursuing this FDT as a payload handoff as something that lives side by side with coreboot tables, as I've had my fill with multiple codepaths doing the same thing in the tree: romcc vs. C env bootblock, CAR teardown in ramstage vs. postcar, resource allocation v3 vs. v4, legacy LAPIC_CPU_INIT vs. parallel_mp, ... It's always a big mess to maintain multiple solutions. I'd hate to introduce more of that...
Currently EDK2 as a USF payload is built with a layer (shimlayer) that translates coreboot tables and then loads the DXE stage as a separate program. That shimlayer is just very poorly implemented: it tries to guess the flash memory mapping and then loads ELF at runtime. A better way would be to use libpayload to parse coreboot tables and pass whatever format EDK2 wants and have cbfstool process the DXE stage ELF (the universal one ;-) ) so that it's easy to find and load. As Libpayload is maintained with coreboot, using it to create a new specialized payload loader is the proper way to deal with payloads requiring a different format.
And completely unrelated to the payload handoff (so it indeed should not impact that decision/discussion)
- FDT as an interface to FSP would be a big improvement for reusability. Currently FSP is totally not reusable and you need code for each SOC, which is exactly the opposite of what Intel USF claims to want to achieve.
- FDT to replace ACPI... For static information that might work, but ACPI has a lot of real code in there that would require actual drivers in Linux to replace it with. It's a lot of work, so it would best be set up as a collaborative effort: let's pick a platform and go for it?
Kind regards Arthur
On Tue, Nov 8, 2022 at 1:14 AM Julius Wernerjwerner@chromium.org wrote:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads...
The only generator code I know is GPLv2.
One could argue that only coreboot should ever generate this format, but I think one goal here is to enable or even encourage new situations.
Yes, I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense. I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea and not worth the cost".
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that.
This isn't just a technical change, for me there is also the more political goal to make more of what Intel does with USF going forward (please seehttps://github.com/UniversalScalableFirmware/Introduction/blob/main/USF_Over... ) maximally useful for and within the coreboot context, ideally while encouraging reuse.
Since Intel has planned to use CBOR and are unlikely to use coreboot tables for anything outside of coreboot I see using FDT as a way to improve the situation for everyone while making players come together more - that's an unusually rare win-win.
I strongly prefer FDT over CBOR. FDT is simple on byte level, which I find important to not lose the simplicity of coreboot tables. FDT can easily be parsed without RAM, CBOR not so much.
I'm a bit confused by the hate for CBOR here to be honest, I think it's a very simple format and not much different from FDT in terms of complexity (maybe the CBOR tag parsing is slightly more complicated, but in return FDT has weird quirks like the memory reserve map, separate strings table and phandles you have to deal with), and I don't think parsing one of them without RAM would be much more cumbersome than the other. But I don't want to replace the coreboot tables with either so it's probably not worth going off on that tangent.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other, I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
ending up with a solution that's less suited for either target
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
The way we use payloads nowadays is often very deeply tied to coreboot,
More so if "often" means number of shipped systems, less so if "often" means the number of individual payloads.
Well, if we want to open that line of argument we also have to consider what's left in the handoff for the payloads where this isn't true. If your payload isn't even accessing CBFS or printing logs to the CBMEM console (both of which are supported directly within libpayload), then what do you still need a handoff for at all? Just to parse the mainboard vendor string?
can hopefully stay pretty isolated and self-contained in the code base
I think there should be just one source of truth, from which other formats are translated/derived.
I mean I generally agree with that sentiment, but then we are at an impasse because I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project. I think keeping these things side-by-side in the code base with a Kconfig to switch between them is the only way we're going to let everyone achieve what they want here, even if it is going to add extra maintenance burden.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch. I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture, and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
Julius Werner wrote:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case? When have you ever had a situation where you'd want to edit coreboot table structures at runtime? (FWIW, even if you wanted to do that for some reason it would still not be very hard to write such a thing for coreboot tables -- the surrounding UI and stuff of this shim payload would probably take the lion's share of the effort, compared to the code to be able to delete/add/change table entries for the known structures.)
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different compilers. "Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
Unless we wanted to replace ACPI with FDT¹ ¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it). We already have support for FDT for kernel handoff in coreboot today, after all.
For the record, I just met with Lean from 9elements about this as well and basically discussed that same position -- that I don't want to block anyone from doing this but that I would like it to remain a compile-time option and would like coreboot tables to continue being available and supported as a first-party citizen going forward (and not "soft deprecated" in favor of FDT, or have the FDT stuff seep into something that can not be compiled out). As I understood, Lean (and Arthur?) want to continue pursuing this implementation under that premise, and then we'll have to see in practice how well we can make them coexist side-by-side without stepping on each other's toes. _______________________________________________ coreboot mailing list --coreboot@coreboot.org To unsubscribe send an email tocoreboot-leave@coreboot.org _______________________________________________ coreboot mailing list --coreboot@coreboot.org To unsubscribe send an email tocoreboot-leave@coreboot.org
coreboot mailing list --coreboot@coreboot.org To unsubscribe send an email tocoreboot-leave@coreboot.org
I agree with the "side by side" idea as well. Thanks to this community, coreboot has always been good at growing with new ideas. This was a really good discussion.
Thanks
Ron
On Tue, Nov 8, 2022 at 11:35 AM Christian Walter < christian.walter@9elements.com> wrote:
Hi,
I wasn't sure if I should also shim in here, but I just wanted to leave my opinion here. I agree with adding FDT as another method to handoff data to a payload. Despite all the advantages that Ron and others pointed out, I think something that passed data into another project e.g. payload here - whatever that is, should never be specific to one project. It should not be the status-quo that other projects, if they want to be loaded as a payload after coreboot need to implement other project specific implementations. What I mean by this is that hands-off's between different projects should always rely on an open standard that is *not* specific to the project.
In this particular case, my opinion is that other payloads implement coreboot tables should not be the defacto standard. coreboot should be encouraged to have a flexible option i.e. FDT to pass data into a payload - so that the payload would not care if that data is coming form coreboot - or any other underlying solution (oreboot?).
Of course I do understand the need for being backwards compatible, so I also would propose the approach to have these two solutions living site-by-site next to each other for some time - and then we as a community can decide how we want to move forward. However, we should never block those kind of approaches to make the open-source firmware would a bit more unified - and we should definitely not do it "just because Intel wants us to do it".
Best,
Chris On 11/8/22 18:25, Jonathan Zhang (Infra) via coreboot wrote:
Hi, following is my perspective:
On FDT as a coreboot payload interface: I fully agree that coreboot table is well established, and we want to maintain the stability of the coreboot payload interface. That being said, FDT offers more than coreboot table does. Ron illustrated that very well, I would like to add that: a. FDT is what Linux kernel expects on most of the embedded devices. b. FDT is one of the two methods that ARM Linux server kernel expects. The other method is UEFI/ACPI. For coreboot to boot LinuxBoot on ARM server, coreboot need to provide either FDT or UEFI interface. So my proposal is to add FDT as a parallel path, but do not retire coreboot table until FDT is very robust. I agree that 2 ways of doing same thing is generally not desired; I suppose that applies to more to coreboot internal, but not necessarily for external facing interfaces.
On FDT as an interface with FSP: such a discussion will not be effective without silicon vendor in the loop. When coreboot community and silicon vendor have trust and rapport with each other, such a discussion will go smoothly.
On FDT vs. ACPI: At least for target OS kernel on server, ACPI will be expected.
Thanks, Jonathan
On Nov 8, 2022, at 1:19 AM, Arthur Heymans arthur@aheymans.xyz arthur@aheymans.xyz wrote:
This Message Is From an External Sender Hi
I appreciate all the solid input on this one!
If coreboot had to be rewritten, I'd argue FDT would be a nicer way of structuring a payload handoff. However that is not the case, and coreboot tables are well established in many different payloads, supporting very different and specialized use cases. I also share the scepticism about the universal payload concept: the reality is that hardware init and payload are tightly coupled. The increased complexity for doing it otherwise may not be worth it, since it involves a lot of runtime decisions based on the payload. Is it really worth overhauling coreboot tables, which only pass limited information like memory map, board id, framebuffer,... ? Maybe... Is it worth maintaining 2 payload handoffs in the tree? My answer is pretty solid to that: *no*. So I won't be pursuing this FDT as a payload handoff as something that lives side by side with coreboot tables, as I've had my fill with multiple codepaths doing the same thing in the tree: romcc vs. C env bootblock, CAR teardown in ramstage vs. postcar, resource allocation v3 vs. v4, legacy LAPIC_CPU_INIT vs. parallel_mp, ... It's always a big mess to maintain multiple solutions. I'd hate to introduce more of that...
Currently EDK2 as a USF payload is built with a layer (shimlayer) that translates coreboot tables and then loads the DXE stage as a separate program. That shimlayer is just very poorly implemented: it tries to guess the flash memory mapping and then loads ELF at runtime. A better way would be to use libpayload to parse coreboot tables and pass whatever format EDK2 wants and have cbfstool process the DXE stage ELF (the universal one ;-) ) so that it's easy to find and load. As Libpayload is maintained with coreboot, using it to create a new specialized payload loader is the proper way to deal with payloads requiring a different format.
And completely unrelated to the payload handoff (so it indeed should not impact that decision/discussion)
- FDT as an interface to FSP would be a big improvement for reusability. Currently FSP is totally not reusable and you need code for each SOC, which is exactly the opposite of what Intel USF claims to want to achieve.
- FDT to replace ACPI... For static information that might work, but ACPI has a lot of real code in there that would require actual drivers in Linux to replace it with. It's a lot of work, so it would best be set up as a collaborative effort: let's pick a platform and go for it?
Kind regards Arthur
On Tue, Nov 8, 2022 at 1:14 AM Julius Werner jwerner@chromium.org jwerner@chromium.org wrote:
- You said that coreboot tables were GPLv2 licensed, and this could
be an adoption problem for payloads...
The only generator code I know is GPLv2.
One could argue that only coreboot should ever generate this format, but I think one goal here is to enable or even encourage new situations.
Yes, I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense. I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea and not worth the cost".
I think the fundamental question here is still: what do we want to achieve, and how does replacing coreboot tables with something else help with that.
This isn't just a technical change, for me there is also the more political goal to make more of what Intel does with USF going forward (please see https://github.com/UniversalScalableFirmware/Introduction/blob/main/USF_Over... ) maximally useful for and within the coreboot context, ideally while encouraging reuse.
Since Intel has planned to use CBOR and are unlikely to use coreboot tables for anything outside of coreboot I see using FDT as a way to improve the situation for everyone while making players come together more - that's an unusually rare win-win.
I strongly prefer FDT over CBOR. FDT is simple on byte level, which I find important to not lose the simplicity of coreboot tables. FDT can easily be parsed without RAM, CBOR not so much.
I'm a bit confused by the hate for CBOR here to be honest, I think it's a very simple format and not much different from FDT in terms of complexity (maybe the CBOR tag parsing is slightly more complicated, but in return FDT has weird quirks like the memory reserve map, separate strings table and phandles you have to deal with), and I don't think parsing one of them without RAM would be much more cumbersome than the other. But I don't want to replace the coreboot tables with either so it's probably not worth going off on that tangent.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other, I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
ending up with a solution that's less suited for either target
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
The way we use payloads nowadays is often very deeply tied to coreboot,
More so if "often" means number of shipped systems, less so if "often" means the number of individual payloads.
Well, if we want to open that line of argument we also have to consider what's left in the handoff for the payloads where this isn't true. If your payload isn't even accessing CBFS or printing logs to the CBMEM console (both of which are supported directly within libpayload), then what do you still need a handoff for at all? Just to parse the mainboard vendor string?
can hopefully stay pretty isolated and self-contained in the code base
I think there should be just one source of truth, from which other formats are translated/derived.
I mean I generally agree with that sentiment, but then we are at an impasse because I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project. I think keeping these things side-by-side in the code base with a Kconfig to switch between them is the only way we're going to let everyone achieve what they want here, even if it is going to add extra maintenance burden.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch. I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture, and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
Julius Werner wrote:
- reflection is easy
- no problem of 'header mismatch' to have the right definition of a struct
- easy to update with a text editor without needing to write code
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case? When have you ever had a situation where you'd want to edit coreboot table structures at runtime? (FWIW, even if you wanted to do that for some reason it would still not be very hard to write such a thing for coreboot tables -- the surrounding UI and stuff of this shim payload would probably take the lion's share of the effort, compared to the code to be able to delete/add/change table entries for the known structures.)
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different compilers. "Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
Unless we wanted to replace ACPI with FDT¹ ¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it). We already have support for FDT for kernel handoff in coreboot today, after all.
For the record, I just met with Lean from 9elements about this as well and basically discussed that same position -- that I don't want to block anyone from doing this but that I would like it to remain a compile-time option and would like coreboot tables to continue being available and supported as a first-party citizen going forward (and not "soft deprecated" in favor of FDT, or have the FDT stuff seep into something that can not be compiled out). As I understood, Lean (and Arthur?) want to continue pursuing this implementation under that premise, and then we'll have to see in practice how well we can make them coexist side-by-side without stepping on each other's toes. _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-- *Christian Walter* *Head of Firmware Development / Cyber Security *
9elements GmbH, Kortumstraße 19-21, 44787 Bochum, Germany Email: christian.walter@9elements.com Phone: *+49 234 68 94 188 <+492346894188>* Mobile: *+49 176 70845047 <+4917670845047>*
Sitz der Gesellschaft: Bochum Handelsregister: Amtsgericht Bochum, HRB 17519 Geschäftsführung: Sebastian Deutsch, Eray Basar
Datenschutzhinweise nach Art. 13 DSGVO https://9elements.com/privacy
As mentioned by Julius, we had a candid discussion yesterday on this, while he still reserves his opinions on FDT, he will not block the implementation. For now we will guard it with kconfig and give it a year of trial to see how it turns out.
So just to be clear, to make sure we weren't taking different things away from that meeting: I said I'm okay with adding the FDT option as a separate, isolated compile-time option for those that want to try it out. But I care that the coreboot tables continue to be supported on equal footing in the long term, and I don't want there to be some pre-formed understanding that they'll be "deprecated eventually" or that payloads will be slowly pressured to move to the FDT handoff, particularly not with any specific timeline attached.
makes sense to me.
On Tue, Nov 8, 2022 at 2:55 PM Julius Werner jwerner@chromium.org wrote:
As mentioned by Julius, we had a candid discussion yesterday on this,
while he still reserves his opinions on FDT, he will not block the implementation. For now we will guard it with kconfig and give it a year of trial to see how it turns out.
So just to be clear, to make sure we weren't taking different things away from that meeting: I said I'm okay with adding the FDT option as a separate, isolated compile-time option for those that want to try it out. But I care that the coreboot tables continue to be supported on equal footing in the long term, and I don't want there to be some pre-formed understanding that they'll be "deprecated eventually" or that payloads will be slowly pressured to move to the FDT handoff, particularly not with any specific timeline attached.
So just to be clear, to make sure we weren't taking different things away from that meeting: I said I'm okay with adding the FDT option as a separate, isolated compile-time option for those that want to try it out. But I care that the coreboot tables continue to be supported on equal footing in the long term, and I don't want there to be some pre-formed understanding that they'll be "deprecated eventually" or that payloads will be slowly pressured to move to the FDT handoff, particularly not with any specific timeline attached.
With this in mind I don't see any reasons why generating the FDT handoff
needs to happen in the coreboot ramstage. A libpayload based payload could just as well generate that FDT handoff and chainload the next payload. All the infrastructure is there in libpayload to cleanly do this. The coreboot code for dealing with FDT is exactly the same as deptcharge code (it comes from that project). This is a better technical solution than to introduce new Kconfig guarded code and maintenance overhead inside the coreboot tree to maintain 2 solutions for the same problem. As a matter of fact that solution is even more 'universal' as you could still swap out payloads without recompiling the coreboot part of things.
Arthur
On Tue, Nov 8, 2022 at 11:55 PM Julius Werner jwerner@chromium.org wrote:
As mentioned by Julius, we had a candid discussion yesterday on this,
while he still reserves his opinions on FDT, he will not block the implementation. For now we will guard it with kconfig and give it a year of trial to see how it turns out.
So just to be clear, to make sure we weren't taking different things away from that meeting: I said I'm okay with adding the FDT option as a separate, isolated compile-time option for those that want to try it out. But I care that the coreboot tables continue to be supported on equal footing in the long term, and I don't want there to be some pre-formed understanding that they'll be "deprecated eventually" or that payloads will be slowly pressured to move to the FDT handoff, particularly not with any specific timeline attached.
Julius Werner wrote:
I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense.
I think it makes some sense because coreboot does what we consider best. Of course we want others to reuse the very best parts of that.
But I also recognize that it is unlikely to happen, so to improve the situation overall (ie. also outside coreboot) I want to consider FDT in coreboot.
I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea
MBR called to disagree.
and not worth the cost".
This is the only interesting discussion.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other,
I find that silly polemic. I see this not as instrumentalizing coreboot but as coreboot accomodating Intel a little while also getting benefit - of course with the longer term goal of Intel and coreboot becoming a better match.
You don't have to care about USF or Intel but do consider that there probably would be no coreboot community within industry were coreboot not to add value to Intel hardware.
Our community is obviously not in danger over this but the better Intel and coreboot go together the more opportunities for really good solutions we may have in the future.
If coreboot doesn't push Intel then Intel will push coreboot (not neccessarily on purpose, just through inertia). You may want to ignore that for now but the coreboot will be dealing with the outcome for some time and now is when we can act.
I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
I understand if you and others want to ignore politics, I did too for a long time.
That said, I wouldn't worry about negative effects. I certainly have no interest to remove coreboot tables nor do I see any reason to do so nor have I seen anyone mention anything in that direction.
Do not make up strawmen.
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
..
I'm a bit confused by the hate for CBOR here to be honest,
..
I don't think parsing one of them without RAM would be much more cumbersome than the other.
..
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
A RAM-less FDT parser probably doesn't need to load the entire tree but merely find one value at a time, perhaps without state between values. The complexity of that in an FDT is low since FDT is flat and has simple primitives. CBOR, well..
--8<-- RFC8949 The initial byte of each encoded data item contains both information about the major type (the high-order 3 bits, described in Section 3.1) and additional information (the low-order 5 bits). With a few exceptions, the additional information's value describes how to load an unsigned integer "argument":
Less than 24: The argument's value is the value of the additional information.
24, 25, 26, or 27: The argument's value is held in the following 1, 2, 4, or 8 bytes, respectively, in network byte order. For major type 7 and additional information value 25, 26, 27, these bytes are not used as an integer argument, but as a floating-point value (see Section 3.3).
28, 29, 30: These values are reserved for future additions to the CBOR format. In the present version of CBOR, the encoded item is not well-formed.
31: No argument value is derived. If the major type is 0, 1, or 6, the encoded item is not well-formed. For major types 2 to 5, the item's length is indefinite, and for major type 7, the byte does not constitute a data item at all but terminates an indefinite- length item; all are described in Section 3.2. -->8--
This is bizarrely complex for storing a simple scalar.
what do you still need a handoff for at all? Just to parse the mainboard vendor string?
Memory map at a minimum.
I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project.
So a way forward where you can ignore FDT is for coreboot tables to stay as-is, and a Kconfig option to make coreboot use its coreboot tables to generate an FDT.
I think keeping these things side-by-side in the code base with a Kconfig to switch between them
Like Arthur I don't appreciate the "side-by-side" narrative, nor an expectation for two independent data sets.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch.
This is supposed to be headed to additional benefit for both coreboot and USF (users). Others will use coreboot in different ways, e.g. without FDT.
I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture,
Noone suggested anything like that. Don't waste our time.
and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
If you mean "it's not a solution for coreboot to just generate both" then I ask why not? Two things can be true at the same time: coreboot tables are simple to generate and the code to generate FDT from coreboot tables is also simple.
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case?
There are several, both outside and within the boot process.
FDT can be meaningfully persisted and so be useful both before and after boot.
When have you ever had a situation where you'd want to edit coreboot table structures at runtime?
You can also consider FDT in coreboot as plumbing to enable new situations that were best avoided with coreboot tables, then of course noone ever wanted to edit them. That doesn't mean that noone would ever want to edit firmware handoff information, especially if we also consider FSP.
Please be a bit more imaginative here.
it would still not be very hard to write such a thing for coreboot tables
Of course. One goal is to have more freedom than that allows, or at a very minimum fill a namespace not (exclusively) defined by coreboot or Intel.
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different compilers. "Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
I don't think it's wrong. I used to, but not anymore.
This is not about 64-bit values which you mentioned in your first email, this is about struct member offsets in the struct.
The structs are not packed so the compiler decides.
This is why both Nico and I are so strong proponents of explicit serialization of the tables. (I'd also like to create new tags for coreboot tables with packed values but that's another matter.)
¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it).
If all handoff interfaces were to use a single format (this would be great) then obviously it can be a single data set, which would make life simpler for a lot of different people.
(and not "soft deprecated" in favor of FDT,
Since this your narrative is baseless it makes it look like you are intentionally acting in bad faith. That's not a good look. Be careful.
or have the FDT stuff seep into something that can not be compiled out).
Like vboot has been for maybe a decade now yes, I agree strongly that this is important to avoid.
without stepping on each other's toes.
I wouldn't worry.
//Peter
I mostly agree with Peter here: "So a way forward where you can ignore FDT is for coreboot tables to stay as-is, and a Kconfig option to make coreboot use its coreboot tables to generate an FDT."
I suspect that, someday, the FDT may contain different information than is contained in coreboot tables. We can not anticipate everything: as needs change, we could find it is not possible to create the necessary FDT from coreboot tables.
For that reason, I think FDT creation should be in coreboot, not in a stage, because then the creation of the FDT will be easiest. And enabling FDT with a kconfig option makes sense. coreboot tables are small, so, thinking about it, it is hard to see a case where we'd ever not have coreboot tables as well.
On Thu, Nov 10, 2022 at 8:23 AM Peter Stuge peter@stuge.se wrote:
Julius Werner wrote:
I think only coreboot should ever generate this format. I'm certainly not trying to make a case for "we should get other firmware projects to adopt coreboot tables" here, I don't think that makes any sense.
I think it makes some sense because coreboot does what we consider best. Of course we want others to reuse the very best parts of that.
But I also recognize that it is unlikely to happen, so to improve the situation overall (ie. also outside coreboot) I want to consider FDT in coreboot.
I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea
MBR called to disagree.
and not worth the cost".
This is the only interesting discussion.
I understand that some people here want to instrumentalize coreboot to try to force Intel to change course on its USF specification, and honestly I don't really care about that one way or the other,
I find that silly polemic. I see this not as instrumentalizing coreboot but as coreboot accomodating Intel a little while also getting benefit - of course with the longer term goal of Intel and coreboot becoming a better match.
You don't have to care about USF or Intel but do consider that there probably would be no coreboot community within industry were coreboot not to add value to Intel hardware.
Our community is obviously not in danger over this but the better Intel and coreboot go together the more opportunities for really good solutions we may have in the future.
If coreboot doesn't push Intel then Intel will push coreboot (not neccessarily on purpose, just through inertia). You may want to ignore that for now but the coreboot will be dealing with the outcome for some time and now is when we can act.
I'm just asking to please not let the fallout of that negatively affect those of us who still just care about using coreboot as a firmware and not as a political cudgel.
I understand if you and others want to ignore politics, I did too for a long time.
That said, I wouldn't worry about negative effects. I certainly have no interest to remove coreboot tables nor do I see any reason to do so nor have I seen anyone mention anything in that direction.
Do not make up strawmen.
I don't think FDT is less suited than coreboot tables where it matters the most - simplicity on byte level.
..
I'm a bit confused by the hate for CBOR here to be honest,
..
I don't think parsing one of them without RAM would be much more cumbersome than the other.
..
FDT is a tokenized hierarchical key-value store with string keys (and a ton of extra quirks), whereas the coreboot tables are just a tagged list of C structures. That's an order of magnitude of difference in simplicity (and binary size).
A RAM-less FDT parser probably doesn't need to load the entire tree but merely find one value at a time, perhaps without state between values. The complexity of that in an FDT is low since FDT is flat and has simple primitives. CBOR, well..
--8<-- RFC8949 The initial byte of each encoded data item contains both information about the major type (the high-order 3 bits, described in Section 3.1) and additional information (the low-order 5 bits). With a few exceptions, the additional information's value describes how to load an unsigned integer "argument":
Less than 24: The argument's value is the value of the additional information.
24, 25, 26, or 27: The argument's value is held in the following 1, 2, 4, or 8 bytes, respectively, in network byte order. For major type 7 and additional information value 25, 26, 27, these bytes are not used as an integer argument, but as a floating-point value (see Section 3.3).
28, 29, 30: These values are reserved for future additions to the CBOR format. In the present version of CBOR, the encoded item is not well-formed.
31: No argument value is derived. If the major type is 0, 1, or 6, the encoded item is not well-formed. For major types 2 to 5, the item's length is indefinite, and for major type 7, the byte does not constitute a data item at all but terminates an indefinite- length item; all are described in Section 3.2. -->8--
This is bizarrely complex for storing a simple scalar.
what do you still need a handoff for at all? Just to parse the mainboard vendor string?
Memory map at a minimum.
I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project.
So a way forward where you can ignore FDT is for coreboot tables to stay as-is, and a Kconfig option to make coreboot use its coreboot tables to generate an FDT.
I think keeping these things side-by-side in the code base with a Kconfig to switch between them
Like Arthur I don't appreciate the "side-by-side" narrative, nor an expectation for two independent data sets.
I'd like to make sure that we can agree to keep coreboot tables as a first-class citizen now and in the future, and don't force a new format on anyone.
I don't think anyone proposed to not generate coreboot tables anymore.
I'm talking about where this is eventually supposed to be headed, not just the first experimental patch.
This is supposed to be headed to additional benefit for both coreboot and USF (users). Others will use coreboot in different ways, e.g. without FDT.
I think we can all agree that there's no point in having two firmware handoff structures that both need to be parsed to get the full picture,
Noone suggested anything like that. Don't waste our time.
and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
If you mean "it's not a solution for coreboot to just generate both" then I ask why not? Two things can be true at the same time: coreboot tables are simple to generate and the code to generate FDT from coreboot tables is also simple.
Well, Arthur also admitted that none of these points are really relevant to coreboot's payload handoff, ;) which I fully agree with.
Currently that may be true, but reflection would pretty much be required for a new payload for manual parameter tweaking during boot.
Yeah, but what's the use case?
There are several, both outside and within the boot process.
FDT can be meaningfully persisted and so be useful both before and after boot.
When have you ever had a situation where you'd want to edit coreboot table structures at runtime?
You can also consider FDT in coreboot as plumbing to enable new situations that were best avoided with coreboot tables, then of course noone ever wanted to edit them. That doesn't mean that noone would ever want to edit firmware handoff information, especially if we also consider FSP.
Please be a bit more imaginative here.
it would still not be very hard to write such a thing for coreboot tables
Of course. One goal is to have more freedom than that allows, or at a very minimum fill a namespace not (exclusively) defined by coreboot or Intel.
As I understand it, cbtable member alignment is currently undefined. That matters when building firmware and payload with different
compilers.
"Just use the coreboot compiler" is not a good answer.
Can we please stop perpetuating this myth? It's just wrong, like I already explained in my first email here.
I don't think it's wrong. I used to, but not anymore.
This is not about 64-bit values which you mentioned in your first email, this is about struct member offsets in the struct.
The structs are not packed so the compiler decides.
This is why both Nico and I are so strong proponents of explicit serialization of the tables. (I'd also like to create new tags for coreboot tables with packed values but that's another matter.)
¹ I wouldn't mind that btw., replacing ACPI.
That would be fantastic! But maybe it's more of a stretch goal. :)
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it).
If all handoff interfaces were to use a single format (this would be great) then obviously it can be a single data set, which would make life simpler for a lot of different people.
(and not "soft deprecated" in favor of FDT,
Since this your narrative is baseless it makes it look like you are intentionally acting in bad faith. That's not a good look. Be careful.
or have the FDT stuff seep into something that can not be compiled out).
Like vboot has been for maybe a decade now yes, I agree strongly that this is important to avoid.
without stepping on each other's toes.
I wouldn't worry.
//Peter _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
I'm trying to make the case for "unifying payload handoff formats between different firmwares is generally not a useful idea
MBR called to disagree.
What's MBR in this context -- Master Boot Record? Kernel handoff and firmware payload handoff are two entirely different ballparks here.
That said, I wouldn't worry about negative effects. I certainly have no interest to remove coreboot tables nor do I see any reason to do so nor have I seen anyone mention anything in that direction.
Do not make up strawmen.
I'm gonna try to ignore this and all the other slight jabs and personal attacks afterwards because I don't think it's productive or going to help us find a compromise. Throughout the whole rest of this thread it seems like everyone had a friendly and respectful discussion despite our technical differences in opinion, let's please try to keep it that way (especially since it seems like we have basically found a compromise that we can all live with already anyway).
Just so you understand where I'm coming from, Arthur's very first comment on the original changelist asked whether it is "a good idea to use FDT and eventually replace coreboot tables", and I think several others (maybe not you) had still hinted at that as a long-term goal a few times. So I'm not trying to "waste your time" with "strawmen", I'm just trying to make sure we're on the same page on this because it is important to me.
A RAM-less FDT parser probably doesn't need to load the entire tree but merely find one value at a time, perhaps without state between values. The complexity of that in an FDT is low since FDT is flat and has simple primitives. CBOR, well..
[...]
This is bizarrely complex for storing a simple scalar.
It just takes an AND and a switch to decode? I honestly think you're overestimating this. I've written a CBOR parser before (and most parts of an FDT parser, for that matter) and it's really not a big deal. In fact looking for a single value might be easier in CBOR because the data is entirely local, whereas with FDT you at least need to juggle the separate string table. (Then again FDT makes it a little easier to skip over unrelated parts of the tree when looking for a single item, so there are certainly trade-offs)
I disagree with abandoning coreboot tables or forcing FDT generator code into coreboot images for those who aren't interested in this project.
So a way forward where you can ignore FDT is for coreboot tables to stay as-is, and a Kconfig option to make coreboot use its coreboot tables to generate an FDT.
Yes, that approach is fine by me. It's not as efficient for your case as generating the FDT directly, of course, but if you want to do it this way I won't complain.
and since the argument for coreboot tables is mostly simplicity / lean code it's not a solution for that side to just generate both.
If you mean "it's not a solution for coreboot to just generate both" then I ask why not? Two things can be true at the same time: coreboot tables are simple to generate and the code to generate FDT from coreboot tables is also simple.
I meant that it wouldn't be a solution to do it unconditionally, because my main argument against FDT is that I don't want that extra bloat for configurations that don't use the FDT. If we have a Kconfig to decide between coreboot-table-only and coreboot-table+FDT, that works for me.
This is not about 64-bit values which you mentioned in your first email, this is about struct member offsets in the struct.
The structs are not packed so the compiler decides.
This is why both Nico and I are so strong proponents of explicit serialization of the tables. (I'd also like to create new tags for coreboot tables with packed values but that's another matter.)
There is no padding in coreboot tables. We intentionally arrange members such that they don't leave padding holes and insert explicit `unused` members where necessary. This is a common pattern in a lot of structures where exact data layout is important (e.g. those representing hardware registers as well). Using __attribute__((packed)) is usually a bad idea and should be avoided unless really necessary (i.e. in cases where individual members need to be explicitly misaligned), because it forces the compiler to assume that all loads may be misaligned and leads to bad code generation on architectures that can't tolerate unaligned accesses in all cases.
FWIW I think this is a completely separate discussion and has nothing to do with what we should do for payload handoff (again, just like with the FSP thing, kernel handoff is a completely different thing that would have very little to share with it).
If all handoff interfaces were to use a single format (this would be great) then obviously it can be a single data set, which would make life simpler for a lot of different people.
The information that is being passed from coreboot to payloads and from the payload to kernels is so fundamentally different that there would be very little overlap (maybe not in the cases where the memory table is the only thing you have in your payload handoff, but certainly in those that actually have enough records to make it worth worrying about the format). Besides, kernel handoff formats tend to differ widely between architectures which I don't think is something we want to duplicate at the payload boundary.
Hi Ron,
On 04.11.22 16:58, ron minnich wrote:
I was involved in the creation of coreboot tables. They were the best we could come up with at the time.
Over time, their limits have been more and more apparent. This is why, in oreboot, we removed the initial support for coreboot tables and moved to FDT.
quick question, please bear my ignorance, I really don't know: Does oreboot do payloads or does it only do LinuxBoot?
Nico
Hi Chris,
On 08.11.22 20:35, Christian Walter wrote:
I wasn't sure if I should also shim in here, but I just wanted to leave my opinion here. I agree with adding FDT as another method to handoff data to a payload. Despite all the advantages that Ron and others pointed out, I think something that passed data into another project e.g. payload here - whatever that is, should never be specific to one project. It should not be the status-quo that other projects, if they want to be loaded as a payload after coreboot need to implement other project specific implementations. What I mean by this is that hands-off's between different projects should always rely on an open standard that is *not* specific to the project.
this, having an open standard, is theoretically the ideal solution. However, depending on the size of the problem, it might be much much more effort to achieve this compared to the effort to work around its absence. For instance, how many payload projects actually want to be loaded by different firmware frameworks? is it 10%, 20%, 90%? I have no idea. If it's few, the projects may lose interest before the new standard is ready. Also, how much trouble is it to adapt a payload? Let's not forget that the goal should be to make the work of firmware developers easier, not harder.
And of course, such a standard shouldn't be decided by a single project or group. If the goal is to get as many projects as possible to imple- ment it, we should get many firmware framework and payload projects together before making a decision. Otherwise, the risk is very high that we work towards an instance of the famous xkcd/927.
Then about FDT: It can only cover a small part of the standard. There is one important point that is often forgotten in this discussion: coreboot tables specify two things! 1. the binary format (this is what FDT can replace) 2. what data is passed (this is what bindings can replace)
Virtually nobody seems to be talking about 2. While IMO it would be the much more important part of an open standard. Replacing 1. is the part that creates churn; agreeing on 2. is mandatory to actually achieve payload compatibility.
I know experimenting with the code is much more fun than creating a standard. But replacing 1. alone won't make an open standard. It would leave us with the 2. part of coreboot tables. Which IIRC, was actually the part that Intel folks didn't want to agree with in the first place.
Nico
Thanks all for the good discussion. There will always be better ideas from time to time, I am very happy to see it happens as long as someone is willing to be in the driving seat on the effort to: - come out with proper specification design - communicate and with other firmware communities and silicon vendors for discussions - align and finalize all inputs together - set a timeline and follow up the implementation to the end
Meanwhile in parallel, as mentioned, I will continue to work on the FDT design with an incremental approach. Still drafting the specification and the code structure, happy to share the spec later for wider feedback. Please also let me know if you are interested in helping out :)
Best Regards, *Lean Sheng Tan*
On Fri, 11 Nov 2022 at 15:31, Nico Huber nico.h@gmx.de wrote:
Hi Chris,
On 08.11.22 20:35, Christian Walter wrote:
I wasn't sure if I should also shim in here, but I just wanted to leave my opinion here. I agree with adding FDT as another method to handoff data to a payload. Despite all the advantages that Ron and others pointed out, I think something that passed data into another project e.g. payload here - whatever that is, should never be specific to one project. It should not be the status-quo that other projects, if they want to be loaded as a payload after coreboot need to implement other project specific implementations. What I mean by this is that hands-off's between different projects should always rely on an open standard that is *not* specific to the project.
this, having an open standard, is theoretically the ideal solution. However, depending on the size of the problem, it might be much much more effort to achieve this compared to the effort to work around its absence. For instance, how many payload projects actually want to be loaded by different firmware frameworks? is it 10%, 20%, 90%? I have no idea. If it's few, the projects may lose interest before the new standard is ready. Also, how much trouble is it to adapt a payload? Let's not forget that the goal should be to make the work of firmware developers easier, not harder.
And of course, such a standard shouldn't be decided by a single project or group. If the goal is to get as many projects as possible to imple- ment it, we should get many firmware framework and payload projects together before making a decision. Otherwise, the risk is very high that we work towards an instance of the famous xkcd/927.
Then about FDT: It can only cover a small part of the standard. There is one important point that is often forgotten in this discussion: coreboot tables specify two things!
- the binary format (this is what FDT can replace)
- what data is passed (this is what bindings can replace)
Virtually nobody seems to be talking about 2. While IMO it would be the much more important part of an open standard. Replacing 1. is the part that creates churn; agreeing on 2. is mandatory to actually achieve payload compatibility.
I know experimenting with the code is much more fun than creating a standard. But replacing 1. alone won't make an open standard. It would leave us with the 2. part of coreboot tables. Which IIRC, was actually the part that Intel folks didn't want to agree with in the first place.
Nico _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
've added the topic to the agenda of the next coreboot leadership meeting. Hopefully some sort of decision can be made there. I'm not trying to push any particular solution, but we need to be able to agree on something and then work towards implementinga single, or even a couple different options
Options: - Mulitboot - FDT - CBOR - Universal Payload - coreboot tables.
I've made a document to capture pros & cons of each: https://docs.google.com/document/d/1rN9PcrGliRCE-q2sa_k49WCInTELyyudN9RQvez5...
I'll try to go through the mailing list and capture arguments for and against each option, but if people could help out and fill things in as well, I'd appreciate it.
We need to be able to reach a decision and stick with it for a while, even if it's not the perfect solution and doesn't make everyone happy.
Martin
Nov 15, 2022, 05:17 by sheng.tan@9elements.com:
Thanks all for the good discussion. There will always be better ideas from time to time, I am very happy to see it happens as long as someone is willing to be in the driving seat on the effort to:
- come out with proper specification design
- communicate and with other firmware communities and silicon vendors for discussions
- align and finalize all inputs together
- set a timeline and follow up the implementation to the end
Meanwhile in parallel, as mentioned, I will continue to work on the FDT design with an incremental approach. Still drafting the specification and the code structure, happy to share the spec later for wider feedback. Please also let me know if you are interested in helping out :)
Best Regards, Lean Sheng Tan
On Fri, 11 Nov 2022 at 15:31, Nico Huber <> nico.h@gmx.de> > wrote:
Hi Chris,
On 08.11.22 20:35, Christian Walter wrote:
I wasn't sure if I should also shim in here, but I just wanted to leave my opinion here. I agree with adding FDT as another method to handoff data to a payload. Despite all the advantages that Ron and others pointed out, I think something that passed data into another project e.g. payload here - whatever that is, should never be specific to one project. It should not be the status-quo that other projects, if they want to be loaded as a payload after coreboot need to implement other project specific implementations. What I mean by this is that hands-off's between different projects should always rely on an open standard that is *not* specific to the project.
this, having an open standard, is theoretically the ideal solution. However, depending on the size of the problem, it might be much much more effort to achieve this compared to the effort to work around its absence. For instance, how many payload projects actually want to be loaded by different firmware frameworks? is it 10%, 20%, 90%? I have no idea. If it's few, the projects may lose interest before the new standard is ready. Also, how much trouble is it to adapt a payload? Let's not forget that the goal should be to make the work of firmware developers easier, not harder.
And of course, such a standard shouldn't be decided by a single project or group. If the goal is to get as many projects as possible to imple- ment it, we should get many firmware framework and payload projects together before making a decision. Otherwise, the risk is very high that we work towards an instance of the famous xkcd/927.
Then about FDT: It can only cover a small part of the standard. There is one important point that is often forgotten in this discussion: coreboot tables specify two things!
- the binary format (this is what FDT can replace)
- what data is passed (this is what bindings can replace)
Virtually nobody seems to be talking about 2. While IMO it would be the much more important part of an open standard. Replacing 1. is the part that creates churn; agreeing on 2. is mandatory to actually achieve payload compatibility.
I know experimenting with the code is much more fun than creating a standard. But replacing 1. alone won't make an open standard. It would leave us with the 2. part of coreboot tables. Which IIRC, was actually the part that Intel folks didn't want to agree with in the first place.
Nico _______________________________________________ coreboot mailing list -- >> coreboot@coreboot.org To unsubscribe send an email to >> coreboot-leave@coreboot.org
-
Arthur Heymans -
Christian Walter -
Felix Held -
Jonathan Zhang (Infra) -
Julius Werner -
Krystian Hebel -
Lean Sheng Tan -
Martin Roth -
Nico Huber -
Paul Menzel -
Peter Stuge -
ron minnich