Rebuilding coreboot image generation
Hey coreboot folks, I'm looking for an approach to make building Chrome OS style coreboot images easier to do with regular coreboot tools, instead of the rather large post-processing pipeline we have in the Chrome OS build system. The rationale is that we also push the Chrome OS capabilities (eg. verified boot) upstream, and actually using them shouldn't depend on checking out yet another custom build environment. I wrote a proposal on how to do that, which can be found and commented at https://docs.google.com/document/d/1o2bFl5HCHDFPccQsOwa-75A8TWojjFiGK3r0yeIc... If this is to be implemented, I'd do that as part of coreboot's build system, and work on supporting the regular image types (with a "simple" bootblock and a fallback/normal switch configuration) we currently have. I appreciate comments and concerns how this fits in with other use cases. Thanks, Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
My main concern with this is that we're introducing another language, with something that may just as well end up being cmos.layout 2.0. I see where the manifest language is coming from. It's sort of like a FMAP dts, but then it really isn't. I understand that we don't want to use a per-board fmap.dts in order to prevent the duplication we ended up having last time we did this (cmos.layout).
From reading the manifest examples, these are the few things that jumped out at as being unwanted: (1) mixing region descriptors with rules. As soon as we agree to move some rules to manifests and some to makefiles, we risk ending up with a mess that's inconsistent between platforms, and hard to track down/debug. (2) Undermining some of the existing infrastructure. We can extend the concept of cbfs-files-y, to fmap-regions-y, whereas the FMAP regions are dynamically declared. Of course, this is in stark contrast to the per-module descriptors (which I also like, assuming they are pure fmap.dts format). However, this inconsistency worries me. (3) It changes everything we've learnt so far about building a firmware image. Some platforms will end up using the new manifests, others will use the existing build infrastructure. I'm worried we'll end up with an inconsistent build system. Who is going to convert all the old hardware to manifests?
Now to the good parts. I mentioned briefly the possibility of a fmap-regions-y macro. Quite honestly, I much better like seeing the fmap.dts in one place in order to get an idea of the flash layout. What I'd like to see come out of this is a dts-like set of flash descriptors, with the rules left in the makefiles. At the very least, dts is not a new language. Do you think we can do that without making a mess out of it? Alex On 11/05/2015 03:42 AM, Patrick Georgi wrote:
Hey coreboot folks,
I'm looking for an approach to make building Chrome OS style coreboot images easier to do with regular coreboot tools, instead of the rather large post-processing pipeline we have in the Chrome OS build system.
The rationale is that we also push the Chrome OS capabilities (eg. verified boot) upstream, and actually using them shouldn't depend on checking out yet another custom build environment.
I wrote a proposal on how to do that, which can be found and commented at https://docs.google.com/document/d/1o2bFl5HCHDFPccQsOwa-75A8TWojjFiGK3r0yeIc...
If this is to be implemented, I'd do that as part of coreboot's build system, and work on supporting the regular image types (with a "simple" bootblock and a fallback/normal switch configuration) we currently have.
I appreciate comments and concerns how this fits in with other use cases.
Thanks, Patrick
What I'd like to see come out of this is a dts-like set of flash descriptors, with the rules left in the makefiles. At the very least, dts is not a new language.
Just wanted to point out (without trying to usurp any of the ongoing discussion in that doc) that we already have an FMAP descriptor language that was implemented in cbfstool half a year ago, because I don't think that got much visibility upstream when it was merged. That's the FMD (flash map descriptor) format some of us are talking about in those comments, which was developed in anticipation of a build system redesign like this. It's aiming to describe FMAPs in a clean, flexible way without the cruft of a format (dts) that was never really meant for that purpose, and has some nice features (like variable-sized sections) built-in. The parser can be seen in util/cbfstool/fmd.c, and a sample file would look something like this (in general, '@offset' is the base offset to the parent section and the number after the space is the size): HOST_FIRMWARE@0xff800000 8M { SI_ALL 2M { SI_DESC 4K SI_ME 0x1ff000 } SI_BIOS { RW_A 0xf0000 { VBLOCK_A 64K RW_MAIN_A RW_FWID_A 0x40 } RW_B 0xf0000 { VBLOCK_B 64K RW_MAIN_B RW_FWID_B 0x40 } RO@4M { RO_VPD 16K FMAP@0x10000 2K RO_FRID 0x40 GBB@0x11000 RO_MAIN 1M } } }
2015-11-06 3:57 GMT+01:00 Alex G. <mr.nuke.me@gmail.com>:
My main concern with this is that we're introducing another language, with something that may just as well end up being cmos.layout 2.0.
I see where the manifest language is coming from. It's sort of like a FMAP dts, but then it really isn't. I understand that we don't want to use a per-board fmap.dts in order to prevent the duplication we ended up having last time we did this (cmos.layout).
From reading the manifest examples, these are the few things that jumped out at as being unwanted: (1) mixing region descriptors with rules. As soon as we agree to move some rules to manifests and some to makefiles, we risk ending up with a mess that's inconsistent between platforms, and hard to track down/debug. All rules end up in manifests. The build system wouldn't add a file to a cbfs _ever_, cbfs-files-y would create the "coreboot" manifest, to be consumed by the later tool.
Rationale: The build system is rather stupid about the order in which to add files, and while adding a proper scheduler that prevents unsuitable ordering (eg adding flexibly laid out files first, which may block files with stricter requirements later), people are rightfully complaining about too much magic in the build system already. (something to work on, but not as part of this effort)
(2) Undermining some of the existing infrastructure. We can extend the concept of cbfs-files-y, to fmap-regions-y, whereas the FMAP regions are dynamically declared. Of course, this is in stark contrast to the per-module descriptors (which I also like, assuming they are pure fmap.dts format). However, this inconsistency worries me. I deliberately didn't go for using dts, because that's a gross hack: When trying to define files to add to CBFS in dts (which I do as a temporary measure on the CrOS build system side), I now use dts properties (key/value pairs) with arbitrary unique keys. Bleh.
And as stated above, gnu make is the wrong tool for the job, even though it _would_ be possible to force it to do the right thing with lots of effort. Readability will suffer, as will execution speed, and in the end, like with certain other parts of the build system the only one touching that stuff without making a mess would be me. _That_ is something I don't want to see a repeat of, no matter of nice it is to see that I can coerce gnu make to do things it wasn't designed for on an ego-stroking level.
(3) It changes everything we've learnt so far about building a firmware image. Some platforms will end up using the new manifests, others will use the existing build infrastructure. I'm worried we'll end up with an inconsistent build system. Who is going to convert all the old hardware to manifests? I'll take care of upstream and CrOS. If only because I want to cut back on the mess we have in the tree now. Most coreboot images right now are pretty simple to convert: They're either CBFS_SIZE == ROM_SIZE, an IFD layout, or one or two boards with some special handling where neither of the other two hold true.
That I'd not have to build an fmap-regions-y for each and every of the boards individually is a plus.
Now to the good parts. I mentioned briefly the possibility of a fmap-regions-y macro. Quite honestly, I much better like seeing the fmap.dts in one place in order to get an idea of the flash layout. Consider we're going for an fmap description like that, and use that and cbfs-files-y for the layout. We still need a way to allocate files going into the various CBFS regions (where current configurations have one or three, and there's a pretty obvious two-CBFS configuration out there). That's another variable to take care of, and there's nothing around really that can make obvious any inconsistencies between those.
The proposal will, in effect, have three manifests on the coreboot side, plus payload and additional projects that may contribute files (eg Chrome EC in the Chrome OS use case). Ignoring the external manifests for a moment, that's - the chipset/board specific descriptor for flash layouts, chosen at build time (and there will be a bunch of common ones, with the option to override for special cases); - the boot method manifest. Chosen by the same family of Kconfig options that choose between simple or fallback/normal bootblocks. These two are static and reside somewhere in the repo. The third one is (what I called) the coreboot manifest, which lists the files. That one is dependent on the build system, and is pretty much a serialization of cbfs-files-y (just that cbfs-files-y now includes all files, instead of some files in cbfs-files-y and some manually added through $(CBFSTOOL) add* invocations in src/arch/*/Makefile.inc). That's the coreboot "build" side. Other projects can contribute their own manifests for the final "link" step. With all that stuff around, the final step is about reading the files and generating the image. Right now, we don't have such a clear description in our build system of what will happen to the final image.
What I'd like to see come out of this is a dts-like set of flash descriptors, with the rules left in the makefiles. At the very least, dts is not a new language. It's also pretty unusable for describing cbfs files.
Do you think we can do that without making a mess out of it? I tried to use the opportunity to cut down on the mess we're already making.
Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
On 11/05/2015 09:55 PM, Julius Werner wrote:
HOST_FIRMWARE@0xff800000 8M { SI_ALL 2M { SI_DESC 4K SI_ME 0x1ff000 } SI_BIOS { RW_A 0xf0000 { VBLOCK_A 64K RW_MAIN_A RW_FWID_A 0x40 } RW_B 0xf0000 { VBLOCK_B 64K RW_MAIN_B RW_FWID_B 0x40 } RO@4M { RO_VPD 16K FMAP@0x10000 2K RO_FRID 0x40 GBB@0x11000 RO_MAIN 1M } } }
Yes, that's exactly what I was referring to! I like that. Alex
2015-11-06 6:55 GMT+01:00 Julius Werner <jwerner@chromium.org>:
HOST_FIRMWARE@0xff800000 8M { SI_ALL 2M { SI_DESC 4K SI_ME 0x1ff000 } SI_BIOS { RW_A 0xf0000 { VBLOCK_A 64K RW_MAIN_A RW_FWID_A 0x40 } RW_B 0xf0000 { VBLOCK_B 64K RW_MAIN_B RW_FWID_B 0x40 } RO@4M { RO_VPD 16K FMAP@0x10000 2K RO_FRID 0x40 GBB@0x11000 RO_MAIN 1M } } } Just wanted to spell out the issues with this approach: First, it doesn't tell you where to put various build collateral. Is ramstage.elf to be added to SI_ME?
Second, this requires more or less one such description file per board and boot method configuration, which is even worse than Alexandru's concern about having a reenactment of horribly maintained cmos.layout files (they're "merely" per board, but otherwise stable). Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
Okay, looks like we're continuing the discussion over here after all.
Rationale: The build system is rather stupid about the order in which to add files, and while adding a proper scheduler that prevents unsuitable ordering (eg adding flexibly laid out files first, which may block files with stricter requirements later), people are rightfully complaining about too much magic in the build system already. (something to work on, but not as part of this effort)
I assume you're talking about things like crosreview.com/289491 as an example for strict requirements. I'm not really familiar with the details there so please correct me if I'm wrong, but if this is a blob that needs to be placed in one exact offset (which it looks like?), then I think we're approaching it from the completely wrong angle trying to stuff it into CBFS anyway. Maybe it was originally shoehorned into CBFS because that was the only thing we had at the time, but if we're rearchitecting things from the ground up to make FMAP a first class citizen this seems to be the right time to reconsider that. The whole point of FMAP is to allow exact placement in the ROM, whereas the whole point of CBFS is to conveniently group files with variable lengths and no strict placement requirements together in an easily manageable way. I think the only placement restriction CBFS should support in that new design is a (mild) minimum alignment, and anything that needs to be placed more exactly than that (including the bootblock and magic Intel blobs) should instead get its own FMAP section.
HOST_FIRMWARE@0xff800000 8M { SI_ALL 2M { SI_DESC 4K SI_ME 0x1ff000 } SI_BIOS { RW_A 0xf0000 { VBLOCK_A 64K RW_MAIN_A RW_FWID_A 0x40 } RW_B 0xf0000 { VBLOCK_B 64K RW_MAIN_B RW_FWID_B 0x40 } RO@4M { RO_VPD 16K FMAP@0x10000 2K RO_FRID 0x40 GBB@0x11000 RO_MAIN 1M } } } Just wanted to spell out the issues with this approach: First, it doesn't tell you where to put various build collateral. Is ramstage.elf to be added to SI_ME?
This was just an example of an existing (and I think oversimplified) Chrome OS ROM that I pulled from an old discussion, to demonstrate the format. We totally can (and should) redesign and rename some existing parts for the new layouts, and we could also add new features to the existing FMD parser if we deem them useful (e.g. some sort of inclusion or macro mechanism to help deduplicate more). Stuff like RO_MAIN and RW_MAIN_B are probably better named RO_CBFS and RW_CBFS_B, and I think the bootblock should be split out into its own section (since it is not a proper CBFS file, and it has strict placement requirements). In my current vision, you would then group all of coreboot's output files (both stages and separate files) into "pre-boot-path-split" and "post-boot-path-split" groups. I think the existing Makefile system would be fine for that, but we could discuss that part separately. Then, whatever controls the Chrome OS specific parts of the build process (I think a Makefile would be enough, but again, that's a separate issue from the ROM layout itself) will 'cbfstool add -r RO_CBFS' the pre-boot-path-split files, 'cbfstool add -r RO_CBFS,RW_CBFS_A,RW_CBFS_B' the post-boot-path-split files, and finally 'cbfstool add -r RW_CBFS_A,RW_CBFS_B' the EC-RW images (which are Chrome OS specific and should therefore probably only come into play at that time). The advantage is that the layout (which may need to be very flexible due to all the weird corner cases that we keep encountering, and may be duplicated a little bit in practice) only tells you *where* the well-known sections are in a single, very easily readable and modifiable place that can be parsed at a glance, while another (more generic) part of the system can decide *what* to put in those sections. I think this would be much easier to use in practice than a solution where the mere decision which section is placed where gets spread out through unordered lines in X different files that all somehow work together to cook up the final layout in a process hardly anyone will really understand (or be able to adjust).
Second, this requires more or less one such description file per board and boot method configuration, which is even worse than Alexandru's concern about having a reenactment of horribly maintained cmos.layout files (they're "merely" per board, but otherwise stable).
I think it probably requires less than two layouts per SoC in practice (one for Chrome OS, one for normal coreboot). In fact most of our current ARM boards could probably share a "default" ARM layout, and for x86 there might be two or three (based on differences like that microcode placement). We could move common differences (like bootblock size) out into a separate Kconfig and throw the .fmd files through CPP to allow even more reuse. Yes, we may still copy a certain pattern in two or three places in the end, and I think that's perfectly fine. These are small, simple to understand files... we have much worse duplication issues in other parts already. The huge upside is that any board *can* easily override the whole thing completely if it has to, just by copying and adjusting the default file into its board directory. It doesn't need to understand which manifests play into which section of the image and how each and every one of them can be overridden (if that's possible at all) to tweak the layout around a certain hardware requirement. It also makes it much easier for an external project that has some special requirements (imagine a new Chrome OS, but not yet integrated into upstream) to easily inject their own layouts to test something new on top of coreboot. Yes, in the end we certainly don't want every board to have their own layout file... but that should be rare in practice, and if we see a certain pattern of common deviations emerge we can find a way to re-unify them after the fact (as it often happens with code as well). I think that's better than forcing everyone into a complex and rigid framework spread out over three parts of the codebase right from the start, because you can never cover all edge cases (especially future ones we don't know yet) in one framework, and because people can only contribute new things if the system is simple enough for everyone to understand and expand.
On Fri, Nov 6, 2015 at 4:09 PM, Julius Werner <jwerner@chromium.org> wrote:
Okay, looks like we're continuing the discussion over here after all.
Rationale: The build system is rather stupid about the order in which to add files, and while adding a proper scheduler that prevents unsuitable ordering (eg adding flexibly laid out files first, which may block files with stricter requirements later), people are rightfully complaining about too much magic in the build system already. (something to work on, but not as part of this effort)
I assume you're talking about things like crosreview.com/289491 as an example for strict requirements. I'm not really familiar with the details there so please correct me if I'm wrong, but if this is a blob that needs to be placed in one exact offset (which it looks like?), then I think we're approaching it from the completely wrong angle trying to stuff it into CBFS anyway. Maybe it was originally shoehorned into CBFS because that was the only thing we had at the time, but if we're rearchitecting things from the ground up to make FMAP a first class citizen this seems to be the right time to reconsider that.
The whole point of FMAP is to allow exact placement in the ROM, whereas the whole point of CBFS is to conveniently group files with variable lengths and no strict placement requirements together in an easily manageable way. I think the only placement restriction CBFS should support in that new design is a (mild) minimum alignment, and anything that needs to be placed more exactly than that (including the bootblock and magic Intel blobs) should instead get its own FMAP section.
That suggestion leads to different software using fmap vs CBFS for finding and locating specifics assets that are required for booting. In fact that would fundamentally break x86 w.r.t. common stage loading because of the xip requirement. I don't think that's a good thing.
HOST_FIRMWARE@0xff800000 8M { SI_ALL 2M { SI_DESC 4K SI_ME 0x1ff000 } SI_BIOS { RW_A 0xf0000 { VBLOCK_A 64K RW_MAIN_A RW_FWID_A 0x40 } RW_B 0xf0000 { VBLOCK_B 64K RW_MAIN_B RW_FWID_B 0x40 } RO@4M { RO_VPD 16K FMAP@0x10000 2K RO_FRID 0x40 GBB@0x11000 RO_MAIN 1M } } } Just wanted to spell out the issues with this approach: First, it doesn't tell you where to put various build collateral. Is ramstage.elf to be added to SI_ME?
This was just an example of an existing (and I think oversimplified) Chrome OS ROM that I pulled from an old discussion, to demonstrate the format. We totally can (and should) redesign and rename some existing parts for the new layouts, and we could also add new features to the existing FMD parser if we deem them useful (e.g. some sort of inclusion or macro mechanism to help deduplicate more). Stuff like RO_MAIN and RW_MAIN_B are probably better named RO_CBFS and RW_CBFS_B, and I think the bootblock should be split out into its own section (since it is not a proper CBFS file, and it has strict placement requirements).
Those requirements are not necessarily known until link time.
In my current vision, you would then group all of coreboot's output files (both stages and separate files) into "pre-boot-path-split" and "post-boot-path-split" groups. I think the existing Makefile system would be fine for that, but we could discuss that part separately. Then, whatever controls the Chrome OS specific parts of the build process (I think a Makefile would be enough, but again, that's a separate issue from the ROM layout itself) will 'cbfstool add -r RO_CBFS' the pre-boot-path-split files, 'cbfstool add -r RO_CBFS,RW_CBFS_A,RW_CBFS_B' the post-boot-path-split files, and finally 'cbfstool add -r RW_CBFS_A,RW_CBFS_B' the EC-RW images (which are Chrome OS specific and should therefore probably only come into play at that time).
But some files need special flags. One of the harder things is identifying all the assets to fit into those 2 buckets. I personally think one stop shopping in a single file is way better than digging through Makefiles.
The advantage is that the layout (which may need to be very flexible due to all the weird corner cases that we keep encountering, and may be duplicated a little bit in practice) only tells you *where* the well-known sections are in a single, very easily readable and modifiable place that can be parsed at a glance, while another (more generic) part of the system can decide *what* to put in those sections. I think this would be much easier to use in practice than a solution where the mere decision which section is placed where gets spread out through unordered lines in X different files that all somehow work together to cook up the final layout in a process hardly anyone will really understand (or be able to adjust).
flash layout vs cbfs files (and their options) are indeed orthogonal. However, Patrick's proposal bridged those two by identifying where files go in the same format. In your case you are making those decisions on *where* things go in Makefiles vs the other file. However, it doesn't handle the options used when adding files effectively.
Second, this requires more or less one such description file per board and boot method configuration, which is even worse than Alexandru's concern about having a reenactment of horribly maintained cmos.layout files (they're "merely" per board, but otherwise stable).
I think it probably requires less than two layouts per SoC in practice (one for Chrome OS, one for normal coreboot). In fact most of our current ARM boards could probably share a "default" ARM layout, and for x86 there might be two or three (based on differences like that microcode placement). We could move common differences (like bootblock size) out into a separate Kconfig and throw the .fmd files through CPP to allow even more reuse.
Yes, we may still copy a certain pattern in two or three places in the end, and I think that's perfectly fine. These are small, simple to understand files... we have much worse duplication issues in other parts already. The huge upside is that any board *can* easily override the whole thing completely if it has to, just by copying and adjusting the default file into its board directory. It doesn't need to understand which manifests play into which section of the image and how each and every one of them can be overridden (if that's possible at all) to tweak the layout around a certain hardware requirement. It also makes it much easier for an external project that has some special requirements (imagine a new Chrome OS, but not yet integrated into upstream) to easily inject their own layouts to test something new on top of coreboot.
Yes, in the end we certainly don't want every board to have their own layout file... but that should be rare in practice, and if we see a certain pattern of common deviations emerge we can find a way to re-unify them after the fact (as it often happens with code as well). I think that's better than forcing everyone into a complex and rigid framework spread out over three parts of the codebase right from the start, because you can never cover all edge cases (especially future ones we don't know yet) in one framework, and because people can only contribute new things if the system is simple enough for everyone to understand and expand.
-- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Whoops... sorry, I meant to keep the discussion on the list. On Fri, Nov 6, 2015 at 3:17 PM, Julius Werner <jwerner@chromium.org> wrote:
That suggestion leads to different software using fmap vs CBFS for finding and locating specifics assets that are required for booting. In fact that would fundamentally break x86 w.r.t. common stage loading because of the xip requirement. I don't think that's a good thing.
Hmm... I don't really understand where you're seeing problems regarding XIP there. I've got to admit I don't understand that area very well though, especially since a lot of that changed recently.
Just to be clear, I didn't mean to say that all XIP stages should go into separate FMAP sections... those can stay in CBFS as they currently are. I didn't intend to include everything XIP with "needs to be placed exactly"... because they theoretically can be placed anywhere (right? or are there limits?), they just need to be explicitly linked to that offset once it has been decided. I know that we used to link the romstage to offset 0x0, take the resulting binary size of that to help layout CBFS, then link it again with the now known offset... seems like all of that moved into cbfstool now, but in theory it should still work just as well (regardless of the order the files get added in and where the stage ends up being placed in the end), right?
I think the case of that microcode blob is different because it seems to need an exact offset (at least that's what the whole hardcoded CONFIG_CPU_MICROCODE_CBFS_LOC thing looks like to me). Same for the bootblock, which always needs to be placed into the reset vector (and hasn't previously been a CBFS "file" anyway). Does that make it clearer or did I misunderstand your concerns?
Those requirements are not necessarily known until link time.
Which ones are you thinking of in particular? CONFIG_CPU_MICROCODE_CBFS_LOC seems to be hardcoded right now (unless I misread how that works). The bootblock size is determined at link time, but I don't think it needs to be... i.e. I think it's fine to just reserve a certain bootblock FMAP section with the maximum size we expect to need, and waste a few K of space if necessary. We could also make it configurable with a Kconfig to allow more fine tuning by preprocessing FMD files. I think that would be much more reasonable and easier to work with than the current model of mushing the bootblock somewhere in the CBFS area in a way that even cbfstool cannot modify/extract again after the fact. It would also allow us to further simplify the CBFS format (removing the need of the master header and the concept that only a certain part of the full CBFS image is space available for files... which was another goal of the CBFS redesign earlier this year that had unfortunately not been finished).
But some files need special flags. One of the harder things is identifying all the assets to fit into those 2 buckets. I personally think one stop shopping in a single file is way better than digging through Makefiles.
Well, but I don't think Patrick's proposal is one-stop-shopping either... rather, it splits both layout and file information across multiple manifests in the same way that CBFS file rules are already split across our Makefiles.
Personally, I'm more concerned with keeping the layout in one place. For the decision of which files are included I think the current split is reasonable... especially since you need to add extra files based on chipset or board much more often than you need to add extra sections, and in an "ideal" CBFS (where no file has magic placement requirements) it doesn't really matter that much how many files get included in the end (as long as the total size is sufficient which usually doesn't seem to be a problem).
But I'd be open to discuss how to handle the files more centrally if you have a good idea for that... I just think that it should be separated from the (central) FMAP layout.
And I didn't notice it was not on the list when it 'a' in gmail... On Fri, Nov 6, 2015 at 5:32 PM, Aaron Durbin <adurbin@google.com> wrote:
On Fri, Nov 6, 2015 at 5:17 PM, Julius Werner <jwerner@chromium.org> wrote:
That suggestion leads to different software using fmap vs CBFS for finding and locating specifics assets that are required for booting. In fact that would fundamentally break x86 w.r.t. common stage loading because of the xip requirement. I don't think that's a good thing.
Hmm... I don't really understand where you're seeing problems regarding XIP there. I've got to admit I don't understand that area very well though, especially since a lot of that changed recently.
Just to be clear, I didn't mean to say that all XIP stages should go into separate FMAP sections... those can stay in CBFS as they currently are. I didn't intend to include everything XIP with "needs to be placed exactly"... because they theoretically can be placed anywhere (right? or are there limits?), they just need to be explicitly linked to that offset once it has been decided. I know that we used to link the romstage to offset 0x0, take the resulting binary size of that to help layout CBFS, then link it again with the now known offset... seems like all of that moved into cbfstool now, but in theory it should still work just as well (regardless of the order the files get added in and where the stage ends up being placed in the end), right?
Yes. That will work. However, we need to pass flags to cbfstool when we do add-stage to make that work.
I think the case of that microcode blob is different because it seems to need an exact offset (at least that's what the whole hardcoded CONFIG_CPU_MICROCODE_CBFS_LOC thing looks like to me). Same for the bootblock, which always needs to be placed into the reset vector (and hasn't previously been a CBFS "file" anyway). Does that make it clearer or did I misunderstand your concerns?
Well there are few subtleties:
1. Some platforms want it in a fixed placed. I'd like that to change, but there is a need currently. Also, if we put microcode in a different place then we'd have to change the code which reads cbfs to get the microcode file later when loading microcode for other cpus, etc. 2. bootlbock is a little different too. While the reset vector is at at the end of the region mapped just below 4GiB we don't have it be a fixed size currently. Because of that you effectively have downward growing stack behavior w.r.t. where things eventually land. That's currently handled by the linker proper. 3. FSP on these newer intel platforms is linked XIP. However, we did add relocation to that as well so that we no longer had to fix the location. But that's only effective for the FSP requirements post cache-as-ram. Currently cache-as-ram on these FSP platforms is brought up in FSP. And identifying the location of FSP w/o a stack is leads to the current fixed location. I'm hoping to rectify this situation going forward because these requirements are unnecessarily binding us, however this is the current situation as of today.
Those requirements are not necessarily known until link time.
Which ones are you thinking of in particular? CONFIG_CPU_MICROCODE_CBFS_LOC seems to be hardcoded right now (unless I misread how that works). The bootblock size is determined at link time, but I don't think it needs to be... i.e. I think it's fine to just reserve a certain bootblock FMAP section with the maximum size we expect to need, and waste a few K of space if necessary. We could also make it configurable with a Kconfig to allow more fine tuning by preprocessing FMD files. I think that would be much more reasonable and easier to work with than the current model of mushing the bootblock somewhere in the CBFS area in a way that even cbfstool cannot modify/extract again after the fact. It would also allow us to further simplify the CBFS format (removing the need of the master header and the concept that only a certain part of the full CBFS image is space available for files... which was another goal of the CBFS redesign earlier this year that had unfortunately not been finished).
It can be changed, yes. However, I was just noting that's not how things work. As noted above bootblock is not the only thing.
But some files need special flags. One of the harder things is identifying all the assets to fit into those 2 buckets. I personally think one stop shopping in a single file is way better than digging through Makefiles.
Well, but I don't think Patrick's proposal is one-stop-shopping either... rather, it splits both layout and file information across multiple manifests in the same way that CBFS file rules are already split across our Makefiles.
I think the coreboot manifest just decided which cbfs region to add a file. The other stuff was flash layout. Sadly, I haven't seen fmd fully rolled out or this so I can't tell with certainty what either looks like. fmd proper is just flash layout and doesn't fix the "what cbfs region and what flags" for each file nor the scheduling for optimal placement.
Personally, I'm more concerned with keeping the layout in one place. For the decision of which files are included I think the current split is reasonable... especially since you need to add extra files based on chipset or board much more often than you need to add extra sections, and in an "ideal" CBFS (where no file has magic placement requirements) it doesn't really matter that much how many files get included in the end (as long as the total size is sufficient which usually doesn't seem to be a problem).
But I'd be open to discuss how to handle the files more centrally if you have a good idea for that... I just think that it should be separated from the (central) FMAP layout.
I think Patrick's concern w/ the layout having a chipset portion was so that one didn't have to touch every mainboard when changing something in the chipset that required such a layout change. I think you suggested #include files and he was opposed to it. I think the disagreement using the c preprocessor is where the design changes stem from (If i'm reading things correctly).
1. Some platforms want it in a fixed placed. I'd like that to change, but there is a need currently. Also, if we put microcode in a different place then we'd have to change the code which reads cbfs to get the microcode file later when loading microcode for other cpus, etc.
Sure, we might have to change code that looks for it... but I'd presume (without knowing that code) that it should be a minor and pretty straight-forward change compared to this big rearchitecture of our build system, right? Also, I'm not saying we should do this on all Intel platforms (although we might, depending on what makes it easier). We could only do it for the ones that need it (Skylake?) and make those use a different FMD file. (About the FMD files, I think there are many small differences that could be solved by either making separate FMD files (and living with a bit of code duplication), or using Kconfig variables and #ifdefs to make one file fit multiple cases. I think taking either of those options to the extreme would be bad, and we'd probably want a middle road of both depending on what works best for a certain situation. I still think that's a good system... the FMD format is dead simple to read/understand, and every developer already knows how #ifdefs and Kconfigs work without having to become familiar with some completely custom system.)
2. bootlbock is a little different too. While the reset vector is at at the end of the region mapped just below 4GiB we don't have it be a fixed size currently. Because of that you effectively have downward growing stack behavior w.r.t. where things eventually land. That's currently handled by the linker proper.
Right. That's why I'm saying I would change that... make the bootblock area a pre-defined size (maybe based on a Kconfig). Sure, we'll waste some space... but that's at most a few kilobytes of ROM, and in return I think it makes the whole thing much easier to understand and work with than the current CBFS-style bootblocks.
3. FSP on these newer intel platforms is linked XIP. However, we did add relocation to that as well so that we no longer had to fix the location. But that's only effective for the FSP requirements post cache-as-ram. Currently cache-as-ram on these FSP platforms is brought up in FSP. And identifying the location of FSP w/o a stack is leads to the current fixed location. I'm hoping to rectify this situation going forward because these requirements are unnecessarily binding us, however this is the current situation as of today.
Is this runtime or compile-time relocation we're talking about? Because I would presume compile-time relocation (which would be good enough for cbfstool) shouldn't make a difference about pre-/post-CAR? Anyway... from a purist's perspective (without knowing any details), this also sounds to me like if the FSP (or those parts of it) absolutely *must* be placed at an exact offset, it doesn't belong in CBFS and should have its own FMAP section. I mean, I think this is the whole reason why we (want to) have an FMAP... we could also keep a whole Chrome OS image inside CBFS if we just carefully adjust the offsets enough, but that would be terribly unmanageable, because CBFS just isn't a good tool for exact placement.
I think the coreboot manifest just decided which cbfs region to add a file. The other stuff was flash layout. Sadly, I haven't seen fmd fully rolled out or this so I can't tell with certainty what either looks like. fmd proper is just flash layout and doesn't fix the "what cbfs region and what flags" for each file nor the scheduling for optimal placement.
Sure, I see FMD as a building block of a whole solution, not the solution itself. (And I'd say that building block is pretty rolled out already... all the code for it is checked in and you can already run 'fmaptool input.fmd output_fmap.bin' today.
I think Patrick's concern w/ the layout having a chipset portion was so that one didn't have to touch every mainboard when changing something in the chipset that required such a layout change. I think you suggested #include files and he was opposed to it. I think the disagreement using the c preprocessor is where the design changes stem from (If i'm reading things correctly).
I'm not married the preprocessor, I certainly agree that there are a lot of ugly things about it. I was suggesting #include as a concept for FMD, not necessarily implemented through CPP. We also don't need to make one FMD file per board (which would then usually just #include a generic one, like memlayout)... we could instead think about symlinks, or using some kind of CONFIG_MAINBOARD_HAS_CUSTOM_FMD... I'm more arguing for general concept than details right now. (The one nice thing that has to be said about CPP, though, is that every developer already knows how it works. I think it has a much lower barrier to entry than any custom mechanism for this we could whip up, and it is quite flexible... I think for the uses I'm suggesting here, those qualities would make it the best choice.)
On Fri, Nov 6, 2015 at 5:17 PM, Julius Werner <jwerner@chromium.org> wrote:
1. Some platforms want it in a fixed placed. I'd like that to change, but there is a need currently. Also, if we put microcode in a different place then we'd have to change the code which reads cbfs to get the microcode file later when loading microcode for other cpus, etc.
Sure, we might have to change code that looks for it... but I'd presume (without knowing that code) that it should be a minor and pretty straight-forward change compared to this big rearchitecture of our build system, right?
Well, some of those code paths are common to many x86 systems. And the biggest thing I don't particularly like about that approach (ignoring how one would need to handle things different at compile time) is the lack of consistency. I don't mind the mix of cbfs and fmap, but I think using both for the granularity of a single "file" is odd. w.r.t. to a "big rearchitecture" I don't think it's that. It was my understanding that the cbfs-file-* entries in the Makefiles would be used to generate the manifest. As it currently stands fmd hasn't been rolled out yet. Both approaches need to parse the file to know fmap location such that we don't *require* a Kconfig and the fmap description to make things line up. The second pass is adding the files (and where they go).
Also, I'm not saying we should do this on all Intel platforms (although we might, depending on what makes it easier). We could only do it for the ones that need it (Skylake?) and make those use a different FMD file.
(About the FMD files, I think there are many small differences that could be solved by either making separate FMD files (and living with a bit of code duplication), or using Kconfig variables and #ifdefs to make one file fit multiple cases. I think taking either of those options to the extreme would be bad, and we'd probably want a middle road of both depending on what works best for a certain situation. I still think that's a good system... the FMD format is dead simple to read/understand, and every developer already knows how #ifdefs and Kconfigs work without having to become familiar with some completely custom system.)
I don't think Patrick's proposal is too complicated either. I think Patrick was attempting to solve both sets of problems (the one FMD solves currently and the cbfs file scheduler). The scheduler problem leverages the description of the fmap.
2. bootlbock is a little different too. While the reset vector is at at the end of the region mapped just below 4GiB we don't have it be a fixed size currently. Because of that you effectively have downward growing stack behavior w.r.t. where things eventually land. That's currently handled by the linker proper.
Right. That's why I'm saying I would change that... make the bootblock area a pre-defined size (maybe based on a Kconfig). Sure, we'll waste some space... but that's at most a few kilobytes of ROM, and in return I think it makes the whole thing much easier to understand and work with than the current CBFS-style bootblocks.
I understand your proposal. It sounds straight forward -- not sure how big of a pain it is. Definitely doable though. Just solves one issue.
3. FSP on these newer intel platforms is linked XIP. However, we did add relocation to that as well so that we no longer had to fix the location. But that's only effective for the FSP requirements post cache-as-ram. Currently cache-as-ram on these FSP platforms is brought up in FSP. And identifying the location of FSP w/o a stack is leads to the current fixed location. I'm hoping to rectify this situation going forward because these requirements are unnecessarily binding us, however this is the current situation as of today.
Is this runtime or compile-time relocation we're talking about? Because I would presume compile-time relocation (which would be good enough for cbfstool) shouldn't make a difference about pre-/post-CAR?
The current situation is that relocation *does not* work now. The assembly code expects it to be at a specific place. There is both compile time and runtime relocation doing for Chrome OS because of the RO, RW-A, and RW-B region. However, the RO one is a hard coded location.
Anyway... from a purist's perspective (without knowing any details), this also sounds to me like if the FSP (or those parts of it) absolutely *must* be placed at an exact offset, it doesn't belong in CBFS and should have its own FMAP section. I mean, I think this is the whole reason why we (want to) have an FMAP... we could also keep a whole Chrome OS image inside CBFS if we just carefully adjust the offsets enough, but that would be terribly unmanageable, because CBFS just isn't a good tool for exact placement.
Again then this becomes an inconsistency on Chrome OS. FSP lives in RO, RW-A, and RW-B regions. RW-A and RW-B will be CBFS. So do make it it partly CBFS and partly FMAP such that the code is consistent to use FMAP APIs throughout? In my opinion that isn't appropriate. From a "change this little bit of code here" thing it's doable. But from looking at source code I standpoint such it's inconsistent. Sure FMAP would be used in certain places, but I'd argue those are in the infrastructure parts that many people porting things don't deal with. I don't want people to head scratch where something should live and then figure out 1. how to access it and 2. how to add it properly. The latter is still not plumbed in anywhere in the build system.
I think the coreboot manifest just decided which cbfs region to add a file. The other stuff was flash layout. Sadly, I haven't seen fmd fully rolled out or this so I can't tell with certainty what either looks like. fmd proper is just flash layout and doesn't fix the "what cbfs region and what flags" for each file nor the scheduling for optimal placement.
Sure, I see FMD as a building block of a whole solution, not the solution itself. (And I'd say that building block is pretty rolled out already... all the code for it is checked in and you can already run 'fmaptool input.fmd output_fmap.bin' today.
No. It's not rolled out. Nothing uses it. header files aren't generated to replace Kconfig, etc. fmd files aren't used for building, etc. Yes, the code is there. The realization of utilizing it as a first-class citizen is not complete. Therefore, it is speculation on everyone's part to know how much duplication there is or not. In either case (sorry beating a dead horse) the scheduling of cbfs file addition is not fixed, but I think your suggestion is: all those special cases punt to fmap and have a fixed size because it makes it easier?
I think Patrick's concern w/ the layout having a chipset portion was so that one didn't have to touch every mainboard when changing something in the chipset that required such a layout change. I think you suggested #include files and he was opposed to it. I think the disagreement using the c preprocessor is where the design changes stem from (If i'm reading things correctly).
I'm not married the preprocessor, I certainly agree that there are a lot of ugly things about it. I was suggesting #include as a concept for FMD, not necessarily implemented through CPP. We also don't need to make one FMD file per board (which would then usually just #include a generic one, like memlayout)... we could instead think about symlinks, or using some kind of CONFIG_MAINBOARD_HAS_CUSTOM_FMD... I'm more arguing for general concept than details right now. (The one nice thing that has to be said about CPP, though, is that every developer already knows how it works. I think it has a much lower barrier to entry than any custom mechanism for this we could whip up, and it is quite flexible... I think for the uses I'm suggesting here, those qualities would make it the best choice.)
I like the CPP, but I still don't think utilizing it w/ FMD solves all problems w.r.t. cbfs file adding.
On Thu, Nov 05, 2015 at 12:42:02PM +0100, Patrick Georgi wrote:
Hey coreboot folks,
I'm looking for an approach to make building Chrome OS style coreboot images easier to do with regular coreboot tools, instead of the rather large post-processing pipeline we have in the Chrome OS build system.
The rationale is that we also push the Chrome OS capabilities (eg. verified boot) upstream, and actually using them shouldn't depend on checking out yet another custom build environment.
I wrote a proposal on how to do that, which can be found and commented at https://docs.google.com/document/d/1o2bFl5HCHDFPccQsOwa-75A8TWojjFiGK3r0yeIc...
It seems to me that similar functionality could be obtained by enhancing (or replacing) cbfstool. If possible, that seems simpler than adding a new tool, a new custom language, and a new layer to the build. For example, instead of a "Chipset manifest" file, I think one could run commands like: $ cbfs2tool coreboot.rom add-region "IFD" --start=0 --end=4K $ cbfs2tool coreboot.rom add-raw build/ifd.bin --region="IFD" --align=bottom --empty=0xff $ cbfs2tool coreboot.rom add-region "ME" --start=4K --end=4M ... The above would require cbfstool (or its replacement) to track some additional metadata (either in the rom itself or in a similarly named file). However, that seems simpler than introducing a new tool. Just my $.02 -Kevin
Again then this becomes an inconsistency on Chrome OS. FSP lives in RO, RW-A, and RW-B regions. RW-A and RW-B will be CBFS. So do make it it partly CBFS and partly FMAP such that the code is consistent to use FMAP APIs throughout? In my opinion that isn't appropriate. From a "change this little bit of code here" thing it's doable. But from looking at source code I standpoint such it's inconsistent. Sure FMAP would be used in certain places, but I'd argue those are in the infrastructure parts that many people porting things don't deal with. I don't want people to head scratch where something should live and then figure out 1. how to access it and 2. how to add it properly. The latter is still not plumbed in anywhere in the build system.
Okay, I guess we disagree on that one, then. In my opinion, the biggest problem and inconsistency here is that we have a CBFS file that needs to be placed at an exact offset (breaking the whole concept of CBFS as a "file system" that can transparently manage its space allocation). I think this is something that was done in the past for lack of a better option, but that can and should be fixed when a better tool (FMAP) is available. In my recollection/perspective, this was the biggest reason we decided to combine CBFS and FMAP in the same image in the first place (as opposed to porting all the Chrome OS / vboot tools over to access CBFS and have the whole Chrome OS image be a single (enhanced) CBFS)... because we acknowledged that FMAP's strengths (global lookup table, rigid regions that can be exactly aligned to custom offsets or erase blocks) are completely different and complementary to CBFS's strengths (organizing many small files in a compact, easy to manage format with automatic, opaque layout). If Intel's architectural requirements force us to have a component that must be linked at an exact spot in the image, I think it would only be consistent to use the tool that we have which is meant for these sorts of things... FMAP. It makes the intention (of placing it *exactly there*) very explicit, and it frees up CBFS from weird requirements that you wouldn't expect (e.g. the RO CBFS must encompass that region, you need to pay attention to the order you add files in, etc.). If that makes the code that handles it more complicated that's unfortunate, and we should try to mitigate it where we can (maybe hide differences with an extra FSP access API, or have all chipsets keep it in its own FMAP section (maybe even in RW?) even though only some really need that, or find a way / ask Intel to separate the location-dependent part out into a separate blob)... but it still seems like a smaller price to me than keeping the separation of duties between FMAP and CBFS muddied and imposing all of those weird restrictions on the latter for this one special case. I should probably stop arguing this because I don't even know the Intel code this is for... if you and all other x86 developers think the FSP really must go in CBFS I'll shut up. It's also really only tangentially related to the original discussion about the build system architecture... but I brought it up because some of the benefits Patrick's approach is trying to achieve (being able to better control CBFS file ordering) should be irrelevant in my opinion, and only even exist because of this problem.
2015-11-09 21:47 GMT+01:00 Julius Werner <jwerner@chromium.org>:
If Intel's architectural requirements force us to have a component that must be linked at an exact spot in the image, I think it would only be consistent to use the tool that we have which is meant for these sorts of things... FMAP. It must be linked to the location it was relocated to. That could even change over time. Just like x86s execute-in-place romstage.
Updates may end up at a different location, and they will likely have a different size. For all we know, updates may grow beyond the space we reserve for FSP in FMAP (which is read-only on Chromebooks). It would be unfortunate to have empty space, just in the wrong region, and run into problems doing an update because of that. I don't think the layout constraints we can impose on files in CBFS are much of a problem, and there's not much overlap: As I understand things, FMAP exists for higher-order partitioning; for things that the current instance of coreboot need not to care about, like a separate copy of coreboot in flash, or chipset data that is none of the CPU firmware's business (ME and the like). FMAP is also suitable for writable fixed-size sections (like MRC cache) that we want to keep far away from the CBFS chain. The other reason (besides file constraints) for this proposal is to enable the payload's build system to describe the files they need to add to CBFS themselves (and without knowing which FMAP regions the payload needs to end up in). depthcharge definitely needs something along those lines so we can get rid of that spaghetti code that is bundle_firmware.py. I think grub2 (and to some degree, seabios) could also benefit from dealing with their configuration files themselves in a structured way. Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
Let's extend your example so it becomes somewhat comparable to my proposal: HOST_FIRMWARE@0xff800000 8M { SI_ALL 2M { SI_DESC 4K SI_ME 0x1ff000 } SI_BIOS { RW_A 0xf0000 { VBLOCK_A 64K RW_MAIN_A CBFS { add-payload -n fallback/payload -f /path/to/whereever/the/payload/resides.elf add-stage -n fallback/ramstage -f $(obj)/cbfs/fallback/ramstage.elf add-stage -n fallback/romstage -f $(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000 } RW_FWID_A 0x40 } RW_B 0xf0000 { VBLOCK_B 64K RW_MAIN_B CBFS { add-payload -n fallback/payload -f /path/to/whereever/the/payload/resides.elf add-stage -n fallback/ramstage -f $(obj)/cbfs/fallback/ramstage.elf add-stage -n fallback/romstage -f $(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000 } RW_FWID_B 0x40 } RO@4M { RO_VPD 16K FMAP@0x10000 2K RO_FRID 0x40 GBB@0x11000 RO_MAIN CBFS { add-payload -n fallback/payload -f /path/to/whereever/the/payload/resides.elf add-stage -n fallback/ramstage -f $(obj)/cbfs/fallback/ramstage.elf add-stage -n fallback/romstage -f $(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000 add-stage -n fallback/verstage -f $(obj)/cbfs/fallback/romstage.elf --xip --align 0x1000 add -n bootblock -f $(obj)/cbfs/fallback/bootblock.bin --base -$(filesize $(obj)/cbfs/fallback/bootblock.bin) } } } } And that for every mainboard, and not transferable to other flash layout schemes. I don't think this is looking better (or even just as good) as my proposal, simple because of maintenance reasons. Patrick 2015-11-06 6:55 GMT+01:00 Julius Werner <jwerner@chromium.org>:
What I'd like to see come out of this is a dts-like set of flash descriptors, with the rules left in the makefiles. At the very least, dts is not a new language.
Just wanted to point out (without trying to usurp any of the ongoing discussion in that doc) that we already have an FMAP descriptor language that was implemented in cbfstool half a year ago, because I don't think that got much visibility upstream when it was merged. That's the FMD (flash map descriptor) format some of us are talking about in those comments, which was developed in anticipation of a build system redesign like this. It's aiming to describe FMAPs in a clean, flexible way without the cruft of a format (dts) that was never really meant for that purpose, and has some nice features (like variable-sized sections) built-in. The parser can be seen in util/cbfstool/fmd.c, and a sample file would look something like this (in general, '@offset' is the base offset to the parent section and the number after the space is the size):
HOST_FIRMWARE@0xff800000 8M { SI_ALL 2M { SI_DESC 4K SI_ME 0x1ff000 } SI_BIOS { RW_A 0xf0000 { VBLOCK_A 64K RW_MAIN_A RW_FWID_A 0x40 } RW_B 0xf0000 { VBLOCK_B 64K RW_MAIN_B RW_FWID_B 0x40 } RO@4M { RO_VPD 16K FMAP@0x10000 2K RO_FRID 0x40 GBB@0x11000 RO_MAIN 1M } } }
-- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
Let's extend your example so it becomes somewhat comparable to my proposal: [...] And that for every mainboard, and not transferable to other flash layout schemes. I don't think this is looking better (or even just as good) as my proposal, simple because of maintenance reasons.
Sorry, I'm not quite sure how to read that... did you literally mean that the cbfstool commands (or something equivalent) would be *in* the .fmd file? That is certainly not what I meant. Instead, I was suggesting to use well-known CBFS "categories" and have another part of the system (e.g. the Makefiles) map files to those. For example when something (a mainboard, an SoC, etc.) declares a file, it would declare that this file belongs to the "pre-boot-split CBFS" category or the "post-boot-split CBFS" category (or for all I care you could also bind them directly to romstage/ramstage/etc. which is I think closer to what you were proposing). Then the part deciding the boot scheme ("vboot" and "traditional coreboot" for now) would know that all "pre-boot-split" files would belong into the well-known CBFS FMAP sections "FALLBACK" and "NORMAL" for traditional coreboot, whereas "post-boot-split" would only go into "NORMAL" (and the vboot version would instead know how to bind those categories to its RO/RW_A/RW_B sections). The .fmd file should *only* define the layout, and all the individual platform pieces that need to add files should *only* declare something abstract about in which context those files are used. Then something else (which is deciding the general boot method) should define how to bind those two together.
So you think we're better off with one place to store the partition layout, a separate one to store what files belong where, and yet another one for the properties of those files, each with their own syntax and tooling? 2015-11-21 1:08 GMT+01:00 Julius Werner <jwerner@chromium.org>:
Let's extend your example so it becomes somewhat comparable to my proposal: [...] And that for every mainboard, and not transferable to other flash layout schemes. I don't think this is looking better (or even just as good) as my proposal, simple because of maintenance reasons.
Sorry, I'm not quite sure how to read that... did you literally mean that the cbfstool commands (or something equivalent) would be *in* the .fmd file? That is certainly not what I meant.
Instead, I was suggesting to use well-known CBFS "categories" and have another part of the system (e.g. the Makefiles) map files to those. For example when something (a mainboard, an SoC, etc.) declares a file, it would declare that this file belongs to the "pre-boot-split CBFS" category or the "post-boot-split CBFS" category (or for all I care you could also bind them directly to romstage/ramstage/etc. which is I think closer to what you were proposing). Then the part deciding the boot scheme ("vboot" and "traditional coreboot" for now) would know that all "pre-boot-split" files would belong into the well-known CBFS FMAP sections "FALLBACK" and "NORMAL" for traditional coreboot, whereas "post-boot-split" would only go into "NORMAL" (and the vboot version would instead know how to bind those categories to its RO/RW_A/RW_B sections). The .fmd file should *only* define the layout, and all the individual platform pieces that need to add files should *only* declare something abstract about in which context those files are used. Then something else (which is deciding the general boot method) should define how to bind those two together.
-- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
Patrick Georgi wrote:
I'm looking for an approach to make building Chrome OS style coreboot images easier to do with regular coreboot tools, instead of the rather large post-processing pipeline we have in the Chrome OS build system.
When designing CBFS we definitely did have fixed locations in mind. Despite its name, CBFS is not intended to function as an opaque file system. And it's certainly possible to implement fixed placement. We do not need to add *another* layer of complexity here, do we? We need to *eliminate* some of the complexity that has crept in. FMAP merely re-invents a block layer, on top of which is supposed to sit what seems to be perceived as a file system layer. Come on - you guys can do better than that. And I think CBFS can do better than that. Can anyone see some concrete problems with Kevin's suggestion? //Peter
2015-11-21 5:08 GMT+01:00 Peter Stuge <peter@stuge.se>:
And it's certainly possible to implement fixed placement.
The main drawback of CBFS, pretty much from the start, was that it creates a single chain of files, which makes it hard to implement certain designs safely: You need to be extra careful when adding files that are supposed to be written to (eg memory training caches) to make sure that they start at a flash block, and keep CBFS data structures out completely. You can't enforce through its structures that some files aren't "overwritten" by others by just reusing the file name, which is relevant even for the fallback/normal scenario: On an x86 system you have the bootblock at the top and the CBFS file chain moving bottom up, so to keep "fallback" safe, you need to ensure that any "normal" updates are written somewhere in the middle of the flash (and in a way that the CBFS file chain is never broken, despite flash block alignments). With fmap, you simply have two (or more) separate chains that can span whatever region is practical.
We do not need to add *another* layer of complexity here, do we? We need to *eliminate* some of the complexity that has crept in.
fmap replaces the master header, which has its own issues, for example its very x86-centric design that led to different interpretation of its fields between coreboot, cbfstool and libpayload while adapting things for non-x86. The data structures are also reasonably simple: It's more or less a list of offset/size/name triplets (plus some flag bits that we mostly ignore). Any hierarchy we show in the fmap region descriptions merely exists because fmap allows overlaps.
Can anyone see some concrete problems with Kevin's suggestion? It doesn't deal with the problem that each command has no visibility into what's going to be added after it.
Some of the constraints that we need to support (fixed placement, alignment, ...) can make the build break for a configuration that would work fine when reordering command invocation. That's an issue we ran into already. This is why I made the proposed flow work from a complete view of what needs to happen, and I tried to keep it out of the build system because people complained already that it's too complex. Given the opposition on the build description side, I'll punt that and have make(1) deal with things (and ignore some duplication in metadata). Both can still be improved later (or turn out not to be an issue after all). Patrick -- Google Germany GmbH, ABC-Str. 19, 20354 Hamburg Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: Hamburg Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle
On Fri, Nov 20, 2015 at 4:18 PM, Patrick Georgi <pgeorgi@google.com> wrote:
So you think we're better off with one place to store the partition layout, a separate one to store what files belong where, and yet another one for the properties of those files, each with their own syntax and tooling?
Yes. I find it strange that you seem to imply this would make it more complex or otherwise undesirable. Your proposal is to use a common language, but spread the files that control all of this (including the layout, which I think is the thing that really needs to be together in one place to be readable) all over the place. I think we agree that declarations of individual files should go in the different source directories that relate to them, I just think that the layout should stay centralized to be easily comprehensible and modifyable.
When designing CBFS we definitely did have fixed locations in mind.
Sure, I'm definitely not trying to argue that... I'm just saying that I think this should change if we have a cleaner option. The old CBFS has done a good job for a long time, but it was designed for a very specific purpose (x86 ROMs, I believe even without IFDs?) and we are now trying to do many more things (ARM boards with different layout requirements, RO/RW split ROMs, individually updateable A/B copies of the same file sets) that it really doesn't support well. Let's face it, problems like "we need to control the order in which we add files because otherwise the fixed location placement breaks" are really ugly and we should try to find a design that removes them completely.
Can anyone see some concrete problems with Kevin's suggestion?
I actually think Kevin's suggestion was pretty much the same thing we're proposing anyway. It's just that he suggested adding regions one at a time per cbfstool invocation, while the .fmd format is designed to read the complete region layout from a file at image creation (which I think is nicer because it makes the layout very easy to read... if you did the thing with commands, you'd effectively still end up having a list of those region-defining commands in some Makefile). The "little additional metadata" that Kevin said we needed *is* the FMAP. Think about it, how else would you store a list of regions with hardcoded bounds? This whole thing really started as an idea for a "CBFS partition layout" (http://www.coreboot.org/User_talk:MrNuke/CBFS_Ideas), until we realized that the difference between any "partition table" format we could come up with and an FMAP was so negligible that we might as well keep the established format with pre-existing tooling (most importantly flashrom support).
Julius Werner wrote:
When designing CBFS we definitely did have fixed locations in mind.
Sure, I'm definitely not trying to argue that... I'm just saying that I think this should change if we have a cleaner option. The old CBFS has done a good job for a long time, but it was designed for a very specific purpose (x86 ROMs, I believe even without IFDs?) and we are now trying to do many more things (ARM boards with different layout requirements, RO/RW split ROMs, individually updateable A/B copies of the same file sets) that it really doesn't support well.
ARM, flash sector alignment and individually updateable files is already in CBFS.
Let's face it, problems like "we need to control the order in which we add files because otherwise the fixed location placement breaks" are really ugly
They aren't caused by CBFS, but by cbfstool.
Can anyone see some concrete problems with Kevin's suggestion?
I actually think Kevin's suggestion was pretty much the same thing we're proposing anyway.
Fair enough. I would like this to first be implemented as he suggested, then adding support to read all of them at once from a file in cbfstool should be a very simple change, while cbfstool and CBFS are still the main tools. I think it is very important to keep existing tooling and existing data formats to stay compatibile and not force people to learn more tooling than neccessary.
it makes the layout very easy to read...
cbfstool print output is also very easy to read, and it's important to me that this continues to be all that is neccessary to grok a coreboot.rom file.
The "little additional metadata" that Kevin said we needed *is* the FMAP. Think about it, how else would you store a list of regions with hardcoded bounds?
It seems like a very simple data structure (name, base, length) that would be easy to store as a file within CBFS, so that a CBFS carries its own metadata. //Peter
participants (6)
-
Aaron Durbin -
Alex G. -
Julius Werner -
Kevin O'Connor -
Patrick Georgi -
Peter Stuge