We've been doing a lot of hacking on buildrom lately. Thanks to everybody, especially Ward and Myles. I wanted to take some time to discuss where we are and particularly where we want to go.
Here is where we are:
* v2 support for 10 platforms (alix1c, db800, dbe61, ga-2761gxdk, m57sli, msm800sev, norwich, serengeti_cheetah, tyan-s2882 and tyan-s2891).
* 5 payloads supported (etherboot, filo, kernel, LAB, memtest). Also, you can plug in a custom ELF if you see fit.
Thats fun, but its more interesting to talk about what we still have left to do:
* v3 support: This is probably the most useful extension we can do in the near term - not only does it serve our v3 hackers, but it also will force us to review some of the buildrom internals (in particular, I am worried about the state of the packages/linuxbios directory).
* OFW: We need to return OFW to working.
* gPXE: Ward is taking the lead on this
* grub2: Another payload we desperately need to get into buildrom - any volunteers?
* New platforms: New platforms always seem to introduce new wrinkles that we haven't thought about. In particular, the VIA platforms are pretty popular, so it would be nice to get support for them.
* buildroot/firmware_linux/etc: I'm still interested in partnering with other build systems in an effort to better scale support for LAB payloads.
* Optimizations: There are always things we can do to clean up - add -j N support for make, reduce memory usage, avoid stupid gmake tricks and the like. I think things are stable enough in the core that we can start to twist some knobs without blowing everything up.
What does everybody else think? What have I missed?
Jordan
jordan, for SiS we really need to get the point of having an normal/fallback LAB payload that runs in 2 MB, one MB for each one.
how hard is this? Does buildrom do fallback in general? I thought it did not.
I still don't know how to do fallback/normal on v3 ...
ron
On 29.11.2007 22:58, ron minnich wrote:
I still don't know how to do fallback/normal on v3 ...
The current v3 fallback/normal code is less than optimal. In case normal boot is specified, it will die() unless normal/initram and normal/stage2 are found. So even in the presence of fallback/initram and fallback/stage2 code will die. In case fallback boot is specified, it will still require normal/stage2 or die() as well.
IMO we should always fail over in both directions: Run normal/* if fallback/* fails and the other way round. The only open question is whether we want to be able to explicitly prohibit execution of normal/*, maybe by retrieving the state of a jumper etc.
I can create a patch on top of my ".o" removal for parsed ELF files.
Regards, Carl-Daniel
On 29/11/07 13:58 -0800, ron minnich wrote:
jordan, for SiS we really need to get the point of having an normal/fallback LAB payload that runs in 2 MB, one MB for each one.
how hard is this? Does buildrom do fallback in general? I thought it did not.
buildrom doesn't do fallback in general. Technically its not very hard, but it isn't very generic - there is a lot of platform specific thinking that needs to happen, and that adds further complexity. Especially if you want to be able to turn it on and off by configuration.
But we'll add it to the list - its good to think about that kind of stuff.
Jordan
On Nov 29, 2007 4:31 PM, Jordan Crouse jordan.crouse@amd.com wrote:
On 29/11/07 13:58 -0800, ron minnich wrote:
jordan, for SiS we really need to get the point of having an normal/fallback LAB payload that runs in 2 MB, one MB for each one.
how hard is this? Does buildrom do fallback in general? I thought it did not.
buildrom doesn't do fallback in general. Technically its not very hard, but it isn't very generic - there is a lot of platform specific thinking that needs to happen, and that adds further complexity. Especially if you want to be able to turn it on and off by configuration.
so it should be payload + normal + fallback + failover or payload_normal + normal + payload_fallback + failover or payload + unfied_normal_fallback + failover
better to put uncompresser into payload again.
YH
* yhlu yinghailu@gmail.com [071130 09:20]:
better to put uncompresser into payload again.
why?
On Thu, Nov 29, 2007 at 11:41:12AM -0700, Jordan Crouse wrote:
We've been doing a lot of hacking on buildrom lately. Thanks to everybody, especially Ward and Myles. I wanted to take some time to discuss where we are and particularly where we want to go.
Here is where we are:
- v2 support for 10 platforms (alix1c, db800, dbe61, ga-2761gxdk, m57sli, msm800sev, norwich, serengeti_cheetah, tyan-s2882 and tyan-s2891).
And adding new platforms is not very difficult.
- gPXE: Ward is taking the lead on this
r70 now has gPXE payload support; it does not work yet though because the gPXE tree is in flux, and (for the moment) no longer has LinuxBIOS support.
- buildroot/firmware_linux/etc: I'm still interested in partnering with
other build systems in an effort to better scale support for LAB payloads.
That would be great. I'm not sure what the best way forward is here though; seems like the major distros have each taken a somewhat separate path here.
- Optimizations: There are always things we can do to clean up - add -j N
support for make, reduce memory usage, avoid stupid gmake tricks and the like. I think things are stable enough in the core that we can start to twist some knobs without blowing everything up.
There's some other stuff that would be nice:
- aborting the build nicely when the generated image is too large - support for different rom chip sizes (in kconfig) - do something about the patching of LinuxBIOS Config files
We talked about some of this before on IRC; I seem to recall Jordan's feeling was that the last 2 points would be easier under v3. Should we focus our efforts there for these improvements?
Thanks, Ward.
On 30/11/07 16:29 -0500, Ward Vandewege wrote:
There's some other stuff that would be nice:
- aborting the build nicely when the generated image is too large
- support for different rom chip sizes (in kconfig)
- do something about the patching of LinuxBIOS Config files
We talked about some of this before on IRC; I seem to recall Jordan's feeling was that the last 2 points would be easier under v3. Should we focus our efforts there for these improvements?
There just isn't any good way to change v2 config files without lots of pain - especially for specifying values on the fly (like rom chip sizes). We currently get away with simple things like payloads and enabling lzma by patching the code, but that is only one Config.lb change away from not applying. Also, the patches scale exponentially. A patch for lzma, a patch for the lab payload, a patch for the lab payload with lzma, and so forth.
So other then patching, our only real option for v2 would be to develop configuration files on the fly with scripts, which is every bit as evil as it sounds.
Not that we really have a great story for v3 either, but we have the opportunity to change that. I think that at a minimum, we will need the ability to override configuration opens with environment variables - ROM size and compression control come to mind as good candidates, but really a generic way to override config without editing text files would be the most ideal.
Jordan
On Mon, Dec 03, 2007 at 09:15:37AM -0700, Jordan Crouse wrote:
There just isn't any good way to change v2 config files without lots of pain
Not that we really have a great story for v3 either,
..
really a generic way to override config without editing text files would be the most ideal.
How about one environment variable that points to another file with options?
This could probably be added pretty easily to the v2 config python thing, as well as v3.
//Peter
On 03/12/07 18:00 +0100, Peter Stuge wrote:
On Mon, Dec 03, 2007 at 09:15:37AM -0700, Jordan Crouse wrote:
There just isn't any good way to change v2 config files without lots of pain
Not that we really have a great story for v3 either,
..
really a generic way to override config without editing text files would be the most ideal.
How about one environment variable that points to another file with options?
This could probably be added pretty easily to the v2 config python thing, as well as v3.
Oh, I like it. Lets make this happen.
Jordan
On Mon, Dec 03, 2007 at 10:41:11AM -0700, Jordan Crouse wrote:
How about one environment variable that points to another file with options?
Oh, I like it. Lets make this happen.
See mainboard() at line 1333 in util/newconfig/config.g.
Does buildrom want to replace/amend Options.lb as well, or is it just Config.lb?
Are we simply looking for a second call to doconfigfile() if a particular environment variable has been set?
//Peter
There just isn't any good way to change v2 config files without lots of pain
Not that we really have a great story for v3 either,
..
really a generic way to override config without editing text files would be the most ideal.
How about one environment variable that points to another file with options?
The unfortunate thing is that either this adds lots more Config.lb files (one for each payload combination or ROM size), or forces you to patch the buildrom-specific file you just introduced.
I think it's more important to figure out how we really want to configure LinuxBIOS (it seems to be a large hurdle for newbies), not buildrom.
Once it's easy to configure LinuxBIOS it will be easy to do it with buildrom.
Myles
On 03/12/07 18:54 +0100, Peter Stuge wrote:
On Mon, Dec 03, 2007 at 10:41:11AM -0700, Jordan Crouse wrote:
How about one environment variable that points to another file with options?
Oh, I like it. Lets make this happen.
See mainboard() at line 1333 in util/newconfig/config.g.
Does buildrom want to replace/amend Options.lb as well, or is it just Config.lb?
I would think that any configuration variable would be eligible to be overriden. Presumably illegal combinations would be rejected?
Are we simply looking for a second call to doconfigfile() if a particular environment variable has been set?
I'm thinking so - the most important thing is that the override file is the final word on the settings.
Jordan
On 03/12/07 11:12 -0700, Myles Watson wrote:
There just isn't any good way to change v2 config files without lots of pain
Not that we really have a great story for v3 either,
..
really a generic way to override config without editing text files would be the most ideal.
How about one environment variable that points to another file with options?
The unfortunate thing is that either this adds lots more Config.lb files (one for each payload combination or ROM size), or forces you to patch the buildrom-specific file you just introduced.
What scared me about changing the configs on the fly was the extreme amount of sed magic we would have to do to get it right. In this case, that fear isn't there since we're not actually modifying any existing files, so we can easily add the config options we want with echo >:
ifeq ($(CONFIG_LZMA),y) echo "NEEDLZMAPLEASE=1" >> $(LBOVERRIDE) endif
That will be pretty generic for most cases - if there is something really special for a particular platform, we can specify a static override file to start and then modify it as above, but that will be the exception, not the rule.
I think it's more important to figure out how we really want to configure LinuxBIOS (it seems to be a large hurdle for newbies), not buildrom.
Agreed - being able to do these things easily with v3 is a priority. We have to think about what we would want to control externally, and then make those knobs easy to turn.
Jordan
There just isn't any good way to change v2 config files without lots of pain
Not that we really have a great story for v3 either,
..
really a generic way to override config without editing text files would be the most ideal.
How about one environment variable that points to another file with options?
The unfortunate thing is that either this adds lots more Config.lb files (one for each payload combination or ROM size), or forces you to patch
the
buildrom-specific file you just introduced.
What scared me about changing the configs on the fly was the extreme amount of sed magic we would have to do to get it right. In this case, that fear isn't there since we're not actually modifying any existing files, so we can easily add the config options we want with echo >:
ifeq ($(CONFIG_LZMA),y) echo "NEEDLZMAPLEASE=1" >> $(LBOVERRIDE) endif
All right. At least it will document what the Config.lb variables mean if they "need" to be tweaked.
Myles
That will be pretty generic for most cases - if there is something really special for a particular platform, we can specify a static override file to start and then modify it as above, but that will be the exception, not the rule.
I think it's more important to figure out how we really want to
configure
LinuxBIOS (it seems to be a large hurdle for newbies), not buildrom.
Agreed - being able to do these things easily with v3 is a priority. We have to think about what we would want to control externally, and then make those knobs easy to turn.
Jordan
Jordan Crouse Systems Software Development Engineer Advanced Micro Devices, Inc.
On Mon, Dec 03, 2007 at 11:29:27AM -0700, Jordan Crouse wrote:
I think it's more important to figure out how we really want to configure LinuxBIOS (it seems to be a large hurdle for newbies), not buildrom.
Agreed - being able to do these things easily with v3 is a priority. We have to think about what we would want to control externally, and then make those knobs easy to turn.
Isn't that easy in v3 already? You just use / modify defconfig files as with the kernel.
Uwe.
- Optimizations: There are always things we can do to clean up - add -j N
support for make, reduce memory usage, avoid stupid gmake tricks and the like. I think things are stable enough in the core that we can start to twist some knobs without blowing everything up.
This patch adds -jN support to speed builds. It passes it as an argument to make for the kernel and uClibc. It breaks the build for busybox, so it isn't passed there.
The default is -j1, or the status quo.
Signed-off-by: Myles Watson myles@pel.cs.byu.edu
On Mon, Dec 03, 2007 at 11:29:27AM -0700, Jordan Crouse wrote:
I think it's more important to figure out how we really want to
configure
LinuxBIOS (it seems to be a large hurdle for newbies), not buildrom.
Agreed - being able to do these things easily with v3 is a priority. We have to think about what we would want to control externally, and
then
make those knobs easy to turn.
Isn't that easy in v3 already? You just use / modify defconfig files as with the kernel.
From my point of view, modifying defconfig files from buildrom is not that
easy. Very small changes frequently require you to run make oldconfig so that you can handle all of the options that were not set, but now need to be set because you changed an option.
Maybe there is a good way to do hierarchical make menuconfig, but then you get the problem that the sources haven't been downloaded yet when you are configuring buildrom.
That said, I haven't used v3 yet.
It seems like in the current state, the addition of new options (a new form of compression, for instance) in v3's config files might require patching all of the .config files in buildrom. That doesn't seem nice.
I'm just hoping that when v3 supports the platforms I'm interested in it will be easy for buildrom to select which images to include and do some rudimentary calculations to see if they'll fit in the ROM.
Thanks,
Myles PS If I'm overlooking the obvious, easy ways to do things I'm all ears. It's easy to end up doing things the hard way if you don't know what the easy way is.
Myles Watson wrote:
- Optimizations: There are always things we can do to clean up - add -j N
support for make, reduce memory usage, avoid stupid gmake tricks and the like. I think things are stable enough in the core that we can start to twist some knobs without blowing everything up.
This patch adds -jN support to speed builds. It passes it as an argument to make for the kernel and uClibc. It breaks the build for busybox, so it isn't passed there.
The default is -j1, or the status quo.
Signed-off-by: Myles Watson myles@pel.cs.byu.edu
Acked-by: Corey Osgood corey.osgood@gmail.com
On Dec 4, 2007 2:29 PM, Myles Watson myles@pel.cs.byu.edu wrote:
That said, I haven't used v3 yet.
give it a try first :-)
thanks
ron
On 04/12/07 23:06 -0500, Corey Osgood wrote:
Myles Watson wrote:
- Optimizations: There are always things we can do to clean up - add -j N
support for make, reduce memory usage, avoid stupid gmake tricks and the like. I think things are stable enough in the core that we can start to twist some knobs without blowing everything up.
This patch adds -jN support to speed builds. It passes it as an argument to make for the kernel and uClibc. It breaks the build for busybox, so it isn't passed there.
The default is -j1, or the status quo.
Signed-off-by: Myles Watson myles@pel.cs.byu.edu
Acked-by: Corey Osgood corey.osgood@gmail.com
r74. Thanks.
That said, I haven't used v3 yet.
give it a try first :-)
thanks
ron
This patch adds support for QEMU to buildrom, for both v2 and v3. It also allows you to build QEMU from sources with the patches.
I've built it for v2 and v3 and it boots in v2, I'm not sure why it builds but doesn't boot in v3. Anyone care to send me their config file that works?
In both v2 and v3 I turned of lzma compression because the decompressor was running out of space, and emulated ROM space was cheap today.
I set it to use revision 539 of v3 because that was the last revision before it was broken on purpose to mirror hardware better.
Myles
Signed-off-by: Myles Watson myles@pel.cs.byu.edu
On 05/12/07 13:28 -0700, Myles Watson wrote:
That said, I haven't used v3 yet.
give it a try first :-)
thanks
ron
This patch adds support for QEMU to buildrom, for both v2 and v3. It also allows you to build QEMU from sources with the patches.
Not attached.
I've built it for v2 and v3 and it boots in v2, I'm not sure why it builds but doesn't boot in v3. Anyone care to send me their config file that works?
In both v2 and v3 I turned of lzma compression because the decompressor was running out of space, and emulated ROM space was cheap today.
I set it to use revision 539 of v3 because that was the last revision before it was broken on purpose to mirror hardware better.
Are you saying you snuck v3 support in when I wasn't looking? Well done.
Jordan
On Wed, Dec 05, 2007 at 01:28:33PM -0700, Myles Watson wrote:
This patch adds support for QEMU to buildrom, for both v2 and v3. It also allows you to build QEMU from sources with the patches.
I've built it for v2 and v3 and it boots in v2, I'm not sure why it builds but doesn't boot in v3. Anyone care to send me their config file that works?
The default config should work fine with current v3 trunk. Just run 'make menuconfig' and save (no changes).
In both v2 and v3 I turned of lzma compression because the decompressor was running out of space, and emulated ROM space was cheap today.
I set it to use revision 539 of v3 because that was the last revision before it was broken on purpose to mirror hardware better.
You can use trunk, qemu build/run has been fixed in v3.
Uwe.
-
Carl-Daniel Hailfinger -
Corey Osgood -
Jordan Crouse -
Myles Watson -
Peter Stuge -
ron minnich -
Stefan Reinauer -
Uwe Hermann -
Ward Vandewege -
yhlu