Joshua Wise <joshua@joshuawise.com> writes:

On Thursday 18 December 2003 7:36 pm, Eric W. Biederman wrote:

...

Brainstorming earlier today I think I have found a way to use an linux kernel for the boot loader and to implement pcbios compatibility without too much cost. The idea is to use a uclinux kernel. And implement a ``user space'' aplication that is a user space shim that makes kernel calls.

The way I implement my bootloader on ARM is like this:

1) First stage assembly loader sets up serial and DRAM. 2) First stage loader probes RAM, and sets up tagged list.

Roughly what LinuxBIOS does.

3) First stage jumps into zImage of special LAB (Linux As Bootldr) kernel. (currently just a 2.6.0 kernel from handhelds.org that has CONFIG_LAB defined.) 4) LAB kernel boots up until it gets ready to jump into init. 5) #ifdef'ed code takes over and calls a LAB main function which does all sorts of cool stuff including giving the user a CLI if requested (usually by holding the iPAQ's joypad down), or autobooting (running a predefined mkdir/mount/armboot sequence.)

Conceivably you could write a subapp for the CLI that does what you want, and put it in the autoboot script.

Somehow I could write a subapp that would make linux look like a normal pcbios, but I can be surprised.

LAB for ARM's zImage is currently ~509kbytes for those who care. (We must keep it below 512KB.)

Ouch! My x86 images are below that, at least before decompression.

...

There are a few nasty details to work out like how to handle services that are expected to work in vm86 mode. But I'm not certain I care. I've tinkered with writing an OS, but I still don't know too much about the x86 architecture, so I couldn't help there. Sorry.

...

Other thoughts? Check out my LAB code, see what you think. It's in handhelds.org anoncvs, module linux/kernel26. Relevant crap is in bootldr/, drivers/bootldr/, and arch/arm/boot/.

I probably will. Doing that stuff inside the kernel does not really feel proper to me. I already have an x86 kernel that can load another kernel from user space. I'm just trying to find a good long term architecture for using the kernel as a bootloader.

...

After I come back from my christmas vacation I am going to have to try it and see how will it will actually work. Take care, have a nice vacation.

Thanks. Eric

On Fri, 2003-12-19 at 05:56, Eric W. Biederman wrote:

Regardless of the actual details what I like is the added requirement that we attempt to support the legacy pcbios interface. It's ugly it's nasty but if we can do that with a Linux kernel driving the hardware we have a single solution which can work for everyone. Some people can strip it down and not include all of the features but...

Or - cf the PXE discussions on etherboot-developers - have the stripped version as the "normal" code and load in the pc bios services as required. Indeed, why not add a "legacy layer" to OSs which need it, so that rather than adding the code to the loader, add it to the *loadee*; the common services can be lean, mean, open source, virtualised or whatever we please.

Richard Smith <rsmith@bitworks.com> writes:

Has anyone tried this to see how small it compiles down to? Probally won't ever

get as small as uclinux but might be useable for somebody.

http://lwn.net/Articles/62858/

uclinux is in the same source tree. It looks like they have patches to compile out all of the recent bloat. If the maintenance continues it looks very promising. But I'm on vacataion at the moment so someone else gets to be the ginny pig. Eric

On 4 Jan 2004, Eric W. Biederman wrote:

I just looked. With everything turned off I have built a 220K kernel. 371K uncompressed but that doesn't count.

this is 2.6 or ... ron

ebiederman@lnxi.com (Eric W. Biederman) writes:

ron minnich <rminnich@lanl.gov> writes:

...

On 4 Jan 2004, Eric W. Biederman wrote:

...

I just looked. With everything turned off I have built a 220K kernel. 371K uncompressed but that doesn't count.

this is 2.6 or ...

2.6.1-rc1-tiny1

Adding in - printk - serial console support - networking support ( Not IP networking) - an eepro100 driver

takes it up to about 306K compressed. With IPv4 support compiled it goes to about 396K compressed.

So the patches for 2.6 exist to make it usable. And there is still room for improvement.

Playing a little more in the nothing selected case I get the kernel to 191K if I compile out support for block devices. After writing the patch that allows that. Eric

Adam Sulmicki <adam@cfar.umd.edu> writes:

you mean just like dosemu runs under linux ???

Right but for BSD and early versions of windows the dependencies were worse. I don't know which services matter though. Eric

Takeshi Sone <ts1@tsn.or.jp> writes:

On Fri, Dec 19, 2003 at 12:49:42AM -0700, Eric W. Biederman wrote:

...

Adam Sulmicki <adam@cfar.umd.edu> writes:

...

you mean just like dosemu runs under linux ???

Right but for BSD and early versions of windows the dependencies were worse. I don't know which services matter though.

Last time I saw the source code of FreeBSD, its _bootloader_ has its own vm86 monitor (in assembly), works in protected mode, and calls all the BIOS calls (video, keyboard, disk,...) in vm86 mode. Also the kernel calls E820, APM, VESA, etc in vm86. Also I saw a recent version of Windows calls BIOS services in vm86.

Maybe we can modify FreeBSD, and have work-around for Windows, but at least I don't think it's the way to go.

Thanks. I knew this was the biggest danger, and from the description it looks like no half measures will do.

My conclusion then was that PCBIOS had to work in real mode.

If that is the case the PCBIOS has hit an evolutionary dead, as the code size cannot increase, there is very little room for change remaining. This explains ACPI. And it makes EFI look much more attractive. I think I see one way out of this pickle, implement the firmware in System Management mode. System calls will be slow and we will need an interrupt reflector but this does allow us to escape the bounds of vm86 mode, and the legacy limitations. And our OS will even have some protection from ``user mode''. System Management mode is essentially big real mode. So code would need to be compiled with .code16gcc, but there is a full 4G available. I know with amd's processors I can switch to 32bit or 64bit protected mode inside of smm mode, Intel's documentation is not clear on that point, so I don't know what we can do there. Depending on what works it may make sense simply to build an smm mode trampoline to get out of vm86 mode.

Maybe we can code it with GCC and .code16gcc hack. If 64KB is not enough, maybe 0xE000 segment can be used so that we have 64KB code and 64KB data segments. And all the "POST" code can be outside the real mode space. So it's not impossible or hard at all.

For just PCBIOS compatibility I agree. Primarily I want that layer to exist as an add-on and an after thought. Something that gives backwards compatibility but allows us to do something better. With only 256KB of room, there simply is not enough room to implement drivers of interesting hardware. Nor is there enough room to implement interesting protocol stacks. And it limits what we can do in the future. Eric

On 20 Dec 2003, Eric W. Biederman wrote:

Ok Then I am up to idea #2. For cooperation, just before I go catch my flight.

here's what I would like: - we support all things bootable that don't need a pcbios linux, plan9, memtest, etc. - we encourage other projects to move away from needing a bios this is mainly xyzbsd - we use emulation for the option roms so that PPC and other non-x86 will work (this is almost done) - we provide (for those who want it) PCBIOS support as an option Making it optional allows vendors to build Windows-proof PCs, and I have had some interest expressed in that concept: if you are Windows-proof, you are audit-proof. A desriable property. we should really make PCBIOS an option of last resort. We should not plan for it as anything but a hack for OSes that are doing the wrong thing -- using a PCBIOS. ron

Hi, When I read the part about being Windows-proof, I *really* liked that idea. Also, about the audit-proof aspect. Would built-in file encryption be a foolish thing to wish for some day? (Maybe when the chips are bigger?) Jorgen ve5jor@rac.ca ps This list is *very* informative, and I am grateful for having access to it. I wish my talents were greater, and especially more current, so that I could contribute. Thanks to all, in the list. Merry Christmas, & Happy Holidays to all. --------------------------------------------------------------------------- On Saturday 20 December 2003 11:21 am, ron minnich wrote: ...... - we provide (for those who want it) PCBIOS support as an option Making it optional allows vendors to build Windows-proof PCs, and I have had some interest expressed in that concept: if you are Windows-proof, you are audit-proof. A desriable property. we should really make PCBIOS an option of last resort. We should not plan for it as anything but a hack for OSes that are doing the wrong thing -- using a PCBIOS. ron _______________________________________________ Linuxbios mailing list Linuxbios@clustermatic.org http://www.clustermatic.org/mailman/listinfo/linuxbios

On Sat, 20 Dec 2003, Another Happy Linux User wrote:

Would built-in file encryption be a foolish thing to wish for some day? (Maybe when the chips are bigger?)

we'll leave that in the hands of the kernel. ron kb2zcw

On Fri, 19 Dec 2003, Stefan Reinauer wrote:

* Eric W. Biederman <ebiederman@lnxi.com> [031219 01:36]:

...

Brainstorming earlier today I think I have found a way to use an linux kernel for the boot loader and to implement pcbios compatibility without too much cost. The idea is to use a uclinux kernel. And implement a ``user space'' aplication that is a user space shim that makes kernel calls.

What functionality is it exactly that we need the uclinux kernel for? We won't do any scheduling, and we won't need source code drivers if we implement a pcbios "emulation" What I liked most about the LinuxBIOS2 approach is that code was only introduced in the tree when it was specifically needed.

I've been assuming that the uclinux approach is for a payload for those who want it. I did not expect to see this in the tree. ron

On 19 Dec 2003, Eric W. Biederman wrote:

The observation has been that any sufficiently general firmware bootloader tends to become an OS, so why not use a real OS.

ah ha! we've come full circle :-) good to hear.

So the things I actually care about are: booting over myrinet, booting over inifinband, booting over quadrics, booting over a pair of bonded nics. Having a good tg3 driver. Getting Lustre as my root filesystem. Unless size issues kill it again I am going to use the linux kernel as my bootloader to do this.

yah, this is why we still put linux in flash here at LANL. ron

On 19 Dec 2003, Eric W. Biederman wrote:

Now if you had more than a proof of concept implementation of most of the pieces I would not be happier, but anyway...

I kind of miss the point. Pink's been booting with linux in ide-flash for a year, and if the flash on pink were large enough, we would have been using that instead. ron

On 19 Dec 2003, Eric W. Biederman wrote:

What Pink does works. I can't throw J. Random kernel at beoboot and have that work. I can't throw memtest86 at it and have that work. I can't use an SMP aware kernel with beoboot. And last I looked beoboot had a bulky user space that makes it hard to squeeze into flash as well. So beoboot with 2 kernel monte works for a certain interesting subset of cases but it does not handle the general case.

ok, gotcha. I don't think I would use beoboot for the general flash case, I think what Andrew Ip set up with kernel+kexec+initrd with some utils is the right model.

Except possibly for the exotic driver case 512KB is a large enough FLASH chip to put in both LinuxBIOS and a kernel. You have to work at it with a 512KB flash chip but it is doable today.

gotcha. thanks ron

Joshua Wise <joshua@joshuawise.com> writes:

On Friday 19 December 2003 4:04 pm, Eric W. Biederman wrote:

...

ron minnich <rminnich@lanl.gov> writes:

...

On 19 Dec 2003, Eric W. Biederman wrote:

...

The observation has been that any sufficiently general firmware bootloader tends to become an OS, so why not use a real OS.

ah ha! we've come full circle :-)

good to hear.

I have never disagreed. Now that we have something that works for the low end limited ROM solutions I can revisit some of the more interesting ideas.

*jumps up and down like a hyperactive two-year-old* LAB can do that stuff!

I would definately like to see LAB used on more than just ARM.

Yes we are reaching the point where we can converge on some of these things. LAB might be the right framework. And if it is something good it will save me the trouble of starting my own project. But it takes more than a hyper active 2 year old to convince me. It might take a hyperactive 2 year old to remind me about interesting ideas though.

...

...

...

So the things I actually care about are: booting over myrinet, booting over inifinband, booting over quadrics, booting over a pair of bonded nics. Having a good tg3 driver. Getting Lustre as my root filesystem. Unless size issues kill it again I am going to use the linux kernel as my bootloader to do this. If you can have it in the kernel you can have it in LAB. LAB's command line API is really easy - I've ported userland apps (mtderase) to LAB in sub-10 minutes.

By and large I don't want a command line interface. I don't want a hardware monitor. I want a configurable but non-interactive boot.

As to size issues, we're at a 512k zImage. We do not have bzImages on ARM, so I could not tell you the size with it.

Compression wise there is not a difference between bzImage and zImage. zImage on x86 has a 640K limit because it loads below 1MB. bzImage breaks that ancient barrier.

512k is with a few ARM-specific drivers, and jffs2. It does not have networking. This is with kernel 2.6.

Hmm. I am pretty certain I have gotten 2.6 down some smaller. Our practical limit with LinuxBIOS etc is in the neighborhood of 384KB.

...

...

yah, this is why we still put linux in flash here at LANL.

Oh and did I say LAB had MTD support? :)

Well I think I have run finally convinced to use the MTD drivers... Mostly I prefer to flash from a production kernel rather than a bootloader, there are more recover options but anyway.

...

Now if you had more than a proof of concept implementation of most of the pieces I would not be happier, but anyway... *jumps up and down* I do I do!!

I will see. Does LAB restrict it's kernel to a very small subset of memory? Or do you use something like kexec? Eric

Joshua Wise <joshua@joshuawise.com> writes:

On Friday 19 December 2003 7:15 pm, Eric W. Biederman wrote:

...

Yes we are reaching the point where we can converge on some of these things. LAB might be the right framework. And if it is something good it will save me the trouble of starting my own project. But it takes more than a hyper active 2 year old to convince me. It might take a hyperactive 2 year old to remind me about interesting ideas though. Right, well then you should see it in action. If you're in the Boston area sometime soon I can give you a demo on an iPAQ, perhaps.

Salt Lake City, and Illinois with my family for Christmas. Though a serial console logfile might be interesting.

...

...

512k is with a few ARM-specific drivers, and jffs2. It does not have networking. This is with kernel 2.6. Hmm. I am pretty certain I have gotten 2.6 down some smaller. Our practical limit with LinuxBIOS etc is in the neighborhood of 384KB. I've done 2.4 in 256k, but it's rather useless like that. If you do not plan to load modules at runtime, you can shave a good bit more off of it. If you write bzip2 compression support (or upport the stuff from kernel 2.4), you can shave even more off of it. I've pulled off 50k with bzip2 (not actually written the code, just did a bzip2 -9 < piggy > piggy.bz2).

The problem is that the bzip2 decompresser is huge, usually bzip2 is a net loss because of the decompresser. But it may be possible to write a tuned version. The cases I have typically worried about are much smaller and I have made huge gains by switching to nrv2b from upx because the decompresser is something like 100 bytes, and the compression is roughly as good as gzip.

If you don't plan to have a framebuffer, you can shave some off of it. If you don't plan to have jffs2 you can shave a lot more off. Little tidbits here and there make the world go 'round.

Quite true.

...

Well I think I have run finally convinced to use the MTD drivers... Mostly I prefer to flash from a production kernel rather than a bootloader, there are more recover options but anyway.

Ah yes, the ancient problem. Instead of read/modify/erase/write, it often turns into read/modify/erase/poweroff. That's Bad. :)

...

I will see. Does LAB restrict it's kernel to a very small subset of memory? Or do you use something like kexec? To boot a secondary kernel I use some code I wrote called armboot, although it's not very arm specific. It does something like this:

1) Load the new kernel into a contiguous vmalloced block. 2) We allocate 64k for a list of things that need to be relocated. We call this a pointer of type "struct physlist", which is 32 bytes. It has four ints: the new address, the old address, the block size, and whether this is the last block. 2) In blocks of the maximum kmalloc size (these blocks have to be contiguous), we kmalloc space for the kernel, and memcpy the kernel into those blocks. We then fill in a struct physlist, and move on to the next struct physlist. We can do this because kmalloc is always contiguous, and we can always map it with virt_to_phys(). 3) We set up another kmalloced block for the tagged list of boot parameters that you need on ARM. 4) We set up one more kmalloced block and copy an assembler function into it, to make sure we don't wipe ourself out while relocating. 5) We flush our data caches. 6) We call the relocated assembler function, which turns off the MMU, jumps into the relocated assembler function's physical address, and does actual relocating. Then we jump into our newly moved zImage. Confused yet?

Nope. Having implemented something similar it sounds sane.

7) If at any point we failed, the system could be in an inconsistent state. You will want to panic() if you fail, because you're leaking memory like a sieve, and if you failed there's probably something bigger wrong.

Ouch.

This looks more difficult than it actually is. The C segment is only about 170 lines, and the assembler bit is 90 lines.

The reason that this works is that kmalloc should allocate from the top of memory down. You need a fair bit of ram - say, 8MB - to prevent the tail from running over other important structures, such as the list of addresses to relocate. But it seems to work well enough, and it looks like it should be fairly portable. The important code is in handhelds.org cvs, module linux/kernel26, files drivers/bootldr/armboot.c and drivers/bootldr/armboot-asm.S.

Ok. I need to get into that kernel tree and take a look. But it sounds similar to my kexec stuff. Which I discuss at least part of the time on fastboot@osdl.org. It sounds compatible enough that we could productively merge implementations, that plus my kexec stuff is still on Andrew Morton todo list ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/must-fix/should-fix-7.txt means it has a fair shot of getting into the stock kernel. On a practical side I think I can boost it's priority high enough after I get back to actually do something. A recent version of kexec patch is at: http://developer.osdl.org/rddunlap/kexec/ Kexec as it is currently structured is actually two system calls callable from user space. sys_kexec_load() load the kernel into a linked list of pages, making certain that when those pages are copied to their final destination nothing will be stomped. And it allocates a chunk of memory with kmalloc for the bit of code that copies the kernel to it's final resting place. This can fail at any time and the system is in a consistent state. sys_reboot(LINUX_REBOOT_CMD_KEXEC) initiates the transfer to the new kernel. The new kernel is started in physical mode. sys_kexec_load() is passed an entry point to jump to, and an array of physical destination address, virtual process space address, and virtual length regions to load. Which allows us to load arbitrary things. The only requirement is that you have enough memory for both kernels simultaneously. For truly high end machines there are some other restrictions because physical mode does not allow access to all of their memory but anyway...

From the descriptions my kexec stuff is a little more general and a little more robust than your armboot, so I'd like to merge your stuff into mine if possible. Now that 2.6.0 is out I can start sending patches again.

- I avoid deliberately avoid vmalloc, the vmalloc area is a limited resource, and I support loading large kernel ramdisk combinations. - I avoid greater than page size memory allocations because random memory fragmentation can make that fail. - I don't leak memory. - I have an implementation that works on SMP machines. The worst part is getting all of the drivers to shut themselves down properly. But I have the appropriate hooks and it doesn't take an extension of the kernel api just lots of driver bug fixes. If even the arm guys are up to using a kernel it should be easier to make progress in this array. On several projects the people I have talked to are two size constrained to even try using a kernel. Eric

Greg Watson <gwatson@lanl.gov> writes:

I would like to see this from the PPC perspective. Since neither filo nor etherboot work on PPC, this might be a good way of booting from, say, a disk. Which would be nice.

I won't argue with that but this will take some time to come to fruition. etherboot should be fairly straight forward to port to the PPC. Currently etherboot supports 3 architectures 4 if you count x86_64. The PPC would not even be the first big endian port. So for a low end solution that works I still recommend etherboot. But if we can avoid the bloat issues the Linux kernel is a good thing to have. Any interest in porting kexec to the ppc? It is not strictly required if you can get your kernel to avoid the memory problem but it is quite useful. Eric

On Fri, 19 Dec 2003, Greg Watson wrote:

I would like to see this from the PPC perspective. Since neither filo nor etherboot work on PPC, this might be a good way of booting from, say, a disk. Which would be nice.

a linux kernel, with a small initrd with modules and kexec, is just about the ideal boot loader setup. cwlinux.com has been selling this configuration for some time. Setting up the same thing on PPC should be easy. We have all the bits we need for a very capable bootloader, and have for 3 years. The only thing missing has been a big enough FLASH part. If the FLASH gets big enough soon enough, our problems will be solved for us. ron