Re: [LinuxBIOS] patch: extending LAR, and removing elf from linuxbios (it is not needed)

[Carl-Daniel Hailfinger]

On 28.08.2007 20:41, ron minnich wrote:
> Round 2.
>
> I had not realized that lar had changed. So all my patches were
> against the wrong files.
>
> This code is against the right files in lar. I have also done a fair
> amount of refactoring as I needed it.
>
> don't get upset about the close of the fd right after mmap; it's legal
> and even recommended.
>
> Comments welcome as usual.
>
> ron
> p.s. I can not get the LARCHIVE problem to NOT happen now. So, until
> we fix it right, let's take my wrong fix. Three of us have tried and
> failed to fix this, so, let's assume it's not trivial.

I have added LARCHIVE debugging code to your patch below to find out why
and where the problem happens. Could you post the log of your patch with
my added debugging? I hope we can see an obvious fix through that.

Other than that, this patch is Acked by me, but please wait for an
additional ack from somebody else.

Carl-Daniel
> This patch is revised based on comments.
>
> It also includes a demo of how to do a PIC initram, and:
>
> EMERGENCY PATCH! Please see patch to lib/lar.c and include/lar.h for the 
> MAGIC constant. This fixes a bug I hit just now.
>
> This patch also includes an EXPERT option for enabling no-ELF mode. 
> The system will default to old behaviour. See Kconfig in the root.
>
> I still wish to kill ELF mode very soon, however.
>
> LAR is a very capable format. With two simple extensions, we can use
> LAR to replace all that we are using ELF for now. This change can
> really make life better:
> we can use streaming decompress instead of the current "uncompress
>
> elf to memory and then copy segments" approach. So we can get rid of
> THIS hardcode:
> #define UNCOMPRESS_AREA (0x400000)
> 2. A simple lar l can show ALL segments, including payload segments
> 3. It's really easy to see where things will go in memory, and catch problems
> 4. We can figure out an ELF input file is bogus BEFORE we flash, not
> AFTER we flash and try to boot it
> 5. did I mention streaming decompress?
> 6. We no longer have to worry about where we decompress the elf in
> memory (this problem was causing trouble when the payload was a linux
> kernel -- it was so big)
> 7. Since we have a load address, we can create this lar entry:
> normal/cmdline
> and specify that it be loaded at a place where linux will find it as
> the cmdline.
> 8. The decision on whether to XIP can be made in the LAR entry, not in
> hardcode. For example, if initram needs to be XIP, set the load
> address to 0xffffffff. Done.
>
> The change is simple. Add a load address and entry point to the lar
> header. Extend the lar tool to parse the elf file and create multiple
> lar segments. It looks like this:
>  normal/payload0 (33192 bytes, lzma compressed to 18088 bytes @0x38
> load @0x100000, entry 0x105258)
>  normal/payload1 (72 bytes, lzma compressed to 47 bytes @0x4718 load
> @0x1225a0, entry 0x105258)
>  normal/option_table (932 bytes @0x4798 load @0, entry 0)
>  normal/stage2 (33308 bytes, lzma compressed to 15474 bytes @0x4b78
> load @0, entry 0)
>  normal/initram (4208 bytes @0x8828 load @0, entry 0)
>  linuxbios.bootblock (16384 bytes @0xfc000 load @0, entry 0)
>
> note that the payload is now payload/segment0, payload/segment1, etc. I've extended
> linuxbios to look for these. Note that you can now see all the things
> that get loaded ;they're no longer hidden in an ELF header somewhere.
> Elf failures are gone!
>
> Note that I've left legacy elf support in, for now, but recommend we
> get rid of it as soon as possible.
>
> patch attached. This is a first pass. lar.c needs some refactoring but
> I want to get the cmdline going. You can now have a linux payload and
> it will uncompress with no problems.
>
> This has been tested with filo and BOCHS.
>
> Signed-off-by: Ronald G. Minnich 
> Index: Kconfig
> ===================================================================
> --- Kconfig	(revision 480)
> +++ Kconfig	(working copy)
> @@ -53,6 +53,25 @@
>    help
>      Append an extra string to the end of the LinuxBIOS version.
>
> +config NOELF
> bool "Don't use ELF for payloads"
> depends EXPERT
> default n
> help
>  Until now, LinuxBIOS has used elf for the payload. There are many problems 
>  this, not least being the inefficiency -- the ELF has to be decompressed to
>  memory and then the segments have to be copied. Plus, lar can't see the segments 
>  in the elf -- to see all segments, you have to extract the elf and run readelf on it.
>  There are problems with collisions of the decompressed ELF location in memory
>  and the segment locations in memory. 
>  Finally, validation of the ELF is done at run time, once you have flashed the 
>  FLASH and rebooted the machine. Boot time is really not the time you want to find 
>  out your ELF payload is broken. 
>  With this option, LinuxBIOS will direct lar to break each elf segment into a LAR 
>  entry. ELF will not be used at all. Note that (for now) LinuxBIOS is backward
>  compatible -- if you put an ELF payload in, LinuxBIOS can still parse it. We hope
>  to remove ELF entirely in the future. 
>
> config BEEPS
>    bool "Enable beeps upon certain LinuxBIOS events"
>    depends EXPERT
> Index: include/lar.h
> ===================================================================
> --- include/lar.h	(revision 480)
> +++ include/lar.h	(working copy)
> @@ -52,9 +52,10 @@
>
> #include <types.h>
>
> -#define MAGIC "LARCHIVE"
> +/* see note in lib/lar.c as to why this is ARCHIVE and not LARCHIVE */
> +#define MAGIC "ARCHIVE"
>  #define MAX_PATHLEN 1024
>
> +/* NOTE -- This and the user-mode lar.h are NOT IN SYNC. Be careful. */
>  struct lar_header {
>    char magic[8];
>    u32 len;
> @@ -62,7 +63,14 @@
>    u32 checksum;
>    u32 compchecksum;
>    u32 offset;
> /* Compression:
> * 0 = no compression
> * 1 = lzma
> * 2 = nrv2b
> */
> u32 compression;
> u32 entry;		/* we might need to make this u64 */
> u32 loadaddress; /* ditto */
>
> };
>
> struct mem_file {
> @@ -70,6 +78,8 @@
>    int len;
>    u32 reallen;
>    u32 compression;
> void *entry;
> void *loadaddress;
>
> };
>
> /* Prototypes. */
> @@ -77,5 +87,6 @@
>  int copy_file(struct mem_file *archive, char *filename, void *where);
>  int run_file(struct mem_file *archive, char *filename, void *where);
>  int execute_in_place(struct mem_file *archive, char *filename);
>
> +int run_address(void *f);
> +void *load_file(struct mem_file *archive, char *filename);
>  #endif /* LAR_H */
> Index: mainboard/emulation/qemu-x86/initram.c
> ===================================================================
> --- mainboard/emulation/qemu-x86/initram.c	(revision 480)
> +++ mainboard/emulation/qemu-x86/initram.c	(working copy)
> @@ -19,10 +19,17 @@
>
> #include <console.h>
>
> +/* This is how we force an absolute call when we are in PIC code (which this file is)
> If there is a nicer way to do this, I'd like to hear it. Ideally, we could tell
> gcc to force abs jumps on certain symbols.
> */
>
> +int (*pk)(int msg_level, const char *fmt, ...) = printk;
>
> int main(void)
>  {
> printk(BIOS_INFO, "RAM init code started.\n");
> printk(BIOS_INFO, "Nothing to do.\n");
>
> pk(BIOS_INFO, "RAM init code started.\n");
>
> pk(BIOS_INFO, "Nothing to do.\n");
>
> return 0;
>
> }
> Index: mainboard/emulation/qemu-x86/Makefile
> ===================================================================
> --- mainboard/emulation/qemu-x86/Makefile	(revision 480)
> +++ mainboard/emulation/qemu-x86/Makefile	(working copy)
> @@ -41,14 +41,17 @@
>  #
>
> INITRAM_OBJ = $(obj)/mainboard/$(MAINBOARDDIR)/initram.o
> +$(obj)/mainboard/$(MAINBOARDDIR)/initram.o: $(src)/mainboard/$(MAINBOARDDIR)/initram.c
> cc -c $(INITCFLAGS)  -fPIE $(src)/mainboard/$(MAINBOARDDIR)/initram.c -o $(obj)/mainboard/$(MAINBOARDDIR)/initram.o
>
> # These are possibly not permanent
> -INITRAM_OBJ += $(obj)/lib/console.o $(obj)/lib/vtxprintf.o $(obj)/lib/uart8250.o $(obj)/arch/x86/post_code.o
> +#INITRAM_OBJ += $(obj)/lib/console.o $(obj)/lib/vtxprintf.o $(obj)/lib/uart8250.o $(obj)/arch/x86/post_code.o
>
> $(obj)/linuxbios.initram: $(obj)/stage0.init $(obj)/stage0.o $(INITRAM_OBJ)
>    $(Q)# initram links against stage0
>    $(Q)printf "  LD      $(subst $(shell pwd)/,,$(@))\n"
> $(Q)$(LD)  -Ttext 0x80000 $(INITRAM_OBJ) \
> --entry=main -o $(obj)/linuxbios.initram.o
> $(Q)$(LD)  $(INITRAM_OBJ) \
> --entry=main -R $(obj)/stage0.o -o $(obj)/linuxbios.initram.o
> $(Q)printf "  OBJCOPY $(subst $(shell pwd)/,,$(@))\n"
> $(Q)$(OBJCOPY) -O binary $(obj)/linuxbios.initram.o \
> $(obj)/linuxbios.initram
> Index: lib/lar.c
>
> --- lib/lar.c	(revision 480)
> +++ lib/lar.c	(working copy)
> @@ -31,6 +31,13 @@
>  #define ntohl(x) (x)
>  #endif
>
> +int run_address(void *f)
> +{
> int (*v) (void);
> v = f;
> return v();
>
> +}
>
> int find_file(struct mem_file *archive, char *filename, struct mem_file *result)
>  {
>    char *walk, *fullname;
> @@ -42,29 +49,103 @@
>
> for (walk = archive->start;
>         (walk - 1) < (char *)(archive->start + archive->len - 1 ); walk += 16) {
> if (strcmp(walk, MAGIC) != 0)
> /* I am leaving this code in here because it is so dangerous. MAGIC is
>  * #define'd to a string. That string lives in data space. All of the 1M linuxbios 
>  * image is a LAR file. Therefore, this search can walk ALL of linuxbios. 
>  * IF the MAGIC string (in code space) just happens to be 16-byte aligned, 
>  * Then the strcmp will succeed, and you will match a non-LAR entry, 
>  * and you are screwed. can this happen? YES!
>  * LAR: Attempting to open 'fallback/initram'.
>  * LAR: Start 0xfff00000 len 0x100000
>  * LAR: current filename is normal/payload
>  * LAR: current filename is normal/option_table
>  * LAR: current filename is normal/stage2
>  * LAR: current filename is normal/initram
>  * LAR: current filename is R: it matches %s @ %p
>  * That garbage is there because the pointer is in the middle of a bunch 
>  * of non-null-terminated junk. The fix is easy, as you can see. 
>  * if (strcmp(walk, MAGIC) != 0)
>  *	continue;
>  * And, yes, this did indeed fix the problem!
>  */
> if (walk[0] != 'L')
> continue;
>
> if (strcmp(&walk[1], MAGIC) != 0)
> 	continue;

Maybe change the code above to add add extra debugging code:
> printk(BIOS_SPEW, "LAR: looking for archive header at %p", walk);
> if (walk[0] != 'L') {
> 	printk(BIOS_SPEW, "LAR: no L magic at %p", walk);
> continue;
> }
>
> if (strcmp(&walk[1], MAGIC) != 0) {
> 	printk(BIOS_SPEW, "LAR: no ARCHIVE magic at %p", walk);
> 	continue;
> }
> printk(BIOS_SPEW, "LAR: full magic at %p", walk);
> header = (struct lar_header *)walk;
> fullname = walk + sizeof(struct lar_header);
>
> printk(BIOS_SPEW, "LAR: current filename is %s\n", fullname);
> printk(BIOS_SPEW, "LAR: search for %s\n", fullname);
>
> // FIXME: check checksum
>
> if (strcmp(fullname, filename) == 0) {
> 	printk(BIOS_SPEW, "LAR: FOUND %s @ %p\n", fullname, header);
> result->start = walk + ntohl(header->offset);
> result->len = ntohl(header->len);
> result->reallen = ntohl(header->reallen);
> result->compression = ntohl(header->compression);
> 	result->entry = (void *)ntohl(header->entry);
> 	result->loadaddress = (void *)ntohl(header->loadaddress);
> 	printk(BIOS_SPEW, 
> 	"start %p len %d reallen %d compression %x entry %p loadaddress %p\n", 
> 		result->start, result->len, result->reallen, 
> 		result->compression, result->entry, result->loadaddress);
> return 0;
>
> }
> // skip file
> walk += (ntohl(header->len) + ntohl(header->offset) -
> 	1) & 0xfffffff0;
> }
>
> printk(BIOS_SPEW, "NO FILE FOUND\n");
> return 1;
>
> }
>
> +void *load_file(struct mem_file *archive, char *filename)
> +{
> int ret;
> struct mem_file result;
> void *where;
> void *entry;
> ret = find_file(archive, filename, &result);
> if (ret) {
> printk(BIOS_INFO, "LAR: load_file: No such file '%s'\n",
>        filename);
> return (void *)-1;
> }
> entry = result.entry;
> where = result.loadaddress;
> printk(BIOS_SPEW, "LAR: Compression algorithm #%i used\n", result.compression);
> /* no compression */
> if (result.compression == 0) {
> memcpy(where, result.start, result.len);
> return entry;
> }
>
> +#ifdef CONFIG_COMPRESSION_LZMA
> /* lzma */
> unsigned long ulzma(unsigned char *src, unsigned char *dst);
> if (result.compression == 1) {
> ulzma(result.start, where);
> return entry;
> }
>
> +#endif
> +#ifdef CONFIG_COMPRESSION_NRV2B
> /* nrv2b */
> unsigned long unrv2b(u8 *src, u8 *dst, unsigned long *ilen_p);
> if (result.compression == 2) {
> int tmp;
> unrv2b(result.start, where, &tmp);
> return entry;
> }
>
> +#endif
> printk(BIOS_INFO, "LAR: Compression algorithm #%i not supported!\n", result.compression);
> return (void *)-1;
>
> +}
>
> +/* FIXME -- most of copy_file should be replaced by load_file */
>  int copy_file(struct mem_file *archive, char *filename, void *where)
>  {
>    int ret;
> @@ -85,7 +166,7 @@
>    }
>  #ifdef CONFIG_COMPRESSION_LZMA
>    /* lzma */
> unsigned long ulzma(unsigned char * src, unsigned char * dst);
> unsigned long ulzma(unsigned char *src, unsigned char *dst);
> if (result.compression == 1) {
> ulzma(result.start, where);
> return 0;
>
> @@ -93,7 +174,7 @@
>  #endif
>  #ifdef CONFIG_COMPRESSION_NRV2B
>    /* nrv2b */
> unsigned long unrv2b(u8 * src, u8 * dst, unsigned long *ilen_p);
> unsigned long unrv2b(u8 *src, u8 *dst, unsigned long *ilen_p);
> if (result.compression == 2) {
> int tmp;
> unrv2b(result.start, where, &tmp);
>
> @@ -113,6 +194,7 @@
>  {
>    int (*v) (void);
>    struct mem_file result;
>
> int ret;
>
> if ((u32) where != 0xFFFFFFFF) {
> if (copy_file(archive, filename, where)) {
>
> @@ -130,9 +212,11 @@
>    	}
>    	where = result.start;
>    }
> printk(BIOS_SPEW, "where is %p\n", where);
> v = where;
> return v();
> ret =  v();
> printk(BIOS_SPEW, "run_file returns with %d\n", ret);
> return ret;
>
> }
>
> /**
> Index: lib/stage2.c
> ===================================================================
> --- lib/stage2.c	(revision 480)
> +++ lib/stage2.c	(working copy)
> @@ -99,5 +99,6 @@
>    write_tables();
>    show_all_devs();
> printk(BIOS_SPEW, "STAGE2 NOW RETURNING\n");
> return 0;
>
> }
> Index: arch/x86/stage1.c
> ===================================================================
> --- arch/x86/stage1.c	(revision 480)
> +++ arch/x86/stage1.c	(working copy)
> @@ -22,12 +22,16 @@
>  #include <io.h>
>  #include <console.h>
>  #include <lar.h>
> +#include <string.h>
>  #include <tables.h>
>  #include <lib.h>
>  #include <mc146818rtc.h>
>  #include <post_code.h>
>
> -#define UNCOMPRESS_AREA 0x60000
> +/* ah, well, what a mess! This is a hard code. FIX ME but how?
> By getting rid of ELF ...
> */
>
> +#define UNCOMPRESS_AREA (0x400000)
>
> /* these prototypes should go into headers */
>  void uart_init(void);
> @@ -48,6 +52,24 @@
>    post_code(0xf2);
>  }
>
> +/* until we get rid of elf */
> +int legacy(struct mem_file *archive, char *name, void *where, struct lb_memory *mem)
> +{
> int ret;
> struct mem_file result;
> int elfboot_mem(struct lb_memory *mem, void *where, int size);
> ret = copy_file(archive, name, where);
> if (ret) {
> printk(BIOS_ERR, "'%s' found, but could not load it.\n", name);
> }
> ret =  elfboot_mem(mem, where, result.reallen);
> printk(BIOS_ERR, "elfboot_mem returns %d\n", ret);
> return -1;
>
> +}
>
> /*
> This function is called from assembler code whith its argument on the
> stack. Force the compiler to generate always correct code for this case.
>
> @@ -57,6 +79,8 @@
>    int ret;
>    struct mem_file archive, result;
>    int elfboot_mem(struct lb_memory *mem, void *where, int size);
>
> void *entry;
>
> int i;
>
> /* we can't statically init this hack. */
> unsigned char faker[64];
>
> @@ -144,20 +168,32 @@
>    printk(BIOS_DEBUG, "Stage2 code done.\n");
>
> ret = find_file(&archive, "normal/payload", &result);
> if (ret) {
> printk(BIOS_ERR, "No such file '%s'.\n", "normal/payload");
> die("FATAL: No payload found.\n");
> if (! ret)
> legacy(&archive, "normal/payload", (void *)UNCOMPRESS_AREA, mem);
> /* new style lar boot. Install all the files in memory.
>  * By convention we take the entry point from the first 
>  * one. Look for a cmdline as well. 
>  */
> for(i = 0, entry = (void *)0; ;i++) {
> char filename[64];
> void *newentry;
> sprintf(filename, "normal/payload/segment%d", i);
> archive.len = *(u32 *)0xfffffff4;
> archive.start =(void *)(0UL-archive.len);
> newentry = load_file(&archive, filename);
> printk("newentry is %p\n", newentry);
> if (newentry == (void *)-1)
> 	break;
> if (! entry)
> 	entry = newentry;
> }
> ret = copy_file(&archive, "normal/payload", (void *)UNCOMPRESS_AREA);
> if (ret) {
> printk(BIOS_ERR, "'%s' found, but could not load it.\n", "normal/payload");
> die("FATAL: No usable payload found.\n");
> }
> printk(BIOS_SPEW, "all loaded, entry %p\n", entry);
> run_address(entry);
> ret =  elfboot_mem(mem, (void *)UNCOMPRESS_AREA, result.reallen);
> die("FATAL: No usable payload found.\n");
> printk(BIOS_ERR, "elfboot_mem returns %d\n", ret);
> die ("FATAL: Last stage returned to LinuxBIOS.\n");
>
> }
> Index: arch/x86/Makefile
>
> --- arch/x86/Makefile	(revision 480)
> +++ arch/x86/Makefile	(working copy)
> @@ -22,7 +22,7 @@
>  ifeq ($(CONFIG_ARCH_X86),y)
>
> INITCFLAGS := $(CFLAGS) -I$(src)/include/arch/x86 -I$(src)/include \
> -I$(obj) -fno-builtin
> -I$(obj) -fno-builtin
>
> SILENT := >/dev/null 2>&1
>
> @@ -78,7 +78,7 @@
>  endif
>    $(Q)printf "  LAR     $(subst $(shell pwd)/,,$(@))\n"
>    $(Q)rm -f $(obj)/linuxbios.rom
> $(Q)cd $(obj)/lar.tmp && ../util/lar/lar $(COMPRESSFLAG) -c \
> $(Q)cd $(obj)/lar.tmp && ../util/lar/lar $(PARSEELF) $(COMPRESSFLAG) -c \
> ../linuxbios.rom \
> $(LARFILES) \
> -s $(ROM_SIZE) -b $(obj)/linuxbios.bootblock
>
> @@ -122,6 +122,11 @@
>  endif
>  endif
>
> +ifeq ($(CONFIG_NOELF), y)
> PARSEELF = -e
>
> +else
> PARSEELF =
>
> +endif
>
> STAGE0_OBJ := $(patsubst %,$(obj)/lib/%,$(STAGE0_LIB_OBJ)) \
>          $(patsubst %,$(obj)/arch/x86/%,$(STAGE0_ARCH_X86_OBJ)) \
> @@ -143,6 +148,9 @@
>    $(Q)printf "  OBJCOPY $(subst $(shell pwd)/,,$(@))\n"
>    $(Q)$(OBJCOPY) -O binary $(obj)/stage0.o $(obj)/stage0.init
> $(Q)printf " OBJCOPY (stage0 link) $(subst $(shell pwd)/,,$(@))\n"
> $(Q)$(OBJCOPY) --prefix-symbols=stage0 $(obj)/stage0.o $(obj)/stage0_link.o
> $(Q)printf "  TEST    $(subst $(shell pwd)/,,$(@))\n"
>  $(Q)test `wc -c < $(obj)/stage0.init` -gt 16128 && \
>   printf "Error. Bootblock got too big.\n" || true
> Index: util/lar/lar.c
>
> --- util/lar/lar.c	(revision 480)
> +++ util/lar/lar.c	(working copy)
> @@ -30,13 +30,14 @@
>  #include <sys/stat.h>
>  #include <sys/mman.h>
>
> +#include "lar.h"
>  #include "lib.h"
> -#include "lar.h"
>
> #define VERSION "v0.9.1"
>  #define COPYRIGHT "Copyright (C) 2006-2007 coresystems GmbH"
>
> static int isverbose = 0;
> +static int iselfparse = 0;
>  static long larsize = 0;
>  static char *bootblock = NULL;
>  enum compalgo algo = none;
> @@ -44,7 +45,7 @@
>  static void usage(char *name)
>  {
>    printf("\nLAR - the LinuxBIOS Archiver " VERSION "\n" COPYRIGHT "\n\n"
>       "Usage: %s [-cxal] archive.lar [[[file1] file2] ...]\n\n", name);
>       "Usage: %s [-ecxal] archive.lar [[[file1] file2] ...]\n\n", name);
> printf("Examples:\n");
>  printf("  lar -c -s 32k -b bootblock myrom.lar foo nocompress:bar\n");
>  printf("  lar -a myrom.lar foo blob:baz\n");
>
> @@ -66,13 +67,18 @@
>    printf("                \ta 'm' suffix to multiple the size by 1024*1024.\n");
>    printf("  -b [bootblock]\tSpecify the bootblock blob\n");
>    printf("  -C [lzma|nrv2b]\tSpecify the compression method to use\n\n");
>
> printf("  -e pre-parse the payload ELF into LAR segments. Recommended\n\n");
>
> printf("General options\n");
> printf("  -v\tEnable verbose mode\n");
> printf("  -V\tShow the version\n");
> printf("  -h\tShow this help\n");
> printf("\n");
>
> +}
>
> +int elfparse(void)
> +{
> return iselfparse;
>
> }
>
> /* Add a human touch to the LAR size by allowing suffixes:
> @@ -209,6 +215,7 @@
>    	{"list", 0, 0, 'l'},
>    	{"size", 1, 0, 's'},
>    	{"bootblock", 1, 0, 'b'},
> {"elfparse", 1, 0, 'e'},
> {"verbose", 0, 0, 'v'},
> {"version", 0, 0, 'V'},
> {"help", 0, 0, 'h'},
>
> @@ -220,7 +227,7 @@
>    	exit(1);
>    }
> while ((opt = getopt_long(argc, argv, "acC:xls:b:vVh?",
> while ((opt = getopt_long(argc, argv, "acC:xels:b:vVh?",
> 		  long_options, &option_index)) != EOF) {
> switch (opt) {
> case 'a':
>
> @@ -240,6 +247,9 @@
>    	case 'l':
>    		larmode = LIST;
>    		break;
> case 'e':
> 	iselfparse = 1;
> 	break;
> case 'x':
> 	larmode = EXTRACT;
> 	break;
> Index: util/lar/lar.h
>
> --- util/lar/lar.h	(revision 480)
> +++ util/lar/lar.h	(working copy)
> @@ -60,6 +60,7 @@
>  typedef uint32_t u32;
>  typedef uint8_t  u8;
>
> +/* NOTE -- This and the linuxbios lar.h are NOT IN SYNC. Be careful. */
>  struct lar_header {
>    char magic[8];
>    u32 len;
> @@ -73,6 +74,8 @@
>     * 2 = nrv2b
>     */
>    u32 compression;
> u32 entry;		/* we might need to make this u64 */
> u32 loadaddress; /* ditto */
>
> };
>
> /**\struct
> Index: util/lar/lib.h
> ===================================================================
> --- util/lar/lib.h	(revision 480)
> +++ util/lar/lib.h	(working copy)
> @@ -38,6 +38,7 @@
>
> /* prototypes for lar.c functions */
>  int verbose(void);
> +int elfparse(void);
>  long get_larsize(void);
>  char *get_bootblock(void);
>
> @@ -50,13 +51,24 @@
>  struct file *get_files(void);
>  void free_files(void);
>
> +/* Prototypes for ELF functions */
> +int iself(char *filebuf);
>
> +/* Prototypes for in-memory LAR operations */
> +int lar_process_name(char *name, char **pfilename, char **ppathname,
> enum compalgo *thisalgo);
>
> +u32 lar_compress(char *ptr, ssize_t size, char *temp, enum compalgo *thisalgo);
> +int lar_add_entry(struct lar *lar, char *pathname, void *data,
> u32 complen, u32 reallen, u32 loadaddress, u32 entry,
> enum compalgo thisalgo);
>
> /* Prototypes for the LAR I/O functions */
> +char *mapfile(char *filename, u32 *size);
>  struct lar * lar_new_archive(const char *archive, unsigned int size);
>  struct lar * lar_open_archive(const char *archive);
>  void lar_close_archive(struct lar *lar);
>
> void lar_list_files(struct lar *lar, struct file *files);
> -int lar_add_file(struct lar *lar, const char *name);
> +int lar_add_file(struct lar *lar, char *name);
>  int lar_add_bootblock(struct lar *lar, const char *bootblock);
>  int lar_extract_files(struct lar *lar, struct file *files);
> Index: util/lar/stream.c
>
> --- util/lar/stream.c	(revision 480)
> +++ util/lar/stream.c	(working copy)
> @@ -32,9 +32,10 @@
>  #include <sys/mman.h>
>  #include <netinet/in.h>
>  #include <libgen.h>
> +#include <elf.h>
>
> +#include "lar.h"
>  #include "lib.h"
> -#include "lar.h"
>
> /**
> \def err(fmt,args...)
>
> @@ -44,7 +45,126 @@
>
> extern enum compalgo algo;
>
> +/* ELF processing */
>  /**
> Given a ptr to data, determine if the data is an ELF image.
> @param filebuf pointer to the data
> @return 1 if ELF, 0 if not
> */
>
> +int iself(char *filebuf)
> +{
> Elf32_Ehdr *ehdr;
> /* validate elf header */
> ehdr = (Elf32_Ehdr *)filebuf;
> if (memcmp(ehdr->e_ident, ELFMAG, 4))
> return 0;
> return 1;
>
> +}
>
> +/**
> Output all the ELF segments for a given file
> @param lar The LAR Archoe
> @param name The LAR name
> @param filebuf The ELF file
> @param filelen Size of the ELF file
> @param algo The recommend compression algorithm
> Return 0 on success, -1 on failure
> */
>
> +int output_elf_segments(struct lar *lar, char *name, char *filebuf,
> 	int filelen, enum compalgo algo)
>
> +{
> int ret;
>    Elf32_Phdr *phdr;
> Elf32_Ehdr *ehdr;
> u32 entry;
> int i;
> int size;
> unsigned char *header;
> char ename[64];
> int headers;
> char *temp;
> enum compalgo thisalgo;
> u32 complen;
> /* Allocate a temporary buffer to compress into - this is unavoidable,
>   because we need to make sure that the compressed data will fit in
>   the LAR, and we won't know the size of the compressed data until
>   we actually compress it */
> temp = calloc(filelen, 1);
> if (temp == NULL) {
> err("Out of memory.\n");
> return -1;
> }
> /* validate elf header */
> ehdr = (Elf32_Ehdr *)filebuf;
> headers = ehdr->e_phnum;
> header  = (unsigned char *)ehdr;
> if (verbose())
> fprintf(stderr, "Type %d machine %d version %d entry %p phoff %d shoff %d flags %#x hsize %d phentsize %d phnum %d s_hentsize %d s_shnum %d \n", 
> ehdr->e_type,
> ehdr->e_machine,
> ehdr->e_version,
> (void *)ehdr->e_entry,
> ehdr->e_phoff,
> ehdr->e_shoff,
> ehdr->e_flags,
> ehdr->e_ehsize,
> ehdr->e_phentsize,
> ehdr->e_phnum,
> ehdr->e_shentsize,
> ehdr->e_shnum);
>    phdr = (Elf32_Phdr *)&(header[ehdr->e_phoff]);
> if (verbose())
> fprintf(stderr, "%s: header %p #headers %d\n", __FUNCTION__, ehdr, headers);
> entry = ehdr->e_entry;
> for(i = 0; i < headers; i++) {
> /* Ignore data that I don't need to handle */
> if (phdr[i].p_type != PT_LOAD) {
> 	if (verbose())
> 		fprintf(stderr, "Dropping non PT_LOAD segment\n");
> 	continue;
> }
> if (phdr[i].p_memsz == 0) {
> 	if (verbose())
> 		fprintf(stderr, "Dropping empty segment\n");
> 	continue;
> }
> thisalgo = algo;
> if (verbose())
> 	fprintf(stderr,  "New segment addr 0x%ulx size 0x%ulx offset 0x%ulx filesize 0x%ulx\n",
> 	(u32)phdr[i].p_paddr, (u32)phdr[i].p_memsz, 
> 	(u32)phdr[i].p_offset, (u32)phdr[i].p_filesz);
> /* Clean up the values */
> size = phdr[i].p_filesz;
> if (phdr[i].p_filesz > phdr[i].p_memsz)  {
> 	size = phdr[i].p_memsz;
> }
> if (verbose()) {
> 	fprintf(stderr, "(cleaned up) New segment addr %p size 0x%#x offset 0x%x\n",
> 		(void *)phdr[i].p_paddr, size, phdr[i].p_offset);
> 	fprintf(stderr, "Copy to %p from %p for %d bytes\n", 
> 		(unsigned char *)phdr[i].p_paddr, 
> 		&header[phdr[i].p_offset], size);
> 	fprintf(stderr, "entry %ux loadaddr %ux\n", 
> 		entry,  phdr[i].p_paddr);
> }
> 	/* ok, copy it out */
> sprintf(ename, "%s/segment%d", name, i);
> complen = lar_compress(filebuf, size, temp, &thisalgo);
> ret = lar_add_entry(lar, ename, &header[phdr[i].p_offset], 
> 		    complen, size, 
> 		    phdr[i].p_paddr, entry, thisalgo);
> }
> return 0;
>
> +out:
> return -1;
>
> +}
>
> +/**
> Given a size, return the offset of the bootblock (including the
> header)
> @param size Size of the LAR archive
>
> @@ -259,7 +379,7 @@
>    }
>
> return lar;
> err:
>
> +err:
>    lar_close_archive(lar);
>
> /* Don't leave a halfbaked LAR laying around */
> @@ -315,7 +435,7 @@
>
> return lar;
> err:
>
> +err:
>    lar_close_archive(lar);
>    return NULL;
>  }
> @@ -535,110 +655,139 @@
>  }
>
> /**
> Add a new file to the LAR archive
> @param lar The LAR archive to write into
> @param name The name of the file to add
> @return  0 on success, or -1 on failure
> Given a pathname in the form [option;]path, determine the file name,
> LAR path name, and compression algorithm.
> @param name name in the [option:][./]path form
> @param pfilename reference pointer to file name -- this is modified
> @param ppathname reference pointer to LAR path name -- this is modified
> @param thisalgo pointer to algorithm, which can be modified by path name
> @return 0 success, or -1 on failure (i.e. a bad name)
> */
>
> -int lar_add_file(struct lar *lar, const char *name)
> +int lar_process_name(char *name, char **pfilename, char **ppathname,
>      enum compalgo *thisalgo)
>
> {
> char *filename, *ptr, *temp;
> char *pathname;
> char *filename = name, *pathname = name;
> enum compalgo thisalgo;
> struct lar_header *header;
> u32 offset;
> int ret, fd, hlen;
> u32 complen;
> int pathlen;
> struct stat s;
> u32 *walk,  csum;
> /* Find the beginning of the available space in the LAR */
> offset = lar_empty_offset(lar);
> thisalgo = algo;
> filename = (char *) name;
> if (!strncmp(name, "nocompress:",11)) {
>   filename += 11;
> thisalgo = none;
> *thisalgo = none;
>
> }
>
> /* this is dangerous */
> if (filename[0] == '.' && filename[1] == '/')
> filename += 2;
>
> pathname = strchr(filename, ':');
>
> if (pathname != NULL) {
>  *pathname = '\0';
>  pathname++;
> *pathname = '\0';
> pathname++;
>
>  if (!strlen(pathname)) {
>    err("Invalid pathname specified.\n");
>    return -1;
>  }
> if (!strlen(pathname)) {
> 	err("Invalid pathname specified.\n");
> 	return -1;
> }
> }
> else
> pathname = filename;
> *pfilename = filename;
> *ppathname = pathname;
> return 0;
>
> +}
>
> +/**
> Given a pathname, open and mmap the file.
> @param filename
> @param size pointer to returned size
> @return ptr to mmap'ed area on success, or MAP_FAILED on failure
> */
>
> +char *mapfile(char *filename, u32 *size)
> +{
>
> int fd;
>
> struct stat s;
>
> char *ptr;
>
> /* Open the file */
>  fd = open(filename, O_RDONLY);
>
> if (fd == -1) {
>   err("Unable to open %s\n", filename);
> return -1;
> return MAP_FAILED;
>
> }
>
> if (fstat(fd, &s)) {
> err("Unable to stat the file %s\n", filename);
> close(fd);
> return -1;
> return MAP_FAILED;
> }
> /* Allocate a temporary buffer to compress into - this is unavoidable,
>   because we need to make sure that the compressed data will fit in
>   the LAR, and we won't know the size of the compressed data until
>   we actually compress it */
> temp = calloc(s.st_size, 1);
> if (temp == NULL) {
> err("Out of memory.\n");
> return -1;
> }
> ptr = mmap(0, s.st_size, PROT_READ, MAP_SHARED, fd, 0);
>
> close(fd);
>
> if (ptr == MAP_FAILED) {
> err("Unable to map %s\n", filename);
> close(fd);
> free(temp);
> return -1;
> return MAP_FAILED;
> }
> *size = s.st_size;
> return ptr;
>
> +}
>
> +/**
> Compress an area according to an algorithm. If the area grows,
> use no compression.
> @param ptr data to be compressed
> @param size size of the data
> @param temp destination of compressed data
> @param thisalgo pointer to algorithm -- this can be modified
> @return  size of compressed data
> */
>
> +u32 lar_compress(char *ptr, ssize_t size, char *temp, enum compalgo *thisalgo)
> +{
> u32 complen;
> compress_functions[*thisalgo](ptr, size, temp, &complen);
> /* Do the compression step */
> compress_functions[thisalgo](ptr, s.st_size, temp, &complen);
> if (complen >= s.st_size && (thisalgo != none)) {
> thisalgo = none;
> compress_functions[thisalgo](ptr, s.st_size, temp, &complen);
> if (complen >= size && (*thisalgo != none)) {
> *thisalgo = none;
> compress_functions[*thisalgo](ptr, size, temp, &complen);
> }
> return complen;
>
> +}
> munmap(ptr, s.st_size);
> close(fd);
>
> +/**
> Add a new entry to the LAR archive
> @param lar The LAR archive to write into
> @param pathname The name of the segment
> @param data The data for the segment
> @param complen The compressed length of the segment
> @param reallen The real (uncompressed) length of the segment
> @param loadaddress The load address of the segment
> @param entry The entry point of the segment
> @param thisalgo The compression algorithm
> @return  0 on success, or -1 on failure
> */
>
> +int lar_add_entry(struct lar *lar, char *pathname, void *data,
>   u32 complen, u32 reallen, u32 loadaddress, u32 entry, 
>   enum compalgo thisalgo)
>
> +{
> struct lar_header *header;
> int ret, hlen;
> int pathlen;
> u32 *walk,  csum;
> u32 offset;
> pathlen = strlen(pathname) + 1 > MAX_PATHLEN ? MAX_PATHLEN : strlen(pathname) + 1;
>
> /* Find the beginning of the available space in the LAR */
>
> offset = lar_empty_offset(lar);
>
> pathlen = strlen(pathname) + 1 > MAX_PATHLEN ?
> MAX_PATHLEN : strlen(pathname) + 1;
>
> /* Figure out how big our header will be */
>  hlen = sizeof(struct lar_header) + pathlen;
>  hlen = (hlen + 15) & 0xFFFFFFF0;
>
> if (offset + hlen + complen >= get_bootblock_offset(lar->size)) {
>   err("Not enough room in the LAR to add the file.\n");
> free(temp);
> return -1;
> }
>
> @@ -651,16 +800,17 @@
>
> memcpy(header, MAGIC, 8);
>    header->compression = htonl(thisalgo);
> header->reallen = htonl(s.st_size);
> header->reallen = htonl(reallen);
> header->len = htonl(complen);
> header->offset = htonl(hlen);
>
> header->loadaddress = htonl(loadaddress);
>
> header->entry = htonl(entry);
> /* Copy the path name */
> strncpy((char *) (lar->map + offset + sizeof(struct lar_header)),
> pathname, pathlen - 1);
>
> /* Copy in the data */
> memcpy(lar->map + (offset + hlen), temp, complen);
>
> memcpy(lar->map + (offset + hlen), data, complen);
>
> /* Figure out the checksum */
>
> @@ -671,7 +821,58 @@
>    	csum += ntohl(*walk);
>    }
>    header->checksum = htonl(csum);
> return 0;
>
> +}
>
> +/**
> Add a new file to the LAR archive
> @param lar The LAR archive to write into
> @param name The name of the file to add
> @return  0 on success, or -1 on failure
> */
>
> +int lar_add_file(struct lar *lar, char *name)
> +{
> char *filename, *ptr, *temp;
> char *pathname;
> enum compalgo thisalgo;
> struct lar_header *header;
> int ret, hlen;
> u32 complen;
> int pathlen;
> u32 size;
> thisalgo = algo;
> lar_process_name(name, &filename, &pathname, &thisalgo);
> ptr = mapfile(filename, &size);
> if (elfparse() && iself(ptr)) {
> output_elf_segments(lar, pathname, ptr, size, thisalgo);
> return 0;
> }
> /* This is legacy stuff. */
> /* Allocate a temporary buffer to compress into - this is unavoidable,
>   because we need to make sure that the compressed data will fit in
>   the LAR, and we won't know the size of the compressed data until
>   we actually compress it */
> temp = calloc(size, 1);
> if (temp == NULL) {
> err("Out of memory.\n");
> munmap(ptr, size);
> return -1;
> }
> complen = lar_compress(ptr, size, temp, &thisalgo);
> munmap(ptr, size);
> ret = lar_add_entry(lar, pathname, temp, complen, size, 0, 0, thisalgo);
> free(temp);
> return 0;
> return ret;
>
> }
> Index: util/lar/Makefile
> ===================================================================
> --- util/lar/Makefile	(revision 480)
> +++ util/lar/Makefile	(working copy)
> @@ -17,7 +17,6 @@
>  ## along with this program; if not, write to the Free Software
>  ## Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA, 02110-1301 USA
>  ##
>
> LAROBJ := lar.o stream.o lib.o
>
> LARDIR := lardir
> @@ -29,7 +28,6 @@
>  LARDIR     += nrv2bdir
>
> LAROBJ_ABS := $(patsubst %,$(obj)/util/lar/%,$(LAROBJ))
>
> lardir:
>    $(Q)printf "  BUILD   LAR\n"
>    $(Q)mkdir -p $(obj)/util/lar