Re: [OpenBIOS] [PATCH 0/8] RFC: OFMEM cleanup/memory reduction patch

[Mark Cave-Ayland]

On 10/04/12 17:43, Blue Swirl wrote:
> > Then we can dramatically reduce the dictionary usage within the file too. I
> > wonder if we could incorporate a simple decompressor based upon something
> > like pdlzip (http://lzip.nongnu.org/pdlzip.html) to perform LZMA
> > decompression/relocation on the dictionary early within the startup process?
>
> ROM image size is not very interesting, but CPU cycles wasted at
> startup are. I think run time memory needs would not be changed by
> compression, the dictionary needs to be uncompressed to memory anyway
> before it can be used.
>
> But we could avoid copying the dictionary when copying ROM to RAM
> initially. Then the dictionary could be copied to OFMEM allocated area
> before Forth start. Some large arrays (FSYS_BUF, dstack, rstack etc.)
> could also be allocated before use. Would this help?

Yes - perhaps I haven't explained this particularly clearly. I'm 
thinking of a 2 stage strategy - the first stage is to copy the 
dictionary to an OFMEM-allocated area on startup. This will save space 
because at the moment the entire dictionary (including space for all the 
definitions used by various bootloaders) is within the OpenBIOS image, 
which is currently 512K in the case of SPARC64. Hence we would only need 
~180K within the image rather than 512K for this.

Secondly, if we're copying the data to an OFMEM-allocated area then why 
don't we just compress it at build time and then decompress it during 
the copy using DEFLATE (see my previous email)? Then we can further 
reduce the dictionary payload size from ~180K to around ~37K, although 
as you rightly point out there could be a small delay on startup - given 
the small size involved (and the fact we can lock the TLB entry like we 
do with the Forth machine), I don't think the penalty will be too bad.

I'm fairly confident I can come up with an experimental patch for this - 
would people mind if we added zlib as a build-time dependency, and 
puff.c to the OpenBIOS codebase? At the very least, if the decompression 
appears too expensive the first stage on its own would still be a good idea.

ATB,

Mark.

[PATCH 0/8] RFC: OFMEM cleanup/memory reduction patch

[Blue Swirl]

On Tue, Apr 10, 2012 at 17:19, Mark Cave-Ayland
 wrote:
> On 10/04/12 17:43, Blue Swirl wrote:
> > > Then we can dramatically reduce the dictionary usage within the file too.
> > > I
> > > wonder if we could incorporate a simple decompressor based upon something
> > > like pdlzip (http://lzip.nongnu.org/pdlzip.html) to perform LZMA
> > > decompression/relocation on the dictionary early within the startup
> > > process?
> >
> > ROM image size is not very interesting, but CPU cycles wasted at
> > startup are. I think run time memory needs would not be changed by
> > compression, the dictionary needs to be uncompressed to memory anyway
> > before it can be used.
> >
> > But we could avoid copying the dictionary when copying ROM to RAM
> > initially. Then the dictionary could be copied to OFMEM allocated area
> > before Forth start. Some large arrays (FSYS_BUF, dstack, rstack etc.)
> > could also be allocated before use. Would this help?
>
> Yes - perhaps I haven't explained this particularly clearly. I'm thinking of
> a 2 stage strategy - the first stage is to copy the dictionary to an
> OFMEM-allocated area on startup. This will save space because at the moment
> the entire dictionary (including space for all the definitions used by
> various bootloaders) is within the OpenBIOS image, which is currently 512K
> in the case of SPARC64. Hence we would only need ~180K within the image
> rather than 512K for this.

Yes, but again, image size savings are not very interesting. Savings
in image loading time is easily lost during decompression.
> Secondly, if we're copying the data to an OFMEM-allocated area then why
> don't we just compress it at build time and then decompress it during the
> copy using DEFLATE (see my previous email)? Then we can further reduce the
> dictionary payload size from ~180K to around ~37K, although as you rightly
> point out there could be a small delay on startup - given the small size
> involved (and the fact we can lock the TLB entry like we do with the Forth
> machine), I don't think the penalty will be too bad.

But only the decompressed result is what matters wrt memory usage,
isn't it? After Forth has started, there shouldn't be any difference
in RAM use, except that the decompressor code takes more space.
> I'm fairly confident I can come up with an experimental patch for this -
> would people mind if we added zlib as a build-time dependency, and puff.c to
> the OpenBIOS codebase? At the very least, if the decompression appears too
> expensive the first stage on its own would still be a good idea.

For maximum compression, bzImage (still remember those?) style
approach could be used for whole image.
> ATB,
>
> Mark.