It's taken me a lot longer to get to grips with moving SPARC32 to OFMEM than expected, mostly due to problems with the increased BSS size of the resulting image causing several large headaches.
Having spent some time looking at where the memory is currently being used, it strikes me that there are 2 main places where we can claim some back:
i) Reduce (remove) the runtime memory allocated to the Forth machine
One interesting aspect of the current design is that we have 2 memory allocation ranges - the Forth machine memory which is used for alloc-mem and free-mem, and also the OFMEM ranges. For example, in SPARC32 this is set to 256K which given the large I/O space requirements for the frame buffer, is about a quarter of the total memory.
I'd like to suggest that we unify the memory management by removing the Forth implementations of alloc-mem/free-mem and replace them with wrappers to the internal ofmem_malloc() and ofmem_free() functions. This would then enable us to have one continuous pool of allocatable memory which could be used for everything.
One slight issue with this is that there are a few references to alloc-mem/free-mem within the internal Forth code. These are mainly for allocating static buffers, and so I believe these could be removed and replaced by functions that simply allocate memory within the dictionary itself.
ii) Avoid re-allocation of memory for the Forth dictionary
At runtime we currently have two copies of the dictionary held within memory - the first is the actual static dictionary data, while the second is the relocated dictionary image within the BSS. In the case of SPARC32 this is responsible for just over 100K of the BSS image size.
Rather than having to allocate a second copy of the dictionary, would it make sense to embed the fixed size dictionary directly within the OpenBIOS image? As an example SPARC32 specifies 256K for the total dictionary size.
I think it should be possible to rewrite the relocation routine to relocate the dictionary 'in place' and then remove the relocation data so that the dictionary can extend normally up to its final limit. The downside of this is that the ROM images will be bigger on disk, but I don't feel that this should be too much of an issue.
I realise that these two ideas are probably fairly controversial, but they should both help considerably reduce the size of the OpenBIOS runtime memory footprint. Thoughts/comments/criticisms?