[LinuxBIOS] LZMA compression of kernel image and p7zip (Carl-Daniel Hailfinger / Stefan Reinauer)

[liran tal]

Hey everyone,

I have stumbled upon posts by Carl-Daniel and Stefan while trying to get
some
information why when I compile a kernel image with 7zip (LZMA) the kernel
image doesn't load up.

The specific post is -
http://www.mail-archive.com/[e-mail]/msg06389.html

I figured that maybe you could help diagnosing this problem as it seems that
I have
found the problem but I am not too sure actually.

I have been given an SDK based on Linux 2.6.10 for an embedded mips 4Kec (TI
Avalanche series) cpu
to build an image from. When I build the image using GZIP (standard) the
kernel boots fine but when I
choose 7ZIP support for better compression, the 'make ram_zimage' procedure
is working successfully
but when the image attempts to run off the device (via nfs) it writes:

Launching kernel decompressor.
        Starting LZMA Uncompression Algorithm.
        Compressed file is LZMA format.
        Kernel decompressor was successful ... launching kernel.

and nothing happens after that.
I've been attempting to figure out what is causing the kernel not to load
and after reading
the post by Carl-Daniel and Stefan about 7zip using different headers than
"normal" LZMA sdk
I thought that this could be the problem.

I understand that this is not the best forum to ask, being a linuxbios
mailing-list
but I would appreciate your comments and any help on this issue as I am
really
trying to understand what is going on.

Thanks,
Sincerely,
Liran.

[LinuxBIOS] LZMA compression of kernel image and p7zip (Carl-Daniel Hailfinger / Stefan Reinauer)

[Carl-Daniel Hailfinger]

Hello Liran,

On 30.08.2007 17:06, liran tal wrote:
> I have stumbled upon posts by Carl-Daniel and Stefan while trying to get
> some information why when I compile a kernel image with 7zip (LZMA)
> the kernel image doesn't load up.
> [...]
> I have been given an SDK based on Linux 2.6.10 for an embedded mips 4Kec (TI
> Avalanche series) cpu to build an image from. When I build the image
> using GZIP (standard) the kernel boots fine but when I
> choose 7ZIP support for better compression, the 'make ram_zimage' procedure
> is working successfully
> but when the image attempts to run off the device (via nfs) it writes:
>     Launching kernel decompressor.
>     Starting LZMA Uncompression Algorithm.
>     Compressed file is LZMA format.
>     Kernel decompressor was successful ... launching kernel.
>
> and nothing happens after that.

Ah, the classic "no logs available" problem because the kernel hangs
during early boot. Since it works with gzip I have to assume there is a
bug either in the decompression routine or some #define has not been
adjusted to the architecture. Less likely, but still possible would be
that the decompression routine is touching memory which is special on
your machine.
> I've been attempting to figure out what is causing the kernel not to load
> and after reading the post by Carl-Daniel and Stefan about 7zip 
> using different headers than "normal" LZMA sdk
> I thought that this could be the problem.

Possible, but rather unlikely. What do the docs for the lzma kernel
patch say? Is it supposed to be used with p7zip or the original SDK?

Carl-Daniel

[LinuxBIOS] LZMA compression of kernel image and p7zip (Carl-Daniel Hailfinger / Stefan Reinauer)

[liran tal]

Hey Carl,

Sorry about the late reply, I've been on and off during flights out of the
country and
haven't got around to check this issue.

I have done some more investigating on this issue but no luck (yet).
It made some sense to me as well that the problem might be with the use of
7ZIP-specific
compression/decompression but I'm not sure of that anymore.
I have attempted to use the LZMA SDK library to compress the image using
that instead of
the 7z application (p7zip) but actually the booloader/kernel denied me from
that image, possibly
because it was written specifically to 7z so I am sure that encoding the
image must be done
using 7z. That's about how to encode the image.

Regarding the decompression which happens in the kernel/bootloader stage -
well the print out is:

Launching kernel decompressor.
     Starting LZMA Uncompression Algorithm.
     Compressed file is LZMA format.
     Kernel decompressor was successful ... launching kernel.

So I am relying on the fact that it decompressed fine and that there is no
bug in the decompression code
(I've been provided with an SDK so this is not some home-brew code that
might contain more errors than usual).

So going over the code more thouroughly - the part that handles the
decompression calls
the kernel_entry function which in turn calls the start_kernel and that to
_text (if I remember the ordering
right) and I am assuming that the problem is there somewhere.

A good thing to remember is that the only difference from this problem to a
working kernel image
is the CONFIG_KERNEL_COMPRESS_7ZIP option which I use.
If I use the default GZIP compression then the kernel image loads just fine.

I guess this could also be some sort of alignment problems, but memory
alignments specific
to the 7z compression is somewhat doubtful, isn't it?

Any insight is welcome,

Regards,
Liran.

On 8/31/07, Carl-Daniel Hailfinger 
wrote:
> Hello Liran,
>
> On 30.08.2007 17:06, liran tal wrote:
> > I have stumbled upon posts by Carl-Daniel and Stefan while trying to get
> > some information why when I compile a kernel image with 7zip (LZMA)
> > the kernel image doesn't load up.
> > [...]
> > I have been given an SDK based on Linux 2.6.10 for an embedded mips 4Kec
>
> (TI
> > Avalanche series) cpu to build an image from. When I build the image
> > using GZIP (standard) the kernel boots fine but when I
> > choose 7ZIP support for better compression, the 'make ram_zimage'
>
> procedure
> > is working successfully
> > but when the image attempts to run off the device (via nfs) it writes:
> >     Launching kernel decompressor.
> >     Starting LZMA Uncompression Algorithm.
> >     Compressed file is LZMA format.
> >     Kernel decompressor was successful ... launching kernel.
> >
> > and nothing happens after that.
>
> Ah, the classic "no logs available" problem because the kernel hangs
> during early boot. Since it works with gzip I have to assume there is a
> bug either in the decompression routine or some #define has not been
> adjusted to the architecture. Less likely, but still possible would be
> that the decompression routine is touching memory which is special on
> your machine.
> > I've been attempting to figure out what is causing the kernel not to
>
> load
> > and after reading the post by Carl-Daniel and Stefan about 7zip
> > using different headers than "normal" LZMA sdk
> > I thought that this could be the problem.
>
> Possible, but rather unlikely. What do the docs for the lzma kernel
> patch say? Is it supposed to be used with p7zip or the original SDK?
>
> Carl-Daniel

[LinuxBIOS] LZMA compression of kernel image and p7zip (Carl-Daniel Hailfinger / Stefan Reinauer)

[Carl-Daniel Hailfinger]

Hi Liran,

I have thought a while about your problem but no obvious solution appeared.

On 04.09.2007 17:50, liran tal wrote:
> I have done some more investigating on this issue but no luck (yet).
> It made some sense to me as well that the problem might be with the use of
> 7ZIP-specific
> compression/decompression but I'm not sure of that anymore.
> I have attempted to use the LZMA SDK library to compress the image using
> that instead of
> the 7z application (p7zip) but actually the booloader/kernel denied me from
> that image, possibly
> because it was written specifically to 7z so I am sure that encoding the
> image must be done
> using 7z. That's about how to encode the image.

OK, so the format seems to be OK.
> Regarding the decompression which happens in the kernel/bootloader stage -
> well the print out is:
>  Launching kernel decompressor.
>  Starting LZMA Uncompression Algorithm.
>  Compressed file is LZMA format.
>  Kernel decompressor was successful ... launching kernel.
>
> So I am relying on the fact that it decompressed fine and that there is no
> bug in the decompression code

At least no obvious bug.
> (I've been provided with an SDK so this is not some home-brew code that
> might contain more errors than usual).

My experience is that SDKs may contain really hard-to-find errors
because sometimes the code is tested, but not reviewed.
> So going over the code more thouroughly - the part that handles the
> decompression calls
> the kernel_entry function which in turn calls the start_kernel and that to
> _text (if I remember the ordering
> right) and I am assuming that the problem is there somewhere.

A possible problem is overlap between the compressed and decompressed
kernel, another possibility is decompression to the wrong address.

Can you try to get both decompression routines to print out where
exactly the decompressed code is located? If possible, printing the end
addr of the decompressed data will be interesting as well.
> A good thing to remember is that the only difference from this problem to a
> working kernel image
> is the CONFIG_KERNEL_COMPRESS_7ZIP option which I use.
> If I use the default GZIP compression then the kernel image loads just fine.

You could try with early serial support to get at least some logs, but I
doubt you'd get that far.
> I guess this could also be some sort of alignment problems, but memory
> alignments specific
> to the 7z compression is somewhat doubtful, isn't it?

Many patches for embedded systems not merged in mainline Linux have had
their share of problems in the past because the review is less thorough
(if any). It has been quite a lot of time since I last looked at the
in-kernel LZMA decompression code and I'm not quite sure how it works
anymore. Hm... another option is a compiler/linker bug.

The decompression scratch pad memory is sometimes quite big and depends
on lots of factors, one of them being dictionary size. Try using the
default values for everything when compressing. That might help as well.

Regards,
Carl-Daniel