r4315, CBFS, supermicro H8DME -> boom

[Ward Vandewege]

So, yesterday I spent some quality time tracking down a rather odd issue I
saw with svn head (r4362):

http://ward.vandewege.net/coreboot/h8dme/minicom-20090618b.cap

As you can see, coreboot somehow garbles serial output right after printing

Clearing initial memory region: Done

and then finally soft resets after entering a CBFS stage:

Stage: load @ 819200/28672 bytes, enter @ c8000

I tracked this down to r4315. Here's a boot log with r4315:

http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap

And here's r4314, which works:

http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap

The weird thing is that I don't have this problem on an h8dmr with K10 cpus
(though I'm still tracking down other issues there, so perhaps they mask this
problem), nor on an m57sli with a dual core AM2 CPU.

I talked to Patrick (the author of r4315) about this yesterday on IRC, and he
said he could not reproduce this on SimNow with amd/serengeti_cheetah.

So, if anyone has thoughts on what might be causing this, they would be very
welcome :)

Thanks,
Ward.

[Carl-Daniel Hailfinger]

On 19.06.2009 16:21, Ward Vandewege wrote:
> I tracked this down to r4315. Here's a boot log with r4315:
> And here's r4314, which works:
>
> So, if anyone has thoughts on what might be causing this, they would be very
> welcome :)

Index: src/arch/i386/init/crt0.S.lb
===================================================================
--- src/arch/i386/init/crt0.S.lb        (revision 4314)
+++ src/arch/i386/init/crt0.S.lb        (revision 4315)
@@ -73,6 +73,10 @@
        movl    $0x4000000, %esp
        movl    %esp, %ebp
        pushl %esi
+#ifdef CONFIG_CBFS
+       pushl $str_coreboot_ram_name
+       call cbfs_and_run_core
+#else
        movl    $_liseg, %esi
        movl    $_iseg,  %edi
        movl    $_eiseg, %ecx
@@ -81,6 +85,7 @@
        pushl %edi
        pushl %esi
        call copy_and_run_core
+#endif

Erm. Shouldn't that be #if instead of #ifdef?

Regards,
Carl-Daniel

[Patrick Georgi]

Carl-Daniel Hailfinger schrieb:
> Index: src/arch/i386/init/crt0.S.lb
>
> --- src/arch/i386/init/crt0.S.lb        (revision 4314)
> +++ src/arch/i386/init/crt0.S.lb        (revision 4315)
> @@ -73,6 +73,10 @@
>         movl    $0x4000000, %esp
>         movl    %esp, %ebp
>         pushl %esi
> +#ifdef CONFIG_CBFS
>   pushl $str_coreboot_ram_name
>   call cbfs_and_run_core
>
> +#else
>         movl    $_liseg, %esi
>         movl    $_iseg,  %edi
>         movl    $_eiseg, %ecx
> @@ -81,6 +85,7 @@
>         pushl %edi
>         pushl %esi
>         call copy_and_run_core
> +#endif
>
> Erm. Shouldn't that be #if instead of #ifdef?

See r4316

Regards,
Patrick

[Carl-Daniel Hailfinger]

On 19.06.2009 16:32, Patrick Georgi wrote:
> Carl-Daniel Hailfinger schrieb:
> > Index: src/arch/i386/init/crt0.S.lb
> >
> > --- src/arch/i386/init/crt0.S.lb        (revision 4314)
> > +++ src/arch/i386/init/crt0.S.lb        (revision 4315)
> > @@ -73,6 +73,10 @@
> >         movl    $0x4000000, %esp
> >         movl    %esp, %ebp
> >         pushl %esi
> > +#ifdef CONFIG_CBFS
> >
> > Erm. Shouldn't that be #if instead of #ifdef?
>
> See r4316

My bad, sorry. I only looked at the isolated change of r4315.

Regards,
Carl-Daniel

[Myles Watson]

> I tracked this down to r4315. Here's a boot log with r4315:
>
> http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap
>
> And here's r4314, which works:
>
> http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap

The URLs are the same, and since 4316 fixes 4315, could you post a link to
the log of 4316?
> > +#ifdef CONFIG_CBFS
> >
> > Erm. Shouldn't that be #if instead of #ifdef?
>
> See r4316

Thanks,
Myles

[Ward Vandewege]

On Fri, Jun 19, 2009 at 08:48:59AM -0600, Myles Watson wrote:
> > I tracked this down to r4315. Here's a boot log with r4315:
> >
> > http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap
> >
> > And here's r4314, which works:
> >
> > http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap
>
> The URLs are the same,

I'm sorry, copy/paste error. This is (working) r4314:

http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap

And broken r4315:

http://ward.vandewege.net/coreboot/h8dme/minicom-20090618p.cap
> and since 4316 fixes 4315, could you post a link to
> the log of 4316?

I have r4320:

http://ward.vandewege.net/coreboot/h8dme/minicom-20090618m.cap

And the problem is the same with head.

Thanks
Ward.

[Myles Watson]

> > > I tracked this down to r4315. Here's a boot log with r4315:
> > >
> > > http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap
> > >
> > > And here's r4314, which works:
> > >
> > > http://ward.vandewege.net/coreboot/h8dme/minicom-20090618q.cap
> >
> > The URLs are the same,
>
> I'm sorry, copy/paste error. This is (working) r4314:

No problem.

There's garbled text in all three logs.  Shouldn't the APs be stopped at
this point?

4314:

Enumerating buses...
Show all devs...Before Phase 3.
Root Device: enabled 1, 0 resources
APIC_CLUSTER: 0: enabled 1, 0 resources
APIC: 00: enabled 1, 0 resources
PCI_DOMAIN: 0000: enabled 1, 0 resources
PCI: 0i0m:a1g8e. 0l:e negntahb l=e d0 010,0 20a 6rceso
uJrucmepsin
gP CtIo:  i0m0a:g0e0..0:
 enabled 1, 0 resources
PCI: C0O0D:E0 1I.N0 :C AeCnHaEb lOeNd  N1O,D E0: 0r1eso
urces
PNP: 002e.0: enabled 0, 3 resources
PNP: 002e.1: enabled 0, 2 resources
PNP: 002eT.r2a:i neDnQaSbRldeWdr P1o,s :2  bruefs_oau:r0c0e0sce
bP3N0P:
 T0r0a2ien.D3Q:S Peonsa:b lMeudt u0a,l C2S PraesssoWu[r4c8e]s :
0P0N0Pc:e 90c082e
.5: enabled 1, 4 resources
PNP: 002e.6: enabled 0, 1 resources

I'm wondering if it is related to the garbling, and it becomes more serious
because of the change in 4315.

If there's a good reason for the garbling, we could try to see what the
other two logs are printing.  It's not fun to decode it, but it looks like
multiple CPUs trying to load the same stuff from CBFS.

Thanks,
Myles

[ron minnich]

so, this may be a pain, but can you yank all but one cpu? I realize
this is asking a lot.

This is multicore, right? So that may not help.

ron

[Myles Watson]

> so, this may be a pain, but can you yank all but one cpu? I realize
> this is asking a lot.
>
> This is multicore, right? So that may not help.

Isn't there a CONFIG_LOGICAL_CPUS setting and a CONFIG_MAX...CPUS setting
that you can use to limit initialization to one CPU?

Thanks,
Myles

[Ward Vandewege]

On Fri, Jun 19, 2009 at 08:31:10AM -0700, ron minnich wrote:
> so, this may be a pain, but can you yank all but one cpu? I realize
> this is asking a lot.
>
> This is multicore, right? So that may not help.

Yeah, it's a 2x dual core setup, so that's going to be hard ;)

Thanks,
Ward.

[Patrick Georgi]

Myles Watson schrieb:
> I'm wondering if it is related to the garbling, and it becomes more serious
> because of the change in 4315.
>
> If there's a good reason for the garbling, we could try to see what the
> other two logs are printing.  It's not fun to decode it, but it looks like
> multiple CPUs trying to load the same stuff from CBFS.

That's actually a good hint.
Where's coreboot_apc loaded to on those systems? CAR or RAM?
If it's loaded to RAM, and all APs load it, things might go bust. Though 
the same thing should have happened with the old nrv2b codepath, too.

Patrick