AMD64 Solo Hang - news

[Stefan Reinauer]

Hi,

it seems that the Solo motherboard is the only motherboard tested so far
that is based on an Athlon64 instead of Opteron CPUs..

The following code in src/northbridge/amd/amdk8/misc_control.c relies on
more than one hypertransport link being available:

cmd = pci_read_config32(dev, 0xdc);
        if((cmd & 0x0000ff00) != 0x02500) {
                cmd &= 0xffff00ff;
                cmd |= 0x00002500;
                pci_write_config32(dev, 0xdc, cmd );
                printk_debug("resetting cpu\n");
                hard_reset();
        }

This implicitly changes CPU0 Link1 FIFO Read Pointer Optimization, not
taking into regard that LDT1 might not be there.

It seems this code should rather check all links to see whether they are
connected and optimize _all_ of the connected.

Is the CPU reset here really needed for the setting to become active?
The BKDG does not state this explicitly ... (Therefore I vote for
removing it)

Stefan

Stefan Reinauer  writes:
> Hi,
>
> it seems that the Solo motherboard is the only motherboard tested so far
> that is based on an Athlon64 instead of Opteron CPUs..
>
> The following code in src/northbridge/amd/amdk8/misc_control.c relies on
> more than one hypertransport link being available:
>     cmd = pci_read_config32(dev, 0xdc);
>     if((cmd & 0x0000ff00) != 0x02500) {
>             cmd &= 0xffff00ff;
>             cmd |= 0x00002500;
>             pci_write_config32(dev, 0xdc, cmd );
>             printk_debug("resetting cpu\n");
>             hard_reset();
>     }
>
> This implicitly changes CPU0 Link1 FIFO Read Pointer Optimization, not
> taking into regard that LDT1 might not be there.
>
> It seems this code should rather check all links to see whether they are
> connected and optimize _all_ of the connected.

Agreed. That part should certainly be more dynamic.  It looks like
I missed that when I was syncing the trees.
> Is the CPU reset here really needed for the setting to become active?

I'm not certain.  But given that it is playing with hypertransport
link settings I would assume so.  But that is worth confirming.
> The BKDG does not state this explicitly ... (Therefore I vote for
> removing it)

I am starting to agree with you, but there must be some reason it is
in there...

Eric

[Stefan Reinauer]

* Eric W. Biederman  [031121 05:06]:
> > it seems that the Solo motherboard is the only motherboard tested so far
> > that is based on an Athlon64 instead of Opteron CPUs..
> >
> > The following code in src/northbridge/amd/amdk8/misc_control.c relies on
> > more than one hypertransport link being available:
> >     cmd = pci_read_config32(dev, 0xdc);
> >     if((cmd & 0x0000ff00) != 0x02500) {
> >             cmd &= 0xffff00ff;
> >             cmd |= 0x00002500;
> >             pci_write_config32(dev, 0xdc, cmd );
> >             printk_debug("resetting cpu\n");
> >             hard_reset();
> >     }
> >
> > This implicitly changes CPU0 Link1 FIFO Read Pointer Optimization, not
> > taking into regard that LDT1 might not be there.
> >
> > It seems this code should rather check all links to see whether they are
> > connected and optimize _all_ of the connected.
>
> Agreed. That part should certainly be more dynamic.  It looks like
> I missed that when I was syncing the trees.

Should this be done for all connected links or coherent links only?
(BKDG suggests setting this to 0x25 for all coherent links and for all
non-coherent links to other AMD chips)

What's the preferred way of fixing this? Walking all links on all nodes
once again? Or adding some configurable hard codes?
> > Is the CPU reset here really needed for the setting to become active?
>
> I'm not certain.  But given that it is playing with hypertransport
> link settings I would assume so.  But that is worth confirming.

I got some update here:
The CPU-Reset/LDTSTOP is needed.
If the read pointer is set to safe 3/7 cycles read accesses get 10
cycles quicker ( 25ns with a 400MHz HT-Device)

Stefan

Stefan Reinauer  writes:
> Eric W. Biederman  [031121 05:06]:
> > > it seems that the Solo motherboard is the only motherboard tested so far
> > > that is based on an Athlon64 instead of Opteron CPUs..
> > >
> > > The following code in src/northbridge/amd/amdk8/misc_control.c relies on
> > > more than one hypertransport link being available:
> > >     cmd = pci_read_config32(dev, 0xdc);
> > >     if((cmd & 0x0000ff00) != 0x02500) {
> > >             cmd &= 0xffff00ff;
> > >             cmd |= 0x00002500;
> > >             pci_write_config32(dev, 0xdc, cmd );
> > >             printk_debug("resetting cpu\n");
> > >             hard_reset();
> > >     }
> > >
> > > This implicitly changes CPU0 Link1 FIFO Read Pointer Optimization, not
> > > taking into regard that LDT1 might not be there.
> > >
> > > It seems this code should rather check all links to see whether they are
> > > connected and optimize _all_ of the connected.
> >
> > Agreed. That part should certainly be more dynamic.  It looks like
> > I missed that when I was syncing the trees.
>
> Should this be done for all connected links or coherent links only?
> (BKDG suggests setting this to 0x25 for all coherent links and for all
> non-coherent links to other AMD chips)
>
> What's the preferred way of fixing this? Walking all links on all nodes
> once again? Or adding some configurable hard codes?

Given where the code is at I was just going to walk the links on the
local chip and assume we were connected to other AMD parts.  It may
be reasonable to move all of this to the general HT link setup if we
can reasonably abstract it into methods.
> > > Is the CPU reset here really needed for the setting to become active?
> >
> > I'm not certain.  But given that it is playing with hypertransport
> > link settings I would assume so.  But that is worth confirming.
>
> I got some update here:
> The CPU-Reset/LDTSTOP is needed.
> If the read pointer is set to safe 3/7 cycles read accesses get 10
> cycles quicker ( 25ns with a 400MHz HT-Device)

Ok That confirms the reset needs to happen.  Note I believe the
commented out hard_reset as part of one of the errata fixes also needs
to happen, and right now one is causing the other.

Eric

Stefan Reinauer  writes:
> Hi,
>
> it seems that the Solo motherboard is the only motherboard tested so far
> that is based on an Athlon64 instead of Opteron CPUs..
>
> The following code in src/northbridge/amd/amdk8/misc_control.c relies on
> more than one hypertransport link being available:
>     cmd = pci_read_config32(dev, 0xdc);
>     if((cmd & 0x0000ff00) != 0x02500) {
>             cmd &= 0xffff00ff;
>             cmd |= 0x00002500;
>             pci_write_config32(dev, 0xdc, cmd );
>             printk_debug("resetting cpu\n");
>             hard_reset();
>     }
>
> This implicitly changes CPU0 Link1 FIFO Read Pointer Optimization, not
> taking into regard that LDT1 might not be there.
>
> It seems this code should rather check all links to see whether they are
> connected and optimize _all_ of the connected.
>
> Is the CPU reset here really needed for the setting to become active?
> The BKDG does not state this explicitly ... (Therefore I vote for
> removing it)

The acid test for needing the reset would be to find a benchmark that
cares about this optimization and see if it just setting the bits is sufficient
or if it requires the hypertransport link to be shut off.  The fact
that the settings persist through reset hint that a reset is required.

I am dropping a #warning into my tree so I won't forget and I will see if
I can find the appropriate benchmark.

Eric