Quick CK804 PCI device numbering question

[Timothy Pearson]

All,

I have been working on porting Coreboot to a new CK804-based K10 
mainboard; it warm boots but will not cold boot due to IRQ/MSI 
configuration issues.

While tracing the IRQ problem I noticed that the CK804 PCI function 
numbers change from the proprietary BIOS to Coreboot:

Proprietary:
-[0000:00]-+-00.0  NVIDIA Corporation CK804 Memory Controller [10de:005e]
            +-01.0  NVIDIA Corporation CK804 ISA Bridge [10de:0051]
            +-01.1  NVIDIA Corporation CK804 SMBus [10de:0052]
            ...etc...

Coreboot:
-[0000:00]-+-01.0  NVIDIA Corporation CK804 Memory Controller [10de:005e]
            +-02.0  NVIDIA Corporation CK804 ISA Bridge [10de:0051]
            +-02.1  NVIDIA Corporation CK804 SMBus [10de:0052]

This, in turn, causes Linux to not detect the CK804 root bridge.

Has anyone else seen this with the CK804 chipset?  Is there a magic 
register somewhere that configures the CK804 to use the "correct" PCI 
function numbers?

The only pertinent quirk of this mainboard is that Asus put the CK804 on 
HT link 1, not 0 or 2 as is more common.

Thanks!

[Timothy Pearson]

On 01/20/2015 01:55 PM, Timothy Pearson wrote:
> All,
>
> I have been working on porting Coreboot to a new CK804-based K10
> mainboard; it warm boots but will not cold boot due to IRQ/MSI
> configuration issues.
>
> While tracing the IRQ problem I noticed that the CK804 PCI function
> numbers change from the proprietary BIOS to Coreboot:
>
> Proprietary:
> -[0000:00]-+-00.0 NVIDIA Corporation CK804 Memory Controller [10de:005e]
> +-01.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> +-01.1 NVIDIA Corporation CK804 SMBus [10de:0052]
> ...etc...
>
> Coreboot:
> -[0000:00]-+-01.0 NVIDIA Corporation CK804 Memory Controller [10de:005e]
> +-02.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> +-02.1 NVIDIA Corporation CK804 SMBus [10de:0052]
>
> This, in turn, causes Linux to not detect the CK804 root bridge.
>
> Has anyone else seen this with the CK804 chipset? Is there a magic
> register somewhere that configures the CK804 to use the "correct" PCI
> function numbers?
>
> The only pertinent quirk of this mainboard is that Asus put the CK804 on
> HT link 1, not 0 or 2 as is more common.
>
> Thanks!

Looks like I was writing too quickly..."function number" above should be 
"device number".  Nothing responds to a PCI vendor ID probe at 
PCI_DEV(0,0,0) which seems to indicate the CK804 itself has renumbered 
its internal devices, but I don't currently know why or how.

[Jonathan A. Kollasch]

On Tue, Jan 20, 2015 at 03:28:45PM -0600, Timothy Pearson wrote:
> On 01/20/2015 01:55 PM, Timothy Pearson wrote:
> > All,
> >
> > I have been working on porting Coreboot to a new CK804-based K10
> > mainboard; it warm boots but will not cold boot due to IRQ/MSI
> > configuration issues.
> >
> > While tracing the IRQ problem I noticed that the CK804 PCI function
> > numbers change from the proprietary BIOS to Coreboot:
> >
> > Proprietary:
> > -[0000:00]-+-00.0 NVIDIA Corporation CK804 Memory Controller [10de:005e]
> > +-01.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> > +-01.1 NVIDIA Corporation CK804 SMBus [10de:0052]
> > ...etc...
> >
> > Coreboot:
> > -[0000:00]-+-01.0 NVIDIA Corporation CK804 Memory Controller [10de:005e]
> > +-02.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> > +-02.1 NVIDIA Corporation CK804 SMBus [10de:0052]
> >
> > This, in turn, causes Linux to not detect the CK804 root bridge.
> >
> > Has anyone else seen this with the CK804 chipset? Is there a magic
> > register somewhere that configures the CK804 to use the "correct" PCI
> > function numbers?
> >
> > The only pertinent quirk of this mainboard is that Asus put the CK804 on
> > HT link 1, not 0 or 2 as is more common.
> >
> > Thanks!
>
> Looks like I was writing too quickly..."function number" above
> should be "device number".  Nothing responds to a PCI vendor ID
> probe at PCI_DEV(0,0,0) which seems to indicate the CK804 itself has
> renumbered its internal devices, but I don't currently know why or
> how.

Coreboot supports dual-ck804 boards, perhaps the ck804 got enumerated at the
secondary offset? (starting at something like bus 0x40 or 0x80)

Jonathan Kollasch

[Timothy Pearson]

On 01/20/2015 03:36 PM, Jonathan A. Kollasch wrote:
> On Tue, Jan 20, 2015 at 03:28:45PM -0600, Timothy Pearson wrote:
> > On 01/20/2015 01:55 PM, Timothy Pearson wrote:
> > > All,
> > >
> > > I have been working on porting Coreboot to a new CK804-based K10
> > > mainboard; it warm boots but will not cold boot due to IRQ/MSI
> > > configuration issues.
> > >
> > > While tracing the IRQ problem I noticed that the CK804 PCI function
> > > numbers change from the proprietary BIOS to Coreboot:
> > >
> > > Proprietary:
> > > -[0000:00]-+-00.0 NVIDIA Corporation CK804 Memory Controller [10de:005e]
> > > +-01.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> > > +-01.1 NVIDIA Corporation CK804 SMBus [10de:0052]
> > > ...etc...
> > >
> > > Coreboot:
> > > -[0000:00]-+-01.0 NVIDIA Corporation CK804 Memory Controller [10de:005e]
> > > +-02.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> > > +-02.1 NVIDIA Corporation CK804 SMBus [10de:0052]
> > >
> > > This, in turn, causes Linux to not detect the CK804 root bridge.
> > >
> > > Has anyone else seen this with the CK804 chipset? Is there a magic
> > > register somewhere that configures the CK804 to use the "correct" PCI
> > > function numbers?
> > >
> > > The only pertinent quirk of this mainboard is that Asus put the CK804 on
> > > HT link 1, not 0 or 2 as is more common.
> > >
> > > Thanks!
> >
> > Looks like I was writing too quickly..."function number" above
> > should be "device number".  Nothing responds to a PCI vendor ID
> > probe at PCI_DEV(0,0,0) which seems to indicate the CK804 itself has
> > renumbered its internal devices, but I don't currently know why or
> > how.
>
> Coreboot supports dual-ck804 boards, perhaps the ck804 got enumerated at the
> secondary offset? (starting at something like bus 0x40 or 0x80)
>
> Jonathan Kollasch

I had thought of that, but the bus number reported is 00 (correct) while 
the device numbers are off by one. I have placed the CK804 on bus 1 for 
debugging but all that does is cause the boot to fail earlier as 
Coreboot itself looks for the CK804 on bus 0; the PCI device numbers 
stay at their incorrectly shifted values.

Is there an example of the lspci output for a dual-CK804 board?  A quick 
Google search didn't turn up anything; I wanted to see if the CK804 
detects the HT link it's on and shifts its device numbering.

Thanks!

[Timothy Pearson]

On 01/20/2015 03:42 PM, Timothy Pearson wrote:
> On 01/20/2015 03:36 PM, Jonathan A. Kollasch wrote:
> > On Tue, Jan 20, 2015 at 03:28:45PM -0600, Timothy Pearson wrote:
> > > On 01/20/2015 01:55 PM, Timothy Pearson wrote:
> > > > All,
> > > >
> > > > I have been working on porting Coreboot to a new CK804-based K10
> > > > mainboard; it warm boots but will not cold boot due to IRQ/MSI
> > > > configuration issues.
> > > >
> > > > While tracing the IRQ problem I noticed that the CK804 PCI function
> > > > numbers change from the proprietary BIOS to Coreboot:
> > > >
> > > > Proprietary:
> > > > -[0000:00]-+-00.0 NVIDIA Corporation CK804 Memory Controller
> > > > [10de:005e]
> > > > +-01.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> > > > +-01.1 NVIDIA Corporation CK804 SMBus [10de:0052]
> > > > ...etc...
> > > >
> > > > Coreboot:
> > > > -[0000:00]-+-01.0 NVIDIA Corporation CK804 Memory Controller
> > > > [10de:005e]
> > > > +-02.0 NVIDIA Corporation CK804 ISA Bridge [10de:0051]
> > > > +-02.1 NVIDIA Corporation CK804 SMBus [10de:0052]
> > > >
> > > > This, in turn, causes Linux to not detect the CK804 root bridge.
> > > >
> > > > Has anyone else seen this with the CK804 chipset? Is there a magic
> > > > register somewhere that configures the CK804 to use the "correct" PCI
> > > > function numbers?
> > > >
> > > > The only pertinent quirk of this mainboard is that Asus put the
> > > > CK804 on
> > > > HT link 1, not 0 or 2 as is more common.
> > > >
> > > > Thanks!
> > >
> > > Looks like I was writing too quickly..."function number" above
> > > should be "device number". Nothing responds to a PCI vendor ID
> > > probe at PCI_DEV(0,0,0) which seems to indicate the CK804 itself has
> > > renumbered its internal devices, but I don't currently know why or
> > > how.
> >
> > Coreboot supports dual-ck804 boards, perhaps the ck804 got enumerated
> > at the
> > secondary offset? (starting at something like bus 0x40 or 0x80)
> >
> > Jonathan Kollasch
>
> I had thought of that, but the bus number reported is 00 (correct) while
> the device numbers are off by one. I have placed the CK804 on bus 1 for
> debugging but all that does is cause the boot to fail earlier as
> Coreboot itself looks for the CK804 on bus 0; the PCI device numbers
> stay at their incorrectly shifted values.
>
> Is there an example of the lspci output for a dual-CK804 board? A quick
> Google search didn't turn up anything; I wanted to see if the CK804
> detects the HT link it's on and shifts its device numbering.
>
> Thanks!

For anyone interested I manually located the offending control register:
CK804 PCI configuration register 0x44 bit 16 (quite possibly a single 
nibble in bits 19-16, though this is untested) controls the device 
numbering offset.  My CK804 initialized that nibble to 1; clearing it 
reset the device numbering

[Scott Duplichan]

Timothy Pearson [mailto:[e-mail]] wrote:

]
]All,
]
]I have been working on porting Coreboot to a new CK804-based K10 
]mainboard; it warm boots but will not cold boot due to IRQ/MSI 
]configuration issues.
]
]While tracing the IRQ problem I noticed that the CK804 PCI function 
]numbers change from the proprietary BIOS to Coreboot:
]
]Proprietary:
]-[0000:00]-+-00.0  NVIDIA Corporation CK804 Memory Controller ][10de:005e]
]            +-01.0  NVIDIA Corporation CK804 ISA Bridge [10de:0051]
]            +-01.1  NVIDIA Corporation CK804 SMBus [10de:0052]
]            ...etc...
]
]Coreboot:
]-[0000:00]-+-01.0  NVIDIA Corporation CK804 Memory Controller ][10de:005e]
]            +-02.0  NVIDIA Corporation CK804 ISA Bridge [10de:0051]
]            +-02.1  NVIDIA Corporation CK804 SMBus [10de:0052]
]
]This, in turn, causes Linux to not detect the CK804 root bridge.
]
]Has anyone else seen this with the CK804 chipset?  Is there a magic 
]register somewhere that configures the CK804 to use the "correct" ]PCI 
]function numbers?
]
]The only pertinent quirk of this mainboard is that Asus put the ]CK804 on 
]HT link 1, not 0 or 2 as is more common.
]
]Thanks!
]
]-- 
]Timothy Pearson
]Raptor Engineering
][phone]
]http://www.raptorengineeringinc.com

If I remember correctly, PCI device IDs are assigned during non-coherent
HyperTransport initialization on these systems. Search a BKDG for "unit id"
for details. The solution may involve changing HT_CHAIN_UNITID_BASE and
related items. Take a look at 'swaplist' in the source code.

Thanks,
Scott