[coreboot] K8 HT architecture

[Tom Sylla]

The HT topology is not really directly reflected in PCI config space.
They are obviously linked, but there is not really a way to "map" a HT
topology of Opteron nodes to a graphical view of config space, it just
doesn't exist. The Opterons just are where they are.

All of the processor PCI devices are on Bus 0, and will always be on
Bus 0. Devices 18-1f are the only ones that will ever exist for
Opterons. The device number that each Opteron responds to is based on
NodeID (0-7), which is set on each Opteron during discovery. There
won't ever be "holes", the NodeIDs must always be contiguous. Node IDs
are not set in stone based on topology, node 0 is always the BSP, but
1-7 can basically be distributed out any of the 3 links to any other
CPUs in any manner. AGESA has a default "discovery method" (I think
breadth first, lowest link number first) but it has options to
over-ride the discovery mechanism to change the order of nodes in a
system. All that matters is that the routing tables are correct and
consistent for the traffic to get where it needs to and without
deadlock. Once that is complete, the processors just show up in PCI as
devices 18-1f (or fewer)

Here is an lpsci from one of our systems with 5 nodes (AMI BIOS with AGESA):

00:01.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge
00:02.0 Host bridge: Broadcom BCM5785 [HT1000] Legacy South Bridge
00:02.1 IDE interface: Broadcom BCM5785 [HT1000] IDE
00:02.2 ISA bridge: Broadcom BCM5785 [HT1000] LPC
00:03.0 USB Controller: Broadcom BCM5785 [HT1000] USB (rev 01)
00:03.1 USB Controller: Broadcom BCM5785 [HT1000] USB (rev 01)
00:03.2 USB Controller: Broadcom BCM5785 [HT1000] USB (rev 01)
00:04.0 VGA compatible controller: XGI Technology Inc. (eXtreme
Graphics Innovation) Volari Z7
00:06.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2)
00:07.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2)
00:08.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2)
00:09.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2)
00:0a.0 PCI bridge: Broadcom HT2100 PCI-Express Bridge (rev a2)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
00:1a.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
00:1a.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
00:1a.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
00:1a.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
00:1b.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
00:1b.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
00:1b.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
00:1b.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
00:1c.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
00:1c.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
00:1c.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
00:1c.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
01:0d.0 PCI bridge: Broadcom BCM5785 [HT1000] PCI/PCI-X Bridge (rev c0)
01:0e.0 IDE interface: Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode)
01:0e.1 IDE interface: Broadcom BCM5785 [HT1000] SATA (PATA/IDE Mode)
03:00.0 PCI bridge: PLX Technology, Inc. Unknown device 8518 (rev aa)
04:01.0 PCI bridge: PLX Technology, Inc. Unknown device 8518 (rev aa)
04:02.0 PCI bridge: PLX Technology, Inc. Unknown device 8518 (rev aa)
04:03.0 PCI bridge: PLX Technology, Inc. Unknown device 8518 (rev aa)
04:04.0 PCI bridge: PLX Technology, Inc. Unknown device 8518 (rev aa)
0c:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit
Ethernet Controller (rev 06)
0c:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit
Ethernet Controller (rev 06)

If we add or remove processors, nothing beside the 18-1f devices will
change (SB Bus numbers, device numbers, etc do not change). When we
add another *non* coherent HT device attached to one of the Opterons,
it gets a new bus number (we start at 20 with ours, but it is
arbitrary). All of the routing associated with HT for both coherent
and non-coherent is contained in the mapping registers and routing
table registers in all of the Opterons. The mapping registers map
mem/io/cfg regions to nodes, and the routing table says how to get to
that node. The ncHT devices can have BARs, and take up memory mapped
IO just the same as another PCI device.

On Thu, Oct 23, 2008 at 10:40 PM, Carl-Daniel Hailfinger
<c-d.hailfinger.devel.2006 at gmx.net> wrote:
> I'm trying to understand how HT is modeled into PCI space so that I can
> propose the "right" way to handle it in the dts.
> Depending on whether I run lspci -t under coreboot or factory BIOS,
> different topologies will be displayed. That means looking at lspci is
> not going to tell me how the hardware really works.

I am a little bit confused by this. What are the exact differences you
see between coreboot and factory? The number of Opterons should be
that same. The position in config space of a particular socket may
change, based on node discovery differences between the BIOSes. Their
is no reason for other devices to move because of the HT changes, but
they may move just by other differences in coreboot.

Tom