[coreboot] 16 GPUs on one board

[Arthur Heymans]

Adam Talbot <ajtalbot1 at gmail.com> writes:
> Arthur: Thanks for the details. I have a board that with give me a "missing memory" beep code with more then 6 GPUs. Now I understand why!
>
> How can I track down how much system DRAM a GPU is using? These are
> all the newest Nvidia Pascal based cards. Mostly GTX 1070's.

The answer is none unless onboard GPU which uses some.

What you are interested in is how much linear address space it uses and
how much you can fit in PCI_MMIO (memory mapped input output) region.

You can use 'lspci -vvv' for that e.g.

00:02.0 VGA compatible controller: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07) (prog-if 00 [VGA controller])
        Subsystem: Lenovo Mobile 4 Series Chipset Integrated Graphics Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at e1000000 (64-bit, non-prefetchable) [size=4M]
        Region 2: Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Region 4: I/O ports at 3400 [size=8]
        [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [d0] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: i915
        Kernel modules: i915

so for instance this device has 2 BAR's (base address reference) in
memory space, the largest one being Region 2 which takes up 256M in
linear memory space starting at the address 0xd0000000 (which is below
4G). (note this is NOT ram)

sidenote: when working in 32bit mode with PAE, the cpu can only address
4G (2^32) of address space. Given that that PCI devices (and other non ram
stuff) take up some place in this 4G address space, you are left with
less than 4G usable ram (if your system has 4G or more). The remainder
of the ram is mapped above 4G which the cpu cannot address in 32bit
mode.

What you want for such a setup is a BIOS/UEFI/boot_firmware that can
allocate these BAR above 4G. coreboot afaik cannot do that.

>
> On an interesting note, one of my oldest motherboards, a Gigabyte GA-970A-UD3 will boot with all 8 cards, but gives me the no VGA beep code. Serial console for the win!
>
> Is this just a BIOS level issue? Or is there some hardware component I should be aware of?
>
> Thanks for the help.
> -Adam
>

-- 
Arthur Heymans