Adam Talbot ajtalbot1@gmail.com writes:
Arthur: Thanks for the details. I have a board that with give me a "missing memory" beep code with more then 6 GPUs. Now I understand why!
How can I track down how much system DRAM a GPU is using? These are all the newest Nvidia Pascal based cards. Mostly GTX 1070's.
The answer is none unless onboard GPU which uses some.
What you are interested in is how much linear address space it uses and how much you can fit in PCI_MMIO (memory mapped input output) region.
You can use 'lspci -vvv' for that e.g.
00:02.0 VGA compatible controller: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07) (prog-if 00 [VGA controller]) Subsystem: Lenovo Mobile 4 Series Chipset Integrated Graphics Controller Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at e1000000 (64-bit, non-prefetchable) [size=4M] Region 2: Memory at d0000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at 3400 [size=8] [virtual] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [d0] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: i915 Kernel modules: i915
so for instance this device has 2 BAR's (base address reference) in memory space, the largest one being Region 2 which takes up 256M in linear memory space starting at the address 0xd0000000 (which is below 4G). (note this is NOT ram)
sidenote: when working in 32bit mode with PAE, the cpu can only address 4G (2^32) of address space. Given that that PCI devices (and other non ram stuff) take up some place in this 4G address space, you are left with less than 4G usable ram (if your system has 4G or more). The remainder of the ram is mapped above 4G which the cpu cannot address in 32bit mode.
What you want for such a setup is a BIOS/UEFI/boot_firmware that can allocate these BAR above 4G. coreboot afaik cannot do that.
On an interesting note, one of my oldest motherboards, a Gigabyte GA-970A-UD3 will boot with all 8 cards, but gives me the no VGA beep code. Serial console for the win!
Is this just a BIOS level issue? Or is there some hardware component I should be aware of?
Thanks for the help. -Adam