[Public]
Hi all,
I've ran into an issue where the 64 bit address window allocated for one of the PCI-e host devices overlaps with a BIOS reserved range of fd00000000-ffffffffff, making that device unusable.
4000000000-7bfffffffff : PCI Bus 0000:00 4000000000-bfffffffff : PCI Bus 0000:0c 4000000000-7fffffffff : 0000:0c:00.0 8000000000-80001fffff : 0000:0c:00.0 c000000000-13fffffffff : PCI Bus 0000:0b c000000000-ffffffffff : 0000:0b:00.0 fd00000000-ffffffffff : Reserved 10000000000-100001fffff : 0000:0b:00.0
8000000000-7ffffffffff : PCI Bus 0000:00 8000000000-ffffffffff : PCI Bus 0000:0c 8000000000-bfffffffff : 0000:0c:00.0 c000000000-c0001fffff : 0000:0c:00.0 fd00000000-ffffffffff : Reserved 10000000000-17fffffffff : PCI Bus 0000:0b 10000000000-13fffffffff : 0000:0b:00.0 14000000000-140001fffff : 0000:0b:00.0
I found that increasing the system ram of the VM to ~256G changes the address window and avoids the issue, and with some digging I think I found the root cause
To begin, QEMU set this address range to be reserved here, and this is passed to Seabios via "etc/e820". https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L865 Separately QEMU pass to Seabios "etc/reserved-memory-end" here which is derived from machine->device_memory https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L1007
In Seabios, "etc/e820" is consumed here, which sets RamSizeOver4G only using the E820_RAM entries, ignoring any E820_RESERVED entries. https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/paravirt.c#L782 Later "etc/reserved-memory-end" and RamSizeOver4G is used to determine the start of the PCI-e address window. https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/pciinit.c#L1138
I think either QEMU should set etc/reserved-memory-end to be after both physical memory and the reserved ranges, or Seabios need to check both etc/820 and etc/reserved-memory-end. But I'm not sure which would be the correct move and indeed how to patch them.
Regards, Yunxiang Li (Teddy)