[Public]
Hi all,
I've ran into an issue where the 64 bit address window allocated for one of the PCI-e host devices overlaps with a BIOS reserved range of fd00000000-ffffffffff, making that device unusable.
4000000000-7bfffffffff : PCI Bus 0000:00 4000000000-bfffffffff : PCI Bus 0000:0c 4000000000-7fffffffff : 0000:0c:00.0 8000000000-80001fffff : 0000:0c:00.0 c000000000-13fffffffff : PCI Bus 0000:0b c000000000-ffffffffff : 0000:0b:00.0 fd00000000-ffffffffff : Reserved 10000000000-100001fffff : 0000:0b:00.0
8000000000-7ffffffffff : PCI Bus 0000:00 8000000000-ffffffffff : PCI Bus 0000:0c 8000000000-bfffffffff : 0000:0c:00.0 c000000000-c0001fffff : 0000:0c:00.0 fd00000000-ffffffffff : Reserved 10000000000-17fffffffff : PCI Bus 0000:0b 10000000000-13fffffffff : 0000:0b:00.0 14000000000-140001fffff : 0000:0b:00.0
I found that increasing the system ram of the VM to ~256G changes the address window and avoids the issue, and with some digging I think I found the root cause
To begin, QEMU set this address range to be reserved here, and this is passed to Seabios via "etc/e820". https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L865 Separately QEMU pass to Seabios "etc/reserved-memory-end" here which is derived from machine->device_memory https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L1007
In Seabios, "etc/e820" is consumed here, which sets RamSizeOver4G only using the E820_RAM entries, ignoring any E820_RESERVED entries. https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/paravirt.c#L782 Later "etc/reserved-memory-end" and RamSizeOver4G is used to determine the start of the PCI-e address window. https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/pciinit.c#L1138
I think either QEMU should set etc/reserved-memory-end to be after both physical memory and the reserved ranges, or Seabios need to check both etc/820 and etc/reserved-memory-end. But I'm not sure which would be the correct move and indeed how to patch them.
Regards, Yunxiang Li (Teddy)
On Tue, Dec 31, 2024 at 09:24:09PM +0000, Li, Yunxiang (Teddy) via SeaBIOS wrote:
[Public]
Hi all,
I've ran into an issue where the 64 bit address window allocated for one of the PCI-e host devices overlaps with a BIOS reserved range of fd00000000-ffffffffff, making that device unusable.
4000000000-7bfffffffff : PCI Bus 0000:00 4000000000-bfffffffff : PCI Bus 0000:0c 4000000000-7fffffffff : 0000:0c:00.0 8000000000-80001fffff : 0000:0c:00.0 c000000000-13fffffffff : PCI Bus 0000:0b c000000000-ffffffffff : 0000:0b:00.0 fd00000000-ffffffffff : Reserved 10000000000-100001fffff : 0000:0b:00.0
8000000000-7ffffffffff : PCI Bus 0000:00 8000000000-ffffffffff : PCI Bus 0000:0c 8000000000-bfffffffff : 0000:0c:00.0 c000000000-c0001fffff : 0000:0c:00.0 fd00000000-ffffffffff : Reserved 10000000000-17fffffffff : PCI Bus 0000:0b 10000000000-13fffffffff : 0000:0b:00.0 14000000000-140001fffff : 0000:0b:00.0
I found that increasing the system ram of the VM to ~256G changes the address window and avoids the issue, and with some digging I think I found the root cause
To begin, QEMU set this address range to be reserved here, and this is passed to Seabios via "etc/e820". https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L865 Separately QEMU pass to Seabios "etc/reserved-memory-end" here which is derived from machine->device_memory https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L1007
In Seabios, "etc/e820" is consumed here, which sets RamSizeOver4G only using the E820_RAM entries, ignoring any E820_RESERVED entries. https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/paravirt.c#L782 Later "etc/reserved-memory-end" and RamSizeOver4G is used to determine the start of the PCI-e address window. https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/pciinit.c#L1138
I think either QEMU should set etc/reserved-memory-end to be after both physical memory and the reserved ranges, or Seabios need to check both etc/820 and etc/reserved-memory-end. But I'm not sure which would be the correct move and indeed how to patch them.
commit 1e1da7a96300 ("check for e820 conflict") should fix that.
take care, Gerd
[Public]
From: Gerd Hoffmann kraxel@redhat.com Sent: Thursday, January 2, 2025 8:52 On Tue, Dec 31, 2024 at 09:24:09PM +0000, Li, Yunxiang (Teddy) via SeaBIOS wrote:
[Public]
Hi all,
I've ran into an issue where the 64 bit address window allocated for one of the
PCI-e host devices overlaps with a BIOS reserved range of fd00000000-ffffffffff, making that device unusable.
4000000000-7bfffffffff : PCI Bus 0000:00 4000000000-bfffffffff : PCI Bus 0000:0c 4000000000-7fffffffff : 0000:0c:00.0 8000000000-80001fffff : 0000:0c:00.0 c000000000-13fffffffff : PCI Bus 0000:0b c000000000-ffffffffff : 0000:0b:00.0 fd00000000-ffffffffff : Reserved 10000000000-100001fffff : 0000:0b:00.0
8000000000-7ffffffffff : PCI Bus 0000:00 8000000000-ffffffffff : PCI Bus 0000:0c 8000000000-bfffffffff : 0000:0c:00.0 c000000000-c0001fffff : 0000:0c:00.0 fd00000000-ffffffffff : Reserved 10000000000-17fffffffff : PCI Bus 0000:0b 10000000000-13fffffffff : 0000:0b:00.0 14000000000-140001fffff : 0000:0b:00.0
I found that increasing the system ram of the VM to ~256G changes the address window and avoids the issue, and with some digging I think I found the root cause
To begin, QEMU set this address range to be reserved here, and this is passed to
Seabios via "etc/e820".
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L865 Separately QEMU pass to Seabios "etc/reserved-memory-end" here which is derived from machine->device_memory https://gitlab.com/qemu-project/qemu/-/blob/master/hw/i386/pc.c#L1007
In Seabios, "etc/e820" is consumed here, which sets RamSizeOver4G only using
the E820_RAM entries, ignoring any E820_RESERVED entries.
https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/paravirt. c#L782 Later "etc/reserved-memory-end" and RamSizeOver4G is used to determine the start of the PCI-e address window. https://gitlab.com/qemu-project/seabios/-/blob/master/src/fw/pciinit.c #L1138
I think either QEMU should set etc/reserved-memory-end to be after both
physical memory and the reserved ranges, or Seabios need to check both etc/820 and etc/reserved-memory-end. But I'm not sure which would be the correct move and indeed how to patch them.
commit 1e1da7a96300 ("check for e820 conflict") should fix that.
take care, Gerd
Thanks! That indeed fixes the issue, I was using 1.16.2 and 1.16.3 fixes it. I was using a weird version of QEMU but I should have tried updating Seabios itself first, d'oh!
Regards, Teddy
-
Gerd Hoffmann -
Li, Yunxiang (Teddy)