[BUG] [Seabios] PCI 64bit BARs on Win2008 - unable to start the device. (ACPI lacks the _DSM method)
Hi,
We have got some issues with 64bit PCI devices support under Windows guests. Windows fails to start any PCI devices with 64bit BARs when size exceeds 512MB with error code 12 (can't allocate resources). In other words it fails when the PCI memory region is located above 4GB.
I tried to investigate this problem and found document from Microsoft: http://msdn.microsoft.com/en-us/windows/hardware/gg462986
It states that the _DSM method must be implemented to support devices above 4GB.
Unfortunately I'm not familiar with ACPI code of seabios, so can't implement the _DSM method. May be you have other ideas what needs to be done to fix it? Any help is appreciated.
Thanks, Alexey
On Wed, Nov 30, 2011 at 02:04:08PM +1300, Alexey Korolev wrote:
Hi,
We have got some issues with 64bit PCI devices support under Windows guests. Windows fails to start any PCI devices with 64bit BARs when size exceeds 512MB with error code 12 (can't allocate resources). In other words it fails when the PCI memory region is located above 4GB.
I tried to investigate this problem and found document from Microsoft: http://msdn.microsoft.com/en-us/windows/hardware/gg462986
It states that the _DSM method must be implemented to support devices above 4GB.
That's not how I read it: Windows Vista always respects the boot configuration of devices above 4 GB, considering the processor’s addressing capability and the version of the Windows operating system that is running.
Unfortunately I'm not familiar with ACPI code of seabios, so can't implement the _DSM method. May be you have other ideas what needs to be done to fix it? Any help is appreciated.
Thanks, Alexey
Could you share some more detail about your setup?
On Wed, Nov 30, 2011 at 02:04:08PM +1300, Alexey Korolev wrote:
Hi,
We have got some issues with 64bit PCI devices support under Windows guests. Windows fails to start any PCI devices with 64bit BARs when size exceeds 512MB with error code 12 (can't allocate resources). In other words it fails when the PCI memory region is located above 4GB.
I tried to investigate this problem and found document from Microsoft: http://msdn.microsoft.com/en-us/windows/hardware/gg462986
It states that the _DSM method must be implemented to support devices above 4GB.
That's not how I read it: Windows Vista always respects the boot configuration of devices above 4 GB, considering the processor’s addressing capability and the version of the Windows operating system that is running.
Well, it's written in a very interesting way. They do recommend to have _DSM method in order to support 64bit. It is quite possible that the source of the problem is somewhere else.
Actually it is also quite interesting to know what is the resource window for 440FX? Perhaps Windows OS does not believe that 440FX supports 64bit PCI resources. But I see no way to check this.
Unfortunately I'm not familiar with ACPI code of seabios, so can't implement the _DSM method. May be you have other ideas what needs to be done to fix it? Any help is appreciated.
Thanks, Alexey
Could you share some more detail about your setup?
Ok Here are logs the faulty device is: 00:05.0. In the first case the BAR1 size is 512MB and it fails in Windows (Linux is fine, the kernel enumerates the bus and assigns the resource above 4GB), In the second case the BAR1 size is 256M and works fine.
The first one causing issues. As PCI memory region for 00.0.5 is not assigned after windows boot. (Note. I also tried to supply a particular physical addresses over 4GB in pciinit.c code but this leads to completely the same behaviour. Windows just don't pick-up the resource.)
--------------------- Seabios log PCI BAR 512MB: ------------------------------------
Changing serial settings was 0/0 now 3/0 Start bios (version pre-0.6.3-20111201_162012-core-3000-3.et.endace.com) enabling shadow ram qemu_cfg_present=1 Find memory size Ram Size=0xe0000000 (0x0000000320000000 high) malloc setup Relocating init from 0x000e01c0 to 0xdffe0980 (size 62800) malloc fixup reloc init ivt init bda init pic init timer CPU Mhz=2801 math cp init init mtrr pci setup === PCI bus & bridge init === PCI: pci_bios_init_bus_rec bus = 0x0 === PCI device probing === PCI probe PCI device 00:00.0 (vd=8086:1237 c=0600) PCI device 00:01.0 (vd=8086:7000 c=0601) PCI device 00:01.1 (vd=8086:7010 c=0101) PCI device 00:01.2 (vd=8086:7020 c=0c03) PCI device 00:01.3 (vd=8086:7113 c=0680) PCI device 00:02.0 (vd=1013:00b8 c=0300) PCI device 00:03.0 (vd=10ec:8139 c=0200) PCI device 00:05.0 (vd=eace:eace c=0280) PCI device 00:06.0 (vd=1af4:1002 c=0500) Found 9 PCI devices (max PCI bus is 00) === PCI new allocation pass #1 === PCI: check devices bus 0 === PCI new allocation pass #2 === PCI: init bases bus 0 (primary) type io max 100 sum 150 base c000 size 100: 1 bar(s), c000 -> c0ff size 20: 2 bar(s), c100 -> c13f size 10: 1 bar(s), c140 -> c14f type mem max 10000 sum 32000 base febc0000 size 10000: 3 bar(s), febc0000 -> febeffff size 1000: 2 bar(s), febf0000 -> febf1fff type prefmem max 2000000 sum 2000000 base fc000000 size 2000000: 1 bar(s), fc000000 -> fdffffff PCI: map device bus 0, bfd 0x0 PCI: map device bus 0, bfd 0x8 PCI: map device bus 0, bfd 0x9 bar 4, addr c140, size 10 [io] PCI: map device bus 0, bfd 0xa bar 4, addr c100, size 20 [io] PCI: map device bus 0, bfd 0xb PCI: map device bus 0, bfd 0x10 bar 0, addr fc000000, size 2000000 [mem] bar 1, addr febf0000, size 1000 [mem] bar 6, addr febc0000, size 10000 [mem] PCI: map device bus 0, bfd 0x18 bar 0, addr c000, size 100 [io] bar 1, addr febf1000, size 100 [mem] bar 6, addr febd0000, size 10000 [mem] PCI: map device bus 0, bfd 0x28 bar 0, addr febe0000, size 10000 [mem] bar 1, addr 0, size 20000000 [mem] PCI: map device bus 0, bfd 0x30 bar 0, addr c120, size 20 [io] PCI: bus=0 devfn=0x00: vendor_id=0x8086 device_id=0x1237 PCI: bus=0 devfn=0x08: vendor_id=0x8086 device_id=0x7000 PCI: bus=0 devfn=0x09: vendor_id=0x8086 device_id=0x7010 PCI: bus=0 devfn=0x0a: vendor_id=0x8086 device_id=0x7020 PCI: bus=0 devfn=0x0b: vendor_id=0x8086 device_id=0x7113 PCI: bus=0 devfn=0x10: vendor_id=0x1013 device_id=0x00b8 PCI: bus=0 devfn=0x18: vendor_id=0x10ec device_id=0x8139 PCI: bus=0 devfn=0x28: vendor_id=0xeace device_id=0xeace PCI: bus=0 devfn=0x30: vendor_id=0x1af4 device_id=0x1002 init smm boot order: 1: /pci@i0cf8/ide@1,1/drive@0/disk@0 2: /pci@i0cf8/ide@1,1/drive@1/disk@0 Found 2 cpu(s) max supported 2 cpu(s) init bios32 init PMM init PNPBIOS table init keyboard init mouse init PIR table init MPTable MP table addr=0x000fda00 MPC table addr=0x000fda10 size=260 init SMBIOS tables SMBIOS ptr=0x000fd9e0 table=0x000fd880 size=346 init ACPI tables ACPI tables: RSDP=0x000fd850 RSDT=0xdfffd740 Scan for VGA option rom Attempting to init PCI bdf 00:02.0 (vd 1013:00b8) Copying option rom (size 35840) from 0xfebc0000 to c0000 Running option rom at c000:0003 Turning on vga text mode console SeaBIOS (version pre-0.6.3-20111201_162012-core-3000-3.et.endace.com)
init usb UHCI init on dev 00:01.2 (io=c100) /dffdf000\ Start thread /dffde000\ Start thread /dffdd000\ Start thread \dffdd000/ End thread init ps2port /dffdd000\ Start thread init lpt Found 0 lpt ports init serial Found 1 serial ports init floppy drives init hard drives ddr 0 ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9) /dffdb000\ Start thread ATA controller 2 at 170/374/0 (irq 15 dev 9) /dffda000\ Start thread init ahci init virtio-blk |dffdb000| ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (51200 MiBytes) |dffdb000| Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0 |dffdb000| Registering bootable: ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (51200 MiBytes) (type:2 prio:1 data:fd800) \dffdb000/ End thread |dffda000| DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD] |dffda000| Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0 |dffda000| Registering bootable: DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD] (type:3 prio:2 data:fd7d0) \dffda000/ End thread |dffde000| set_address 0xdffe0150 |dffde000| config_usb: 0xdffdccec |dffde000| device rev=0100 cls=00 sub=00 proto=00 size=08 |dffde000| usb_hid_init 0xdffdccec \dffde000/ End thread \dffdf000/ End thread |dffdd000| PS2 keyboard initialized \dffdd000/ End thread All threads complete. Scan for option roms Attempting to init PCI bdf 00:00.0 (vd 8086:1237) Attempting to init PCI bdf 00:01.0 (vd 8086:7000) Attempting to init PCI bdf 00:01.2 (vd 8086:7020) Attempting to init PCI bdf 00:01.3 (vd 8086:7113) Attempting to init PCI bdf 00:03.0 (vd 10ec:8139) Copying option rom (size 61440) from 0xfebd0000 to c9000 Running option rom at c900:0003 pmm call arg1=1 pmm01: handle=18ae1000 pmm call arg1=0 pmm00: length=f00 handle=18ae1000 flags=2 pmm call arg1=1 pmm01: handle=18ae2004 pmm call arg1=0 pmm00: length=4000 handle=18ae2004 flags=2 Attempting to init PCI bdf 00:05.0 (vd eace:eace) Attempting to init PCI bdf 00:06.0 (vd 1af4:1002) Searching bootorder for: /pci@i0cf8/*@3 Registering bootable: iPXE (PCI 00:03.0) (type:128 prio:9999 data:c9000372) Searching bootorder for: /rom@genroms/vapic.bin Registering bootable: Legacy option rom (type:129 prio:9999 data:ca000003) Mapping hd drive 0x000fd800 to 0 drive 0x000fd800: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=104857600 Mapping cd drive 0x000fd7d0 Running option rom at ca00:0003 ebda moved from 9fc00 to 9f400 finalize PMM malloc finalize Returned 53248 bytes of ZoneHigh e820 map has 7 items: 0: 0000000000000000 - 000000000009f400 = 1 RAM 1: 000000000009f400 - 00000000000a0000 = 2 RESERVED 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED 3: 0000000000100000 - 00000000dfffd000 = 1 RAM 4: 00000000dfffd000 - 00000000e0000000 = 2 RESERVED 5: 00000000fffbc000 - 0000000100000000 = 2 RESERVED 6: 0000000100000000 - 0000000420000000 = 1 RAM locking shadow ram Jump to int19 enter handle_19:
-------- PCI INFO after boot ---------------------- Bus 0, device 0, function 0: Host bridge: PCI device 8086:1237 id "" Bus 0, device 1, function 0: ISA bridge: PCI device 8086:7000 id "" Bus 0, device 1, function 1: IDE controller: PCI device 8086:7010 BAR4: I/O at 0xc140 [0xc14f]. id "" Bus 0, device 1, function 2: USB controller: PCI device 8086:7020 IRQ 5. BAR4: I/O at 0xc100 [0xc11f]. id "" Bus 0, device 1, function 3: Bridge: PCI device 8086:7113 IRQ 9. id "" Bus 0, device 2, function 0: VGA controller: PCI device 1013:00b8 BAR0: 32 bit prefetchable memory at 0xfc000000 [0xfdffffff]. BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 3, function 0: Ethernet controller: PCI device 10ec:8139 IRQ 11. BAR0: I/O at 0xc000 [0xc0ff]. BAR1: 32 bit memory at 0xfebf1000 [0xfebf10ff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "net0" Bus 0, device 5, function 0: Network controller: PCI device eace:eace IRQ 10. BAR0: 32 bit memory at 0xfebe0000 [0xfebeffff]. BAR1: 64 bit prefetchable memory at 0xffffffffffffffff [0x1ffffffe]. id "" Bus 0, device 6, function 0: RAM controller: PCI device 1af4:1002 IRQ 10. BAR0: I/O at 0xc120 [0xc13f]. id "balloon0"
--------------------- Seabios log PCI BAR 256MB: ------------------------------------
Changing serial settings was 0/0 now 3/0 Start bios (version pre-0.6.3-20111201_162012-core-3000-3.et.endace.com) enabling shadow ram qemu_cfg_present=1 Find memory size Ram Size=0xe0000000 (0x0000000320000000 high) malloc setup Relocating init from 0x000e01c0 to 0xdffe0980 (size 62800) malloc fixup reloc init ivt init bda init pic init timer CPU Mhz=2801 math cp init init mtrr pci setup === PCI bus & bridge init === PCI: pci_bios_init_bus_rec bus = 0x0 === PCI device probing === PCI probe PCI device 00:00.0 (vd=8086:1237 c=0600) PCI device 00:01.0 (vd=8086:7000 c=0601) PCI device 00:01.1 (vd=8086:7010 c=0101) PCI device 00:01.2 (vd=8086:7020 c=0c03) PCI device 00:01.3 (vd=8086:7113 c=0680) PCI device 00:02.0 (vd=1013:00b8 c=0300) PCI device 00:03.0 (vd=10ec:8139 c=0200) PCI device 00:05.0 (vd=eace:eace c=0280) PCI device 00:06.0 (vd=1af4:1002 c=0500) Found 9 PCI devices (max PCI bus is 00) === PCI new allocation pass #1 === PCI: check devices bus 0 === PCI new allocation pass #2 === PCI: init bases bus 0 (primary) type io max 100 sum 150 base c000 size 100: 1 bar(s), c000 -> c0ff size 20: 2 bar(s), c100 -> c13f size 10: 1 bar(s), c140 -> c14f type mem max 10000 sum 32000 base febc0000 size 10000: 3 bar(s), febc0000 -> febeffff size 1000: 2 bar(s), febf0000 -> febf1fff type prefmem max 2000000 sum 2000000 base fc000000 size 2000000: 1 bar(s), fc000000 -> fdffffff PCI: map device bus 0, bfd 0x0 PCI: map device bus 0, bfd 0x8 PCI: map device bus 0, bfd 0x9 bar 4, addr c140, size 10 [io] PCI: map device bus 0, bfd 0xa bar 4, addr c100, size 20 [io] PCI: map device bus 0, bfd 0xb PCI: map device bus 0, bfd 0x10 bar 0, addr fc000000, size 2000000 [mem] bar 1, addr febf0000, size 1000 [mem] bar 6, addr febc0000, size 10000 [mem] PCI: map device bus 0, bfd 0x18 bar 0, addr c000, size 100 [io] bar 1, addr febf1000, size 100 [mem] bar 6, addr febd0000, size 10000 [mem] PCI: map device bus 0, bfd 0x28 bar 0, addr febe0000, size 10000 [mem] bar 1, addr 0, size 10000000 [mem] PCI: map device bus 0, bfd 0x30 bar 0, addr c120, size 20 [io] PCI: bus=0 devfn=0x00: vendor_id=0x8086 device_id=0x1237 PCI: bus=0 devfn=0x08: vendor_id=0x8086 device_id=0x7000 PCI: bus=0 devfn=0x09: vendor_id=0x8086 device_id=0x7010 PCI: bus=0 devfn=0x0a: vendor_id=0x8086 device_id=0x7020 PCI: bus=0 devfn=0x0b: vendor_id=0x8086 device_id=0x7113 PCI: bus=0 devfn=0x10: vendor_id=0x1013 device_id=0x00b8 PCI: bus=0 devfn=0x18: vendor_id=0x10ec device_id=0x8139 PCI: bus=0 devfn=0x28: vendor_id=0xeace device_id=0xeace PCI: bus=0 devfn=0x30: vendor_id=0x1af4 device_id=0x1002 init smm boot order: 1: /pci@i0cf8/ide@1,1/drive@0/disk@0 2: /pci@i0cf8/ide@1,1/drive@1/disk@0 Found 2 cpu(s) max supported 2 cpu(s) init bios32 init PMM init PNPBIOS table init keyboard init mouse init PIR table init MPTable MP table addr=0x000fda00 MPC table addr=0x000fda10 size=260 init SMBIOS tables SMBIOS ptr=0x000fd9e0 table=0x000fd880 size=346 init ACPI tables ACPI tables: RSDP=0x000fd850 RSDT=0xdfffd740 Scan for VGA option rom Attempting to init PCI bdf 00:02.0 (vd 1013:00b8) Copying option rom (size 35840) from 0xfebc0000 to c0000 Running option rom at c000:0003 Turning on vga text mode console SeaBIOS (version pre-0.6.3-20111201_162012-core-3000-3.et.endace.com)
init usb UHCI init on dev 00:01.2 (io=c100) /dffdf000\ Start thread /dffde000\ Start thread /dffdd000\ Start thread \dffdd000/ End thread init ps2port /dffdd000\ Start thread init lpt Found 0 lpt ports init serial Found 1 serial ports init floppy drives init hard drives ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9) /dffdb000\ Start thread ATA controller 2 at 170/374/0 (irq 15 dev 9) /dffda000\ Start thread init ahci init virtio-blk |dffdb000| ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (51200 MiBytes) |dffdb000| Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0 |dffdb000| Registering bootable: ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (51200 MiBytes) (type:2 prio:1 data:fd800) \dffdb000/ End thread |dffda000| DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD] |dffda000| Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0 |dffda000| Registering bootable: DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD] (type:3 prio:2 data:fd7d0) \dffda000/ End thread |dffde000| set_address 0xdffe0150 |dffde000| config_usb: 0xdffdccec |dffde000| device rev=0100 cls=00 sub=00 proto=00 size=08 |dffde000| usb_hid_init 0xdffdccec \dffde000/ End thread \dffdf000/ End thread |dffdd000| PS2 keyboard initialized \dffdd000/ End thread All threads complete. Scan for option roms Attempting to init PCI bdf 00:00.0 (vd 8086:1237) Attempting to init PCI bdf 00:01.0 (vd 8086:7000) Attempting to init PCI bdf 00:01.2 (vd 8086:7020) Attempting to init PCI bdf 00:01.3 (vd 8086:7113) Attempting to init PCI bdf 00:03.0 (vd 10ec:8139) Copying option rom (size 61440) from 0xfebd0000 to c9000 Running option rom at c900:0003 pmm call arg1=1 pmm01: handle=18ae1000 pmm call arg1=0 pmm00: length=f00 handle=18ae1000 flags=2 pmm call arg1=1 pmm01: handle=18ae2004 pmm call arg1=0 pmm00: length=4000 handle=18ae2004 flags=2 Attempting to init PCI bdf 00:05.0 (vd eace:eace) Attempting to init PCI bdf 00:06.0 (vd 1af4:1002) Searching bootorder for: /pci@i0cf8/*@3 Registering bootable: iPXE (PCI 00:03.0) (type:128 prio:9999 data:c9000372) Searching bootorder for: /rom@genroms/vapic.bin Registering bootable: Legacy option rom (type:129 prio:9999 data:ca000003) Mapping hd drive 0x000fd800 to 0 drive 0x000fd800: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=104857600 Mapping cd drive 0x000fd7d0 Running option rom at ca00:0003 ebda moved from 9fc00 to 9f400 finalize PMM malloc finalize Returned 53248 bytes of ZoneHigh e820 map has 7 items: 0: 0000000000000000 - 000000000009f400 = 1 RAM 1: 000000000009f400 - 00000000000a0000 = 2 RESERVED 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED 3: 0000000000100000 - 00000000dfffd000 = 1 RAM 4: 00000000dfffd000 - 00000000e0000000 = 2 RESERVED 5: 00000000fffbc000 - 0000000100000000 = 2 RESERVED 6: 0000000100000000 - 0000000420000000 = 1 RAM locking shadow ram Jump to int19 enter handle_19: NULL Booting from Hard Disk... Booting from 0000:7c00 unimplemented handle_1aXX:441: a=0000bb00 b=0000aa55 c=0000ffff d=00000080 ds=0000 es=0000 ss=0000 si=00007e00 di=00000800 bp=000007be sp=00007bfe cs=0000 ip=06e7 f=3246 unimplemented handle_1aXX:441: a=0000bb00 b=00000000 c=00000000 d=000009a0 ds=07c0 es=09a0 ss=0000 si=00007be4 di=00000800 bp=000007be sp=00007c00 cs=07c0 ip=00db f=3246 unimplemented handle_1aXX:441: a=0000bb00 b=00000000 c=0000005e d=00000000 ds=07c0 es=2000 ss=0000 si=00003808 di=00000000 bp=000007be sp=00007bda cs=07c0 ip=0f54 f=3246 unimplemented handle_1aXX:441: a=0000bb00 b=00000000 c=00000000 d=00000000 ds=0000 es=0000 ss=23de si=00000000 di=00000000 bp=00000000 sp=00001462 cs=2000 ip=0a39 f=3256 unimplemented handle_1aXX:441: a=0000bb00 b=00000000 c=00000000 d=00000000 ds=0000 es=0000 ss=23de si=00000000 di=00000000 bp=00000000 sp=00001462 cs=2000 ip=0a39 f=3256 invalid handle_legacy_disk:841: a=00001500 b=00000000 c=00000000 d=00000000 ds=0000 es=0000 ss=23de si=00000000 di=00000000 bp=00000000 sp=00001462 cs=2000 ip=0a1d f=3246 invalid handle_legacy_disk:841: a=00001500 b=00000000 c=00000000 d=00000001 ds=0000 es=0000 ss=23de si=00000000 di=00000000 bp=00000000 sp=00001462 cs=2000 ip=0a1d f=3246
--------------------- Seabios log PCI BAR 256MB: ------------------------------------
Bus 0, device 0, function 0: Host bridge: PCI device 8086:1237 id "" Bus 0, device 1, function 0: ISA bridge: PCI device 8086:7000 id "" Bus 0, device 1, function 1: IDE controller: PCI device 8086:7010 BAR4: I/O at 0xc140 [0xc14f]. id "" Bus 0, device 1, function 2: USB controller: PCI device 8086:7020 IRQ 5. BAR4: I/O at 0xc100 [0xc11f]. id "" Bus 0, device 1, function 3: Bridge: PCI device 8086:7113 IRQ 9. id "" Bus 0, device 2, function 0: VGA controller: PCI device 1013:00b8 BAR0: 32 bit prefetchable memory at 0xfc000000 [0xfdffffff]. BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 3, function 0: Ethernet controller: PCI device 10ec:8139 IRQ 11. BAR0: I/O at 0xc000 [0xc0ff]. BAR1: 32 bit memory at 0xfebf1000 [0xfebf10ff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "net0" Bus 0, device 5, function 0: Network controller: PCI device eace:eace IRQ 10. BAR0: 32 bit memory at 0xfebe0000 [0xfebeffff]. BAR1: 64 bit prefetchable memory at 0xe0000000 [0xefffffff]. id "" Bus 0, device 6, function 0: RAM controller: PCI device 1af4:1002 IRQ 10. BAR0: I/O at 0xc120 [0xc13f]. id "balloon0"
Hi,
Ok Here are logs the faulty device is: 00:05.0. In the first case the BAR1 size is 512MB and it fails in Windows (Linux is fine, the kernel enumerates the bus and assigns the resource above 4GB), In the second case the BAR1 size is 256M and works fine.
=== PCI new allocation pass #1 === PCI: check devices bus 0
Hmm, with 512 MB seabios should have paniced here with:
PCI: out of address space
PCI: map device bus 0, bfd 0x28 bar 0, addr febe0000, size 10000 [mem] bar 1, addr 0, size 20000000 [mem]
Somehow seabios didn't recognise the bar correctly it seems (both 512 and 256 MB cases look the same). For the 256 MB case seabios should have mapped the bar @ 0xe0000000.
cheers, Gerd
Hi,
PCI: map device bus 0, bfd 0x28 bar 0, addr febe0000, size 10000 [mem] bar 1, addr 0, size 20000000 [mem]
Somehow seabios didn't recognise the bar correctly it seems (both 512 and 256 MB cases look the same). For the 256 MB case seabios should have mapped the bar @ 0xe0000000.
... and it should also have figured it is prefetchable memory. Was pci config space messed up somehow? What does 'lspci -v' say once you've booted the machine with linux?
What qemu version you are running? What kind of device is this? Emulated? Code somewhere? Or a real device passed through to the guest?
cheers, Gerd
Hi Gerd,
We have very early prototype of data acquisition device, with quite large MMIO buffer. It is an emulated device. We are running the 0.15 release. 0.15 doesn't work correctly with 64bit BARs so I've already added some hacks to Seabios to let OS to choose the memory region. Thus you see bar 1, addr 0 in seabios log. Sorry that I haven't specified all this initially. I just want to make 64bit PCI bar working properly. Linux guests works correctly (except early versions - not investigated this yet). At the moment I have some issues with windows which relies on ACPI _CRS.
Thanks, Alexey
Hi,
PCI: map device bus 0, bfd 0x28 bar 0, addr febe0000, size 10000 [mem] bar 1, addr 0, size 20000000 [mem]
Somehow seabios didn't recognise the bar correctly it seems (both 512 and 256 MB cases look the same). For the 256 MB case seabios should have mapped the bar @ 0xe0000000.
... and it should also have figured it is prefetchable memory. Was pci config space messed up somehow? What does 'lspci -v' say once you've booted the machine with linux?
What qemu version you are running? What kind of device is this? Emulated? Code somewhere? Or a real device passed through to the guest?
cheers, Gerd
On 12/05/11 05:29, Alexey Korolev wrote:
Hi Gerd,
We have very early prototype of data acquisition device, with quite large MMIO buffer. It is an emulated device. We are running the 0.15 release. 0.15 doesn't work correctly with 64bit BARs so I've already added some hacks to Seabios to let OS to choose the memory region. Thus you see bar 1, addr 0 in seabios log.
I'd strongly suggest to move forward to qemu 1.0. Memory region handling has seen a major rewrite in 1.0 (memory api patches by avi). Chances are good that the 64bit bar bugs in qemu have been fixed meanwhile.
I have experimental patches which add a 64bit bar to the qxl device and seabios handles it just fine (although memory-backed not mmio), except that there is no support yet to map 64bit bars above 4G.
It shouldn't be that hard to add the latter though. seabios needs two more pci_region_type (PCI_REGION_TYPE_MEM_64 and PCI_REGION_TYPE_PREFMEM_64) to track and map 64bit bars separately. And a address space window where it can map 64bit bars to.
Sorry that I haven't specified all this initially. I just want to make 64bit PCI bar working properly. Linux guests works correctly (except early versions - not investigated this yet). At the moment I have some issues with windows which relies on ACPI _CRS.
... and a _CRS entry for the 64bit bar address space window of course.
cheers, Gerd
Hi Gerd,
I'd strongly suggest to move forward to qemu 1.0. Memory region handling has seen a major rewrite in 1.0 (memory api patches by avi). Chances are good that the 64bit bar bugs in qemu have been fixed meanwhile.
Thanks, will try it. Hope it will be better.
I have experimental patches which add a 64bit bar to the qxl device and seabios handles it just fine (although memory-backed not mmio), except that there is no support yet to map 64bit bars above 4G.
It shouldn't be that hard to add the latter though. seabios needs two more pci_region_type (PCI_REGION_TYPE_MEM_64 and PCI_REGION_TYPE_PREFMEM_64) to track and map 64bit bars separately. And a address space window where it can map 64bit bars to.
Right. This is a thing I'm thinking about now. I seems that the specifying 0x0 address in 64bit BARS is a bad idea. At least older versions of Linux just hang as soon as requested range does not fit in first 4GB. So the only option would be specifying the particular address range. It seems this works for everything - yet.
Sorry that I haven't specified all this initially. I just want to make 64bit PCI bar working properly. Linux guests works correctly (except early versions - not investigated this yet). At the moment I have some issues with windows which relies on ACPI _CRS.
... and a _CRS entry for the 64bit bar address space window of course.
Yes.
Cheers, Alexey
Hi,
It shouldn't be that hard to add the latter though. seabios needs two more pci_region_type (PCI_REGION_TYPE_MEM_64 and PCI_REGION_TYPE_PREFMEM_64) to track and map 64bit bars separately. And a address space window where it can map 64bit bars to.
Right. This is a thing I'm thinking about now. I seems that the specifying 0x0 address in 64bit BARS is a bad idea. At least older versions of Linux just hang as soon as requested range does not fit in first 4GB. So the only option would be specifying the particular address range. It seems this works for everything - yet.
I wonder if there any particular reason to separate prefetchable a non-prefetchable memory regions in pciinit? Extra two more regions would make code more complex.
Regards, Alexey
On Wed, Dec 14, 2011 at 05:30:23PM +1300, Alexey Korolev wrote:
Hi,
It shouldn't be that hard to add the latter though. seabios needs two more pci_region_type (PCI_REGION_TYPE_MEM_64 and PCI_REGION_TYPE_PREFMEM_64) to track and map 64bit bars separately. And a address space window where it can map 64bit bars to.
Right. This is a thing I'm thinking about now. I seems that the specifying 0x0 address in 64bit BARS is a bad idea. At least older versions of Linux just hang as soon as requested range does not fit in first 4GB. So the only option would be specifying the particular address range. It seems this works for everything - yet.
I wonder if there any particular reason to separate prefetchable a non-prefetchable memory regions in pciinit? Extra two more regions would make code more complex.
If I understand this correctly, the prefmem and normal mem separation is for pci bridges. Devices behind a bridge need to have all their prefmem regions consecutive and not overlapping with all the normal mem ranges. That way the bridge device can advertise just one prefmem region and one normal mem region which encapsulates all the devices behind it.
-Kevin
On 12/14/11 05:30, Alexey Korolev wrote:
Hi,
It shouldn't be that hard to add the latter though. seabios needs two more pci_region_type (PCI_REGION_TYPE_MEM_64 and PCI_REGION_TYPE_PREFMEM_64) to track and map 64bit bars separately. And a address space window where it can map 64bit bars to.
Right. This is a thing I'm thinking about now. I seems that the specifying 0x0 address in 64bit BARS is a bad idea. At least older versions of Linux just hang as soon as requested range does not fit in first 4GB. So the only option would be specifying the particular address range. It seems this works for everything - yet.
I wonder if there any particular reason to separate prefetchable a non-prefetchable memory regions in pciinit? Extra two more regions would make code more complex.
Oh yes, there is. Which reminds me that the whole thing isn't that easy unfortunaly ...
The reason are pci bridges. They have two memory regions, one for prefetchable and one for non-prefetchable memory. All devices behind a pci bridge must have the bars within the bridges memory regions, thats why they are grouped together.
This also implies that a 32bit and a 64bit bar (of the same type) behind a pci bridge must be mapped next to each other, so moving up 64bit bars unconditionally isn't going to fly. Devices behind bridges need some extra care, only when all bars are 64bit capable they can actually be mapped above 4G.
cheers, Gerd
I wonder if there any particular reason to separate prefetchable a non-prefetchable memory regions in pciinit? Extra two more regions would make code more complex.
Oh yes, there is. Which reminds me that the whole thing isn't that easy unfortunaly ...
The reason are pci bridges. They have two memory regions, one for prefetchable and one for non-prefetchable memory. All devices behind a pci bridge must have the bars within the bridges memory regions, thats why they are grouped together.
This also implies that a 32bit and a 64bit bar (of the same type) behind a pci bridge must be mapped next to each other, so moving up 64bit bars unconditionally isn't going to fly. Devices behind bridges need some extra care, only when all bars are 64bit capable they can actually be mapped above 4G.
The qemu actually does not simulate PCI bridges at all. So good question shall we bother about this? I did some preliminary tests: 64bit BARs are working quite well for linux 2.6.18 - 3.0 and windows 2008.
Also I've found important detail that according to PCI architecture specification, the bridges can describe 64bit ranges for prefetchable type of memory only. So it's very unlikely that devices exporting 64bit non-prefetchable BARs. I guess we just need to add one extra type.
Even if there are two separate prefetchable memory regions (32 bit and 64bit), it won't be a problem as there is no bridge on the 440FX inside the virtual machine.
Cheers, Alexey
On 12/16/11 05:41, Alexey Korolev wrote:
I wonder if there any particular reason to separate prefetchable a non-prefetchable memory regions in pciinit? Extra two more regions would make code more complex.
Oh yes, there is. Which reminds me that the whole thing isn't that easy unfortunaly ...
The reason are pci bridges. They have two memory regions, one for prefetchable and one for non-prefetchable memory. All devices behind a pci bridge must have the bars within the bridges memory regions, thats why they are grouped together.
This also implies that a 32bit and a 64bit bar (of the same type) behind a pci bridge must be mapped next to each other, so moving up 64bit bars unconditionally isn't going to fly. Devices behind bridges need some extra care, only when all bars are 64bit capable they can actually be mapped above 4G.
The qemu actually does not simulate PCI bridges at all. So good question shall we bother about this?
Yes. There is work in progress to add pci-express and a more modern chipset emulation to qemu.
I think for now it is fine if we simply map everything behind a bridge below 4G and hash out the details (i.e. whenever we can move the bridge memory window above 4G or not) later.
I did some preliminary tests: 64bit BARs are working quite well for linux 2.6.18 - 3.0 and windows 2008.
Nice.
Also I've found important detail that according to PCI architecture specification, the bridges can describe 64bit ranges for prefetchable type of memory only. So it's very unlikely that devices exporting 64bit non-prefetchable BARs. I guess we just need to add one extra type.
Agree.
Even if there are two separate prefetchable memory regions (32 bit and 64bit), it won't be a problem as there is no bridge on the 440FX inside the virtual machine.
Yes, devices attached to the root bus don't have to care about bridge windows.
cheers, Gerd
Isaku san,
I've just added you to discussion. There are some issues with PCI 64bit support in Windows. Windows fails to assign the resource if it doesn't fit in first 4GB window.
I really don't know why it happens. One of the possibilities is related to lack of _DSM method in ACPI.
Another guesse could be related to the fact that 440FX only supports 32bit PCI bus interface and windows may limit PCI address range to first 4GB for PCI devices under this bridge. I remember you were working on Q35 chipset simulation, I wonder if it is working and would it be possible to try?
Thanks, Alexey
On Wed, Nov 30, 2011 at 02:04:08PM +1300, Alexey Korolev wrote:
Hi,
We have got some issues with 64bit PCI devices support under Windows guests. Windows fails to start any PCI devices with 64bit BARs when size exceeds 512MB with error code 12 (can't allocate resources). In other words it fails when the PCI memory region is located above 4GB.
I tried to investigate this problem and found document from Microsoft: http://msdn.microsoft.com/en-us/windows/hardware/gg462986
It states that the _DSM method must be implemented to support devices above 4GB.
That's not how I read it: Windows Vista always respects the boot configuration of devices above 4 GB, considering the processor’s addressing capability and the version of the Windows operating system that is running.
Unfortunately I'm not familiar with ACPI code of seabios, so can't implement the _DSM method. May be you have other ideas what needs to be done to fix it? Any help is appreciated.
Thanks, Alexey
Could you share some more detail about your setup?
On Thu, Dec 01, 2011 at 06:49:54PM +1300, Alexey Korolev wrote:
Isaku san,
I've just added you to discussion. There are some issues with PCI 64bit support in Windows. Windows fails to assign the resource if it doesn't fit in first 4GB window.
I really don't know why it happens. One of the possibilities is related to lack of _DSM method in ACPI.
Another guesse could be related to the fact that 440FX only supports 32bit PCI bus interface and windows may limit PCI address range to first 4GB for PCI devices under this bridge. I remember you were working on Q35 chipset simulation, I wonder if it is working and would it be possible to try?
Thanks, Alexey
Maybe the range above 4G needs to be declared in the _CRS resource?
Hi Michael,
Thank you for good advice, you are right. When I added new range above 4GB in _CRS the problem has gone. QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x00000000, // Address Space Granularity 0x100000000, // Address Range Minimum 0x3FFFFFFFF, // Address Range Maximum 0x00000000, // Address Translation Offset 0x400000000, // Address Length ,, , AddressRangeMemory, TypeStatic)
The only big problem with this range - as soon as I have more than 3GB of RAM, windows will boot in BSOD. The problem relates to memory range intersection. Unfortunately it is not possible to predict how many GB of RAM the virtual machine could have - so it's difficult to specify a particular region. Do you have any ideas what can be done to solve this problem?
Regards, Alexey
On Thu, Dec 01, 2011 at 06:49:54PM +1300, Alexey Korolev wrote:
Isaku san,
I've just added you to discussion. There are some issues with PCI 64bit support in Windows. Windows fails to assign the resource if it doesn't fit in first 4GB window.
I really don't know why it happens. One of the possibilities is related to lack of _DSM method in ACPI.
Another guesse could be related to the fact that 440FX only supports 32bit PCI bus interface and windows may limit PCI address range to first 4GB for PCI devices under this bridge. I remember you were working on Q35 chipset simulation, I wonder if it is working and would it be possible to try?
Thanks, Alexey
Maybe the range above 4G needs to be declared in the _CRS resource?
On Mon, Dec 05, 2011 at 05:20:32PM +1300, Alexey Korolev wrote:
Hi Michael,
Thank you for good advice, you are right. When I added new range above 4GB in _CRS the problem has gone. QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x00000000, // Address Space Granularity 0x100000000, // Address Range Minimum 0x3FFFFFFFF, // Address Range Maximum 0x00000000, // Address Translation Offset 0x400000000, // Address Length ,, , AddressRangeMemory, TypeStatic)
The only big problem with this range - as soon as I have more than 3GB of RAM, windows will boot in BSOD. The problem relates to memory range intersection. Unfortunately it is not possible to predict how many GB of RAM the virtual machine could have - so it's difficult to specify a particular region. Do you have any ideas what can be done to solve this problem?
Regards, Alexey
Two possible ideas: 1. Pass the value in from qemu 2. Get a range toward the upper end of the memory, around 1<<40
On Thu, Dec 01, 2011 at 06:49:54PM +1300, Alexey Korolev wrote:
Isaku san,
I've just added you to discussion. There are some issues with PCI 64bit support in Windows. Windows fails to assign the resource if it doesn't fit in first 4GB window.
I really don't know why it happens. One of the possibilities is related to lack of _DSM method in ACPI.
Another guesse could be related to the fact that 440FX only supports 32bit PCI bus interface and windows may limit PCI address range to first 4GB for PCI devices under this bridge. I remember you were working on Q35 chipset simulation, I wonder if it is working and would it be possible to try?
Thanks, Alexey
Maybe the range above 4G needs to be declared in the _CRS resource?
On 05/12/11 19:31, Michael S. Tsirkin wrote:
On Mon, Dec 05, 2011 at 05:20:32PM +1300, Alexey Korolev wrote:
Hi Michael,
Thank you for good advice, you are right. When I added new range above 4GB in _CRS the problem has gone. QWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x00000000, // Address Space Granularity 0x100000000, // Address Range Minimum 0x3FFFFFFFF, // Address Range Maximum 0x00000000, // Address Translation Offset 0x400000000, // Address Length ,, , AddressRangeMemory, TypeStatic)
The only big problem with this range - as soon as I have more than 3GB of RAM, windows will boot in BSOD. The problem relates to memory range intersection. Unfortunately it is not possible to predict how many GB of RAM the virtual machine could have - so it's difficult to specify a particular region. Do you have any ideas what can be done to solve this problem?
Regards, Alexey
Two possible ideas:
- Pass the value in from qemu
Do you mean to create the 64bit region dynamically? I've tried to obtain RAM size somehow inside DSL code, but the ACPI spec doesn't tell how to do that.
- Get a range toward the upper end of the memory, around 1<<40
Yes. This solution works great. And this is very simple as well. Just wondering if it will be good as a complete solution?
Hi,
Two possible ideas:
- Pass the value in from qemu
Do you mean to create the 64bit region dynamically? I've tried to obtain RAM size somehow inside DSL code, but the ACPI spec doesn't tell how to do that.
There is a global variable IIRC. You could also check the e820 table passed in from qemu.
- Get a range toward the upper end of the memory, around 1<<40
Yes. This solution works great. And this is very simple as well. Just wondering if it will be good as a complete solution?
I would have tried this first too ;)
Care to share the patches so we can review & merge them?
thanks, Gerd
- Get a range toward the upper end of the memory, around 1<<40
Yes. This solution works great. And this is very simple as well. Just wondering if it will be good as a complete solution?
I would have tried this first too ;)
Care to share the patches so we can review& merge them?
No worries. Just need to complete and test the code and I'll post RFC patches shortly.
-
Alexey Korolev -
Gerd Hoffmann -
Kevin O'Connor -
Michael S. Tsirkin