Large 32 bit BAR's

[Wannes Bouwen (Nokia)]

Hello,

I'm running a x86 QEMU with Ubuntu. On this system we want to emulate a PCIe device which contains a large 32-bit BAR (2 GiB). This is not working and I get following error:

SeaBIOS (version rel-1.14.0-0-g155821a-20210629_105355-sharpie)
BUILD: gcc: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) binutils: version 2.27-34.base.el7
No Xen hypervisor found.
Running on QEMU (q35)
Running on KVM
Found QEMU fw_cfg
QEMU fw_cfg DMA interface supported
qemu/e820: addr 0x0000000000000000 len 0x0000000080000000 [RAM]
qemu/e820: addr 0x0000000100000000 len 0x0000000180000000 [RAM]
Relocating init from 0x000d4de0 to 0x7ffa9420 (size 93008)
Moving pm_base to 0x600
kvmclock: at 0xeb040 (msr 0x4b564d01)
kvmclock: stable tsc, 2992 MHz
CPU Mhz=2992 (kvmclock)
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
PCI: pci_bios_init_bus_rec bdf = 0x20
PCI: primary bus = 0x0
PCI: secondary bus = 0xff -> 0x1
PCI: pci_bios_init_bus_rec bus = 0x1
PCI: subordinate bus = 0x0 -> 0x1
=== PCI device probing ===
Found 12 PCI devices (max PCI bus is 01)
PCIe: using q35 mmconfig at 0xb0000000
=== PCI new allocation pass #1 ===
PCI: check devices
PCI: secondary bus 1 size 00000000 type io
PCI: secondary bus 1 size 100000000 type mem
PCI: secondary bus 1 size 100000000 type prefmem
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c0ff
PCI: 32: 00000000c0000000 - 00000000fec00000
PCI: out of 32bit address space

I've had a look at the code and it seems that 32-bit BAR's have a limited memory region available (less than 1 GiB). Is there a possibility to use such large 32-bit BAR's in combination with SeaBios by e.g. mapping them in the higher memory region where 64-bit BAR's are mapped?

Thanks in advance!

Kind regards,

Wannes

[Daniel Paziyski]

Hello,

For a device with such a big 32 bit BAR, it would be a challenge to fit 
it inside the 32 bit window, since the RAM region available on <4GB must 
be specially tuned with help from the emulator (which, as far as I am 
aware, isn't available) to give space for BAR allocations there. Is 
there a possibility for that emulated PCIe device to be configured to 
expose a 64 bit BAR instead?

Regards,

Daniel

On 4/12/23 08:23, Wannes Bouwen (Nokia) wrote:
> Hello,
>
> I’m running a x86 QEMU with Ubuntu. On this system we want to emulate 
> a PCIe device which contains a large 32-bit BAR (2 GiB). This is not 
> working and I get following error:
>
> SeaBIOS (version rel-1.14.0-0-g155821a-20210629_105355-sharpie)
>
> BUILD: gcc: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) binutils: version 
> 2.27-34.base.el7
>
> No Xen hypervisor found.
>
> Running on QEMU (q35)
>
> Running on KVM
>
> Found QEMU fw_cfg
>
> QEMU fw_cfg DMA interface supported
>
> qemu/e820: addr 0x0000000000000000 len 0x0000000080000000 [RAM]
>
> qemu/e820: addr 0x0000000100000000 len 0x0000000180000000 [RAM]
>
> Relocating init from 0x000d4de0 to 0x7ffa9420 (size 93008)
>
> Moving pm_base to 0x600
>
> kvmclock: at 0xeb040 (msr 0x4b564d01)
>
> kvmclock: stable tsc, 2992 MHz
>
> CPU Mhz=2992 (kvmclock)
>
> === PCI bus & bridge init ===
>
> PCI: pci_bios_init_bus_rec bus = 0x0
>
> PCI: pci_bios_init_bus_rec bdf = 0x20
>
> PCI: primary bus = 0x0
>
> PCI: secondary bus = 0xff -> 0x1
>
> PCI: pci_bios_init_bus_rec bus = 0x1
>
> PCI: subordinate bus = 0x0 -> 0x1
>
> === PCI device probing ===
>
> Found 12 PCI devices (max PCI bus is 01)
>
> PCIe: using q35 mmconfig at 0xb0000000
>
> === PCI new allocation pass #1 ===
>
> PCI: check devices
>
> PCI: secondary bus 1 size 00000000 type io
>
> PCI: secondary bus 1 size 100000000 type mem
>
> PCI: secondary bus 1 size 100000000 type prefmem
>
> === PCI new allocation pass #2 ===
>
> PCI: IO: c000 - c0ff
>
> PCI: 32: 00000000c0000000 - 00000000fec00000
>
> PCI: out of 32bit address space
>
> I’ve had a look at the code and it seems that 32-bit BAR’s have a 
> limited memory region available (less than 1 GiB). Is there a 
> possibility to use such large 32-bit BAR’s in combination with SeaBios 
> by e.g. mapping them in the higher memory region where 64-bit BAR’s 
> are mapped?
>
> Thanks in advance!
>
> Kind regards,
>
> Wannes
>
> SeaBIOS mailing list [e-mail]
> To unsubscribe send an email [e-mail]

[Wannes Bouwen (Nokia)]

Hello Daniel,

No, unfortunately this cannot be changed. So there is no possibility to map the 32 bit BAR in a higher memory region (where the 64 bit BAR’s are located)?

Kind regards,

Wannes

From: Daniel Paziyski 
Sent: Wednesday, April 12, 2023 9:19 AM
To: Wannes Bouwen (Nokia) ; [e-mail]
Subject: Re: [SeaBIOS] Large 32 bit BAR's

U ontvangt niet vaak e-mail van [e-mail]. Meer informatie over waarom dit belangrijk ishttps://aka.ms/LearnAboutSenderIdentification

CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See the URL nok.it/ext for additional information.

Hello,

For a device with such a big 32 bit BAR, it would be a challenge to fit it inside the 32 bit window, since the RAM region available on <4GB must be specially tuned with help from the emulator (which, as far as I am aware, isn't available) to give space for BAR allocations there. Is there a possibility for that emulated PCIe device to be configured to expose a 64 bit BAR instead?

Regards,

Daniel

On 4/12/23 08:23, Wannes Bouwen (Nokia) wrote:
Hello,

I’m running a x86 QEMU with Ubuntu. On this system we want to emulate a PCIe device which contains a large 32-bit BAR (2 GiB). This is not working and I get following error:

SeaBIOS (version rel-1.14.0-0-g155821a-20210629_105355-sharpie)
BUILD: gcc: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) binutils: version 2.27-34.base.el7
No Xen hypervisor found.
Running on QEMU (q35)
Running on KVM
Found QEMU fw_cfg
QEMU fw_cfg DMA interface supported
qemu/e820: addr 0x0000000000000000 len 0x0000000080000000 [RAM]
qemu/e820: addr 0x0000000100000000 len 0x0000000180000000 [RAM]
Relocating init from 0x000d4de0 to 0x7ffa9420 (size 93008)
Moving pm_base to 0x600
kvmclock: at 0xeb040 (msr 0x4b564d01)
kvmclock: stable tsc, 2992 MHz
CPU Mhz=2992 (kvmclock)
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
PCI: pci_bios_init_bus_rec bdf = 0x20
PCI: primary bus = 0x0
PCI: secondary bus = 0xff -> 0x1
PCI: pci_bios_init_bus_rec bus = 0x1
PCI: subordinate bus = 0x0 -> 0x1
=== PCI device probing ===
Found 12 PCI devices (max PCI bus is 01)
PCIe: using q35 mmconfig at 0xb0000000
=== PCI new allocation pass #1 ===
PCI: check devices
PCI: secondary bus 1 size 00000000 type io
PCI: secondary bus 1 size 100000000 type mem
PCI: secondary bus 1 size 100000000 type prefmem
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c0ff
PCI: 32: 00000000c0000000 - 00000000fec00000
PCI: out of 32bit address space

I’ve had a look at the code and it seems that 32-bit BAR’s have a limited memory region available (less than 1 GiB). Is there a possibility to use such large 32-bit BAR’s in combination with SeaBios by e.g. mapping them in the higher memory region where 64-bit BAR’s are mapped?

Thanks in advance!

Kind regards,

Wannes

_______________________________________________

SeaBIOS mailing list -- [e-mail]

To unsubscribe send an email to [e-mail]

[Daniel Paziyski]

Hello Wannes,

Sorry, I don't think that's possible, because there's no way you can set 
up a 32 bit BAR to point to anywhere higher than 4GB. The only thing I 
could recommend is to mess with QEMU to see if it has some feature for 
leaving such a huge free space on lower memory for BAR allocations, and 
modify SeaBIOS so you can actually use it, since it is limited to use as 
low as 0x80000000/0xc0000000 if running on the I440FX chipset, or right 
after the PCI ECAM space, which is usually at 0xb0000000 + whatever size 
it is.

Best Regards,

Daniel

On 4/12/23 09:24, Wannes Bouwen (Nokia) wrote:
> Hello Daniel,
>
> No, unfortunately this cannot be changed. So there is no possibility 
> to map the 32 bit BAR in a higher memory region (where the 64 bit 
> BAR’s are located)?
>
> Kind regards,
>
> Wannes
>
> *From:*Daniel Paziyski 
> *Sent:* Wednesday, April 12, 2023 9:19 AM
> *To:* Wannes Bouwen (Nokia) ; [e-mail]
> *Subject:* Re: [SeaBIOS] Large 32 bit BAR's
>
> U ontvangt niet vaak e-mail van [e-mail]. Meer 
> informatie over waarom dit belangrijk is 
> https://aka.ms/LearnAboutSenderIdentification
>
> *CAUTION:*This is an external email. Please be very careful when 
> clicking links or opening attachments. See the URL nok.it/ext for 
> additional information.
>
> Hello,
>
> For a device with such a big 32 bit BAR, it would be a challenge to 
> fit it inside the 32 bit window, since the RAM region available on 
> <4GB must be specially tuned with help from the emulator (which, as 
> far as I am aware, isn't available) to give space for BAR allocations 
> there. Is there a possibility for that emulated PCIe device to be 
> configured to expose a 64 bit BAR instead?
>
> Regards,
>
> Daniel
>
> On 4/12/23 08:23, Wannes Bouwen (Nokia) wrote:
> Hello,
>
> I’m running a x86 QEMU with Ubuntu. On this system we want to
> emulate a PCIe device which contains a large 32-bit BAR (2 GiB).
> This is not working and I get following error:
>
> SeaBIOS (version rel-1.14.0-0-g155821a-20210629_105355-sharpie)
>
> BUILD: gcc: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) binutils:
> version 2.27-34.base.el7
>
> No Xen hypervisor found.
>
> Running on QEMU (q35)
>
> Running on KVM
>
> Found QEMU fw_cfg
>
> QEMU fw_cfg DMA interface supported
>
> qemu/e820: addr 0x0000000000000000 len 0x0000000080000000 [RAM]
>
> qemu/e820: addr 0x0000000100000000 len 0x0000000180000000 [RAM]
>
> Relocating init from 0x000d4de0 to 0x7ffa9420 (size 93008)
>
> Moving pm_base to 0x600
>
> kvmclock: at 0xeb040 (msr 0x4b564d01)
>
> kvmclock: stable tsc, 2992 MHz
>
> CPU Mhz=2992 (kvmclock)
>
> === PCI bus & bridge init ===
>
> PCI: pci_bios_init_bus_rec bus = 0x0
>
> PCI: pci_bios_init_bus_rec bdf = 0x20
>
> PCI: primary bus = 0x0
>
> PCI: secondary bus = 0xff -> 0x1
>
> PCI: pci_bios_init_bus_rec bus = 0x1
>
> PCI: subordinate bus = 0x0 -> 0x1
>
> === PCI device probing ===
>
> Found 12 PCI devices (max PCI bus is 01)
>
> PCIe: using q35 mmconfig at 0xb0000000
>
> === PCI new allocation pass #1 ===
>
> PCI: check devices
>
> PCI: secondary bus 1 size 00000000 type io
>
> PCI: secondary bus 1 size 100000000 type mem
>
> PCI: secondary bus 1 size 100000000 type prefmem
>
> === PCI new allocation pass #2 ===
>
> PCI: IO: c000 - c0ff
>
> PCI: 32: 00000000c0000000 - 00000000fec00000
>
> PCI: out of 32bit address space
>
> I’ve had a look at the code and it seems that 32-bit BAR’s have a
> limited memory region available (less than 1 GiB). Is there a
> possibility to use such large 32-bit BAR’s in combination with
> SeaBios by e.g. mapping them in the higher memory region where
> 64-bit BAR’s are mapped?
>
> Thanks in advance!
>
> Kind regards,
>
>
> Wannes
>
>
>
> _______________________________________________
>
> SeaBIOS mailing list [e-mail]
>
> To unsubscribe send an email [e-mail]

[Daniel Paziyski]

(note for the previous message: the latter case for the Q35 chipset)

On 4/12/23 09:41, Daniel Paziyski wrote:
> Hello Wannes,
>
> Sorry, I don't think that's possible, because there's no way you can 
> set up a 32 bit BAR to point to anywhere higher than 4GB. The only 
> thing I could recommend is to mess with QEMU to see if it has some 
> feature for leaving such a huge free space on lower memory for BAR 
> allocations, and modify SeaBIOS so you can actually use it, since it 
> is limited to use as low as 0x80000000/0xc0000000 if running on the 
> I440FX chipset, or right after the PCI ECAM space, which is usually at 
> 0xb0000000 + whatever size it is.
>
> Best Regards,
>
> Daniel
>
> On 4/12/23 09:24, Wannes Bouwen (Nokia) wrote:
> > Hello Daniel,
> >
> > No, unfortunately this cannot be changed. So there is no possibility 
> > to map the 32 bit BAR in a higher memory region (where the 64 bit 
> > BAR’s are located)?
> >
> > Kind regards,
> >
> > Wannes
> >
> > *From:*Daniel Paziyski 
> > *Sent:* Wednesday, April 12, 2023 9:19 AM
> > *To:* Wannes Bouwen (Nokia) ; 
> > [e-mail]
> > *Subject:* Re: [SeaBIOS] Large 32 bit BAR's
> >
> > U ontvangt niet vaak e-mail van [e-mail]. Meer 
> > informatie over waarom dit belangrijk is 
> > https://aka.ms/LearnAboutSenderIdentification
> >
> > *CAUTION:*This is an external email. Please be very careful when 
> > clicking links or opening attachments. See the URL nok.it/ext for 
> > additional information.
> >
> > Hello,
> >
> > For a device with such a big 32 bit BAR, it would be a challenge to 
> > fit it inside the 32 bit window, since the RAM region available on 
> > <4GB must be specially tuned with help from the emulator (which, as 
> > far as I am aware, isn't available) to give space for BAR allocations 
> > there. Is there a possibility for that emulated PCIe device to be 
> > configured to expose a 64 bit BAR instead?
> >
> > Regards,
> >
> > Daniel
> >
> > On 4/12/23 08:23, Wannes Bouwen (Nokia) wrote:
> > Hello,
> >
> > I’m running a x86 QEMU with Ubuntu. On this system we want to
> > emulate a PCIe device which contains a large 32-bit BAR (2 GiB).
> > This is not working and I get following error:
> >
> > SeaBIOS (version rel-1.14.0-0-g155821a-20210629_105355-sharpie)
> >
> > BUILD: gcc: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) binutils:
> > version 2.27-34.base.el7
> >
> > No Xen hypervisor found.
> >
> > Running on QEMU (q35)
> >
> > Running on KVM
> >
> > Found QEMU fw_cfg
> >
> > QEMU fw_cfg DMA interface supported
> >
> > qemu/e820: addr 0x0000000000000000 len 0x0000000080000000 [RAM]
> >
> > qemu/e820: addr 0x0000000100000000 len 0x0000000180000000 [RAM]
> >
> > Relocating init from 0x000d4de0 to 0x7ffa9420 (size 93008)
> >
> > Moving pm_base to 0x600
> >
> > kvmclock: at 0xeb040 (msr 0x4b564d01)
> >
> > kvmclock: stable tsc, 2992 MHz
> >
> > CPU Mhz=2992 (kvmclock)
> >
> > === PCI bus & bridge init ===
> >
> > PCI: pci_bios_init_bus_rec bus = 0x0
> >
> > PCI: pci_bios_init_bus_rec bdf = 0x20
> >
> > PCI: primary bus = 0x0
> >
> > PCI: secondary bus = 0xff -> 0x1
> >
> > PCI: pci_bios_init_bus_rec bus = 0x1
> >
> > PCI: subordinate bus = 0x0 -> 0x1
> >
> > === PCI device probing ===
> >
> > Found 12 PCI devices (max PCI bus is 01)
> >
> > PCIe: using q35 mmconfig at 0xb0000000
> >
> > === PCI new allocation pass #1 ===
> >
> > PCI: check devices
> >
> > PCI: secondary bus 1 size 00000000 type io
> >
> > PCI: secondary bus 1 size 100000000 type mem
> >
> > PCI: secondary bus 1 size 100000000 type prefmem
> >
> > === PCI new allocation pass #2 ===
> >
> > PCI: IO: c000 - c0ff
> >
> > PCI: 32: 00000000c0000000 - 00000000fec00000
> >
> > PCI: out of 32bit address space
> >
> > I’ve had a look at the code and it seems that 32-bit BAR’s have a
> > limited memory region available (less than 1 GiB). Is there a
> > possibility to use such large 32-bit BAR’s in combination with
> > SeaBios by e.g. mapping them in the higher memory region where
> > 64-bit BAR’s are mapped?
> >
> > Thanks in advance!
> >
> > Kind regards,
> >
> >
> > Wannes
> >
> >
> >
> > _______________________________________________
> >
> > SeaBIOS mailing list [e-mail]
> >
> > To unsubscribe send an email [e-mail]

[Gerd Hoffmann]

Hi,
> I'm running a x86 QEMU with Ubuntu. On this system we want to emulate
> a PCIe device which contains a large 32-bit BAR (2 GiB). This is not
> working and I get following error:

Why a 32-bit bar, not a 64-bit bar?

A 2G 32-bit bar simply can't be mapped anywhere on x86.  It must be
below 4G, and it must be 2G-aligned.  Which leaves exactly two options:
0G -> 2G or 2G -> 4G.  Both overlap with resources critical for x86
boot, the former with the real mode address space (0-1M) and the latter
with lapic, ioapic, hpet, tpm etc just below 4G.

take care,
  Gerd

[Wannes Bouwen (Nokia)]

Hello,

Thanks for the answers. We'll indeed investigate if we can use a 64-bit BAR.

Kind regards,

Wannes

-----Original Message-----
From: Gerd Hoffmann  
Sent: Wednesday, April 12, 2023 10:04 AM
To: Wannes Bouwen (Nokia) 
Cc: [e-mail]
Subject: Re: [SeaBIOS] Large 32 bit BAR's

[U ontvangt vaak geen e-mail van [e-mail]. Informatie over waarom dit belangrijk is op https://aka.ms/LearnAboutSenderIdentification]

CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See the URL nok.it/ext for additional information.

Hi,
> I'm running a x86 QEMU with Ubuntu. On this system we want to emulate 
> a PCIe device which contains a large 32-bit BAR (2 GiB). This is not 
> working and I get following error:

Why a 32-bit bar, not a 64-bit bar?

A 2G 32-bit bar simply can't be mapped anywhere on x86.  It must be below 4G, and it must be 2G-aligned.  Which leaves exactly two options:
0G -> 2G or 2G -> 4G.  Both overlap with resources critical for x86 boot, the former with the real mode address space (0-1M) and the latter with lapic, ioapic, hpet, tpm etc just below 4G.

take care,
  Gerd

[Peter Stuge]

Gerd Hoffmann wrote:
> A 2G 32-bit bar simply can't be mapped anywhere on x86.  It must be
> below 4G,

Agree..
> and it must be 2G-aligned.

..but I can't find this requirement. Why do you say?

The one mention of alignment in PCI 3.0 is for the Expansion ROM BAR,
where *the device* can indicate its required alignment.

So why couldn't a 2G BAR be mapped at 1G?

If must size-align after all then there is at minimum one creative
solution achieved by dividing the one 2G BAR into two 1G BARs, that
can then be mapped at 1G and 2G. This could probably be hidden from
the rest of the device by the bus interface. Just be prepared that
firmware can map the two BARs in any order!

//Peter

[Daniel Paziyski]

The way the size is reported by the BAR register by checking if the device
decodes certain bits or not implies that the address must be naturally
aligned, i.e. aligned to the size.

El vie, 14 abr 2023 17:38, Peter Stuge  escribió:
> Gerd Hoffmann wrote:
> > A 2G 32-bit bar simply can't be mapped anywhere on x86.  It must be
> > below 4G,
>
> Agree..
> > and it must be 2G-aligned.
>
> ..but I can't find this requirement. Why do you say?
>
> The one mention of alignment in PCI 3.0 is for the Expansion ROM BAR,
> where *the device* can indicate its required alignment.
>
> So why couldn't a 2G BAR be mapped at 1G?
>
> If must size-align after all then there is at minimum one creative
> solution achieved by dividing the one 2G BAR into two 1G BARs, that
> can then be mapped at 1G and 2G. This could probably be hidden from
> the rest of the device by the bus interface. Just be prepared that
> firmware can map the two BARs in any order!
>
> //Peter
> _______________________________________________
> SeaBIOS mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]