[SeaBIOS] Re: [PATCH v2 4/4] only enable 64bit pci io window when RAM >64G

[Igor Mammedov]

On Tue, 16 Jul 2024 13:56:08 +0200
Fiona Ebner  wrote:
> Am 12.07.24 um 15:26 schrieb Igor Mammedov:
> > On Fri, 12 Jul 2024 14:24:40 +0200
> > Fiona Ebner  wrote:
> > > Hi,
> > >
> > > Am 21.06.24 um 14:05 schrieb Gerd Hoffmann:
> > > > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
> > > > > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> > > > > 64-bit pci io window, failing during boot with errors like:
> > > >
> > > > Well.  Why people would use *that* ubuntu version is not clear to me.
> > > > It's *loooooong* out of support.  Even the LTS version from that year
> > > > (16.04) is not supported any more.  But it is at least available for
> > > > download still,  so I gave it a spin.
> > > >
> > > > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
> > > > 44 phys-bits).  Test patch below.
> > > >
> > > > take care,
> > > >   Gerd
> > > >
> > > > diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
> > > > index bb44dc296047..a43876a931c9 100644
> > > > --- a/src/fw/pciinit.c
> > > > +++ b/src/fw/pciinit.c
> > > > @@ -1189,11 +1189,16 @@ pci_setup(void)
> > > >  if (CPUPhysBits) {
> > > >      pci_mem64_top = 1LL << CPUPhysBits;
> > > >    if (CPUPhysBits > 46) {
> > > >        // Old linux kernels have trouble dealing with more than 46
> > > >        // phys-bits, so avoid that for now.  Seems to be a bug in the
> > > >        // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
> > > >        pci_mem64_top = 1LL << 46;
> > > >    if (CPUPhysBits > 44) {
> > > >        // Old linux kernels have trouble dealing with more than 44/46
> > > >        // phys-bits. Seems to be a bug in the virtio-pci driver.
> > > >        //   46:  centos-7, ubuntu-18.04
> > > >        //   44:  ubuntu-16.04
> > > >        // Limit the used address space to mitigate the bug, except we are
> > > >        // running in a guest with more than 1TB of memory installed.
> > > >        if (RamSizeOver4G < (1LL << 40)) {
> > > >            pci_mem64_top = 1LL << 44;
> > > >        }
> > > >    }
> > > > }
> > >
> > > we've also had two reports about issues with 32-bit guests now[0][1]
> > > (both had '-m 4096' in the QEMU commandline), which I was able to
> > > reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
> > > QEMU commandline is below[2].
> >
> > is it also reproducible with upstream kernel?
> > if yes, it would be better to fix that on guest kernel side,
> > rather than in SeaBIOS which has no idea what guest OS is going to be
> > running after it.
>
> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE
> boots fine (but the PAE was the one installed by default). I built a
> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't.
> > > Unfortunately, it still fails to boot, even with the "limit address
> > > space used for pci devices, part two" patch applied on top of rel-1.16.3
> > > (and using current QEMU master).
> > >
> > > It boots fine with '-m 3583', but not anymore with '-m 3584'. So is
> > > bumping the limit for the check necessary after all?
> > >
> > > Best Regards,
> > > Fiona
> > >
> >
> > well, with current QEMU master branch (I assume it haven't even got topic patch yet)
> >   ./qemu-system-x86_64 --enable-kvm -cpu host -smp 4 -snapshot -m 4096 -M pc-i440fx-6.1 winxp_x86_build2600
> > following CLI boots just fine an 32-bit XP on Haswell host.
> >
> > Also using 'host' with ancient OS, is basically asking for trouble.
> > If it works for user with old 'XP contemporary' CPU model, user should use that
> > instead or workarounds (aka it's management task to configure CLI in
> > compatible with OS manner).
> >
> > From suspicions config options in that post I see 'viommu' and 'vmgenid',
> > are you sure XP even knows what to do with that, perhaps it triggers BSOD.
>
> No idea why the user enabled viommu for XP. vmgenid is added by our
> management stack by default and I don't remember it having ever caused
> problems. The user said the VM booted fine after adding lm=off to the
> CPU options. I'm not super interested in the Windows case to be honest O:)
> > > [2]:
> > > > ./qemu-system-x86_64 \
> > > >   -accel 'kvm' \
> > > >   -cpu 'host' \
> > > >   -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \
> > > >   -mon 'chardev=qmp,mode=control' \
> > > >   -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
> > > >   -mon 'chardev=qmp-event,mode=control' \
> > > >   -pidfile /var/run/qemu-server/121.pid \
> > > >   -smp '4,sockets=1,cores=4,maxcpus=4' \
> > > >   -nodefaults \
> > > >   -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \
> > > >   -m 4096 \
> > > >   -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
> > > >   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
> > > >   -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
> > > >   -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
> > > >   -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
> > > >   -machine 'type=pc'
> >
> > can you try to boot with q35 + intel iommu enabled
> > and/or force virtio into legacy mode.
>
> I tried
> > -device 'intel-iommu' \
> >   -machine 'type=q35'
>
> and intel_iommu=on in the guest's kernel cmdline, but it still failed
> with kernels with PAE.
>
> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci
> line made it work however (also with pc machine) :)

does it also help in Windows case?

Perhaps it's a better workaround (compared to lm=off or going back to
older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side,
as it directly targets bug in guest's virtio-driver.

Can we put this config tweak into libosinfo somehow, so that provisioning
tools/mgmt could all reuse that to properly configure virtio for PAE kernels?
> Best Regards,
> Fiona