regression: cannot boot from nvme anymore, disabling 64bit pci io window helps

[Marek Marczykowski-Górecki]

Hi,

After an update some time ago, a VM wit 8GB RAM cannot boot from nvme anymore.

QEMU cmdline:
/usr/bin/qemu-system-x86_64 \
 -chardev ringbuf,id=serial0,logfile=serial0,logappend=on \
 -serial chardev:serial0 \
 -m 8192 \
 -enable-kvm \
 -vnc :2,share=force-shared \
 -blockdev driver=file,node-name=hd0-file,filename=disk.qcow2,cache.no-flush=on,discard=unmap \
 -blockdev driver=qcow2,node-name=hd0,file=hd0-file,cache.no-flush=on,discard=unmap \
 -device nvme,id=hd0-device,drive=hd0,serial=hd0 \
 -chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \
 -bios /home/user/seabios/out/bios.bin

There is nothing fancy on the disk, just standard grub2 setup, but also
I believe it doesn't matter, as it doesn't even reach grub.

When it fails, the seabios log contains:
  PCI: 64: 0000038000000000 - 0000038040000000                                            
  PCI: map device bdf=00:04.0  bar 0, addr 38000000000, size 00004000 [mem]               
  ...
  Searching bootorder for: /pci@i0cf8/isa@1/fdc@03f0/floppy@0
  ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
  ATA controller 2 at 170/374/0 (irq 15 dev 9)
  Searching bootorder for: HALT
  Can not map memory bar over 4Gig
  ...
  Boot failed: could not read the boot disk                                               

Working version has this instead:

  PCI: map device bdf=00:04.0  bar 0, addr febf0000, size 00004000 [mem]                  
  ...
  Booting from Hard Disk...
  Booting from 0000:7c00   

00:04.0 is the nvme.

I'm attaching full logs for both cases.

Bisect shows:

96a8d130a8c2e908e357ce62cd713f2cc0b0a2eb is the first bad commit
commit 96a8d130a8c2e908e357ce62cd713f2cc0b0a2eb (HEAD)
Author: Gerd Hoffmann <[e-mail]>
Date:   Fri Sep 9 10:17:15 2022 +0200

    be less conservative with the 64bit pci io window

    Current seabios code will only enable and use the 64bit pci io window in
    case it runs out of space in the 32bit pci mmio window below 4G.

    This patch will also enable the 64bit pci io window when
      (a) RAM above 4G is present, and
      (b) the physical address space size is known, and
      (c) seabios is running on a 64bit capable processor.

    This operates with the assumption that guests which are ok with memory
    above 4G most likely can handle mmio above 4G too.

    In case the 64bit pci io window is enabled also assign more memory to
    prefetchable pci bridge windows and the complete 64bit pci io window.

    The total mmio window size is 1/8 of the physical address space.
    Minimum bridge windows size is 1/256 of the total mmio window size.

    Signed-off-by: Gerd Hoffmann <[e-mail]>


I've tried the recent "[PATCH] limit address space used for pci devices,
part two" patch from the ML, but it doesn't help. OTOH, the series with
"only enable 64bit pci io window when RAM >64G" does help.


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[Rudolf Marek]

Dne 14. 07. 24 v 15:50 Marek Marczykowski-Górecki napsal(a):
> Can not map memory bar over 4Gig

It always made me wonder why we can't run SeaBIOS in 64-bit mode ;) or with 32-bit compatibility mode but long mode with identity paging enabled with some 64-bit stubs
to access mem > 4GiB - if it would be easier.

Thanks,
Rudolf

[Gerd Hoffmann]

Hi,
> After an update some time ago, a VM wit 8GB RAM cannot boot from nvme anymore.

From 7b54087dbd7f201a6cdb2794518c71a84aa0d8a6 Mon Sep 17 00:00:00 2001
From: Gerd Hoffmann <[e-mail]>
Date: Mon, 15 Jul 2024 10:53:23 +0200
Subject: [PATCH] pciinit: ignore nvme for 64-bit mmio window

Add nvme drives to the list of devices which are not
moved to the 64-bit mmio window.

Signed-off-by: Gerd Hoffmann <[e-mail]>
---
 src/fw/pciinit.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
index b3e359d7fe62..f6aa1a554853 100644
--- a/src/fw/pciinit.c
+++ b/src/fw/pciinit.c
@@ -764,6 +764,8 @@ static void pci_region_migrate_64bit_entries(struct pci_region *from,
             continue;
         if (entry->dev->class == PCI_CLASS_SERIAL_USB)
             continue;
+        if (entry->dev->class == PCI_CLASS_STORAGE_NVME)
+            continue;
         // Move from source list to destination list.
         hlist_del(&entry->node);
         hlist_add(&entry->node, last);
-- 
2.45.2

[Marek Marczykowski-Górecki]

On Mon, Jul 15, 2024 at 10:59:12AM +0200, Gerd Hoffmann wrote:
> Hi,
> > After an update some time ago, a VM wit 8GB RAM cannot boot from nvme anymore.
>
> From 7b54087dbd7f201a6cdb2794518c71a84aa0d8a6 Mon Sep 17 00:00:00 2001
> From: Gerd Hoffmann <[e-mail]>
> Date: Mon, 15 Jul 2024 10:53:23 +0200
> Subject: [PATCH] pciinit: ignore nvme for 64-bit mmio window
>
> Add nvme drives to the list of devices which are not
> moved to the 64-bit mmio window.
>
> Signed-off-by: Gerd Hoffmann <[e-mail]>

Thanks, I can confirm this helps too.

Tested-By: Marek Marczykowski-Górecki <[e-mail]>
> ---
>  src/fw/pciinit.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
> index b3e359d7fe62..f6aa1a554853 100644
> --- a/src/fw/pciinit.c
> +++ b/src/fw/pciinit.c
> @@ -764,6 +764,8 @@ static void pci_region_migrate_64bit_entries(struct pci_region *from,
>              continue;
>          if (entry->dev->class == PCI_CLASS_SERIAL_USB)
>              continue;
> +        if (entry->dev->class == PCI_CLASS_STORAGE_NVME)
> +            continue;
>          // Move from source list to destination list.
>          hlist_del(&entry->node);
>          hlist_add(&entry->node, last);
> -- 
> 2.45.2