Re: [SeaBIOS] >256 Virtio-net-pci hotplug Devices

[Marcel Apfelbaum]

On 22/07/2017 2:57, Kinsella, Ray wrote:
> Hi Marcel

Hi Ray,
> On 21/07/2017 01:33, Marcel Apfelbaum wrote:
> > On 20/07/2017 3:44, Kinsella, Ray wrote:
> > That's strange. Please ensure the virtio devices are working in
> > virtio 1.0 mode (disable-modern=0,disable-legacy=1).
> > Let us know any problems you see.
> > > Not sure what yet, I will try scaling it with hotplugging tomorrow.
> >
> > Updates?
>
> I have managed to scale it to 128 devices.
> The kernel does complain about IO address space exhaustion.
>
> [   83.697956] pci 0000:80:00.0: BAR 13: no space for [io  size 0x1000]
> [   83.700958] pci 0000:80:00.0: BAR 13: failed to assign [io  size 0x1000]
> [   83.701689] pci 0000:80:00.1: BAR 13: no space for [io  size 0x1000]
> [   83.702378] pci 0000:80:00.1: BAR 13: failed to assign [io  size 0x1000]
> [   83.703093] pci 0000:80:00.2: BAR 13: no space for [io  size 0x1000]
>
> I was surprised that I am running out of IO address space, as I am 
> disabling legacy virtio. I assumed that this would remove the need for 
> SeaBIOS to allocate the PCI Express Root Port IO address space.

Indeed, SeeBIOS does not reserve IO ports in this case, but Linux kernel
still decides ""it knows better" and tries to allocate IO resources
anyway. It does not affect the "modern" virtio-net devices because
they don't need IO ports anyway.

One way to work around the error message is to have the PCIe Root Port
have the corresponding IO headers read-only since IO support is
optional. I tried this some time ago, I'll get back to it.
> In any 
> case - it doesn't stop the virtio-net device coming up and working as 
> expected.

Right.
> [  668.692081] virtio_net virtio103 enp141s0f4: renamed from eth101
> [  668.707114] virtio_net virtio130 enp144s0f7: renamed from eth128
> [  668.719795] virtio_net virtio129 enp144s0f6: renamed from eth127
>
> I encountered some issues in vhost, due to open file exhaustion but 
> resolved these with 'ulimit' in the usual way - burned alot of time on 
> that today.
>
> When scaling up to 512 Virtio-net devices SeaBIOS appears to really slow 
> down when configuring PCI Config space - haven't manage to get this to 
> work yet.

Adding SeaBIOS mailing list and maintainers, maybe there is a known
issue about 500+ PCI devices configuration.
> > Not really. All you have to do is to add a property to the pxb-pci/pxb
> > devices: pci_domain=x; then update the ACPI table to include the pxb
> > domain. You also have to tweak a little the pxb-pcie/pxb devices
> > to not share the bus numbers if pci_domain > 0.
>
> Thanks for information, will add to the list.

Is also on my todo list :)

Thanks,
Marcel
> Ray K
> \

>256 Virtio-net-pci hotplug Devices

[Kevin O'Connor]

On Sun, Jul 23, 2017 at 07:28:01PM +0300, Marcel Apfelbaum wrote:
> On 22/07/2017 2:57, Kinsella, Ray wrote:
> > When scaling up to 512 Virtio-net devices SeaBIOS appears to really slow
> > down when configuring PCI Config space - haven't manage to get this to
> > work yet.

If there is a slowdown in SeaBIOS, it would help to produce a log with
timing information - see:
https://www.seabios.org/Debugging#Timing_debug_messages

It may also help to increase the debug level in SeaBIOS to get more
fine grained timing reports.

-Kevin

>256 Virtio-net-pci hotplug Devices

[Kinsella, Ray]

So as it turns out at 512 devices, it is nothing to do SeaBIOS, it was the Kernel again.
It is taking quite a while to startup, a little over two hours (7489 seconds).
The main culprits appear to be enumerating/initializing the PCI Express ports and enabling interrupts.

The PCI Express Root Ports are taking a long time to enumerate/ initializing.
42 minutes in total=2579/60=64 ports in total, 40 seconds each.

[   50.612822] pci_bus 0000:80: root bus resource [bus 80-c1]
[  172.345361] pci 0000:80:00.0: PCI bridge to [bus 81]
...
[ 2724.734240] pci 0000:80:08.0: PCI bridge to [bus c1]
[ 2751.154702] ACPI: Enabled 2 GPEs in block 00 to 3F

I assume the 1 hour (3827 seconds) below is being spent enabling interrupts.

[ 2899.394288] ACPI: PCI Interrupt Link [GSIG] enabled at IRQ 22
[ 2899.531324] ACPI: PCI Interrupt Link [GSIH] enabled at IRQ 23
[ 2899.534778] ACPI: PCI Interrupt Link [GSIE] enabled at IRQ 20
[ 6726.914388] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 6726.937932] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 6726.964699] Linux agpgart interface v0.103

There finally there is another 20 minutes to find in the boot.

[ 7489.202589] virtio_net virtio515 enp193s0f0: renamed from eth513

Poky (Yocto Project Reference Distro) 2.3 qemux86-64 ttyS0

qemux86-64 login: root

I will remove the virtio-net-pci devices and hotplug them instead.
In theory it should improve boot time, at expense of incurring some of these costs at runtime.

Ray K

-----Original Message-----
From: Kevin O'Connor [mailto:[e-mail]] 
Sent: Sunday, July 23, 2017 1:05 PM
To: Marcel Apfelbaum ; Kinsella, Ray 
Cc: [e-mail]; [e-mail]; Gerd Hoffmann ; Michael Tsirkin 
Subject: Re: >256 Virtio-net-pci hotplug Devices

On Sun, Jul 23, 2017 at 07:28:01PM +0300, Marcel Apfelbaum wrote:
> On 22/07/2017 2:57, Kinsella, Ray wrote:
> > When scaling up to 512 Virtio-net devices SeaBIOS appears to really 
> > slow down when configuring PCI Config space - haven't manage to get 
> > this to work yet.

If there is a slowdown in SeaBIOS, it would help to produce a log with timing information - see:
https://www.seabios.org/Debugging#Timing_debug_messages

It may also help to increase the debug level in SeaBIOS to get more fine grained timing reports.

-Kevin