[SeaBIOS] hotplug failure issue on pci-bridge

[Laszlo Ersek]

On 07/12/18 07:43, Liu, Jing2 wrote:
> Yep, thanks for the advice.
> But hotplugging on pci-bridge is the actual use case
> request so we would better solve and fix this.

It doesn't take a bugfix, but a feature. The firmware needs to be told
to reserve PCI resources in advance, in preparation for the hotplug. The
firmware might reserve some amount of resources by default, but those
defaults cannot be arbitrarily large. The reservation sizes need to come
from QEMU.

The most robust method for this was deemed to be placing a vendor
capability in the hotplug controller's PCI config space. In QEMU this is
implemented with the pci_bridge_qemu_reserve_cap_init() function. Right
now, the function is only called from the PCI Express Root Port device
model, in "hw/pci-bridge/gen_pcie_root_port.c".

You can cold-plug a PCI Express Root Port in the Q35 root complex
(pcie.0), reserving the MMIO resources you want, cold-plug a PCIE-PCI
Bridge in that root port, and then hot-plug the desired endpoint into
that PCIE-PCI Bridge. This is one of the exact examples that
"docs/pcie_pci_bridge.txt" provides. (The other example is when the
PCIE-PCI bridge itself is hot-plugged into the root port, for which bus
number reservation is necessary too, at the root port level.)

If you want more than that, e.g. do something similar on i440fx, that
will take QEMU work as well, not just SeaBIOS.

Laszlo

> 
> On 7/11/2018 6:38 PM, Laszlo Ersek wrote:
>> On 07/11/18 05:12, Liu, Jing2 wrote:
>>> Hi,
>>>
>>> Recently, we tried some hotplug issues. The case is: when hotplug a
>>> device (e.g. iGPU) onto pci-bridge after guest booting up, guest reports
>>> "BAR 2: no space for [mem size 0x40000000 64bit pref]" etc.
>>>
>>> Seabios checks all the devices under the pci-bridge when qemu launching
>>> the guest, and only allocates "size=ALIGN(sum, align)" of memory space
>>> for pci-bridge mem and pref-mem windows. So if we hotplug a big pci
>>> device like Intel GPU which needs 256M mem/pref-mem or bigger, it will
>>> fail.
>>>
>>>
>>> If my understanding is right, we may need some other logic of the memory
>>> allocation in seabios?
>>>
>>> Looking forward to the advice.
>>
>> I suggest using the Q35 machine type, and hot-plugging the device into a
>> PCI Express Root Port ("pcie-root-port"). The latter has properties
>> dedicated to reserving various PCI resources specifically for hotplug
>> purposes:
>>
>> pcie-root-port.mem-reserve=size
>> pcie-root-port.pref32-reserve=size
>> pcie-root-port.bus-reserve=uint32
>> pcie-root-port.pref64-reserve=size
>> pcie-root-port.io-reserve=size
>>
>> In order to address the issue you report at the top, you would use
>>
>>    -device pcie-root-port,bus=pcie.0,id=root-port-XXX,pref64-reserve=1G
>>
>> then hot-plug the device into "root-port-XXX".
>>
>>
>> Please see the following two files in the QEMU tree:
>> - docs/pcie.txt
>> - docs/pcie_pci_bridge.txt
>>
>> The first provides guidelines on the PCI Express hierarchy in general,
>> and also on hotplug in particular (see section 5).
>>
>> The second is relevant here because it describes the vendor capability
>> that QEMU and SeaBIOS (and OVMF) use, for passing the resource
>> reservation hints from QEMU to guest firmware.
>>
>> The 2nd document mainly focuses on hot-plugging a PCIE-PCI bridge into a
>> PCIE root port, and on reserving a bus number range for the
>> sub-hierarchy behind said hot-plugged  PCIE-PCI bridge. However, the
>> reservation mechanism is the same for other types of PCI resources, and
>> for other types of child devices that are hot-plugged into root ports.
>>
>> HTH,
>> Laszlo
>>