PCIe downstream ports (Root Ports and switches Downstream Ports) appear to firmware as PCI-PCI bridges and a 4K IO space is allocated for them even if there is no device behind them requesting IO space, all that for hotplug purpose.
However, PCIe devices can work without IO, so there is no need to allocate IO space for hotplug.
Signed-off-by: Marcel Apfelbaum marcel@redhat.com --- v1 -> v2: - Addressed Gerd's comments: - move pci_find_capability out of the loop and call it only once. - more descriptive names for variables
Notes: - This patch fixes a 15 PCIe Root ports limitation when used with virtio pci-express devices having no IO space requirements: <qemu cmd line> -M q35 `for i in {1..15}; do echo -device ioh3420,chassis=$i,id=b$i -device virtio-net-pci,bus=b$i,disable-legacy=on; done` - There is a patch on QEMU devel list that tackles the problem from another angle by allowing PCIe downstream ports to not forward IO requests. https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg04478.html - The mentioned patch is of course not enough and also requires management software intervention.
Thanks, Marcel
src/fw/pciinit.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c index 7b8aab7..c31c2fa 100644 --- a/src/fw/pciinit.c +++ b/src/fw/pciinit.c @@ -645,9 +645,8 @@ pci_region_create_entry(struct pci_bus *bus, struct pci_device *dev, return entry; }
-static int pci_bus_hotplug_support(struct pci_bus *bus) +static int pci_bus_hotplug_support(struct pci_bus *bus, u8 pcie_cap) { - u8 pcie_cap = pci_find_capability(bus->bus_dev, PCI_CAP_ID_EXP, 0); u8 shpc_cap;
if (pcie_cap) { @@ -727,7 +726,8 @@ static int pci_bios_check_devices(struct pci_bus *busses) */ parent = &busses[0]; int type; - int hotplug_support = pci_bus_hotplug_support(s); + u8 pcie_cap = pci_find_capability(s->bus_dev, PCI_CAP_ID_EXP, 0); + int hotplug_support = pci_bus_hotplug_support(s, pcie_cap); for (type = 0; type < PCI_REGION_TYPE_COUNT; type++) { u64 align = (type == PCI_REGION_TYPE_IO) ? PCI_BRIDGE_IO_MIN : PCI_BRIDGE_MEM_MIN; @@ -736,7 +736,8 @@ static int pci_bios_check_devices(struct pci_bus *busses) if (pci_region_align(&s->r[type]) > align) align = pci_region_align(&s->r[type]); u64 sum = pci_region_sum(&s->r[type]); - if (!sum && hotplug_support) + int resource_optional = pcie_cap && (type == PCI_REGION_TYPE_IO); + if (!sum && hotplug_support && !resource_optional) sum = align; /* reserve min size for hot-plug */ u64 size = ALIGN(sum, align); int is64 = pci_bios_bridge_region_is64(&s->r[type],
On Mon, Dec 07, 2015 at 02:05:14PM +0200, Marcel Apfelbaum wrote:
PCIe downstream ports (Root Ports and switches Downstream Ports) appear to firmware as PCI-PCI bridges and a 4K IO space is allocated for them even if there is no device behind them requesting IO space, all that for hotplug purpose.
However, PCIe devices can work without IO, so there is no need to allocate IO space for hotplug.
Signed-off-by: Marcel Apfelbaum marcel@redhat.com
Acked-by: Michael S. Tsirkin mst@redhat.com
v1 -> v2:
- Addressed Gerd's comments:
- move pci_find_capability out of the loop and call it only once.
- more descriptive names for variables
Notes:
- This patch fixes a 15 PCIe Root ports limitation when used with virtio pci-express devices having no IO space requirements: <qemu cmd line> -M q35 `for i in {1..15}; do echo -device ioh3420,chassis=$i,id=b$i -device virtio-net-pci,bus=b$i,disable-legacy=on; done`
- There is a patch on QEMU devel list that tackles the problem from another angle by allowing PCIe downstream ports to not forward IO requests. https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg04478.html
- The mentioned patch is of course not enough and also requires management software intervention.
Thanks, Marcel
src/fw/pciinit.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c index 7b8aab7..c31c2fa 100644 --- a/src/fw/pciinit.c +++ b/src/fw/pciinit.c @@ -645,9 +645,8 @@ pci_region_create_entry(struct pci_bus *bus, struct pci_device *dev, return entry; }
-static int pci_bus_hotplug_support(struct pci_bus *bus) +static int pci_bus_hotplug_support(struct pci_bus *bus, u8 pcie_cap) {
u8 pcie_cap = pci_find_capability(bus->bus_dev, PCI_CAP_ID_EXP, 0); u8 shpc_cap;
if (pcie_cap) {
@@ -727,7 +726,8 @@ static int pci_bios_check_devices(struct pci_bus *busses) */ parent = &busses[0]; int type;
int hotplug_support = pci_bus_hotplug_support(s);
u8 pcie_cap = pci_find_capability(s->bus_dev, PCI_CAP_ID_EXP, 0);int hotplug_support = pci_bus_hotplug_support(s, pcie_cap); for (type = 0; type < PCI_REGION_TYPE_COUNT; type++) { u64 align = (type == PCI_REGION_TYPE_IO) ? PCI_BRIDGE_IO_MIN : PCI_BRIDGE_MEM_MIN;@@ -736,7 +736,8 @@ static int pci_bios_check_devices(struct pci_bus *busses) if (pci_region_align(&s->r[type]) > align) align = pci_region_align(&s->r[type]); u64 sum = pci_region_sum(&s->r[type]);
if (!sum && hotplug_support)
int resource_optional = pcie_cap && (type == PCI_REGION_TYPE_IO);if (!sum && hotplug_support && !resource_optional) sum = align; /* reserve min size for hot-plug */ u64 size = ALIGN(sum, align); int is64 = pci_bios_bridge_region_is64(&s->r[type],-- 2.1.0
On Mon, Dec 07, 2015 at 02:05:14PM +0200, Marcel Apfelbaum wrote:
PCIe downstream ports (Root Ports and switches Downstream Ports) appear to firmware as PCI-PCI bridges and a 4K IO space is allocated for them even if there is no device behind them requesting IO space, all that for hotplug purpose.
However, PCIe devices can work without IO, so there is no need to allocate IO space for hotplug.
Thanks. Applied.
-Kevin
-
Kevin O'Connor -
Marcel Apfelbaum -
Michael S. Tsirkin