On 03/10/2015 08:23 AM, Alexey Kardashevskiy wrote:
On 03/10/2015 06:21 AM, Marcel Apfelbaum wrote:
On 03/09/2015 06:55 PM, Gerd Hoffmann wrote:
On Mo, 2015-03-09 at 18:26 +0200, Marcel Apfelbaum wrote:
On 03/09/2015 04:19 PM, Gerd Hoffmann wrote:
Hi,My series is based on commit 09d219a. Try please on top of this commit.
Ok, that works. Going to play with that now ;)
Good luck! ... and tell me what you think :) If you need any help with the command line of the pxb device, let me know,.
First thing I've noticed: You need to define a numa node so you can pass a valid numa node to the pxb-device. Guess that is ok as the whole point of this is to assign pci devices to numa nodes. More complete test instructions would be nice though.
Exactly, this is by design. But you can also use it without specifying the NUMA node...
A detailed command line would be:
[qemu-bin + storage options] -bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/ -m 2G -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 -numa node,nodeid=0,cpus=0,memdev=ram-node0 -object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 -numa node,nodeid=1,cpus=1,memdev=ram-node1 -device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd -device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device e1000,bus=bridge2,addr=0x3 -device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive if=none,id=drive0,file=[img] -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1
I replayed this patchset on top of 09d219a "acpi: update generated files" and got this:
qemu-system-x86_64: -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0: NUMA node binding are not supported by this QEMU qemu-system-x86_64: -object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1: NUMA node binding are not supported by this QEMU
Hi,
Please check your configuration (after you run ./configure script). See if you have a line like this: - NUMA host support yes
This is my exact command line:
/scratch/alexey/p/qemu-build/x86_x86_64/x86_64-softmmu/qemu-system-x86_64 \ -L /home/alexey/p/qemu/pc-bios/ \ -hda x86/fc19_24GB_x86.qcow2 \ -enable-kvm \ -kernel x86/vmlinuz-3.12.11-201.fc19.x86_64 \ -initrd x86/initramfs-3.12.11-201.fc19.x86_64.img \ -append "root=/dev/sda3 console=ttyS0" \ -nographic \ -nodefaults \ -chardev stdio,id=id2,signal=off,mux=on \ -device isa-serial,id=id3,chardev=id2 \ -mon id=id4,chardev=id2,mode=readline \ -m 2G \ -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 \ -numa node,nodeid=0,cpus=0,memdev=ram-node0 \ -object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 \ -numa node,nodeid=1,cpus=1,memdev=ram-node1 \ -device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 \ -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd \ -device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 \ -device e1000,bus=bridge2,addr=0x3 \ -device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 \ -drive if=none,id=drive0,file=debian_lenny_powerpc_desktop.qcow2 \ -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 \
What am I missing here?
See above, check for NUMA host support
What I actually wanted to find out (instead of asking what I am doing now) is is this PXB device a PCI device sitting on the same PCI host bus adapter (1) or it is a separate PHB (2) with its own PCI domain (new XXXX in XXXX:00:00.0 PCI address)? I would think it is (1) but then what exactly do you call "A primary PCI bus" here (that's my ignorance speaking, yes :) )? Thanks.
You are right, the PXB is a device on the piix host-bridge bus and its bus uses the same PCI domain. However, the bus behind is exposed through ACPI as Primary PCI bus and starts a new PCI hierarchy.
You have a similar approach on Intel 450x chipset: http://www.intel.com/design/chipsets/datashts/243771.htm Look for 82454NX PCI Expander Bridge (PXB)
Thanks, Marcel
Here you have:
- 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node
in host, but you can and should put it in different host NUMA nodes)
- a pxb host bridge attached to NUMA 1 with an e1000 behind it
- a pxb host bridge attached to NUMA 0 with an e1000 behind it
- a pxb host bridge not attached to any NUMA with a hard drive behind it.
As you can see, since you already "decide" NUMA mapping at command line, it is "natural" also to attach the pxbs to the NUMA nodes.
Second thing: Booting with an unpatched seabios has bad effects:
[root@localhost ~]# cat /proc/iomem 00000000-000fffff : PCI Bus 0000:10 00000000-00000fff : reserved 00001000-0009fbff : System RAM 0009fc00-0009ffff : reserved 000c0000-000c91ff : Video ROM 000c9800-000ca1ff : Adapter ROM 000ca800-000ccbff : Adapter ROM 000f0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-3ffdffff : System RAM 01000000-0174bde4 : Kernel code 0174bde5-01d30cff : Kernel data 01eaa000-0202afff : Kernel bss 3ffe0000-3fffffff : reserved fd000000-fdffffff : 0000:00:02.0 fd000000-fdffffff : bochs-drm febc0000-febdffff : 0000:00:03.0 febc0000-febdffff : e1000 febf0000-febf0fff : 0000:00:02.0 febf0000-febf0fff : bochs-drm fec00000-fec003ff : IOAPIC 0 fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fee00000-fee00fff : Local APIC feffc000-feffffff : reserved fffc0000-ffffffff : reserved
"PCI Bus 0000:10" is bogus and "PCI Bus 0000:00" isn't there at all.
Yes, you shouldn't use pxb if you are not using the corresponding SeaBIOS. However, as I understand we always attach a SeaBIOS binary with a QEMU release, so we should be OK with this.
And this is the reason I wanted bios support *before* the PXB device implementation, but anyway, even if we have them in the same time, as long as the release has both pxb and BIOS with pxb support, is OK. (I think...)
I appreciate you looking into this and if you need further assistance don't hesitate to mail me! :)
Thanks, Marcel
cheers, Gerd