On Fri, Jul 13, 2012 at 5:49 PM, Vasilis Liaskovitis vasilis.liaskovitis@profitbricks.com wrote:
On Thu, Jul 12, 2012 at 08:04:56PM +0000, Blue Swirl wrote:
On Wed, Jul 11, 2012 at 10:31 AM, Vasilis Liaskovitis vasilis.liaskovitis@profitbricks.com wrote:
This is v2 of the ACPI memory hotplug prototype for x86_64 target.
I think the concept of DIMMs (what about SIMMs? SODIMMs? I liked memslot) would be useful for most targets, but hotplugging may be limited to x86 only. It would be nice to keep these two separate or as loosely coupled as possible.
agreed. what specific usecases besides hotplugging are you thinking about?
Most real boards have some kind of RAM module slots. Now this is implemented with -m option, but a generic memory slot model would be more accurate. Also the memory layout needs to be communicated to BIOS somehow unless we want to spend cycles for BIOS memory probes. The NUMA fw_cfg memory description should be usable for most cases even for embedded UP machines.
Also are there non-acpi hotplug platforms?
Some enterprise-class Sparc and PPC machines support memory hotplug.
I am trying to keep generic dimm manipulation functions (e.g. population / depopulation and searching) in hw/dimm[.ch]. Currently the x86-acpi_piix4 "backend" registers a callback for hot-add / hot-remove. In theory other hotplug backends can hook in.
btw I don't mind using "-memslot" (I think someone during v1 mentioned -dimm), we just need some consensus on the naming.
Changes v1->v2
- memory map is automatically calculated for hotplug dimms. Dimms are added from
top-of-memory skipping the pci hole at [PCI_HOLE_START, 4G).
- Renamed from "-memslot" to "-dimm". Commands changed to "dimm_add", "dimm_del".
- Seabios ejection array reduced to a byte. Use extraction macros for dimm ssdt.
- additional SRAT paravirt info does not break previous SRAT fw_cfg layout.
- Documentation of new acpi_piix4 registers and paravirt data.
- add ACPI _OST support for _OST enabled guests. This allows qemu to receive
notification for success / failure of memory hot-add and hot-remove operations. Guest needs to support _OST (https://lkml.org/lkml/2012/6/25/321)
- add monitor info command to report total guest memory (initial + hot-added)
- add command line options and monitor commands for batch dimm creation/population
Overview:
Dimm devices are modeled with a new qemu command line
"-dimm id=name,size=sz,node=pxm,populated=on|off"
As already mentioned, the starting physical address for all dimms is calculated automatically from top of memory, skipping the pci hole at [PCI_HOLE_START, 4G). Node is defining numa proximity for this dimm. When not defined it defaults to zero. "-dimm id=dimm0,size=512M,node=0,populated=off" will define a 512M memory slot belonging to numa node 0.
Dimms are added or removed with a new hmp command "dimm_add/dimm_del": Hot-add syntax: "dimm_add id" Hot-remove syntax: "dimm_del id"
Issues:
- Live migration works as long as populated field is changed to "on" for
hotplugged dimms at the destination qemu command line (patch 12/21 lifts this requirement). The DimmState structure does not yet define a VMStateDescription, but i assume this is the preferred way to pass state for migration.
- Dimms are abstracted as qdevices attached to the main system bus. However,
memory hotplugging has its own side channel ignoring main_system_bus's hotplug incapability. A cleaner integration is still needed, probably attaching memory devices as children-links of an acpi-capable device (in the pc case acpi_piix4) instead of the system bus (TBD). Then device_add/device_del instead of new commands can hopefully be used.
Comments/review welcome.
series is based on uq/master for qemu-kvm, and master for seabios. Can be found also at: http://github.com/vliaskov/qemu-kvm/commits/memhp-v2 http://github.com/vliaskov/seabios/commits/memhp-v2
Vasilis Liaskovitis (14): dimm: Implement memory device abstraction acpi_piix4: Implement memory device hotplug registers pc: calculate dimm physical addresses and adjust memory map pc: Add dimm paravirt SRAT info Implement "-dimm" command line option Implement dimm_add and dimm_del commands for hmp and qmp fix live-migration when "populated=on" is missing Implement memory hotplug notification lists acpi_piix4: _OST dimm support acpi_piix4: Update dimm state on VM reboot acpi_piix4: Update dimm bitmap state on hot-remove fail Implement "info memtotal" and "query-memtotal" Implement -dimms, -dimmspop command line options Implement mem_increase, mem_decrease hmp/qmp commands
arch_init.c | 23 ++- docs/specs/acpi_hotplug.txt | 46 +++++ docs/specs/fwcfg.txt | 28 +++ hmp-commands.hx | 67 +++++++ hmp.c | 24 +++ hmp.h | 2 + hw/Makefile.objs | 2 +- hw/acpi_piix4.c | 131 ++++++++++++- hw/dimm.c | 449 +++++++++++++++++++++++++++++++++++++++++++ hw/dimm.h | 72 +++++++ hw/pc.c | 94 +++++++++- hw/pc.h | 6 + hw/pc_piix.c | 18 ++- monitor.c | 35 ++++ monitor.h | 5 + qapi-schema.json | 38 ++++ qemu-config.c | 70 +++++++ qemu-options.hx | 15 ++ qmp-commands.hx | 137 +++++++++++++ sysemu.h | 1 + vl.c | 122 ++++++++++++- 21 files changed, 1368 insertions(+), 17 deletions(-) create mode 100644 docs/specs/acpi_hotplug.txt create mode 100644 docs/specs/fwcfg.txt create mode 100644 hw/dimm.c create mode 100644 hw/dimm.h
Vasilis Liaskovitis (7): Add ACPI_EXTRACT_DEVICE* macros Add SSDT memory device support acpi-dsdt: Implement functions for memory hotplug. acpi: generate hotplug memory devices. pciinit: Fix pcimem_start value acpi_dsdt: Support _OST dimm method acpi_dsdt: Revert internal dimm state on _OST failure
Makefile | 2 +- src/acpi-dsdt.dsl | 120 ++++++++++++++++++++++++++++++++++++- src/acpi.c | 158 +++++++++++++++++++++++++++++++++++++++++++++++-- src/pciinit.c | 2 +- src/ssdt-mem.dsl | 69 +++++++++++++++++++++ tools/acpi_extract.py | 28 +++++++++ 6 files changed, 369 insertions(+), 10 deletions(-) create mode 100644 src/ssdt-mem.dsl