[PATCH v2 0/4] support booting more than 255 CPUs with QEMU
Changelog since: v1: * s/count_cpu/apic_id_init/ * merge handle_x2apic() into apic_id_init() RFC: * move out max-cpus check out of mptable_setup() * factor out CPU counting/apic ID detection in separate function * return back accidentially deleted debug message with APIC ID * drop unused code in smp_setup() According to SDM, if CPUs have APIC ID more than 254 firmware should pass control to OS in x2APIC mode. This series adds x2APIC bootstrap initialization. QEMU side of x2APIC support: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01094.html Igor Mammedov (3): paravirt: disable legacy bios tables in case of more than 255 CPUs support booting with more than 255 CPUs cleanup smp_setup() Kevin O'Connor (1): smp: refactor present CPU APIC ID detection and counting src/fw/paravirt.c | 6 ++++-- src/fw/smp.c | 57 ++++++++++++++++++++++++++++++++++++------------------- src/x86.h | 1 + 3 files changed, 42 insertions(+), 22 deletions(-) -- 1.8.3.1
MPTable doesn't support more than 255 CPUs and QEMU supplies an alternative MADT table which guest will use instead of it. So do not install legacy tables if more than 254 CPUs are provided Signed-off-by: Igor Mammedov <imammedo@redhat.com> --- src/fw/paravirt.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/fw/paravirt.c b/src/fw/paravirt.c index 8ed4380..9b07aaa 100644 --- a/src/fw/paravirt.c +++ b/src/fw/paravirt.c @@ -154,8 +154,10 @@ qemu_platform_setup(void) smp_setup(); // Create bios tables - pirtable_setup(); - mptable_setup(); + if (MaxCountCPUs <= 255) { + pirtable_setup(); + mptable_setup(); + } smbios_setup(); if (CONFIG_FW_ROMFILE_LOAD) { -- 1.8.3.1
From: Kevin O'Connor <kevin@koconnor.net> Signed-off-by: "Kevin O'Connor" <kevin@koconnor.net> Signed-off-by: Igor Mammedov <imammedo@redhat.com> --- v2: * s/count_cpu/apic_id_init/ * call apic_id_init() after sending SIPI, it will be needed for switching BSP into x2APIC mode --- src/fw/smp.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/src/fw/smp.c b/src/fw/smp.c index 579acdb..f2cc827 100644 --- a/src/fw/smp.c +++ b/src/fw/smp.c @@ -46,27 +46,34 @@ int apic_id_is_present(u8 apic_id) return !!(FoundAPICIDs[apic_id/32] & (1ul << (apic_id % 32))); } +static int +apic_id_init(void) +{ + CountCPUs++; + + // Track found apic id for use in legacy internal bios tables + u32 eax, ebx, ecx, cpuid_features; + cpuid(1, &eax, &ebx, &ecx, &cpuid_features); + u8 apic_id = ebx>>24; + FoundAPICIDs[apic_id/32] |= 1 << (apic_id % 32); + + return apic_id; +} + void VISIBLE32FLAT handle_smp(void) { if (!CONFIG_QEMU) return; - // Detect apic_id - u32 eax, ebx, ecx, cpuid_features; - cpuid(1, &eax, &ebx, &ecx, &cpuid_features); - u8 apic_id = ebx>>24; + // Track this CPU and detect the apic_id + int apic_id = apic_id_init(); dprintf(DEBUG_HDL_smp, "handle_smp: apic_id=%d\n", apic_id); // MTRR setup int i; for (i=0; i<smp_mtrr_count; i++) wrmsr(smp_mtrr[i].index, smp_mtrr[i].val); - - // Set bit on FoundAPICIDs - FoundAPICIDs[apic_id/32] |= (1 << (apic_id % 32)); - - CountCPUs++; } // Atomic lock for shared stack across processors. @@ -91,11 +98,6 @@ smp_setup(void) return; } - // mark the BSP initial APIC ID as found, too: - u8 apic_id = ebx>>24; - FoundAPICIDs[apic_id/32] |= (1 << (apic_id % 32)); - CountCPUs = 1; - // Setup jump trampoline to counter code. u64 old = *(u64*)BUILD_AP_BOOT_ADDR; // ljmpw $SEG_BIOS, $(entry_smp - BUILD_BIOS_ADDR) @@ -123,6 +125,9 @@ smp_setup(void) u32 sipi_vector = BUILD_AP_BOOT_ADDR >> 12; writel(APIC_ICR_LOW, 0x000C4600 | sipi_vector); + // Put BSP into APIC ID bitmap and CPUs counter + apic_id_init(); + // Wait for other CPUs to process the SIPI. u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1; while (cmos_smp_count != CountCPUs) -- 1.8.3.1
SDM[*1] says that if there are CPUs with APIC ID greater than 254, BIOS is to pass control to OS in x2APIC mode. Use the fact that QEMU passes in "etc/max-cpus" max possible "APIC ID + 1" to detect need for x2APIC mode. Also instead of CMOS_BIOS_SMP_COUNT which is limited to 256 CPUs use a new rom file "etc/boot-cpus" that QEMU supporting more than 256 CPUs will provide. *1) SDM: Volume 3: EXTENDED XAPIC (X2APIC): Initialization by System Software Signed-off-by: Igor Mammedov <imammedo@redhat.com> --- v2: * merge handle_x2apic() into apic_id_init() --- src/fw/smp.c | 27 +++++++++++++++++++++------ src/x86.h | 1 + 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/src/fw/smp.c b/src/fw/smp.c index f2cc827..dfb8425 100644 --- a/src/fw/smp.c +++ b/src/fw/smp.c @@ -19,6 +19,9 @@ #define APIC_LINT1 ((u8*)BUILD_APIC_ADDR + 0x360) #define APIC_ENABLED 0x0100 +#define MSR_IA32_APIC_BASE 0x01B +#define MSR_LOCAL_APIC_ID 0x802 +#define MSR_IA32_APICBASE_EXTD (1ULL << 10) /* Enable x2APIC mode */ static struct { u32 index; u64 val; } smp_mtrr[32]; static u32 smp_mtrr_count; @@ -49,13 +52,24 @@ int apic_id_is_present(u8 apic_id) static int apic_id_init(void) { + u32 apic_id; CountCPUs++; - // Track found apic id for use in legacy internal bios tables u32 eax, ebx, ecx, cpuid_features; cpuid(1, &eax, &ebx, &ecx, &cpuid_features); - u8 apic_id = ebx>>24; - FoundAPICIDs[apic_id/32] |= 1 << (apic_id % 32); + apic_id = ebx>>24; + if (MaxCountCPUs < 256) { // xAPIC mode + // Track found apic id for use in legacy internal bios tables + FoundAPICIDs[apic_id/32] |= 1 << (apic_id % 32); + } else if (ecx & CPUID_X2APIC) { + // switch to x2APIC mode + u64 apic_base = rdmsr(MSR_IA32_APIC_BASE); + wrmsr(MSR_IA32_APIC_BASE, apic_base | MSR_IA32_APICBASE_EXTD); + apic_id = rdmsr(MSR_LOCAL_APIC_ID); + } else { + // x2APIC is masked by CPUID + apic_id = -1; + } return apic_id; } @@ -125,12 +139,14 @@ smp_setup(void) u32 sipi_vector = BUILD_AP_BOOT_ADDR >> 12; writel(APIC_ICR_LOW, 0x000C4600 | sipi_vector); + MaxCountCPUs = romfile_loadint("etc/max-cpus", 1); // Put BSP into APIC ID bitmap and CPUs counter apic_id_init(); // Wait for other CPUs to process the SIPI. - u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1; - while (cmos_smp_count != CountCPUs) + u16 boot_cpus_count = romfile_loadint("etc/boot-cpus", + rtc_read(CMOS_BIOS_SMP_COUNT) + 1); + while (boot_cpus_count != CountCPUs) asm volatile( // Release lock and allow other processors to use the stack. " movl %%esp, %1\n" @@ -146,7 +162,6 @@ smp_setup(void) // Restore memory. *(u64*)BUILD_AP_BOOT_ADDR = old; - MaxCountCPUs = romfile_loadint("etc/max-cpus", 0); if (!MaxCountCPUs || MaxCountCPUs < CountCPUs) MaxCountCPUs = CountCPUs; diff --git a/src/x86.h b/src/x86.h index 53378e9..a770e6f 100644 --- a/src/x86.h +++ b/src/x86.h @@ -68,6 +68,7 @@ static inline void wbinvd(void) #define CPUID_MSR (1 << 5) #define CPUID_APIC (1 << 9) #define CPUID_MTRR (1 << 12) +#define CPUID_X2APIC (1 << 21) static inline void __cpuid(u32 index, u32 *eax, u32 *ebx, u32 *ecx, u32 *edx) { asm("cpuid" -- 1.8.3.1
MaxCountCPUs could never be 0 or less CountCPUs anymore, remove code that wouldn't be executed. Signed-off-by: Igor Mammedov <imammedo@redhat.com> --- src/fw/smp.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/fw/smp.c b/src/fw/smp.c index dfb8425..cb40ec6 100644 --- a/src/fw/smp.c +++ b/src/fw/smp.c @@ -162,9 +162,6 @@ smp_setup(void) // Restore memory. *(u64*)BUILD_AP_BOOT_ADDR = old; - if (!MaxCountCPUs || MaxCountCPUs < CountCPUs) - MaxCountCPUs = CountCPUs; - dprintf(1, "Found %d cpu(s) max supported %d cpu(s)\n", CountCPUs, MaxCountCPUs); } -- 1.8.3.1
On Wed, May 11, 2016 at 12:03:37PM +0200, Igor Mammedov wrote:
Changelog since: v1: * s/count_cpu/apic_id_init/ * merge handle_x2apic() into apic_id_init() RFC: * move out max-cpus check out of mptable_setup() * factor out CPU counting/apic ID detection in separate function * return back accidentially deleted debug message with APIC ID * drop unused code in smp_setup()
According to SDM, if CPUs have APIC ID more than 254 firmware should pass control to OS in x2APIC mode. This series adds x2APIC bootstrap initialization.
QEMU side of x2APIC support: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01094.html
Thanks Igor. The first 3 patches look good to me. I'll commit to SeaBIOS once the corresponding QEMU support is accepted. -Kevin
On Mo, 2016-05-16 at 21:00 -0400, Kevin O'Connor wrote:
On Wed, May 11, 2016 at 12:03:37PM +0200, Igor Mammedov wrote:
Changelog since: v1: * s/count_cpu/apic_id_init/ * merge handle_x2apic() into apic_id_init() RFC: * move out max-cpus check out of mptable_setup() * factor out CPU counting/apic ID detection in separate function * return back accidentially deleted debug message with APIC ID * drop unused code in smp_setup()
According to SDM, if CPUs have APIC ID more than 254 firmware should pass control to OS in x2APIC mode. This series adds x2APIC bootstrap initialization.
QEMU side of x2APIC support: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01094.html
Thanks Igor. The first 3 patches look good to me. I'll commit to SeaBIOS once the corresponding QEMU support is accepted.
Ping. What is the status of this in qemu? qemu 2.7 soft freeze is coming, and it is about time to plan the seabios update (1.9.3 release). cheers, Gerd
On Mon, 20 Jun 2016 10:45:37 +0200 Gerd Hoffmann <kraxel@redhat.com> wrote:
On Mo, 2016-05-16 at 21:00 -0400, Kevin O'Connor wrote:
On Wed, May 11, 2016 at 12:03:37PM +0200, Igor Mammedov wrote:
Changelog since: v1: * s/count_cpu/apic_id_init/ * merge handle_x2apic() into apic_id_init() RFC: * move out max-cpus check out of mptable_setup() * factor out CPU counting/apic ID detection in separate function * return back accidentially deleted debug message with APIC ID * drop unused code in smp_setup()
According to SDM, if CPUs have APIC ID more than 254 firmware should pass control to OS in x2APIC mode. This series adds x2APIC bootstrap initialization.
QEMU side of x2APIC support: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01094.html
Thanks Igor. The first 3 patches look good to me. I'll commit to SeaBIOS once the corresponding QEMU support is accepted.
Ping. What is the status of this in qemu? qemu 2.7 soft freeze is coming, and it is about time to plan the seabios update (1.9.3 release). I need to dust RFC off and respin as Paolo wanted to sync APIC ID other way than in RFC and also on list there is v2 ACPI refactoring that x2APIC enablement depends on.
Laszlo, wrt OVMF does suggested here fw_Cfg interface look ok to you?
cheers, Gerd
Hi,
I need to dust RFC off and respin as Paolo wanted to sync APIC ID other way than in RFC and also on list there is v2 ACPI refactoring that x2APIC enablement depends on.
Soft freeze is only one week away. Do you expect this to land in 2.7 nevertheless? cheers, Gerd
On Mon, 20 Jun 2016 13:51:28 +0200 Gerd Hoffmann <kraxel@redhat.com> wrote:
Hi,
I need to dust RFC off and respin as Paolo wanted to sync APIC ID other way than in RFC and also on list there is v2 ACPI refactoring that x2APIC enablement depends on.
Soft freeze is only one week away. Do you expect this to land in 2.7 nevertheless? considering its dependency on not yet finished KVM conterpart it probably won't make to 2.7
cheers, Gerd
On Mi, 2016-06-22 at 16:50 +0200, Igor Mammedov wrote:
On Mon, 20 Jun 2016 13:51:28 +0200 Gerd Hoffmann <kraxel@redhat.com> wrote:
Hi,
I need to dust RFC off and respin as Paolo wanted to sync APIC ID other way than in RFC and also on list there is v2 ACPI refactoring that x2APIC enablement depends on.
Soft freeze is only one week away. Do you expect this to land in 2.7 nevertheless? considering its dependency on not yet finished KVM conterpart it probably won't make to 2.7
Ok, so I guess I'll do a 1.9.3 stable release with what we have in the 1.9 branch right now, which is just this: kraxel@nilsson ~/projects/seabios (master)# git shortlog rel-1.9.2..1.9-stable Alex Williamson (1): fw/pci: Add support for mapping Intel IGD via QEMU If there are any other suggestions for cherry-picks please speak up now. cheers, Gerd
On 24/06/2016 09:28, Gerd Hoffmann wrote:
On Mi, 2016-06-22 at 16:50 +0200, Igor Mammedov wrote:
On Mon, 20 Jun 2016 13:51:28 +0200 Gerd Hoffmann <kraxel@redhat.com> wrote:
Hi,
I need to dust RFC off and respin as Paolo wanted to sync APIC ID other way than in RFC and also on list there is v2 ACPI refactoring that x2APIC enablement depends on.
Soft freeze is only one week away. Do you expect this to land in 2.7 nevertheless? considering its dependency on not yet finished KVM conterpart it probably won't make to 2.7
Ok, so I guess I'll do a 1.9.3 stable release with what we have in the 1.9 branch right now, which is just this:
kraxel@nilsson ~/projects/seabios (master)# git shortlog rel-1.9.2..1.9-stable Alex Williamson (1): fw/pci: Add support for mapping Intel IGD via QEMU
If there are any other suggestions for cherry-picks please speak up now.
I would like to have "[PATCH v3] fw/msr_feature_control: add support to set MSR_IA32_FEATURE_CONTROL", please. Paolo
Hi,
If there are any other suggestions for cherry-picks please speak up now.
I would like to have "[PATCH v3] fw/msr_feature_control: add support to set MSR_IA32_FEATURE_CONTROL", please.
Hmm, the qemu side for that one isn't merged yet, right? Is it queued to land in 2.7 already? cheers, Gerd
On 24/06/2016 10:06, Gerd Hoffmann wrote:
Hi,
If there are any other suggestions for cherry-picks please speak up now.
I would like to have "[PATCH v3] fw/msr_feature_control: add support to set MSR_IA32_FEATURE_CONTROL", please.
Hmm, the qemu side for that one isn't merged yet, right? Is it queued to land in 2.7 already?
The KVM side has been merged, and the LMCE patches are on track for being merged in 2.7 during soft freeze. Even if LMCE patches were not merged, I would extract the patch that creates the fw_cfg file and use it for nested virtualization only. Paolo
On Fri, Jun 24, 2016 at 10:12:12AM +0200, Paolo Bonzini wrote:
On 24/06/2016 10:06, Gerd Hoffmann wrote:
If there are any other suggestions for cherry-picks please speak up now.
I would like to have "[PATCH v3] fw/msr_feature_control: add support to set MSR_IA32_FEATURE_CONTROL", please.
Hmm, the qemu side for that one isn't merged yet, right? Is it queued to land in 2.7 already?
The KVM side has been merged, and the LMCE patches are on track for being merged in 2.7 during soft freeze. Even if LMCE patches were not merged, I would extract the patch that creates the fw_cfg file and use it for nested virtualization only.
The v3 patch looks fine to me. Normally I prefer to commit to SeaBIOS after the corresponding QEMU commit, but if it is easier to commit this one to SeaBIOS first I'm also fine with that. Gerd - I'll leave this one for you to commit to master as you see fit. -Kevin
On 06/20/16 13:15, Igor Mammedov wrote:
On Mon, 20 Jun 2016 10:45:37 +0200 Gerd Hoffmann <kraxel@redhat.com> wrote:
On Mo, 2016-05-16 at 21:00 -0400, Kevin O'Connor wrote:
On Wed, May 11, 2016 at 12:03:37PM +0200, Igor Mammedov wrote:
Changelog since: v1: * s/count_cpu/apic_id_init/ * merge handle_x2apic() into apic_id_init() RFC: * move out max-cpus check out of mptable_setup() * factor out CPU counting/apic ID detection in separate function * return back accidentially deleted debug message with APIC ID * drop unused code in smp_setup()
According to SDM, if CPUs have APIC ID more than 254 firmware should pass control to OS in x2APIC mode. This series adds x2APIC bootstrap initialization.
QEMU side of x2APIC support: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01094.html
Thanks Igor. The first 3 patches look good to me. I'll commit to SeaBIOS once the corresponding QEMU support is accepted.
Ping. What is the status of this in qemu? qemu 2.7 soft freeze is coming, and it is about time to plan the seabios update (1.9.3 release). I need to dust RFC off and respin as Paolo wanted to sync APIC ID other way than in RFC and also on list there is v2 ACPI refactoring that x2APIC enablement depends on.
Laszlo, wrt OVMF does suggested here fw_Cfg interface look ok to you?
I checked your v2 series quickly, with more attention to "etc/boot-cpus" and patch #3. I also re-read your argument in <20160509172828.0d839294@nial.brq.redhat.com>:
Reasons, I've skipped CPUID[0xb]:edx is not because it's currently not supported but rather: 1st: we already have 'etc/max-cpus' which is 'max possible APIC ID + 1', so reuse it. 2nd: beside complicating(making it longer, timewise) AP bootstrap, detection at runtime wouldn't work in case of CPU hotplug as it won't detect x2APIC CPUs if they are not yet present. real HW might work around this issue just hardcodding what APIC mode it wants as possible APIC IDs are known in advance for a particular board.
I think it should be okay. For OVMF I have no idea yet how we're going to accommodate this. Minimally it will depend on fixing <https://github.com/tianocore/edk2/issues/87> in UefiCpuPkg. But, the "etc/boot-cpus" interface looks sane to me. Thanks Laszlo
participants (5)
-
Gerd Hoffmann -
Igor Mammedov -
Kevin O'Connor -
Laszlo Ersek -
Paolo Bonzini