[PATCH v5 0/4] support booting more than 255 CPUs with QEMU

[Igor Mammedov]

Changelog since:
  v4:
    * (Kevin) drop
      [PATCH v4 3/5] error out if present cpus count changed during SMP bringup
      and resolve conflicts caused by this in followup patches
  v3:
    * don't move CPU counting around, leave it as is and
      consolidate apic_id accounting only
    * add and use qemu_*_present_cpus_count() helpers to init/get boot
      cpus count at boot time and only qemu_get_present_cpus_count()
      at resume time
    * fix S3 wakeup stuck in resume vector
    * If cpu were hotplugged during smp_scan, report it to user
      instead of silently hanging if it came online while smp_scan()
      waits for APs
  v2:
    * rebase on top of current master /smp_scan() changes/
  v1:
    * s/count_cpu/apic_id_init/
    * merge handle_x2apic() into apic_id_init()
  RFC:
    * move out max-cpus check out of mptable_setup()
    * factor out CPU counting/apic ID detection in separate function
    * return back accidentially deleted debug message with APIC ID
    * drop unused code in smp_setup()

According to SDM, if CPUs have APIC ID more than 254
firmware should pass control to OS in x2APIC mode.
This series adds x2APIC bootstrap initialization.

QEMU side of x2APIC support:
"[PATCH v3 00/13] pc: q35: x2APIC support in kvm_apic mode"
https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg02865.html

Note:
S3 wakeup works as expected if linux guest is running with IRQ remapping enabled.
However it seems that kernel is buggy with IRQ remapping disabled as kernel
disables CPUs with APIC ID > 254 and on resume from S3 hangs somewere after
getting control from Seabios.

Cc: [e-mail]
Cc: [e-mail]
Cc: [e-mail]
Cc: [e-mail]
Cc: [e-mail]
Cc: [e-mail]
Cc: [e-mail]

Igor Mammedov (3):
  paravirt: disable legacy bios tables in case of more than 255 CPUs
  add helpers to read etc/boot-cpus at resume time
  support booting with more than 255 CPUs

Kevin O'Connor (1):
  smp: consolidate CPU APIC ID detection and accounting

src/fw/paravirt.h |  3 +++
 src/x86.h         |  1 +
 src/fw/paravirt.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
 src/fw/smp.c      | 52 +++++++++++++++++++++++++++++++++++++---------------
 4 files changed, 83 insertions(+), 17 deletions(-)

[PATCH v5 1/4] paravirt: disable legacy bios tables in case of more than 255 CPUs

[Igor Mammedov]

MPTable doesn't support more than 255 CPUs and
QEMU supplies an alternative MADT table which
guest will use instead of it. So do not install
legacy tables if more than 254 CPUs are provided

Signed-off-by: Igor Mammedov 
Acked-by: Michael S. Tsirkin 
---
 src/fw/paravirt.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/fw/paravirt.c b/src/fw/paravirt.c
index 73a08f0..33a471b 100644
--- a/src/fw/paravirt.c
+++ b/src/fw/paravirt.c
@@ -164,8 +164,10 @@ qemu_platform_setup(void)
     smp_setup();

// Create bios tables
-    pirtable_setup();
-    mptable_setup();
+    if (MaxCountCPUs <= 255) {
+        pirtable_setup();
+        mptable_setup();
+    }
     smbios_setup();

if (CONFIG_FW_ROMFILE_LOAD) {

[PATCH v5 2/4] smp: consolidate CPU APIC ID detection and accounting

[Igor Mammedov]

From: Kevin O'Connor

Signed-off-by: "Kevin O'Connor" 
Signed-off-by: Igor Mammedov 
---
v3:
  * fix hang on resume path by resetting CountCPUs on every smp_scan
v2:
  * s/count_cpu/apic_id_init/
  * call apic_id_init() after sending SIPI,
    it will be needed for switching BSP into x2APIC mode
---
 src/fw/smp.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/src/fw/smp.c b/src/fw/smp.c
index 719d51d..31bcc6a 100644
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -55,23 +55,29 @@ int apic_id_is_present(u8 apic_id)
     return !!(FoundAPICIDs[apic_id/32] & (1ul << (apic_id % 32)));
 }

+static int
+apic_id_init(void)
+{
+    // Track found apic id for use in legacy internal bios tables
+    u32 eax, ebx, ecx, cpuid_features;
+    cpuid(1, &eax, &ebx, &ecx, &cpuid_features);
+    u8 apic_id = ebx>>24;
+    FoundAPICIDs[apic_id/32] |= 1 << (apic_id % 32);
+    return apic_id;
+}
+
 void VISIBLE32FLAT
 handle_smp(void)
 {
     if (!CONFIG_QEMU)
         return;

-    // Detect apic_id
-    u32 eax, ebx, ecx, cpuid_features;
-    cpuid(1, &eax, &ebx, &ecx, &cpuid_features);
-    u8 apic_id = ebx>>24;
-    dprintf(DEBUG_HDL_smp, "handle_smp: apic_id=%d\n", apic_id);
+    // Track this CPU and detect the apic_id
+    int apic_id = apic_id_init();
+    dprintf(DEBUG_HDL_smp, "handle_smp: apic_id=0x%x\n", apic_id);

smp_write_msrs();

-    // Set bit on FoundAPICIDs
-    FoundAPICIDs[apic_id/32] |= (1 << (apic_id % 32));
-
     CountCPUs++;
 }

@@ -94,8 +100,7 @@ smp_scan(void)
     }

// mark the BSP initial APIC ID as found, too:
-    u8 apic_id = ebx>>24;
-    FoundAPICIDs[apic_id/32] |= (1 << (apic_id % 32));
+    apic_id_init();
     CountCPUs = 1;

// Setup jump trampoline to counter code.

[PATCH v5 3/4] add helpers to read etc/boot-cpus at resume time

[Igor Mammedov]

Signed-off-by: Igor Mammedov 
---
 src/fw/paravirt.h |  3 +++
 src/fw/paravirt.c | 38 ++++++++++++++++++++++++++++++++++++++
 src/fw/smp.c      | 11 ++++++-----
 3 files changed, 47 insertions(+), 5 deletions(-)

diff --git a/src/fw/paravirt.h b/src/fw/paravirt.h
index ed8e5f1..6f26fd0 100644
--- a/src/fw/paravirt.h
+++ b/src/fw/paravirt.h
@@ -52,4 +52,7 @@ void qemu_preinit(void);
 void qemu_platform_setup(void);
 void qemu_cfg_init(void);

+u16 qemu_init_present_cpus_count(void);
+u16 qemu_get_present_cpus_count(void);
+
 #endif
diff --git a/src/fw/paravirt.c b/src/fw/paravirt.c
index 33a471b..125066d 100644
--- a/src/fw/paravirt.c
+++ b/src/fw/paravirt.c
@@ -319,6 +319,44 @@ qemu_romfile_add(char *name, int select, int skip, int size)
     romfile_add(&qfile->file);
 }

+static int
+qemu_romfile_get_fwcfg_entry(char *name, int *select)
+{
+    struct romfile_s *file = romfile_find(name);
+    if (!file)
+        return 0;
+    struct qemu_romfile_s *qfile;
+    qfile = container_of(file, struct qemu_romfile_s, file);
+    if (select)
+        *select = qfile->select;
+    return file->size;
+}
+
+static int boot_cpus_sel;
+static int boot_cpus_file_sz;
+
+u16
+qemu_init_present_cpus_count(void)
+{
+    u16 smp_count = romfile_loadint("etc/boot-cpus",
+                                    rtc_read(CMOS_BIOS_SMP_COUNT) + 1);
+    boot_cpus_file_sz =
+        qemu_romfile_get_fwcfg_entry("etc/boot-cpus", &boot_cpus_sel);
+    return smp_count;
+}
+
+u16
+qemu_get_present_cpus_count(void)
+{
+    u16 smp_count;
+    if (!boot_cpus_file_sz) {
+        smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
+    } else {
+        qemu_cfg_read_entry(&smp_count, boot_cpus_sel, boot_cpus_file_sz);
+    }
+    return smp_count;
+}
+
 struct e820_reservation {
     u64 address;
     u64 length;
diff --git a/src/fw/smp.c b/src/fw/smp.c
index 31bcc6a..9c404b9 100644
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -12,6 +12,7 @@
 #include "stacks.h" // yield
 #include "util.h" // smp_setup, msr_feature_control_setup
 #include "x86.h" // wrmsr
+#include "paravirt.h" // qemu_*_present_cpus_count

#define APIC_ICR_LOW ((u8*)BUILD_APIC_ADDR + 0x300)
 #define APIC_SVR     ((u8*)BUILD_APIC_ADDR + 0x0F0)
@@ -131,8 +132,8 @@ smp_scan(void)
     writel(APIC_ICR_LOW, 0x000C4600 | sipi_vector);

// Wait for other CPUs to process the SIPI.
-    u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
-    while (cmos_smp_count != CountCPUs)
+    u16 expected_cpus_count = qemu_get_present_cpus_count();
+    while (expected_cpus_count != CountCPUs)
         asm volatile(
             // Release lock and allow other processors to use the stack.
             "  movl %%esp, %1\n"
@@ -159,9 +160,9 @@ smp_setup(void)
         return;

MaxCountCPUs = romfile_loadint("etc/max-cpus", 0);
-    u8 cmos_smp_count = rtc_read(CMOS_BIOS_SMP_COUNT) + 1;
-    if (MaxCountCPUs < cmos_smp_count)
-        MaxCountCPUs = cmos_smp_count;
+    u16 smp_count = qemu_init_present_cpus_count();
+    if (MaxCountCPUs < smp_count)
+        MaxCountCPUs = smp_count;

smp_scan();
 }

[PATCH v5 4/4] support booting with more than 255 CPUs

[Igor Mammedov]

SDM[*1] says that if there are CPUs with APIC ID
greater than 254, BIOS is to pass control to OS
in x2APIC mode. Use the fact that QEMU passes in
"etc/max-cpus" max possible "APIC ID + 1" to
detect need for x2APIC mode. Also instead of
CMOS_BIOS_SMP_COUNT which is limited to 256 CPUs
use a new rom file "etc/boot-cpus" that QEMU
supporting more than 256 CPUs will provide.

*1) SDM: Volume 3: EXTENDED XAPIC (X2APIC):
     Initialization by System Software

Signed-off-by: Igor Mammedov 
---
v4:
  * use qemu_*_present_cpus_count() helpers to init/get boot
    cpus count at boot time and only qemu_get_present_cpus_count()
    at resume time
v2:
  * merge handle_x2apic() into apic_id_init()
---
 src/x86.h    |  1 +
 src/fw/smp.c | 24 ++++++++++++++++++++----
 2 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/src/x86.h b/src/x86.h
index 53378e9..a770e6f 100644
--- a/src/x86.h
+++ b/src/x86.h
@@ -68,6 +68,7 @@ static inline void wbinvd(void)
 #define CPUID_MSR (1 << 5)
 #define CPUID_APIC (1 << 9)
 #define CPUID_MTRR (1 << 12)
+#define CPUID_X2APIC (1 << 21)
 static inline void __cpuid(u32 index, u32 *eax, u32 *ebx, u32 *ecx, u32 *edx)
 {
     asm("cpuid"
diff --git a/src/fw/smp.c b/src/fw/smp.c
index 9c404b9..bcbad69 100644
--- a/src/fw/smp.c
+++ b/src/fw/smp.c
@@ -20,6 +20,9 @@
 #define APIC_LINT1   ((u8*)BUILD_APIC_ADDR + 0x360)

#define APIC_ENABLED 0x0100
+#define MSR_IA32_APIC_BASE 0x01B
+#define MSR_LOCAL_APIC_ID 0x802
+#define MSR_IA32_APICBASE_EXTD (1ULL << 10) /* Enable x2APIC mode */

static struct { u32 index; u64 val; } smp_msr[32];
 static u32 smp_msr_count;
@@ -59,11 +62,21 @@ int apic_id_is_present(u8 apic_id)
 static int
 apic_id_init(void)
 {
-    // Track found apic id for use in legacy internal bios tables
     u32 eax, ebx, ecx, cpuid_features;
     cpuid(1, &eax, &ebx, &ecx, &cpuid_features);
-    u8 apic_id = ebx>>24;
-    FoundAPICIDs[apic_id/32] |= 1 << (apic_id % 32);
+    u32 apic_id = ebx>>24;
+    if (MaxCountCPUs < 256) { // xAPIC mode
+        // Track found apic id for use in legacy internal bios tables
+        FoundAPICIDs[apic_id/32] |= 1 << (apic_id % 32);
+    } else if (ecx & CPUID_X2APIC) {
+        // switch to x2APIC mode
+        u64 apic_base = rdmsr(MSR_IA32_APIC_BASE);
+        wrmsr(MSR_IA32_APIC_BASE, apic_base | MSR_IA32_APICBASE_EXTD);
+        apic_id = rdmsr(MSR_LOCAL_APIC_ID);
+    } else {
+        // x2APIC is masked by CPUID
+        apic_id = -1;
+    }
     return apic_id;
 }

@@ -101,7 +114,6 @@ smp_scan(void)
     }

// mark the BSP initial APIC ID as found, too:
-    apic_id_init();
     CountCPUs = 1;

// Setup jump trampoline to counter code.
@@ -131,6 +143,10 @@ smp_scan(void)
     u32 sipi_vector = BUILD_AP_BOOT_ADDR >> 12;
     writel(APIC_ICR_LOW, 0x000C4600 | sipi_vector);

+    // switch to x2APIC mode after sending SIPI so that
+    // x2APIC and xAPIC mode could share AP wake up code
+    apic_id_init();
+
     // Wait for other CPUs to process the SIPI.
     u16 expected_cpus_count = qemu_get_present_cpus_count();
     while (expected_cpus_count != CountCPUs)

[Kevin O'Connor]

On Thu, Oct 13, 2016 at 02:38:24PM +0200, Igor Mammedov wrote:
[...]
> According to SDM, if CPUs have APIC ID more than 254
> firmware should pass control to OS in x2APIC mode.
> This series adds x2APIC bootstrap initialization.

Thanks.  Looks good to me - just waiting on QEMU side.

-Kevin

[Igor Mammedov]

On Tue, 18 Oct 2016 19:13:29 -0400
"Kevin O'Connor"  wrote:
> On Thu, Oct 13, 2016 at 02:38:24PM +0200, Igor Mammedov wrote:
> [...]
> > According to SDM, if CPUs have APIC ID more than 254
> > firmware should pass control to OS in x2APIC mode.
> > This series adds x2APIC bootstrap initialization.
>
> Thanks.  Looks good to me - just waiting on QEMU side.

QEMU side has been merged

commit 080ac219cc7d9c55adf925c3545b7450055ad625
pc: Add 'etc/boot-cpus' fw_cfg file for machine with more than 255 CPUs
> -Kevin

[Kevin O'Connor]

On Tue, Oct 25, 2016 at 12:44:35PM +0200, Igor Mammedov wrote:
> On Tue, 18 Oct 2016 19:13:29 -0400
> "Kevin O'Connor"  wrote:
> > On Thu, Oct 13, 2016 at 02:38:24PM +0200, Igor Mammedov wrote:
> > [...]
> > > According to SDM, if CPUs have APIC ID more than 254
> > > firmware should pass control to OS in x2APIC mode.
> > > This series adds x2APIC bootstrap initialization.
> >
> > Thanks.  Looks good to me - just waiting on QEMU side.
>
> QEMU side has been merged
>
> commit 080ac219cc7d9c55adf925c3545b7450055ad625
> pc: Add 'etc/boot-cpus' fw_cfg file for machine with more than 255 CPUs

Thanks.  I committed this series.

I plan on releasing SeaBIOS v1.10 tomorrow.

-Kevin