[SeaBIOS] [RFC seabios PATCH] enumerate APIC IDs directly from CPUs

[Eduardo Habkost]

On Thu, Jul 19, 2012 at 11:28:54AM -0300, Eduardo Habkost wrote:
> On Thu, Jul 19, 2012 at 12:58:46PM +0300, Gleb Natapov wrote:
> > On Tue, Jul 17, 2012 at 06:56:30PM -0300, Eduardo Habkost wrote:
> > > This patch is an attempt to fix the non-continguous-APIC-ID problem without the
> > > FW_CFG_LAPIC_INFO approach I have sent proposed last week.
> > > 
> > > Basically, this changes Seabios to probe for APIC IDs directly from the
> > > CPUs on boot, instead of getting it using fw_cfg, store the found APIC
> > > IDs on a bitmap, and use that information whe building the MADT, SRAT,
> > > and SSDT ACPI tables.
> > > 
> > > To do this properly, we have to decide the meaning of "CPU IDs" in the
> > > QEMU<->Seabios interfaces, too. I see two possible approaches:
> > > 
> > > 1) Have Seabios and QEMU agree on a a "CPU identifier", that is
> > >    independent from the APIC ID.
> > > 2) Always use the Initial APIC ID on all communication between QEMU and
> > >    Seabios.
> > > 
> > We need to be prepared to support more than 255 cpus. With > 255 cpus
> > comes x2apic and x2apic has 32bit apic ids. HW does not have to support
> > all of the bits though, but potentially all the bitmasks can grow
> > prohibitedly large.
> 
> I see only two solutions:
> 
> - Specify an interface/convention for QEMU and Seabios agree upon a "CPU
>   identifier" <=> x2apic <=> LAPIC ID mapping for all CPUs.
> - Specify new NUMA-information and CPU hotplug interfaces (or extend the
>   existing ones) based on x2apic ID, when Seabios start supporting
>   x2apic.
> 
> I am not particularly inclined towards any of those two solutions. I
> dislike them equally.  :-)

Oh, it is simpler than I have expected. x2APIC specification:

"The local APIC ID is initialized by hardware with a 32 bit ID (x2APIC
ID). The lowest 8 bits of the x2APIC ID is the legacy local xAPIC ID,
and is stored in the upper 8 bits of the APIC register for access in
xAPIC mode."

And the ACPI specification:

"Logical processors with APIC ID values of 255 and greater are required
to have a Processor Device object and must convey the processor’s APIC
information to OSPM using the Processor Local X2APIC structure. Logical
processors with APIC ID values less than 255 must use the Processor
Local APIC structure to convey their APIC information to OSPM."

That means the x2APIC ID and the xAPIC ID are interchangeable, for
values <= 255. That means the QEMU<=>Seabios communication can be safely
based on "APIC IDs" without any ambiguity.

The CPU hotplug interface is a bit of a problem because it is based on a
256-bit bitmap. But on the day it gets extended to support more than 256
CPUs, it can safely be based on APIC IDs and still keep compatibility
with systems without x2APIC.

So, now I am strongly inclined towards the second option from the list
above: just use APIC IDs everywhere to identify CPUs when QEMU and
Seabios communicate with each other, and QEMU can completely ignore the
"Processor ID" used by Seabios.

> 
> Note that I am more worried about the QEMU<->Seabios interfaces. The
> APIC ID bitmap on smp.c, for example, is just an implementation detail:
> if we make Seabios support x2apic, that code can be changed to use a
> different data structure instead.
> 
[...]
> > > To try to make things less likely to break on the common, non-hotplug
> > > case, this patch makes the Processor IDs be chosen this way:
> > > 
> > > - The CPUs present on boot get contiguous Processor IDs (even if the
> > >   APIC IDs are not contiguous);
> > > - The remaining Processor declarations are going to associated to the
> > >   remaining APIC IDs (immediately after the last present APIC ID),
> > >   sequentially. This means that hotplugged CPUs may not get contiguous
> > >   Processor declarations if their APIC IDs are not contiguous.
> > > 
> > I am curious what will happen if cpu will be hot plugged, than hibernate
> > and resume is done. After resume hot plugged cpu will have different
> > Processor ID in ACPI. This may or may not be a problem.
> 
> True. Keeping those tables stable after hotplug and hibernate may be a
> challenge.
> 
> Maybe it would be easier to just leave holes on the MADT and SSDT tables
> (making APIC ID and Processor ID always match), and hope no OS will be
> confused by the holes.

I am inclined to try this approach first (keep APIC ID == ACPI Processor
ID), to keep things simple in Seabios. I am hoping no OS will have
problems with the holes in the list of enabled Processor IDs.

-- 
Eduardo