Hello!
Looks like MSR_IA32_FEATURE_CONTROL is not set by firmware on CPU
hotplug. This crashes Windows kvm (cpu=host) guests on processors with
SGX, as win expects to see bit #18 on a new CPU in this register. Linux
got this issue patched in 5.6 (1db2a6e1e29f) to set msr 3a itself in
case firmware forgot to lock it.
Host kernel tracing events looks like this:
echo 1 > /sys/kernel/tracing/events/kvm/kvm_msr/enable
...
cat /sys/kernel/tracing/trace | grep 3a
CPU 0/KVM-16856 [007] ....... 380.398696: kvm_msr: msr_write
3a = 0x40005 <Guest firmware write>
CPU 3/KVM-16859 [001] ....... 380.398914: kvm_msr: msr_write
3a = 0x40005 <Guest firmware write>
CPU 2/KVM-16858 [006] ....... 380.398964: kvm_msr: msr_write
3a = 0x40005 <Guest firmware write>
CPU 1/KVM-16857 [004] ....... 380.399007: kvm_msr: msr_write
3a = 0x40005 <Guest firmware write>
CPU 0/KVM-16856 [001] ....... 384.497714: kvm_msr: msr_read
3a = 0x40005 <Guest OS read>
CPU 0/KVM-16856 [001] ....... 384.497716: kvm_msr: msr_read
3a = 0x40005
CPU 1/KVM-16857 [007] ....... 384.934791: kvm_msr: msr_read
3a = 0x40005
CPU 1/KVM-16857 [007] ....... 384.934793: kvm_msr: msr_read
3a = 0x40005
CPU 2/KVM-16858 [002] ....... 384.977871: kvm_msr: msr_read
3a = 0x40005
CPU 2/KVM-16858 [002] ....... 384.977873: kvm_msr: msr_read
3a = 0x40005
CPU 3/KVM-16859 [006] ....... 385.021217: kvm_msr: msr_read
3a = 0x40005
CPU 3/KVM-16859 [006] ....... 385.021220: kvm_msr: msr_read
3a = 0x40005
<here vCPU is hotplugged and firmware should also do msr_write>
CPU 4/KVM-17500 [002] ....... 453.733743: kvm_msr: msr_read
3a = 0x0 <- new vcpu, Windows reads and expects to see 0x40005
here instead of default 0>
CPU 4/KVM-17500 [002] ....... 453.733745: kvm_msr: msr_read
3a = 0x0
<windows crash>
I tried to fix that myself, but it got me nowhere.
I see that after SIPI CPUs start at entry_smp -> handle_smp(), but this
is only for the boot. Looks like ASL code manages hotplug. But I was not
able to follow if it even executes bios code or just jumps to somewhere
in the OS.
I would appreciate any hints how can I set MSR_IA32_FEATURE_CONTROL on
newly hotplugged CPUs. I can also test you patches if you have a fix in
mind.
Kind regards
Andrey
