Attention is currently required from: Arthur Heymans, Christian Walter, Johnny Lin, Jonathan Zhang, Lean Sheng Tan, Nico Huber, Patrick Rudolph, Tim Chu.
Shuo Liu has posted comments on this change by Shuo Liu. ( https://review.coreboot.org/c/coreboot/+/83538?usp=email )
Change subject: soc/intel/xeon_sp: Reserve FSP MMIO high window ......................................................................
Patch Set 1:
(5 comments)
Patchset:
PS1:
Just for the future, please disable CONSOLE_USE_ANSI_ESCAPES and use triple backticks ``` for quo […]
Good catch, thanks!
Commit Message:
https://review.coreboot.org/c/coreboot/+/83538/comment/a9468286_e98964dc?usp... : PS1, Line 7: soc/intel/xeon_sp: Reserve FSP MMIO high window
Only for SPR, right? btw. […]
GNR and later SoC also fit.
The detailed reasons are as below,
SPR: The reserved MMIO space for HQM1/CPM1 forms holes for SoCs actually without HQM1/CPM1. GNR and later: IIO stack's MMIO window will be further assigned to PCI domains under it. There will be MMIO ranges not for PCI domain usage, and thus forming holes.
https://review.coreboot.org/c/coreboot/+/83538/comment/f359aa4d_dc58d71b?usp... : PS1, Line 19: is especially important : on systems with 2 or more sockets, where each socket has multiple : domains.
How so? In what scenario is this important? What part of coreboot […]
On multiple socket system, each socket will have IIO stacks and PCI domains under them, and thus there will be possible holes between the PCI domain MMIO windows. The more PCI domains there are, the more holes there could be, which might need more UC MTRR to cover (if the default type is chosen as WB).
P.S. In our LinuxBoot payload, if using x86_setup_mtrrs_with_detect_no_above_4gb, I cannot boot in SPR. I didn't debug into the details, but from the payload compatibility point of view, setting MTRR >4G helps here.
``` diff --git a/src/soc/intel/xeon_sp/spr/cpu.c b/src/soc/intel/xeon_sp/spr/cpu.c index ad099ab70b..f5dd07dde9 100644 --- a/src/soc/intel/xeon_sp/spr/cpu.c +++ b/src/soc/intel/xeon_sp/spr/cpu.c @@ -216,7 +216,7 @@ static void set_max_turbo_freq(void) */ static void pre_mp_init(void) { - x86_setup_mtrrs_with_detect(); + x86_setup_mtrrs_with_detect_no_above_4gb(); x86_mtrr_check(); } ```
``` [DEBUG] MTRR: Physical address space: [DEBUG] 0x0000000000000000 - 0x000000000009ffff size 0x000a0000 type 6 [DEBUG] 0x00000000000a0000 - 0x00000000000bffff size 0x00020000 type 0 [DEBUG] 0x00000000000c0000 - 0x000000007fffffff size 0x7ff40000 type 6 [DEBUG] 0x0000000080000000 - 0x00000000ffffffff size 0x80000000 type 0 [DEBUG] 0x0000000100000000 - 0x000000207fffffff size 0x1f80000000 type 6 [DEBUG] 0x0000200ffffa2000 - 0x0000200fffffffff size 0x0005e000 type 0 [DEBUG] 0x0000201ffffe0000 - 0x0000201fffffffff size 0x00020000 type 0 [DEBUG] 0x0000202ffffe0000 - 0x0000202fffffffff size 0x00020000 type 0 [DEBUG] 0x0000203ffffe0000 - 0x0000203fffffffff size 0x00020000 type 0 [DEBUG] 0x0000204ffff80000 - 0x0000204fffffffff size 0x00080000 type 0 [DEBUG] 0x0000205ffff80000 - 0x0000205fffffffff size 0x00080000 type 0 [DEBUG] 0x0000206ffffa0000 - 0x0000206fffffffff size 0x00060000 type 0 [DEBUG] 0x0000207ffffa0000 - 0x0000207fffffffff size 0x00060000 type 0 [DEBUG] 0x0000208ffffa0000 - 0x0000208fffffffff size 0x00060000 type 0 [DEBUG] 0x0000209ffffa0000 - 0x0000209fffffffff size 0x00060000 type 0 [DEBUG] 0x000020bffffe0000 - 0x000020bfffffffff size 0x00020000 type 0 [DEBUG] 0x000020cffffe0000 - 0x000020cfffffffff size 0x00020000 type 0 [DEBUG] 0x000020dffffe0000 - 0x000020dfffffffff size 0x00020000 type 0 [DEBUG] 0x000020effff80000 - 0x000020efffffffff size 0x00080000 type 0 [DEBUG] 0x000020fffff80000 - 0x000020ffffffffff size 0x00080000 type 0 [DEBUG] 0x0000210ffffa0000 - 0x0000210fffffffff size 0x00060000 type 0 [DEBUG] 0x0000211ffffa0000 - 0x0000211fffffffff size 0x00060000 type 0 [DEBUG] 0x0000212ffffa0000 - 0x0000212fffffffff size 0x00060000 type 0 [DEBUG] 0x0000213ffffa0000 - 0x0000213fffffffff size 0x00060000 type 0 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x250 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x258 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x259 0x0000000000000000 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x268 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x269 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x26a 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x26b 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x26c 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x26d 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x26e 0x0606060606060606 [DEBUG] apic_id 0x0: MTRR: Fixed MSR 0x26f 0x0606060606060606 [SPEW ] apic_id 0x0 call enable_fixed_mtrr() [DEBUG] apic_id 0x0 setup mtrr for CPU physical address size: 52 bits [DEBUG] MTRR: default type WB/UC MTRR counts: 1/1. [DEBUG] MTRR: UC selected as default type. [DEBUG] MTRR: 0 base 0x0000000000000000 mask 0x000fffff80000000 type 6 ```
``` Decompressing Linux... Parsing ELF... Performing relocations... done. Booting the kernel. Linux version 5.10.50 (root@yv3-s2-osf) (gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4), GNU ld version 2.30-108.el8_5.1) #2 SMP Wed Mar 22 13:11:01 CST 2023 Command line: loglevel=7 earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 x86/split lock detection: warning about user-space split_locks ... x86/fpu: Enabled xstate features 0x6e7, context size is 2448 bytes, using 'compacted' format. BIOS-provided physical RAM map: BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved BIOS-e820: [mem 0x0000000000001000-0x000000000009ffff] usable BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved BIOS-e820: [mem 0x0000000000100000-0x000000006354dfff] usable BIOS-e820: [mem 0x000000006354e000-0x000000008fffffff] reserved BIOS-e820: [mem 0x00000000957fc000-0x00000000957fffff] reserved BIOS-e820: [mem 0x000000009f7fc000-0x000000009f7fffff] reserved BIOS-e820: [mem 0x00000000a93fc000-0x00000000a93fffff] reserved BIOS-e820: [mem 0x00000000b2ffc000-0x00000000b2ffffff] reserved BIOS-e820: [mem 0x00000000bcbfc000-0x00000000bcbfffff] reserved BIOS-e820: [mem 0x00000000c67fc000-0x00000000c67fffff] reserved BIOS-e820: [mem 0x00000000c6ffc000-0x00000000c6ffffff] reserved BIOS-e820: [mem 0x00000000c77fc000-0x00000000c77fffff] reserved BIOS-e820: [mem 0x00000000c7ffc000-0x00000000c7ffffff] reserved BIOS-e820: [mem 0x00000000c87fc000-0x00000000c87fffff] reserved BIOS-e820: [mem 0x00000000d13fc000-0x00000000d13fffff] reserved BIOS-e820: [mem 0x00000000d97fc000-0x00000000d97fffff] reserved BIOS-e820: [mem 0x00000000e17fc000-0x00000000e17fffff] reserved BIOS-e820: [mem 0x00000000e97fc000-0x00000000e97fffff] reserved BIOS-e820: [mem 0x00000000f17fc000-0x00000000f17fffff] reserved BIOS-e820: [mem 0x00000000f97fc000-0x00000000f97fffff] reserved BIOS-e820: [mem 0x00000000f9ffc000-0x00000000f9ffffff] reserved BIOS-e820: [mem 0x00000000fa7fc000-0x00000000fa7fffff] reserved BIOS-e820: [mem 0x00000000faffc000-0x00000000faffffff] reserved BIOS-e820: [mem 0x00000000fb7fc000-0x00000000fb7fffff] reserved BIOS-e820: [mem 0x00000000fd000000-0x00000000fdffffff] reserved BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved BIOS-e820: [mem 0x0000000100000000-0x000000207fffffff] usable printk: bootconsole [earlyser0] enabled NX (Execute Disable) protection: active SMBIOS 3.0 present. DMI: Intel Archer City CRB/Archer City CRB, BIOS 24.05-519-g4c66d85fec9c-dirty 07/23/2024 tsc: Detected 2000.000 MHz processor last_pfn = 0x2080000 max_arch_pfn = 0x10000000000 x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT WARNING: BIOS bug: CPU MTRRs don't cover all of memory, losing 129024MB of RAM. ------------[ cut here ]------------ WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/mtrr/cleanup.c:978 mtrr_trim_uncached_memory+0x2b9/0x2f9 CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.50 #2 Hardware name: Intel Archer City CRB/Archer City CRB, BIOS 24.05-519-g4c66d85fec9c-dirty 07/23/2024 RIP: 0010:mtrr_trim_uncached_memory+0x2b9/0x2f9 Code: ff ff 48 01 c3 48 85 db 0f 84 8d fd ff ff 48 89 de 48 c7 c7 12 b9 14 90 48 c1 ee 14 e8 0e 24 83 ff 83 3d 4b 11 08 00 00 75 02 <0f> 0b 48 c7 c7 86 b9 14 90 41 bd 01 00 00 00 e8 f1 23 83 ff e8 43 RSP: 0000:ffffffff90203e30 EFLAGS: 00010046 ORIG_RAX: 0000000000000000 RAX: 000000000000004f RBX: 0000001f80000000 RCX: 00000000ffffefff RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff90a24a4c RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffefff R10: ffffffff90203c70 R11: ffffffff90203c68 R12: 0000000002080000 R13: ffffffff90442de0 R14: 000000000000000a R15: ffffffff904415c0 FS: 0000000000000000(0000) GS:ffffffff90386000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ff1100013a7f0ff8 CR3: 000000013a7f0000 CR4: 00000000000416a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 00000000000f0400 Call Trace: ? 0xffffffff8f600000 ? setup_arch+0x4bb/0xaed ? printk+0x53/0x6a ? start_kernel+0x55/0x507 ? load_ucode_intel_bsp+0x1c/0x4d ? secondary_startup_64_no_verify+0xc2/0xcb random: get_random_bytes called from init_oops_id+0x1d/0x2c with crng_init=0 ---[ end trace 0e56686fd458f0c5 ]--- update e820 for mtrr modified physical RAM map: modified: [mem 0x0000000000000000-0x0000000000000fff] reserved modified: [mem 0x0000000000001000-0x000000000009ffff] usable modified: [mem 0x00000000000a0000-0x00000000000fffff] reserved modified: [mem 0x0000000000100000-0x000000006354dfff] usable modified: [mem 0x000000006354e000-0x000000008fffffff] reserved modified: [mem 0x00000000957fc000-0x00000000957fffff] reserved modified: [mem 0x000000009f7fc000-0x000000009f7fffff] reserved modified: [mem 0x00000000a93fc000-0x00000000a93fffff] reserved modified: [mem 0x00000000b2ffc000-0x00000000b2ffffff] reserved modified: [mem 0x00000000bcbfc000-0x00000000bcbfffff] reserved modified: [mem 0x00000000c67fc000-0x00000000c67fffff] reserved modified: [mem 0x00000000c6ffc000-0x00000000c6ffffff] reserved modified: [mem 0x00000000c77fc000-0x00000000c77fffff] reserved modified: [mem 0x00000000c7ffc000-0x00000000c7ffffff] reserved modified: [mem 0x00000000c87fc000-0x00000000c87fffff] reserved modified: [mem 0x00000000d13fc000-0x00000000d13fffff] reserved modified: [mem 0x00000000d97fc000-0x00000000d97fffff] reserved modified: [mem 0x00000000e17fc000-0x00000000e17fffff] reserved modified: [mem 0x00000000e97fc000-0x00000000e97fffff] reserved modified: [mem 0x00000000f17fc000-0x00000000f17fffff] reserved modified: [mem 0x00000000f97fc000-0x00000000f97fffff] reserved modified: [mem 0x00000000f9ffc000-0x00000000f9ffffff] reserved modified: [mem 0x00000000fa7fc000-0x00000000fa7fffff] reserved modified: [mem 0x00000000faffc000-0x00000000faffffff] reserved modified: [mem 0x00000000fb7fc000-0x00000000fb7fffff] reserved modified: [mem 0x00000000fd000000-0x00000000fdffffff] reserved modified: [mem 0x00000000fee00000-0x00000000fee00fff] reserved modified: [mem 0x00000000ff000000-0x000000207fffffff] reserved last_pfn = 0x6354e max_arch_pfn = 0x10000000000 Memory KASLR using RDRAND RDTSC... x2apic: enabled by BIOS, switching to x2apic ops Using GB pages for direct mapping ... Initmem setup node 0 [mem 0x0000000000001000-0x000000006354dfff] DMA zone: 28769 pages in unavailable ranges DMA32 zone: 19122 pages in unavailable ranges BUG: unable to handle page fault for address: ff24b56eba60cff8 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page BAD Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 0 Comm: swapper Tainted: G W 5.10.50 #2 Hardware name: Intel Archer City CRB/Archer City CRB, BIOS 24.05-519-g4c66d85fec9c-dirty 07/23/2024 RIP: 0010:fill_pud+0xa/0x62 Code: 80 e1 08 b8 01 00 00 00 75 15 48 c1 ef 09 48 8b 42 08 83 e7 3f 48 0f a3 38 0f 92 c0 0f b6 c0 c3 41 54 49 89 f4 55 53 48 89 fb <48> f7 07 9f ff ff ff 75 40 e8 9e ca 5b 00 48 89 c7 48 89 c5 e8 5f RSP: 0000:ffffffff90203e60 EFLAGS: 00010006 RAX: 00000000000001ff RBX: ff24b56eba60cff8 RCX: 0000000001ffffff RDX: 800000013a605165 RSI: ffffffffff600000 RDI: ff24b56eba60cff8 RBP: 00000000634ed000 R08: ffffffffff600000 R09: 800000013a605165 R10: ffffffff90203b68 R11: ffffffff90203b60 R12: ffffffffff600000 R13: ffffffff8f600000 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffffffff90386000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ff24b56eba60a920 CR3: 000000013a60a000 CR4: 00000000000616b0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 00000000000f0400 Call Trace: ? set_pte_vaddr_p4d+0x24/0x35 ? __native_set_fixmap+0x21/0x28 ? map_vsyscall+0x35/0x56 ? setup_arch+0xa00/0xaed ? printk+0x53/0x6a ? start_kernel+0x55/0x507 ? load_ucode_intel_bsp+0x1c/0x4d ? secondary_startup_64_no_verify+0xc2/0xcb CR2: ff24b56eba60cff8 ---[ end trace 0e56686fd458f0c6 ]--- RIP: 0010:fill_pud+0xa/0x62 Code: 80 e1 08 b8 01 00 00 00 75 15 48 c1 ef 09 48 8b 42 08 83 e7 3f 48 0f a3 38 0f 92 c0 0f b6 c0 c3 41 54 49 89 f4 55 53 48 89 fb <48> f7 07 9f ff ff ff 75 40 e8 9e ca 5b 00 48 89 c7 48 89 c5 e8 5f RSP: 0000:ffffffff90203e60 EFLAGS: 00010006 RAX: 00000000000001ff RBX: ff24b56eba60cff8 RCX: 0000000001ffffff RDX: 800000013a605165 RSI: ffffffffff600000 RDI: ff24b56eba60cff8 RBP: 00000000634ed000 R08: ffffffffff600000 R09: 800000013a605165 R10: ffffffff90203b68 R11: ffffffff90203b60 R12: ffffffffff600000 R13: ffffffff8f600000 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffffffff90386000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ff24b56eba60a920 CR3: 000000013a60a000 CR4: 00000000000616b0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 00000000000f0400 Kernel panic - not syncing: Attempted to kill the idle task! ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]--- ```
File src/soc/intel/xeon_sp/uncore.c:
https://review.coreboot.org/c/coreboot/+/83538/comment/8b268374_285ba94c?usp... : PS1, Line 368: * Add MMIO high window.
That's what the code says. Please don't state the obvious in comments. […]
Done
https://review.coreboot.org/c/coreboot/+/83538/comment/37402f72_6e427b7d?usp... : PS1, Line 375: index++;
This not stable because mmapvtd_read_resources() mixes generic PCI […]
Yes, mmapvtd will have both anonymous static resources (starting from 0x0) and PCI resources (0x180, the VTd BAR). But here what we are actually adding is named static resources. One option is to make the named static resources starting from 0x1000, since the PCI configuration space will be only 4KB (0x000 - 0xfff).
However, maybe a more thoroughgoing change is to totally move the static resources (named and anonymous) both to start from 0x1000, where the named resources are listed first, the anonymous resources follows. I tentative test this, and the log is as below,
After - ``` [SPEW ] dev: PCI: 00:00:00.0, index: 0x180, base: 0x957fc000, size: 0x1000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x1001, base: 0x0, size: 0xa0000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x1002, base: 0x100000, size: 0x636fffff [SPEW ] dev: PCI: 00:00:00.0, index: 0x1003, base: 0x637fffff, size: 0x14000001 [SPEW ] dev: PCI: 00:00:00.0, index: 0x1004, base: 0x78000000, size: 0x8000000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x1006, base: 0x77800000, size: 0x800000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x1007, base: 0x78000000, size: 0x8000000 [INFO ] Available memory above 4GB: 129024M [SPEW ] dev: PCI: 00:00:00.0, index: 0x1009, base: 0x100000000, size: 0x1f80000000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x100a, base: 0x80000000, size: 0x10000000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x100b, base: 0xfee00000, size: 0x1000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x100c, base: 0xa0000, size: 0x20000 [SPEW ] dev: PCI: 00:00:00.0, index: 0x100d, base: 0xc0000, size: 0x40000 ```
Before - ``` [SPEW ] dev: PCI: 00:00:00.0, index: 0x180, base: 0x957fc000, size: 0x1000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x0, base: 0x0, size: 0xa0000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x1, base: 0x100000, size: 0x636fffff[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x2, base: 0x637fffff, size: 0x14000001[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x3, base: 0x78000000, size: 0x8000000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x5, base: 0x77800000, size: 0x800000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x6, base: 0x78000000, size: 0x8000000[0m [INFO ] Available memory above 4GB: 129024M[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x8, base: 0x100000000, size: 0x1f80000000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0x9, base: 0x80000000, size: 0x10000000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0xa, base: 0xfee00000, size: 0x1000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0xb, base: 0xa0000, size: 0x20000[0m [SPEW ] dev: PCI: 00:00:00.0, index: 0xc, base: 0xc0000, size: 0x40000[0m ```