Benjamin Doron has uploaded this change for review. ( https://review.coreboot.org/c/coreboot/+/79160?usp=email )
Change subject: Documentation: Improve x86_64 ......................................................................
Documentation: Improve x86_64
* Move to own markdown document * Better describe current implementation * Update TODOs
Change-Id: Ia5ba51be629a8c878aad64d3297176457cf8e855 Signed-off-by: Patrick Rudolph siro@das-labor.org --- M Documentation/arch/x86/index.md A Documentation/arch/x86/x86_64.md 2 files changed, 159 insertions(+), 90 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/60/79160/1
diff --git a/Documentation/arch/x86/index.md b/Documentation/arch/x86/index.md index c7115b4..328b934 100644 --- a/Documentation/arch/x86/index.md +++ b/Documentation/arch/x86/index.md @@ -3,93 +3,4 @@ This section contains documentation about coreboot on x86 architecture.
* [x86 PAE support](pae.md) - -## State of x86_64 support -At the moment there's only experimental x86_64 support. -The `emulation/qemu-i440fx` and `emulation/qemu-q35` boards do support -*ARCH_RAMSTAGE_X86_64* , *ARCH_POSTCAR_X86_64* and *ARCH_ROMSTAGE_X86_64*. - -In order to add support for x86_64 the following assumptions were made: -* The CPU supports long mode -* All memory returned by malloc must be below 4GiB in physical memory -* All code that is to be run must be below 4GiB in physical memory -* The high dword of pointers is always zero -* The reference implementation is qemu -* The CPU supports 1GiB hugepages -* x86 payloads are loaded below 4GiB in physical memory and are jumped - to in *protected mode* - -## Assumptions for all stages using the reference implementation -* 0-4GiB are identity mapped using 2MiB-pages as WB -* Memory above 4GiB isn't accessible -* page tables reside in memory mapped ROM -* A stage can install new page tables in RAM - -## Page tables -A `pagetables` cbfs file is generated based on an assembly file. - -To generate the static page tables it must know the physical address where to -place the file. - -The page tables contains the following structure: -* PML4E pointing to PDPE -* PDPE with *$n* entries each pointing to PDE -* *$n* PDEs with 512 entries each - -At the moment *$n* is 4, which results in identity mapping the lower 4 GiB. - -## Basic x86_64 support -Basic support for x86_64 has been implemented for QEMU mainboard target. - -## Reference implementation -The reference implementation is -* [QEMU i440fx](../../mainboard/emulation/qemu-i440fx.md) -* [QEMU Q35](../../mainboard/emulation/qemu-q35.md) - -## TODO -* Identity map memory above 4GiB in ramstage - -## Future work - -1. Fine grained page tables for SMM: - * Must not have execute and write permissions for the same page. - * Must allow only that TSEG pages can be marked executable - * Must reside in SMRAM -2. Support 64bit PCI BARs above 4GiB -3. Place and run code above 4GiB - -## Porting other boards -* Fix compilation errors -* Test how well CAR works with x86_64 and paging -* Improve mode switches -* Test libgfxinit / VGA Option ROMs / FSP - -## Known bugs on real hardware - -According to Intel x86_64 mode hasn't been validated in CAR environments. -Until now it could be verified on various Intel platforms and no issues have -been found. - -## Known bugs on KVM enabled qemu - -The `x86_64` reference code runs fine in qemu soft-cpu, but has serious issues -when using KVM mode on some machines. The workaround is to *not* place -page-tables in ROM, as done in -[CB:49228](https://review.coreboot.org/c/coreboot/+/49228). - -Here's a list of known issues: - -* After entering long mode, the FPU doesn't work anymore, including accessing - MMX registers. It works fine before entering long mode. It works fine when - switching back to protected mode. Other registers, like SSE registers, are - working fine. -* Reading from virtual memory, when the page tables are stored in ROM, causes - the MMU to abort the "page table walking" mechanism when the lower address - bits of the virtual address to be translated have a specific pattern. - Instead of loading the correct physical page, the one containing the - page tables in ROM will be loaded and used, which breaks code and data as - the page table doesn't contain the expected data. This in turn leads to - undefined behaviour whenever the 'wrong' address is being read. -* Disabling paging in compatibility mode crashes the CPU. -* Returning from long mode to compatibility mode crashes the CPU. -* Entering long mode crashes on AMD host platforms. +* [x86 long mode support](x86_64.md) diff --git a/Documentation/arch/x86/x86_64.md b/Documentation/arch/x86/x86_64.md new file mode 100644 index 0000000..fb59c3a --- /dev/null +++ b/Documentation/arch/x86/x86_64.md @@ -0,0 +1,158 @@ +# x86_64 architecture documentation + +This section documents coreboot's experimental x86_64 support. +When enabled, every coreboot stage is built for x86_64, +contrary to UEFI's implementation that only runs some stages in x86_64. +On UEFI the PEI phase, which is x86_32, brings up DRAM and installs +page tables for the x86_64 DXE and BDS phases. + +coreboot's x86_64 implementation relies on modern compiler support and +uses a *"hack"* which places page tables in ROM, which is OK as long as the CPU +doesn't set *dirty* bits, making page tables available to all stages without +the need for extra RAM. + +### Hardware requirements for x86_64 support +* The CPU must support long mode (x86_64). +* On Intel platforms the BIOS region must be large enough to contain the + page tables (24 KiB minimum) besides the coreboot code itself, which is + about 20% bigger than the x86_32 code. +* The BIOS region of the SPI flash is memory mapped at reset. + +### Toolchain requirements for x86_64 support +* The compiler must support generating code for the *large memory model* + (-mcmodel=large). It's supported since GCC 4.4. + + The large memory model causes the compiler to emit 64bit addressing + instructions, which are not only slower, but also increase code size. + +* The coreboot toolchain must generate page tables at build time that identity + map 0-4GiB using 2MiB-pages as WB. Due to this page layout the memory above + 4GiB isn't accessible, as it'd be the case for protected mode. + The page tables are placed in the CBFS and thus reside in memory mapped ROM. + +* All x86 coreboot payloads must be loaded below 4GiB in physical memory and + must have *protected mode* entry points. When jumping to the payload coreboot + will drop from long mode back to protected mode to keep compatibility with + these payloads. + +### Comparison to UEFI +On UEFI the SEC and PEI phases (similar to coreboot's bootblock and romstage) +are run in x86_32 mode. The following (guessed) reasons are likely: +* There's no need for x86_64 as memory hasn't been trained yet. The whole 4GiB + address space, including CAR, memory mapped SPI flash and PCI BARs, are + accessible in x86_32. +* When the EFI specification was written compilers did not support + *large memory model*, required in CAR when using a 1:1 page mapping +* Code is 20% bigger in x86_64 due to *large memory model* where pointers and + function calls always use 8 byte addressing. However flash size was very + limited, compared to today's flash chips, when the EFI spec was written. + +### Current software constraints for x86_64 support +The following constraints are coreboot limitations as it was intended to run in +protected mode only. The code still assumes 32bit pointers in some places and thus: +* The high dword of pointers must always be zero. +* All memory returned by malloc must be below 4GiB in physical memory. +* All code that is to be run must be below 4GiB in physical memory. + +## Supported boards +On the supported boards you can enable x86_64 compilation by setting the +Kconfig `CONFIG_USE_EXP_X86_64_SUPPORT`. + +* QEMU x86 boards +* Intel Sandy Bridge boards +* Intel Ironlake boards +* Intel X4X boards +* Intel Coffee Lake boards + +## Protected mode wrappers +On some platforms binary blobs are run to initialize parts of the hardware. +When these binary blobs have been compiled for x86_32, then coreboot must +switch to protected mode in order to call and run the blobs. Once the invoked +blobs finish running, coreboot needs to switch back to long mode. + +A wrapper has been developed to easily support such platforms. The file +`src/arch/x86/include/mode_switch.h` contains functions to call a function +(or blob) in protected mode, where up to two 32bit arguments can be passed. +The return value is propagated back to the caller. + +## Reference implementation +The reference implementation is +* [QEMU i440fx](../../mainboard/emulation/qemu-i440fx.md) +* [QEMU Q35](../../mainboard/emulation/qemu-q35.md) + +## Page tables +Page tables are generated by a tool in `util/pgtblgen/pgtblgen`. It writes +the page tables to a file which is then included into the CBFS as file called +`pagetables`. + +To generate the static page tables it must know the physical address where to +place the file. The page tables must be placed in flash that is memory mapped at +reset and thus can be read in bootblock's assembly code without additional +setup. + +The page tables contain the following structure: +* PML4E pointing to PDPE +* PDPE with *$n* entries each pointing to PDE +* *$n* PDEs with 512 entries each + +At the moment *$n* is 4, which results in identity mapping the lower 4 GiB. + +As coreboot's x86_64 support uses a *"hack"* which places page tables in ROM, +the CPU must not write to page tables at all. Usually the CPU sets the +*accessed bit* and *dirty bit*, which can be prevented by setting these bits +in the page table at creation time. + +## TODO +* Support more platforms +* Fix running VGA Option ROMs +* Fix running MRC.bin (Sandy Bridge / Haswell boards) +* Identity map memory above 4GiB in ramstage + - This requires a complete memory map to be present in postcar stage +* Fine grained page tables for SMM: + * Must not have execute and write permissions for the same page. + * Must only allow TSEG pages to be marked as executable. + * Must reside in SMRAM. + * Must be placed together with SMM rmodule. +* Support 64bit PCI BARs above 4GiB +* Place and run code above 4GiB +* Jump to compatible payloads in long mode + +## Porting other boards +* Fix compilation errors +* Test how well CAR works with x86_64 and paging +* Improve mode switches +* Test libgfxinit / VGA Option ROMs / FSP + +## Known bugs on real hardware + +According to Intel x86_64 mode hasn't been validated in CAR environments. +However, coreboot developers working on x86_64 support have tried this on +various Intel platforms, and so far haven't found any issues with CAR when +running in x86_64 mode. + +## Known bugs on KVM enabled qemu + +The `x86_64` reference code runs fine in qemu soft-cpu, but has serious issues +when using KVM mode on some machines. The workaround is to *not* place +page-tables in ROM, as done in +[CB:49228](https://review.coreboot.org/c/coreboot/+/49228). + +Here's a list of known issues: + +* After entering long mode, the FPU doesn't work anymore, including accessing + MMX registers. It works fine before entering long mode. It works fine when + switching back to protected mode. Other registers, like SSE registers, are + working fine. +* Reading from virtual memory, when the page tables are stored in ROM, causes + the MMU to abort the "page table walking" mechanism when the lower address + bits of the virtual address to be translated have a specific pattern. + Instead of loading the correct physical page, the one containing the + page tables in ROM will be loaded and used, which breaks code and data as + the page table doesn't contain the expected data. This in turn leads to + undefined behaviour whenever the 'wrong' address is being read. +* Disabling paging in compatibility mode crashes the CPU. +* Returning from long mode to compatibility mode crashes the CPU. +* Entering long mode crashes on AMD host platforms. +* Code using the R_AMD64_32S ELF relocation must not be placed above 2GiB in + memory. To prevent generating this relocation type use the -mcmodel=large + compiler flag.
-
Benjamin Doron (Code Review)