Hi!
I have a bit of a newbie question. We're using Coreboot in VMs in our custom virtualization stack for x86. We are mapping the whole Coreboot image into guest physical memory so its end coincides with 4G. This makes Coreboot's reset vector entry point line up with where the hardware wants to start executing.
This works, but I wonder whether this is the intended way to use Coreboot or whether there is some more elaborate way to do it. Does all of Coreboot have to be mapped at the end of 4G? Or is there any documentation on this?
Thanks for any pointers, Julian
Hi Julian,
November 9, 2021 6:27 PM, "Julian Stecklina" julian.stecklina@cyberus-technology.de wrote:
This works, but I wonder whether this is the intended way to use Coreboot or whether there is some more elaborate way to do it. Does all of Coreboot have to be mapped at the end of 4G? Or is there any documentation on this?
The normal layout for coreboot on x86 is indeed that it's mapped in its entirety at "below 4GB" because that's how x86 generally works.
The exceptions to that rule are: - emulation/qemu* because, like in your case, hardware is virtual with no/little need for initialization - AMD Zen family, where their coprocessor initializes RAM before x86 even enters the picture
They still map everything to 4GB there because that's the most convenient approach given how everything else on coreboot for x86 works (and on Zen, coreboot is still on that single flash part so why go through the trouble of splitting things up?), but you might be able to take some ideas from those for your own platform.
Is there any particular concern with having coreboot in that memory region on your platform that makes you ask this or is this just some general curiosity?
Regards, Patrick
Hi Patrick,
On Tue, 2021-11-09 at 17:43 +0000, Patrick Georgi via coreboot wrote:
Is there any particular concern with having coreboot in that memory region on your platform that makes you ask this or is this just some general curiosity?
Thanks for confirming that we are not doing anything obviously wrong. That's already very helpful.
I was asking, because we see coreboot memcpy'ing to ROM. We map it to 0xFF00_0000 and it's copying 0xee8 bytes of memory from 0xff20_0300 to 0xff20_0300 (the same memory). So this is basically a NOP, but very weird. It happens in the bootblock:
coreboot-4.14-a0aee78c8261804e498b3c31bf4b855fb7e7d1cd Wed Nov 10 08:18:54 UTC 2021 bootblock starting (log level: 8)... FMAP: Found "FLASH" version 1.1 at 0x200000. FMAP: base = 0xff000000 size = 0x1000000 #areas = 3 FMAP: area COREBOOT found @ 200200 (14679552 bytes) CBFS: mcache @0x00014e00 built for 8 files, used 0x1e0 of 0x2000 bytes CBFS: Found 'fallback/romstage' @0x80 size 0x3ba0 in mcache @0x00014e2c WARN - MOVS to ROM at RIP ffffdc2f RCX ee8 ff200300->ff200300
The warning is from us. Improvising a stack trace at this point yields:
cbfs_prog_stage_load -> cbfs_load_and_decompress.isra.5 -> rdev_readat -> mdev_readat -> mdev -> memcpy
In cbfs_prog_stage_load, I can see this code:
size_t fsize = cbfs_load_and_decompress(&rdev, prog_start(pstage), prog_size(pstage), compression, file_hash);
And here prog_start() definitely returns the pointer 0xff20_0300, which points into ROM. Passing a pointer to ROM as a writable pointer feels wrong. Given that the code just does memcpy with src=dst in this buffer, this all is harmless, but it feels a bit like "works by accident".
Am I missing something?
Julian
Hi Julian,
Am 10.11.21 um 10:02 schrieb Julian Stecklina:
CBFS: Found 'fallback/romstage' @0x80 size 0x3ba0 in mcache @0x00014e2c WARN - MOVS to ROM at RIP ffffdc2f RCX ee8 ff200300->ff200300
The warning is from us. Improvising a stack trace at this point yields:
cbfs_prog_stage_load -> cbfs_load_and_decompress.isra.5 -> rdev_readat -> mdev_readat -> mdev -> memcpy
In cbfs_prog_stage_load, I can see this code:
size_t fsize = cbfs_load_and_decompress(&rdev, prog_start(pstage), prog_size(pstage), compression, file_hash);
And here prog_start() definitely returns the pointer 0xff20_0300, which points into ROM. Passing a pointer to ROM as a writable pointer feels wrong. Given that the code just does memcpy with src=dst in this buffer, this all is harmless, but it feels a bit like "works by accident".
Am I missing something?
no, I think you are not. It really seems odd. In `src/lib/cbfs.c:553` there is a comment:
/* Hacky way to not load programs over read only media. The stages * that would hit this path initialize themselves. */
AIUI, this "hacky way" should be taken, so cbfs_load_and_decompress() would never be called. First, I'd check if the path is tried, i.e. check the .config file if these are set to `y`:
* CONFIG_NO_XIP_EARLY_STAGES * CONFIG_BOOT_DEVICE_MEMORY_MAPPED
If they are, it's still possible that something fails and it falls back to cbfs_load_and_decompress(). In that case, further debugging might be needed.
Cheers, Nico
Am 10.11.21 um 10:43 schrieb Nico Huber:
Hi Julian,
Am 10.11.21 um 10:02 schrieb Julian Stecklina:
CBFS: Found 'fallback/romstage' @0x80 size 0x3ba0 in mcache @0x00014e2c WARN - MOVS to ROM at RIP ffffdc2f RCX ee8 ff200300->ff200300
The warning is from us. Improvising a stack trace at this point yields:
cbfs_prog_stage_load -> cbfs_load_and_decompress.isra.5 -> rdev_readat -> mdev_readat -> mdev -> memcpy
In cbfs_prog_stage_load, I can see this code:
size_t fsize = cbfs_load_and_decompress(&rdev, prog_start(pstage), prog_size(pstage), compression, file_hash);
And here prog_start() definitely returns the pointer 0xff20_0300, which points into ROM. Passing a pointer to ROM as a writable pointer feels wrong. Given that the code just does memcpy with src=dst in this buffer, this all is harmless, but it feels a bit like "works by accident".
Am I missing something?
no, I think you are not. It really seems odd. In `src/lib/cbfs.c:553` there is a comment:
/* Hacky way to not load programs over read only media. The stages * that would hit this path initialize themselves. */
AIUI, this "hacky way" should be taken, so cbfs_load_and_decompress() would never be called. First, I'd check if the path is tried, i.e. check the .config file if these are set to `y`:
- CONFIG_NO_XIP_EARLY_STAGES
Sorry, this one is inverted in the condition, it should be set to no / unset, of course.
Nico
- CONFIG_BOOT_DEVICE_MEMORY_MAPPED
If they are, it's still possible that something fails and it falls back to cbfs_load_and_decompress(). In that case, further debugging might be needed.
Cheers, Nico
Hi Nico,
On Wed, 2021-11-10 at 10:54 +0100, Nico Huber wrote:
AIUI, this "hacky way" should be taken, so cbfs_load_and_decompress() would never be called. First, I'd check if the path is tried, i.e. check the .config file if these are set to `y`:
- CONFIG_NO_XIP_EARLY_STAGES
Sorry, this one is inverted in the condition, it should be set to no / unset, of course.
Nico
- CONFIG_BOOT_DEVICE_MEMORY_MAPPED
Both of thise are disabled for qemu platforms (which we use). The Qemu platforms explicitly select BOOT_DEVICE_NOT_SPI_FLASH which in turn disables CONFIG_BOOT_DEVICE_MEMORY_MAPPED.
If they are, it's still possible that something fails and it falls back to cbfs_load_and_decompress(). In that case, further debugging might be needed.
Julian
Hi Nico,
with the following patch, the Qemu coreboot image indeed does not write to ROM anymore.
The question is whether this is behavior that is also considered broken on qemu?
diff --git a/src/mainboard/emulation/qemu-i440fx/Kconfig b/src/mainboard/emulation/qemu-i440fx/Kconfig index 19f0fca5abe9..eba8d00bd761 100644 --- a/src/mainboard/emulation/qemu-i440fx/Kconfig +++ b/src/mainboard/emulation/qemu-i440fx/Kconfig @@ -15,7 +15,7 @@ config BOARD_SPECIFIC_OPTIONS select MAINBOARD_FORCE_NATIVE_VGA_INIT select HAVE_ASAN_IN_ROMSTAGE select NO_SMM - select BOOT_DEVICE_NOT_SPI_FLASH + select BOOT_DEVICE_SPI_FLASH
Julian
On Thu, 2021-11-11 at 12:20 +0100, Julian Stecklina wrote:
Hi Nico,
On Wed, 2021-11-10 at 10:54 +0100, Nico Huber wrote:
AIUI, this "hacky way" should be taken, so cbfs_load_and_decompress() would never be called. First, I'd check if the path is tried, i.e. check the .config file if these are set to `y`:
- CONFIG_NO_XIP_EARLY_STAGES
Sorry, this one is inverted in the condition, it should be set to no / unset, of course.
Nico
- CONFIG_BOOT_DEVICE_MEMORY_MAPPED
Both of thise are disabled for qemu platforms (which we use). The Qemu platforms explicitly select BOOT_DEVICE_NOT_SPI_FLASH which in turn disables CONFIG_BOOT_DEVICE_MEMORY_MAPPED.
If they are, it's still possible that something fails and it falls back to cbfs_load_and_decompress(). In that case, further debugging might be needed.
Julian
Hi,
On 11.11.21 14:05, Julian Stecklina wrote:
with the following patch, the Qemu coreboot image indeed does not write to ROM anymore.
The question is whether this is behavior that is also considered broken on qemu?
diff --git a/src/mainboard/emulation/qemu-i440fx/Kconfig b/src/mainboard/emulation/qemu-i440fx/Kconfig index 19f0fca5abe9..eba8d00bd761 100644 --- a/src/mainboard/emulation/qemu-i440fx/Kconfig +++ b/src/mainboard/emulation/qemu-i440fx/Kconfig @@ -15,7 +15,7 @@ config BOARD_SPECIFIC_OPTIONS select MAINBOARD_FORCE_NATIVE_VGA_INIT select HAVE_ASAN_IN_ROMSTAGE select NO_SMM
- select BOOT_DEVICE_NOT_SPI_FLASH
- select BOOT_DEVICE_SPI_FLASH
this wouldn't be correct as QEMU doesn't emulate SPI, IIRC. However, selecting BOOT_DEVICE_MEMORY_MAPPED directly would reflect reality. Not sure if it's that easy, though ;)
Nico
On Thu, 2021-11-11 at 15:21 +0100, Nico Huber wrote:
Hi,
On 11.11.21 14:05, Julian Stecklina wrote:
with the following patch, the Qemu coreboot image indeed does not write to ROM anymore.
The question is whether this is behavior that is also considered broken on qemu?
diff --git a/src/mainboard/emulation/qemu-i440fx/Kconfig b/src/mainboard/emulation/qemu-i440fx/Kconfig index 19f0fca5abe9..eba8d00bd761 100644 --- a/src/mainboard/emulation/qemu-i440fx/Kconfig +++ b/src/mainboard/emulation/qemu-i440fx/Kconfig @@ -15,7 +15,7 @@ config BOARD_SPECIFIC_OPTIONS select MAINBOARD_FORCE_NATIVE_VGA_INIT select HAVE_ASAN_IN_ROMSTAGE select NO_SMM - select BOOT_DEVICE_NOT_SPI_FLASH + select BOOT_DEVICE_SPI_FLASH
this wouldn't be correct as QEMU doesn't emulate SPI, IIRC. However, selecting BOOT_DEVICE_MEMORY_MAPPED directly would reflect reality. Not sure if it's that easy, though ;)
Just selecting BOOT_DEVICE_MEMORY_MAPPED in the qemu Kconfigs works just fine. I'll try to send a patch to Gerrit, if I can defeat the tooling for that. :)
Julian
On Thu, 2021-11-11 at 16:43 +0100, Julian Stecklina wrote:
On Thu, 2021-11-11 at 15:21 +0100, Nico Huber wrote:
this wouldn't be correct as QEMU doesn't emulate SPI, IIRC. However, selecting BOOT_DEVICE_MEMORY_MAPPED directly would reflect reality. Not sure if it's that easy, though ;)
Just selecting BOOT_DEVICE_MEMORY_MAPPED in the qemu Kconfigs works just fine. I'll try to send a patch to Gerrit, if I can defeat the tooling for that. :)
I defeated the tooling: https://review.coreboot.org/c/coreboot/+/59474
Julian
Julian Stecklina wrote:
I was asking, because we see coreboot memcpy'ing to ROM. We map it to 0xFF00_0000 and it's copying 0xee8 bytes of memory from 0xff20_0300 to 0xff20_0300 (the same memory). So this is basically a NOP, but very weird.
Right after reset it wouldn't be a NOP, but this seems to be a lot later.
(Real mode f000:0000 after reset maps to physical ffff0000.)
CBFS: Found 'fallback/romstage' @0x80 size 0x3ba0 in mcache @0x00014e2c WARN - MOVS to ROM at RIP ffffdc2f RCX ee8 ff200300->ff200300
I guess this is in 32-bit mode. coreboot leaves real mode quickly.
Anyway, memory mapped sounds correct for QEMU. :)
//Peter
-
Julian Stecklina -
Nico Huber -
Nico Huber -
patrick＠coreboot.org -
Peter Stuge