Assigning PCI resources clobbers console with io-mapped / memory-mapped UART
We ran into an issue where we were not getting a full coreboot log on Denverton with the Harcuvar CRB, where it just abruptly stops the serial console output during the assignment of the PCI resources. Root Device assign_resources, bus 0 link: 0 DOMAIN: 0000 assign_resources, bus 0 link: 0 PCI: 00:09.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 01 io PCI: 00:09.0 24 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 01 prefmem PCI: 00:09.0 20 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 01 mem PCI: 00:09.0 10 <- [0x00df700000 - 0x00df71ffff] size 0x00020000 gran 0x11 mem64 PCI: 00:0e.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 02 io PCI: 00:0e.0 24 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 02 prefmem PCI: 00:0e.0 20 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 02 mem PCI: 00:0e.0 10 <- [0x00df720000 - 0x00df73ffff] size 0x00020000 gran 0x11 mem64 PCI: 00:10.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 03 io PCI: 00:10.0 24 <- [0x00dc000000 - 0x00ddffffff] size 0x02000000 gran 0x14 bus 03 prefmem PCI: 00:10.0 20 <- [0x00de000000 - 0x00de8fffff] size 0x00900000 gran 0x14 bus 03 mem PCI: 00:10.0 10 <- [0x00df740000 - 0x00df75ffff] size 0x00020000 gran 0x11 mem64 PCI: 00:10.0 assign_resources, bus 3 link: 0 PCI: 03:00.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 04 io PCI: 03:00.0 24 <- [0x00dc000000 - 0x00ddffffff] size 0x02000000 gran 0x14 bus 04 prefmem PCI: 03:00.0 20 <- [0x00de000000 - 0x00de8fffff] size 0x00900000 gran 0x14 bus 04 mem PCI: 03:00.0 assign_resources, bus 4 link: 0 PCI: 04:00.0 10 <- [0x00dc000000 - 0x00ddffffff] size 0x02000000 gran 0x19 prefmem PCI: 04:00.0 14 <- [0x00de820000 - 0x00de823fff] size 0x00004000 gran 0x0e mem PCI: 04:00.0 18 <- [0x00de000000 - 0x00de7fffff] size 0x00800000 gran 0x17 mem PCI: 04:00.0 30 <- [0x00de800000 - 0x00de81ffff] size 0x00020000 gran 0x11 romem PCI: 03:00.0 assign_resources, bus 4 link: 0 PCI: 00:10.0 assign_resources, bus 3 link: 0 PCI: 00:12.0 10 <- [0x00df77b000 - 0x00df77b3ff] size 0x00000400 gran 0x0a mem64 PCI: 00:14.0 10 <- [0x00df774000 - 0x00df775fff] size 0x00002000 gran 0x0d mem PCI: 00:14.0 14 <- [0x00df77c000 - 0x00df77c0ff] size 0x00000100 gran 0x08 mem PCI: 00:14.0 18 <- [0x0000001c40 - 0x0000001c47] size 0x00000008 gran 0x03 io PCI: 00:14.0 1c <- [0x0000001c60 - 0x0000001c63] size 0x00000004 gran 0x02 io PCI: 00:14.0 20 <- [0x0000001c00 - 0x0000001c1f] size 0x00000020 gran 0x05 io PCI: 00:14.0 24 <- [0x00df77a000 - 0x00df77a7ff] size 0x00000800 gran 0x0b mem PCI: 00:15.0 10 <- [0x00df760000 - 0x00df76ffff] size 0x00010000 gran 0x10 mem64 PCI: 00:16.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 05 io PCI: 00:16.0 24 <- [0x00dea00000 - 0x00deefffff] size 0x00500000 gran 0x14 bus 05 prefmem PCI: 00:16.0 20 <- [0x00df500000 - 0x00df5fffff] size 0x00100000 gran 0x14 bus 05 mem PCI: 00:16.0 assign_resources, bus 5 link: 0 PCI: 05:00.0 10 <- [0x00dea00000 - 0x00debfffff] size 0x00200000 gran 0x15 prefmem64 PCI: 05:00.0 20 <- [0x00dee00000 - 0x00dee03fff] size 0x00004000 gran 0x0e prefmem64 PCI: 05:00.0 30 <- [0x00df500000 - 0x00df57ffff] size 0x00080000 gran 0x13 romem PCI: 05:00.1 10 <- [0x00dec00000 - 0x00dedfffff] size 0x00200000 gran 0x15 prefmem64 PCI: 05:00.1 20 <- [0x00dee04000 - 0x00dee07fff] size 0x00004000 gran 0x0e prefmem64 PCI: 05:00.1 30 <- [0x00df580000 - 0x00df5fffff] size 0x00080000 gran 0x13 romem PCI: 00:16.0 assign_resources, bus 5 link: 0 PCI: 00:17.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 06 io PCI: 00:17.0 24 <- [0x00df000000 - 0x00df4fffff] size 0x00500000 gran 0x14 bus 06 prefmem PCI: 00:17.0 20 <- [0x00df600000 - 0x00df6fffff] size 0x00100000 gran 0x14 bus 06 mem PCI: 00:17.0 assign_resources, bus 6 link: 0 PCI: 06:00.0 10 <- [0x00df000000 - 0x00df1fffff] size 0x00200000 gran 0x15 prefmem64 PCI: 06:00.0 20 <- [0x00df400000 - 0x00df403fff] size 0x00004000 gran 0x0e prefmem64 PCI: 06:00.0 30 <- [0x00df600000 - 0x00df67ffff] size 0x00080000 gran 0x13 romem PCI: 06:00.1 10 <- [0x00df200000 - 0x00df3fffff] size 0x00200000 gran 0x15 prefmem64 PCI: 06:00.1 20 <- [0x00df404000 - 0x00df407fff] size 0x00004000 gran 0x0e prefmem64 PCI: 06:00.1 30 <- [0x00df680000 - 0x00df6fffff] size 0x00080000 gran 0x13 romem PCI: 00:17.0 assign_resources, bus 6 link: 0 PCI: 00:18.0 10 <- [0x00df776000 - 0x00df776fff] size 0x00001000 gran 0x0c mem64 The boot continues, there's just no more console output from coreboot past this point. The payload launches fine after coreboot is finished. If we use a Linux kernel payload, it still successfully sends all of its kernel console output to the serial console beginning immediately after where the coreboot console output stopped. The last device that has its resources assigned that is displayed in the log is D24, F0 (PCI 00:18.0). The next device in the list to have its resources assigned is D26, F0 (PCI 00:1a.0), which is UART 0, which is the UART used for the serial console. So this is starting to make some sense. Since the BARs are assigned in order, and the first BAR of the UART device is the I/O BAR at offset 0x10, it's pretty clear that changing the I/O space resource assigned to the UART is what's clobbering the console. I haven't looked at it yet, but I'm assuming that the UART driver just caches the base address early on and doesn't know that the base address changed during PCI resource assignment later in the boot cycle. Of course, not only do we lose the console output, but also the UART driver is still writing the rest of the log to an I/O address that is no longer associated with the UART, but now potentially assigned to some other PCI device. Fortunately, with Denverton, the UART mode can be changed from the default Non Legacy Mode to Legacy Mode, which appears to have resolved the issue in this particular case. However, I can see this as being a general problem on any/all platforms that don't have a legacy UART (or a UART that can be put in legacy mode). Any platform with UARTs that strictly use I/O and/or memory mapped registers that are accessed via PCI BARs have the potential of having the BARs programmed with different addresses during PCI resource assignment, thus terminating the console output and otherwise potentially causing other bad things to happen. I thought it best to bring this up to the coreboot community, since this could easily be a problem on any/all platforms that don't have a legacy UART (or a UART that can be put into legacy mode). Has anybody else run into this and/or thought about a general solution to this potential problem? - Jay Jay Talbott Principal Consulting Engineer SysPro Consulting, LLC 3057 E. Muirfield St. Gilbert, AZ 85298 (480) 704-8045 (480) 445-9895 (FAX) <mailto:JayTalbott@sysproconsulting.com> JayTalbott@sysproconsulting.com <http://www.sysproconsulting.com/> http://www.sysproconsulting.com
On Thu, Mar 29, 2018 at 3:36 PM, Jay Talbott <JayTalbott@sysproconsulting.com> wrote:
We ran into an issue where we were not getting a full coreboot log on Denverton with the Harcuvar CRB, where it just abruptly stops the serial console output during the assignment of the PCI resources.
Root Device assign_resources, bus 0 link: 0
DOMAIN: 0000 assign_resources, bus 0 link: 0
PCI: 00:09.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 01 io
PCI: 00:09.0 24 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 01 prefmem
PCI: 00:09.0 20 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 01 mem
PCI: 00:09.0 10 <- [0x00df700000 - 0x00df71ffff] size 0x00020000 gran 0x11 mem64
PCI: 00:0e.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 02 io
PCI: 00:0e.0 24 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 02 prefmem
PCI: 00:0e.0 20 <- [0x00dfffffff - 0x00dffffffe] size 0x00000000 gran 0x14 bus 02 mem
PCI: 00:0e.0 10 <- [0x00df720000 - 0x00df73ffff] size 0x00020000 gran 0x11 mem64
PCI: 00:10.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 03 io
PCI: 00:10.0 24 <- [0x00dc000000 - 0x00ddffffff] size 0x02000000 gran 0x14 bus 03 prefmem
PCI: 00:10.0 20 <- [0x00de000000 - 0x00de8fffff] size 0x00900000 gran 0x14 bus 03 mem
PCI: 00:10.0 10 <- [0x00df740000 - 0x00df75ffff] size 0x00020000 gran 0x11 mem64
PCI: 00:10.0 assign_resources, bus 3 link: 0
PCI: 03:00.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 04 io
PCI: 03:00.0 24 <- [0x00dc000000 - 0x00ddffffff] size 0x02000000 gran 0x14 bus 04 prefmem
PCI: 03:00.0 20 <- [0x00de000000 - 0x00de8fffff] size 0x00900000 gran 0x14 bus 04 mem
PCI: 03:00.0 assign_resources, bus 4 link: 0
PCI: 04:00.0 10 <- [0x00dc000000 - 0x00ddffffff] size 0x02000000 gran 0x19 prefmem
PCI: 04:00.0 14 <- [0x00de820000 - 0x00de823fff] size 0x00004000 gran 0x0e mem
PCI: 04:00.0 18 <- [0x00de000000 - 0x00de7fffff] size 0x00800000 gran 0x17 mem
PCI: 04:00.0 30 <- [0x00de800000 - 0x00de81ffff] size 0x00020000 gran 0x11 romem
PCI: 03:00.0 assign_resources, bus 4 link: 0
PCI: 00:10.0 assign_resources, bus 3 link: 0
PCI: 00:12.0 10 <- [0x00df77b000 - 0x00df77b3ff] size 0x00000400 gran 0x0a mem64
PCI: 00:14.0 10 <- [0x00df774000 - 0x00df775fff] size 0x00002000 gran 0x0d mem
PCI: 00:14.0 14 <- [0x00df77c000 - 0x00df77c0ff] size 0x00000100 gran 0x08 mem
PCI: 00:14.0 18 <- [0x0000001c40 - 0x0000001c47] size 0x00000008 gran 0x03 io
PCI: 00:14.0 1c <- [0x0000001c60 - 0x0000001c63] size 0x00000004 gran 0x02 io
PCI: 00:14.0 20 <- [0x0000001c00 - 0x0000001c1f] size 0x00000020 gran 0x05 io
PCI: 00:14.0 24 <- [0x00df77a000 - 0x00df77a7ff] size 0x00000800 gran 0x0b mem
PCI: 00:15.0 10 <- [0x00df760000 - 0x00df76ffff] size 0x00010000 gran 0x10 mem64
PCI: 00:16.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 05 io
PCI: 00:16.0 24 <- [0x00dea00000 - 0x00deefffff] size 0x00500000 gran 0x14 bus 05 prefmem
PCI: 00:16.0 20 <- [0x00df500000 - 0x00df5fffff] size 0x00100000 gran 0x14 bus 05 mem
PCI: 00:16.0 assign_resources, bus 5 link: 0
PCI: 05:00.0 10 <- [0x00dea00000 - 0x00debfffff] size 0x00200000 gran 0x15 prefmem64
PCI: 05:00.0 20 <- [0x00dee00000 - 0x00dee03fff] size 0x00004000 gran 0x0e prefmem64
PCI: 05:00.0 30 <- [0x00df500000 - 0x00df57ffff] size 0x00080000 gran 0x13 romem
PCI: 05:00.1 10 <- [0x00dec00000 - 0x00dedfffff] size 0x00200000 gran 0x15 prefmem64
PCI: 05:00.1 20 <- [0x00dee04000 - 0x00dee07fff] size 0x00004000 gran 0x0e prefmem64
PCI: 05:00.1 30 <- [0x00df580000 - 0x00df5fffff] size 0x00080000 gran 0x13 romem
PCI: 00:16.0 assign_resources, bus 5 link: 0
PCI: 00:17.0 1c <- [0x000000ffff - 0x000000fffe] size 0x00000000 gran 0x0c bus 06 io
PCI: 00:17.0 24 <- [0x00df000000 - 0x00df4fffff] size 0x00500000 gran 0x14 bus 06 prefmem
PCI: 00:17.0 20 <- [0x00df600000 - 0x00df6fffff] size 0x00100000 gran 0x14 bus 06 mem
PCI: 00:17.0 assign_resources, bus 6 link: 0
PCI: 06:00.0 10 <- [0x00df000000 - 0x00df1fffff] size 0x00200000 gran 0x15 prefmem64
PCI: 06:00.0 20 <- [0x00df400000 - 0x00df403fff] size 0x00004000 gran 0x0e prefmem64
PCI: 06:00.0 30 <- [0x00df600000 - 0x00df67ffff] size 0x00080000 gran 0x13 romem
PCI: 06:00.1 10 <- [0x00df200000 - 0x00df3fffff] size 0x00200000 gran 0x15 prefmem64
PCI: 06:00.1 20 <- [0x00df404000 - 0x00df407fff] size 0x00004000 gran 0x0e prefmem64
PCI: 06:00.1 30 <- [0x00df680000 - 0x00df6fffff] size 0x00080000 gran 0x13 romem
PCI: 00:17.0 assign_resources, bus 6 link: 0
PCI: 00:18.0 10 <- [0x00df776000 - 0x00df776fff] size 0x00001000 gran 0x0c mem64
The boot continues, there’s just no more console output from coreboot past this point. The payload launches fine after coreboot is finished. If we use a Linux kernel payload, it still successfully sends all of its kernel console output to the serial console beginning immediately after where the coreboot console output stopped.
The last device that has its resources assigned that is displayed in the log is D24, F0 (PCI 00:18.0). The next device in the list to have its resources assigned is D26, F0 (PCI 00:1a.0), which is UART 0, which is the UART used for the serial console. So this is starting to make some sense…
Since the BARs are assigned in order, and the first BAR of the UART device is the I/O BAR at offset 0x10, it’s pretty clear that changing the I/O space resource assigned to the UART is what’s clobbering the console. I haven’t looked at it yet, but I’m assuming that the UART driver just caches the base address early on and doesn’t know that the base address changed during PCI resource assignment later in the boot cycle.
Of course, not only do we lose the console output, but also the UART driver is still writing the rest of the log to an I/O address that is no longer associated with the UART, but now potentially assigned to some other PCI device.
Fortunately, with Denverton, the UART mode can be changed from the default Non Legacy Mode to Legacy Mode, which appears to have resolved the issue in this particular case.
However, I can see this as being a general problem on any/all platforms that don’t have a legacy UART (or a UART that can be put in legacy mode). Any platform with UARTs that strictly use I/O and/or memory mapped registers that are accessed via PCI BARs have the potential of having the BARs programmed with different addresses during PCI resource assignment, thus terminating the console output and otherwise potentially causing other bad things to happen.
I thought it best to bring this up to the coreboot community, since this could easily be a problem on any/all platforms that don’t have a legacy UART (or a UART that can be put into legacy mode). Has anybody else run into this and/or thought about a general solution to this potential problem?
You already have the partial solution in denverton, though it's really messy and not sure where it came from: src/soc/intel/denverton_ns/uart.c src/soc/intel/skylake/uart.c has the currently preferred solution (look for pch_uart_read_resources()). Alternatively you could use a dynamic uart_platform_baseptr() implementation which returns the updated addresses.
- Jay
Jay Talbott Principal Consulting Engineer SysPro Consulting, LLC 3057 E. Muirfield St. Gilbert, AZ 85298 (480) 704-8045 (480) 445-9895 (FAX) JayTalbott@sysproconsulting.com
http://www.sysproconsulting.com
-- coreboot mailing list: coreboot@coreboot.org https://mail.coreboot.org/mailman/listinfo/coreboot
participants (2)
-
Aaron Durbin -
Jay Talbott