Re: [coreboot] Out of MTRRs

[Jay Talbott]

Hi Nico,

What's the status for getting the patches merged that Aaron referenced below?

Do you think they will resolve the issue that I am seeing?

Also, just a note that the three ”No more variable MTRRs: 10" messages that occur during postcar are in the log regardless of how much memory I have in the system.

Thanks,

- Jay

Jay Talbott
Principal Consulting Engineer
SysPro Consulting, LLC
3057 E. Muirfield St.
Gilbert, AZ 85298
[phone]
[phone] (FAX)
[e-mail]
http://www.sysproconsulting.com
> -----Original Message-----
> From: Aaron Durbin [mailto:[e-mail]]
> Sent: Monday, March 19, 2018 10:22 AM
> To: Jay Talbott
> Cc: Coreboot
> Subject: Re: [coreboot] Out of MTRRs
>
> On Mon, Mar 19, 2018 at 10:55 AM, Jay Talbott
>  wrote:
> > See below…
> > Jay
>
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=79fff000 End=7a000000
> (Size 1000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7a000000 End=7a800000
> (Size 800000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7a800000 End=7ac00000
> (Size 400000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7ac00000 End=7ae00000
> (Size 200000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7ae00000 End=7af00000
> (Size 100000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7af00000 End=7af80000
> (Size 80000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7af80000 End=7afc0000
> (Size 40000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7afc0000 End=7afe0000
> (Size 20000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7afe0000 End=7aff0000
> (Size 10000)
> [Mon Mar 19 09:41:59.840 2018] MTRR Range: Start=7aff0000 End=7aff8000
> (Size 8000)
> [Mon Mar 19 09:41:59.840 2018] No more variable MTRRs: 10
> [Mon Mar 19 09:41:59.840 2018] No more variable MTRRs: 10
> [Mon Mar 19 09:41:59.840 2018] No more variable MTRRs: 10
>
> This is from postcar loading. I don't understand why the code is is
> adding that many MTRR entries during postcar loading. The only that
> should be added is the ranges needed to load and run ramstage.  The
> postcar MTRR code *does not* combine entries. It just determines the
> number of MTRRs to meet what is sent in. It seems 0x79fff000--7aff8000
> is being requested. That's a pretty bad request given the current
> constraints. Do you know why that was requested? And why it can't be
> more naturally aligned?
>
> Below is the final MTRR calculation in ramstage:
>
> [Mon Mar 19 09:42:04.473 2018] MTRR: Physical address space:
> [Mon Mar 19 09:42:04.473 2018] 0x0000000000000000 - 0x00000000000a0000
> size 0x000a0000 type 6
> [Mon Mar 19 09:42:04.473 2018] 0x00000000000a0000 - 0x00000000000c0000
> size 0x00020000 type 0
> [Mon Mar 19 09:42:04.473 2018] 0x00000000000c0000 - 0x000000007b800000
> size 0x7b740000 type 6
> [Mon Mar 19 09:42:04.473 2018] 0x000000007b800000 - 0x00000000c0000000
> size 0x44800000 type 0
> [Mon Mar 19 09:42:04.473 2018] 0x00000000c0000000 - 0x00000000d0000000
> size 0x10000000 type 1
> [Mon Mar 19 09:42:04.473 2018] 0x00000000d0000000 - 0x0000000100000000
> size 0x30000000 type 0
> [Mon Mar 19 09:42:04.473 2018] 0x0000000100000000 - 0x000000047f000000
> size 0x37f000000 type 6
> [Mon Mar 19 09:42:04.473 2018] MTRR: Fixed MSR 0x250 0x0606060606060606
> [Mon Mar 19 09:42:04.473 2018] MTRR: Fixed MSR 0x258 0x0606060606060606
> [Mon Mar 19 09:42:04.473 2018] MTRR: Fixed MSR 0x259 0x0000000000000000
> [Mon Mar 19 09:42:04.473 2018] MTRR: Fixed MSR 0x268 0x0606060606060606
> [Mon Mar 19 09:42:04.473 2018] MTRR: Fixed MSR 0x269 0x0606060606060606
> [Mon Mar 19 09:42:04.473 2018] MTRR: Fixed MSR 0x26a 0x0606060606060606
> [Mon Mar 19 09:42:04.473 2018] MTRR: Fixed MSR 0x26b
> 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x26c 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x26d
> 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x26e
> 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x26f 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] call enable_fixed_mtrr()
> [Mon Mar 19 09:42:04.480 2018] CPU physical address size: 39 bits
> [Mon Mar 19 09:42:04.480 2018] MTRR: default type WB/UC MTRR counts:
> 6/11.
> [Mon Mar 19 09:42:04.480 2018] MTRR: WB selected as default type.
> [Mon Mar 19 09:42:04.480 2018] MTRR: 0 base 0x000000007b800000 mask
> 0x0000007fff800000 type 0
> [Mon Mar 19 09:42:04.480 2018] MTRR: 1 base 0x000000007c000000 mask
> 0x0000007ffc000000 type 0
> [Mon Mar 19 09:42:04.480 2018] MTRR: 2 base 0x0000000080000000 mask
> 0x0000007fc0000000 type 0
> [Mon Mar 19 09:42:04.480 2018] MTRR: 3 base 0x00000000c0000000 mask
> 0x0000007ff0000000 type 1
> [Mon Mar 19 09:42:04.480 2018] MTRR: 4 base 0x00000000d0000000 mask
> 0x0000007ff0000000 type 0
> [Mon Mar 19 09:42:04.480 2018] MTRR: 5 base 0x00000000e0000000 mask
> 0x0000007fe0000000 type 0
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x250 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x258 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x259 0x0000000000000000
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x268 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x269 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x26a 0x0606060606060606
> [Mon Mar 19 09:42:04.480 2018] MTRR: Fixed MSR 0x26b
> 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x26c 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x26d
> 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x26e
> 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x26f 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: TEMPORARY Physical address space:
> [Mon Mar 19 09:42:04.486 2018] call enable_fixed_mtrr()
> [Mon Mar 19 09:42:04.486 2018] 0x0000000000000000 - 0x00000000000a0000
> size 0x000a0000 type 6
> [Mon Mar 19 09:42:04.486 2018] 0x00000000000a0000 - 0x00000000000c0000
> size 0x00020000 type 0
> [Mon Mar 19 09:42:04.486 2018] 0x00000000000c0000 - 0x000000007b800000
> size 0x7b740000 type 6
> [Mon Mar 19 09:42:04.486 2018] 0x000000007b800000 - 0x00000000ffe00000
> size 0x84600000 type 0
> [Mon Mar 19 09:42:04.486 2018] 0x00000000ffe00000 - 0x0000000100000000
> size 0x00200000 type 5
> [Mon Mar 19 09:42:04.486 2018] 0x0000000100000000 - 0x000000047f000000
> size 0x37f000000 type 6
> [Mon Mar 19 09:42:04.486 2018] CPU physical address size: 39 bits
> [Mon Mar 19 09:42:04.486 2018] MTRR: Removing WRCOMB type. WB/UC
> MTRR
> counts: 13/11 > 8.
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x250 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x250 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x258 0x0606060606060606
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x259 0x0000000000000000
> [Mon Mar 19 09:42:04.486 2018] MTRR: Fixed MSR 0x268 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x269 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26a 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26b
> 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26c 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26d
> 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26e
> 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26f 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x258 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] call enable_fixed_mtrr()
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x259 0x0000000000000000
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x268 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x269 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26a 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26b
> 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26c 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26d
> 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26e
> 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] MTRR: Fixed MSR 0x26f 0x0606060606060606
> [Mon Mar 19 09:42:04.859 2018] CPU physical address size: 39 bits
> [Mon Mar 19 09:42:04.859 2018] call enable_fixed_mtrr()
> [Mon Mar 19 09:42:04.859 2018] MTRR: default type WB/UC MTRR counts:
> 13/11.
> [Mon Mar 19 09:42:04.859 2018] MTRR: UC selected as default type.
> [Mon Mar 19 09:42:04.859 2018] CPU physical address size: 39 bits
> [Mon Mar 19 09:42:04.859 2018] MTRR: 0 base 0x0000000000000000 mask
> 0x0000007fc0000000 type 6
> [Mon Mar 19 09:42:04.859 2018] MTRR: 1 base 0x0000000040000000 mask
> 0x0000007fe0000000 type 6
> [Mon Mar 19 09:42:04.865 2018] MTRR: 2 base 0x0000000060000000 mask
> 0x0000007ff0000000 type 6
> [Mon Mar 19 09:42:04.865 2018] MTRR: 3 base 0x0000000070000000 mask
> 0x0000007ff8000000 type 6
> [Mon Mar 19 09:42:04.865 2018] MTRR: 4 base 0x0000000078000000 mask
> 0x0000007ffc000000 type 6
> [Mon Mar 19 09:42:04.865 2018] MTRR: 5 base 0x000000007b800000 mask
> 0x0000007fff800000 type 0
> [Mon Mar 19 09:42:04.865 2018] MTRR: 6 base 0x00000000ffe00000 mask
> 0x0000007fffe00000 type 5
> [Mon Mar 19 09:42:04.865 2018] MTRR: 7 base 0x0000000100000000 mask
> 0x0000007f00000000 type 6
> [Mon Mar 19 09:42:04.865 2018] Taking a reserved OS MTRR.
> [Mon Mar 19 09:42:04.865 2018] MTRR: 8 base 0x0000000200000000 mask
> 0x0000007e00000000 type 6
> [Mon Mar 19 09:42:04.865 2018] Taking a reserved OS MTRR.
> [Mon Mar 19 09:42:04.865 2018] MTRR: 9 base 0x0000000400000000 mask
> 0x0000007f80000000 type 6
> [Mon Mar 19 09:42:04.865 2018] Taking a reserved OS MTRR.
> [Mon Mar 19 09:42:04.865 2018] ERROR: Not enough MTRRs available! MTRR
> indexis 10 with 10 MTTRs in total.
> [Mon Mar 19 09:42:04.865 2018] Not enough MTRRs: 11 vs 10
> [Mon Mar 19 09:42:04.865 2018] Unable to insert temporary MTRR range:
> 0x00000000ffe00000 - 0x0000000100000000 size 0x00200000 type 5
> [Mon Mar 19 09:42:04.865 2018]
> [Mon Mar 19 09:42:04.865 2018] MTRR check
> [Mon Mar 19 09:42:04.865 2018] Fixed MTRRs   : Enabled
> [Mon Mar 19 09:42:04.865 2018] Variable MTRRs: Enabled
>
> Looks like the temporary mapping is causing all the spam. You could do
> a couple fo things: don't try calling mtrr_use_temp_range().
> Additionally you could set your settings differently such that it
> reduces the MTRR pressure; something like requesting different io hole
> sizes, etc.
>
> 0x0000000100000000 - 0x000000047f000000 size 0x37f000000 type 6
>
> That one hurts a lot. Count the bits set in the size. That's 9 entries
> right there.
>
> Nico has some patches outstanding that could possibly better optimize
> this condition. https://review.coreboot.org/#/c/coreboot/+/21915/ and
> 2 more on top.

Out of MTRRs

[Nico Huber]

Hello Jay,

On 23.03.2018 16:29, Jay Talbott wrote:
> What's the status for getting the patches merged that Aaron referenced
> below?

I've just updated the first two. The last one is only a source optimi-
zation that probably won't get merged (unless I find a better way to
explain how it works).
> Do you think they will resolve the issue that I am seeing?

I don't know. As you top posted, I haven't looked at it yet. I'll have a
look. But it seems very likely that you'll have to fix something about
your memory map.

Nico

Out of MTRRs

[Nico Huber]

On 31.03.2018 17:31, Nico Huber wrote:
> On 23.03.2018 16:29, Jay Talbott wrote:
> > Do you think they will resolve the issue that I am seeing?
>
> I don't know. As you top posted, I haven't looked at it yet. I'll have a
> look. But it seems very likely that you'll have to fix something about
> your memory map.

Yes, it should help. Can you please send a log of the same configuration
as used for the last log but with (at least) the first two of my patches
applied?

I would expect the UC-default case to succeed, then, because we can
squash the MTRRs above 4GiB together.

Another thought that came about the temporary ROM caching: In your case
2MiB ROM are set up (is that correct? [1]). I think this is why nobody
else run into this issue, most have a big (e.g. 16MiB) SPI flash that is
easier to handle in the WB-default case. But the code that adds the tem-
porary caching confuses me anyway:

(from src/soc/intel/skylake/romstage/romstage_fsp20.c)
>     /* Cache the ROM as WP just below 4GiB. */
>     postcar_frame_add_mtrr(&pcf, 0xFFFFFFFF - CONFIG_ROM_SIZE + 1,
>                             CONFIG_ROM_SIZE, MTRR_TYPE_WRPROT);

Here (and for many other platforms in coreboot), CONFIG_ROM_SIZE is
used. Beside your problem of the hard to cache smaller ROM sizes, it
could also be too big. e.g. with a 32MiB ROM, we would mark MMIO space
as cacheable. Aaron, what do you think? would it hurt to always mark
16MiB on Intel? IIRC, 16MiB is the maximum that is memory mapped of
the boot ROM, and I don't think it would hurt if the ROM is actually
smaller.

Nico

[1] This line of your log seems to suggest that you actually have 16MiB:
> SF: Detected FAST_SPI Hardware Sequencer with sector size 0x1000, total 0x1000000

So I suspect your CONFIG_ROM_SIZE is off.

Out of MTRRs

[Aaron Durbin]

On Sat, Mar 31, 2018 at 10:21 AM, Nico Huber  wrote:
> On 31.03.2018 17:31, Nico Huber wrote:
> > On 23.03.2018 16:29, Jay Talbott wrote:
> > > Do you think they will resolve the issue that I am seeing?
> >
> > I don't know. As you top posted, I haven't looked at it yet. I'll have a
> > look. But it seems very likely that you'll have to fix something about
> > your memory map.
>
> Yes, it should help. Can you please send a log of the same configuration
> as used for the last log but with (at least) the first two of my patches
> applied?
>
> I would expect the UC-default case to succeed, then, because we can
> squash the MTRRs above 4GiB together.
>
> Another thought that came about the temporary ROM caching: In your case
> 2MiB ROM are set up (is that correct? [1]). I think this is why nobody
> else run into this issue, most have a big (e.g. 16MiB) SPI flash that is
> easier to handle in the WB-default case. But the code that adds the tem-
> porary caching confuses me anyway:
>
> (from src/soc/intel/skylake/romstage/romstage_fsp20.c)
> >     /* Cache the ROM as WP just below 4GiB. */
> >     postcar_frame_add_mtrr(&pcf, 0xFFFFFFFF - CONFIG_ROM_SIZE + 1,
> >                             CONFIG_ROM_SIZE, MTRR_TYPE_WRPROT);
>
> Here (and for many other platforms in coreboot), CONFIG_ROM_SIZE is
> used. Beside your problem of the hard to cache smaller ROM sizes, it
> could also be too big. e.g. with a 32MiB ROM, we would mark MMIO space
> as cacheable. Aaron, what do you think? would it hurt to always mark
> 16MiB on Intel? IIRC, 16MiB is the maximum that is memory mapped of
> the boot ROM, and I don't think it would hurt if the ROM is actually
> smaller.

I think that's definitely viable. That'd fix some of these messages.
To be fair, the MTRR calculation does end up working. It's just trying
really hard temporarily add things in.
> Nico
>
> [1] This line of your log seems to suggest that you actually have 16MiB:
> > SF: Detected FAST_SPI Hardware Sequencer with sector size 0x1000, total
> > 0x1000000
>
> So I suspect your CONFIG_ROM_SIZE is off.