Problems with CAR setup on Tiger Lake

[Michał Żygowski]

Dear coreboot community,

I have a Tiger Lake UP3 RVP and I try to build a working coreboot on it,
however facing an early stuck during CAR setup. Tried different approaches:

- native coreboot's CAR setup - the last seen post code is 0x26
- FSP-T CAR setup - the last seen post code is 0x7F (which is
TempRamInit Exit event according to FSP integration guide), FSP from
public repo, Client variant

Used microcode from original RVP firmware.

Are there any patches that I have to apply to make it working?

Best regards,

-- 
Michał Żygowski
Firmware Engineer
https://3mdeb.com | @3mdeb_com

[Tim Wawrzynczak]

Dear Michał,

Do you have a config for us to look at for your board? Are you
using INTEL_CAR_NEM_ENHANCED ?

Based on the post code you observed last, I might guess that you overfilled
the cache when doing the cache fill
operation, e.g. the next few lines in cache_as_ram.S after post-code 0x26;
doing so will cause a MCE.
Do you know how much data you're trying to place in the cache
(CONFIG_DCACHE_RAM_SIZE) and also how much LLC your SKU has?

FYI, Tiger Lake has some new requirements for CAR setup when using eNEM
mode, which
should be taken care of by the default soc/intel/tigerlake/Kconfig file. Do
you have local changes to that?

You can also see mb/google/volteer for an example of a functional TGL
coreboot board, feel free to ask me
any other questions you have.

Cheers,
 - Tim


On Tue, Jan 19, 2021 at 8:58 AM Michał Żygowski <[e-mail]>
wrote:
> Dear coreboot community,
>
> I have a Tiger Lake UP3 RVP and I try to build a working coreboot on it,
> however facing an early stuck during CAR setup. Tried different approaches:
>
> - native coreboot's CAR setup - the last seen post code is 0x26
> - FSP-T CAR setup - the last seen post code is 0x7F (which is
> TempRamInit Exit event according to FSP integration guide), FSP from
> public repo, Client variant
>
> Used microcode from original RVP firmware.
>
> Are there any patches that I have to apply to make it working?
>
> Best regards,

[Michał Żygowski]

On 1/19/21 5:10 PM, Tim Wawrzynczak via coreboot wrote:
> Dear Michał,
>
> Do you have a config for us to look at for your board? Are you
> using INTEL_CAR_NEM_ENHANCED ?

Hi Tim,

Sure, see attached, yes INTEL_CAR_NEM_ENHANCED is selected.
> Based on the post code you observed last, I might guess that you
> overfilled the cache when doing the cache fill
> operation, e.g. the next few lines in cache_as_ram.S after post-code
> 0x26; doing so will cause a MCE.
> Do you know how much data you're trying to place in the cache
> (CONFIG_DCACHE_RAM_SIZE) and also how much LLC your SKU has?

Not sure about LLC size, but yeah, caching whole SPI (32MB) may be too
big. Maybe this is the problem. This is a Super SKU.
CONFIG_DCACHE_RAM_SIZE=0x80000
> FYI, Tiger Lake has some new requirements for CAR setup when using
> eNEM mode, which
> should be taken care of by the default soc/intel/tigerlake/Kconfig
> file. Do you have local changes to that?

No changes except the FMD layout to give more space for
COREBOOT/FW_MAIN_A/FW_MAIN_B by sacrificing RW_LEGACY.
> You can also see mb/google/volteer for an example of a functional TGL
> coreboot board, feel free to ask me
> any other questions you have.

Will visit it and check it.
> Cheers,
>  - Tim

Regards,

-- 
Michał Żygowski
Firmware Engineer
https://3mdeb.com | @3mdeb_com
> On Tue, Jan 19, 2021 at 8:58 AM Michał Żygowski
> <[e-mail] <mailto:[e-mail]>> wrote:
>
> Dear coreboot community,
>
> I have a Tiger Lake UP3 RVP and I try to build a working coreboot
>     on it,
>     however facing an early stuck during CAR setup. Tried different
>     approaches:
>
> - native coreboot's CAR setup - the last seen post code is 0x26
>     - FSP-T CAR setup - the last seen post code is 0x7F (which is
>     TempRamInit Exit event according to FSP integration guide), FSP from
>     public repo, Client variant
>
> Used microcode from original RVP firmware.
>
> Are there any patches that I have to apply to make it working?
>
> Best regards,
>
> _______________________________________________
> coreboot mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Tim Wawrzynczak]

On Tue, Jan 19, 2021 at 9:27 AM Michał Żygowski <[e-mail]>
wrote:
> On 1/19/21 5:10 PM, Tim Wawrzynczak via coreboot wrote:
> > Dear Michał,
> >
> > Do you have a config for us to look at for your board? Are you
> > using INTEL_CAR_NEM_ENHANCED ?
>
> Hi Tim,
>
> Sure, see attached, yes INTEL_CAR_NEM_ENHANCED is selected.
> > Based on the post code you observed last, I might guess that you
> > overfilled the cache when doing the cache fill
> > operation, e.g. the next few lines in cache_as_ram.S after post-code
> > 0x26; doing so will cause a MCE.
> > Do you know how much data you're trying to place in the cache
> > (CONFIG_DCACHE_RAM_SIZE) and also how much LLC your SKU has?
>
> Not sure about LLC size, but yeah, caching whole SPI (32MB) may be too
> big. Maybe this is the problem. This is a Super SKU.
> CONFIG_DCACHE_RAM_SIZE=0x80000

That would absolutely be a problem :) The biggest LLC we've seen on the UP3
parts
is 12MB, IIRC, starting as low as ~4MB, IIRC. The goal of eNEM is to keep
all of the
modifiable data (e.g. stack) in cache ways that are marked as 'no-evict',
but also to
keep code in cache when feasible; however the code is kept in cache ways
that are
marked as evictable, because in CAR, obviously the code comes from SPI
flash, so
it is OK to flush out those cache lines because the data has a backing
store.

To make this scheme work, you need to ensure that you are not using more
than
(LLC Size - 0.5 MiB) of the LLC during CAR, or you will (probably) get a
MCE,
and hang with just a few post codes to debug with :)

-Tim
> > FYI, Tiger Lake has some new requirements for CAR setup when using
> > eNEM mode, which
> > should be taken care of by the default soc/intel/tigerlake/Kconfig
> > file. Do you have local changes to that?
> > No changes except the FMD layout to give more space for
> > COREBOOT/FW_MAIN_A/FW_MAIN_B by sacrificing RW_LEGACY.
> >
> > You can also see mb/google/volteer for an example of a functional TGL
> > coreboot board, feel free to ask me
> > any other questions you have.
> > Will visit it and check it.
> >
> > Cheers,
> >  - Tim
> > Regards,
> > On Tue, Jan 19, 2021 at 8:58 AM Michał Żygowski
> > <[e-mail] <mailto:[e-mail]>> wrote:
> >
> > Dear coreboot community,
> >
> > I have a Tiger Lake UP3 RVP and I try to build a working coreboot
> >     on it,
> >     however facing an early stuck during CAR setup. Tried different
> >     approaches:
> >
> > - native coreboot's CAR setup - the last seen post code is 0x26
> >     - FSP-T CAR setup - the last seen post code is 0x7F (which is
> >     TempRamInit Exit event according to FSP integration guide), FSP from
> >     public repo, Client variant
> >
> > Used microcode from original RVP firmware.
> >
> > Are there any patches that I have to apply to make it working?
> >
> > Best regards,
> >
> > _______________________________________________
> > coreboot mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
>
> _______________________________________________
> coreboot mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Furquan Shaikh]

On Tue, Jan 19, 2021 at 9:17 AM Tim Wawrzynczak via coreboot
<[e-mail]> wrote:
> On Tue, Jan 19, 2021 at 9:27 AM Michał Żygowski <[e-mail]> wrote:
> > On 1/19/21 5:10 PM, Tim Wawrzynczak via coreboot wrote:
> > > Dear Michał,
> > >
> > > Do you have a config for us to look at for your board? Are you
> > > using INTEL_CAR_NEM_ENHANCED ?
> >
> > Hi Tim,
> >
> > Sure, see attached, yes INTEL_CAR_NEM_ENHANCED is selected.
> > > Based on the post code you observed last, I might guess that you
> > > overfilled the cache when doing the cache fill
> > > operation, e.g. the next few lines in cache_as_ram.S after post-code
> > > 0x26; doing so will cause a MCE.
> > > Do you know how much data you're trying to place in the cache
> > > (CONFIG_DCACHE_RAM_SIZE) and also how much LLC your SKU has?
> >
> > Not sure about LLC size, but yeah, caching whole SPI (32MB) may be too
> > big. Maybe this is the problem. This is a Super SKU.
> > CONFIG_DCACHE_RAM_SIZE=0x80000
>
> That would absolutely be a problem :) The biggest LLC we've seen on the UP3 parts
> is 12MB, IIRC, starting as low as ~4MB, IIRC. The goal of eNEM is to keep all of the
> modifiable data (e.g. stack) in cache ways that are marked as 'no-evict', but also to
> keep code in cache when feasible; however the code is kept in cache ways that are
> marked as evictable, because in CAR, obviously the code comes from SPI flash, so
> it is OK to flush out those cache lines because the data has a backing store.
>
> To make this scheme work, you need to ensure that you are not using more than
> (LLC Size - 0.5 MiB) of the LLC during CAR, or you will (probably) get a MCE,
> and hang with just a few post codes to debug with :)
>
> -Tim
> > > FYI, Tiger Lake has some new requirements for CAR setup when using
> > > eNEM mode, which
> > > should be taken care of by the default soc/intel/tigerlake/Kconfig
> > > file. Do you have local changes to that?
> > > No changes except the FMD layout to give more space for
> > > COREBOOT/FW_MAIN_A/FW_MAIN_B by sacrificing RW_LEGACY.
> > >
> > > You can also see mb/google/volteer for an example of a functional TGL
> > > coreboot board, feel free to ask me
> > > any other questions you have.
> > > Will visit it and check it.
> > >
> > > Cheers,
> > >  - Tim
> > > Regards,
> > > On Tue, Jan 19, 2021 at 8:58 AM Michał Żygowski
> > > <[e-mail] <mailto:[e-mail]>> wrote:
> > >
> > > Dear coreboot community,
> > >
> > > I have a Tiger Lake UP3 RVP and I try to build a working coreboot
> > >     on it,
> > >     however facing an early stuck during CAR setup. Tried different
> > >     approaches:
> > >
> > > - native coreboot's CAR setup - the last seen post code is 0x26
> > >     - FSP-T CAR setup - the last seen post code is 0x7F (which is
> > >     TempRamInit Exit event according to FSP integration guide), FSP from
> > >     public repo, Client variant
> > >
> > > Used microcode from original RVP firmware.

Do you know the microcode version that you are using? There was an
issue with the older versions that caused a hang during NEM setup. As
per the commit message in
https://review.coreboot.org/c/coreboot/+/45094, it looks like you need
0x56 or newer microcode for enhanced NEM to work.

A quick way to verify if it is the microcode version issue: Select
INTEL_CAR_NEM instead of INTEL_CAR_NEM_ENHANCED for your board and see
if it gets any further in the boot.
> > > Are there any patches that I have to apply to make it working?
> > >
> > > Best regards,
> > >
> > > _______________________________________________
> > > coreboot mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> >
> > _______________________________________________
> > coreboot mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
>
> _______________________________________________
> coreboot mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Michał Żygowski]

Hi Furquan,
> Do you know the microcode version that you are using? There was an
> issue with the older versions that caused a hang during NEM setup. As
> per the commit message in
> https://review.coreboot.org/c/coreboot/+/45094, it looks like you need
> 0x56 or newer microcode for enhanced NEM to work.
>
> A quick way to verify if it is the microcode version issue: Select
> INTEL_CAR_NEM instead of INTEL_CAR_NEM_ENHANCED for your board and see
> if it gets any further in the boot.

I have tried a few microcodes:
- the one shipped originally with RVP platform is CpuSignature:
000806C0h, Revision: 00000064h, Date: 02.03.2020 so the revision is
newer that 0x56.
- also tried from TGL_BIOS_ww51_2020 BIOS images which has 4 blobs:
    CpuSignature: 000806D1h, Revision: 00000008h, Date: 30.11.2020
    CpuSignature: 000806D0h, Revision: 0000004Eh, Date: 01.12.2020
    CpuSignature: 000806C1h, Revision: 00000072h, Date: 20.11.2020
    CpuSignature: 000806C2h, Revision: 00000002h, Date: 04.12.2020

It seems it have different steppings, so don't know if it really works.

I will try to select INTEL_CAR_NEM with any of these. Thank you for the
insights.

Best regards,

-- 
Michał Żygowski
Firmware Engineer
https://3mdeb.com | @3mdeb_com

[Michal Zygowski]

Hi Furquan, Tim,

On 20.01.2021 10:38, Michał Żygowski wrote:
> Hi Furquan,
> > Do you know the microcode version that you are using? There was an
> > issue with the older versions that caused a hang during NEM setup. As
> > per the commit message in
> > https://review.coreboot.org/c/coreboot/+/45094, it looks like you need
> > 0x56 or newer microcode for enhanced NEM to work.
> >
> > A quick way to verify if it is the microcode version issue: Select
> > INTEL_CAR_NEM instead of INTEL_CAR_NEM_ENHANCED for your board and see
> > if it gets any further in the boot.
>
> I have tried a few microcodes:
> - the one shipped originally with RVP platform is CpuSignature:
> 000806C0h, Revision: 00000064h, Date: 02.03.2020 so the revision is
> newer that 0x56.
> - also tried from TGL_BIOS_ww51_2020 BIOS images which has 4 blobs:
>     CpuSignature: 000806D1h, Revision: 00000008h, Date: 30.11.2020
>     CpuSignature: 000806D0h, Revision: 0000004Eh, Date: 01.12.2020
>     CpuSignature: 000806C1h, Revision: 00000072h, Date: 20.11.2020
>     CpuSignature: 000806C2h, Revision: 00000002h, Date: 04.12.2020
>
> It seems it have different steppings, so don't know if it really works.
>
> I will try to select INTEL_CAR_NEM with any of these. Thank you for the
> insights.
>
> Best regards,
> Selecting INTEL_CAR_NEM did the trick. However, I am facing an issue
> with FSP memory training now. TGL UP3 RVP has 2 DIMMs, but the memory
> FSP params are set for LPDDR4. Is it intentional?

Best regards,

-- 
Michał Żygowski
Firmware Engineer
https://3mdeb.com | @3mdeb_com

[Furquan Shaikh]

On Fri, Jan 29, 2021 at 5:08 AM Michal Zygowski
<[e-mail]> wrote:
> Hi Furquan, Tim,
>
> On 20.01.2021 10:38, Michał Żygowski wrote:
> > Hi Furquan,
> > > Do you know the microcode version that you are using? There was an
> > > issue with the older versions that caused a hang during NEM setup. As
> > > per the commit message in
> > > https://review.coreboot.org/c/coreboot/+/45094, it looks like you need
> > > 0x56 or newer microcode for enhanced NEM to work.
> > >
> > > A quick way to verify if it is the microcode version issue: Select
> > > INTEL_CAR_NEM instead of INTEL_CAR_NEM_ENHANCED for your board and see
> > > if it gets any further in the boot.
> >
> > I have tried a few microcodes:
> > - the one shipped originally with RVP platform is CpuSignature:
> > 000806C0h, Revision: 00000064h, Date: 02.03.2020 so the revision is
> > newer that 0x56.
> > - also tried from TGL_BIOS_ww51_2020 BIOS images which has 4 blobs:
> >     CpuSignature: 000806D1h, Revision: 00000008h, Date: 30.11.2020
> >     CpuSignature: 000806D0h, Revision: 0000004Eh, Date: 01.12.2020
> >     CpuSignature: 000806C1h, Revision: 00000072h, Date: 20.11.2020
> >     CpuSignature: 000806C2h, Revision: 00000002h, Date: 04.12.2020
> >
> > It seems it have different steppings, so don't know if it really works.
> >
> > I will try to select INTEL_CAR_NEM with any of these. Thank you for the
> > insights.
> >
> > Best regards,
> > Selecting INTEL_CAR_NEM did the trick. However, I am facing an issue
> > with FSP memory training now. TGL UP3 RVP has 2 DIMMs, but the memory
> > FSP params are set for LPDDR4. Is it intentional?

Looks like tglrvp board in coreboot assumes that:
1. Memory down is being used:
https://review.coreboot.org/cgit/coreboot.git/tree/src/mainboard/intel/tglrv...
2. LP4x is being used:
https://review.coreboot.org/cgit/coreboot.git/tree/src/mainboard/intel/tglrv...

If the TGL RVP variant you have is using a different memory technology
and topology, then you will have to provide relevant `spd_info`
structure and `variant_mem_params()` implementation. I believe there
are different RVP variants with different topologies. So, you will
have to add support for your own variant under
mainboard/intel/tglrvp/variants/tglrvp_up3_ddr4 or something similar.
> Best regards,
>
> _______________________________________________
> coreboot mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]