denverton: crash when using coreboot 4.17

[Sumo]

Hi,

My denverton system is crashing after the FSM memory init, probably when
jumping to POSTCAR. The following lines are shown before the crash:

[DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
[DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031. filesize:
0x6060 memsize: 0xc358
[DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
[DEBUG]  BS: romstage times (exec / console): total (unknown) / 1350 ms

Below are my CAR settings:
# CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
CONFIG_USE_DENVERTON_NS_FSP_CAR=y
CONFIG_ARCH_POSTCAR_X86_32=y
CONFIG_POSTCAR_STAGE=y
CONFIG_CARDBUS_PLUGIN_SUPPORT=y
CONFIG_FSP_CAR=y
CONFIG_POSTCAR_CONSOLE=y

Everything is fine when using coreboot v4.14.

I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but things got
even worse - in this case nothing is shown in the console output.

Any suggestions?

Thanks,
Sumo

[Paul Menzel]

Dear Suma,

Am 11.06.22 um 00:16 schrieb Sumo:
> My denverton system is crashing after the FSM memory init, probably when
> jumping to POSTCAR. The following lines are shown before the crash:
>
> [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031. filesize:
> 0x6060 memsize: 0xc358
> [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> [DEBUG]  BS: romstage times (exec / console): total (unknown) / 1350 ms
>
> Below are my CAR settings:
> # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> CONFIG_ARCH_POSTCAR_X86_32=y
> CONFIG_POSTCAR_STAGE=y
> CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> CONFIG_FSP_CAR=y
> CONFIG_POSTCAR_CONSOLE=y
>
> Everything is fine when using coreboot v4.14.
>
> I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but things got
> even worse - in this case nothing is shown in the console output.
>
> Any suggestions?

It’d be great if you could bisect.

Kind regards,

Paul

PS: In your address book please spell coreboot lowercase. ;-)

[Sumo]

Thanks Paul, I'll try bisect.

Do we have any instructions of how to use the normal/fallback coreboot
stage prefixes? During the bisect it will be painful always bricking the
board and having to use a flash programmer for restoring instead of using
flashrom.

Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then reuse a
prebuilt coreboot.rom with fallback stages included? Anything else needed
besides having a cmos layout?

Kind regards,
Sumo

On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel  wrote:
> Dear Suma,
>
> Am 11.06.22 um 00:16 schrieb Sumo:
> > My denverton system is crashing after the FSM memory init, probably when
> > jumping to POSTCAR. The following lines are shown before the crash:
> >
> > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031. filesize:
> > 0x6060 memsize: 0xc358
> > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > [DEBUG]  BS: romstage times (exec / console): total (unknown) / 1350 ms
> >
> > Below are my CAR settings:
> > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > CONFIG_ARCH_POSTCAR_X86_32=y
> > CONFIG_POSTCAR_STAGE=y
> > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > CONFIG_FSP_CAR=y
> > CONFIG_POSTCAR_CONSOLE=y
> >
> > Everything is fine when using coreboot v4.14.
> >
> > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but things
>
> got
> > even worse - in this case nothing is shown in the console output.
> >
> > Any suggestions?
>
> It’d be great if you could bisect.
>
> Kind regards,
>
> Paul
>
> PS: In your address book please spell coreboot lowercase. ;-)

[Sumo]

Hi,

Update: I found the bad commit:

commit 5315e96abfa5b45fcd53149df5ebaa069a830558
Author: Arthur Heymans 
Date:   Fri May 14 11:22:31 2021 +0200

arch/x86/postcar: Use a separate stack for C execution

Add a stack in .bss for C execution. This will make it easier to move
    the setup of MTRRs in C code.

Hehehe, this confirmed my suspicion that the problem was in the
POSTCAR...should have done a grep for postcar in git log instead of the
painful bisect :D

So denverton_ns is broken due to this commit, how can we proceed? I guess a
git revert is not an option right, since this change looks like a base for
future implementations... Please let me know if you need more tests.

Kind regards,
Sumo

On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> Thanks Paul, I'll try bisect.
>
> Do we have any instructions of how to use the normal/fallback coreboot
> stage prefixes? During the bisect it will be painful always bricking the
> board and having to use a flash programmer for restoring instead of using
> flashrom.
>
> Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then reuse a
> prebuilt coreboot.rom with fallback stages included? Anything else needed
> besides having a cmos layout?
>
> Kind regards,
> Sumo
>
> On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel  wrote:
> > Dear Suma,
> >
> > Am 11.06.22 um 00:16 schrieb Sumo:
> > > My denverton system is crashing after the FSM memory init, probably when
> > > jumping to POSTCAR. The following lines are shown before the crash:
> > >
> > > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031. filesize:
> > > 0x6060 memsize: 0xc358
> > > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > [DEBUG]  BS: romstage times (exec / console): total (unknown) / 1350 ms
> > >
> > > Below are my CAR settings:
> > > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > CONFIG_ARCH_POSTCAR_X86_32=y
> > > CONFIG_POSTCAR_STAGE=y
> > > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > CONFIG_FSP_CAR=y
> > > CONFIG_POSTCAR_CONSOLE=y
> > >
> > > Everything is fine when using coreboot v4.14.
> > >
> > > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but things
> >
> > got
> > > even worse - in this case nothing is shown in the console output.
> > >
> > > Any suggestions?
> >
> > It’d be great if you could bisect.
> >
> > Kind regards,
> >
> > Paul
> >
> > PS: In your address book please spell coreboot lowercase. ;-)

[Arthur Heymans]

Hi

Can you try selecting "NO_FSP_TEMP_RAM_EXIT" in the denverton_ns Kconfig?
That way coreboot uses native code to tear down CAR (so without FSP
TempRamExit)
INTEL_CAR_NEM_ENHANCED also needs to be select in the FSP_CAR option for
that to work (so it knows how to tear down CAR).

Kind regards
Arthur

On Mon, Jun 13, 2022 at 4:46 PM Sumo  wrote:
> Hi,
>
> Update: I found the bad commit:
>
> commit 5315e96abfa5b45fcd53149df5ebaa069a830558
> Author: Arthur Heymans 
> Date:   Fri May 14 11:22:31 2021 +0200
> arch/x86/postcar: Use a separate stack for C execution
>
> Add a stack in .bss for C execution. This will make it easier to move
> the setup of MTRRs in C code.
>
> Hehehe, this confirmed my suspicion that the problem was in the
> POSTCAR...should have done a grep for postcar in git log instead of the
> painful bisect :D
>
> So denverton_ns is broken due to this commit, how can we proceed? I guess
> a git revert is not an option right, since this change looks like a base
> for future implementations... Please let me know if you need more tests.
>
> Kind regards,
> Sumo
>
> On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> > Thanks Paul, I'll try bisect.
> >
> > Do we have any instructions of how to use the normal/fallback coreboot
> > stage prefixes? During the bisect it will be painful always bricking the
> > board and having to use a flash programmer for restoring instead of using
> > flashrom.
> >
> > Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then reuse a
> > prebuilt coreboot.rom with fallback stages included? Anything else needed
> > besides having a cmos layout?
> >
> > Kind regards,
> > Sumo
> >
> > On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel 
> > wrote:
> > > Dear Suma,
> > >
> > > Am 11.06.22 um 00:16 schrieb Sumo:
> > > > My denverton system is crashing after the FSM memory init, probably
> > >
> > > when
> > > > jumping to POSTCAR. The following lines are shown before the crash:
> > > >
> > > > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031. filesize:
> > > > 0x6060 memsize: 0xc358
> > > > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > > [DEBUG]  BS: romstage times (exec / console): total (unknown) / 1350 ms
> > > >
> > > > Below are my CAR settings:
> > > > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > > CONFIG_ARCH_POSTCAR_X86_32=y
> > > > CONFIG_POSTCAR_STAGE=y
> > > > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > > CONFIG_FSP_CAR=y
> > > > CONFIG_POSTCAR_CONSOLE=y
> > > >
> > > > Everything is fine when using coreboot v4.14.
> > > >
> > > > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but things
> > >
> > > got
> > > > even worse - in this case nothing is shown in the console output.
> > > >
> > > > Any suggestions?
> > >
> > > It’d be great if you could bisect.
> > >
> > > Kind regards,
> > >
> > > Paul
> > >
> > > PS: In your address book please spell coreboot lowercase. ;-)
>
> coreboot mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Sumo]

Hi Arthur,

Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes the
issue, which seems pretty odd since I haven't enabled
INTEL_CAR_NEM_ENHANCED. According to your explanation
INTEL_CAR_NEM_ENHANCED is required right?

But I have also tried adding INTEL_CAR_NEM_ENHANCED which worked as well.
However when selecting CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED the
makefile complained about the FSP_CAR dependency which I have then enabled
in Kconfig also.
(INTEL_CAR_NEM_ENHANCED is enabled only
when USE_DENVERTON_NS_CAR_NEM_ENHANCED is set)

diff --git a/src/soc/intel/denverton_ns/Kconfig
b/src/soc/intel/denverton_ns/Kconfig
index 92fc065a..cd5e13b8 100644
--- a/src/soc/intel/denverton_ns/Kconfig
+++ b/src/soc/intel/denverton_ns/Kconfig
@@ -20,6 +20,7 @@ config CPU_SPECIFIC_OPTIONS
        select CPU_INTEL_FIRMWARE_INTERFACE_TABLE
        select CPU_SUPPORTS_PM_TIMER_EMULATION
        select DEBUG_GPIO
+       select NO_FSP_TEMP_RAM_EXIT
        select FSP_M_XIP
        select FSP_T_XIP if FSP_CAR
        select HAVE_INTEL_FSP_REPO
@@ -163,6 +164,7 @@ config USE_DENVERTON_NS_CAR_NEM_ENHANCED
        bool "Enhanced Non-evict mode"
        select SOC_INTEL_COMMON_BLOCK_CAR
        select INTEL_CAR_NEM_ENHANCED
+       select FSP_CAR
        help
          A current limitation of NEM (Non-Evict mode) is that code and
data sizes
          are derived from the requirement to not write out any modified
cache line.

The log output of both tries are almost the same...
By configuring coreboot this way, the Temp RAM FSP is not used? So for the
coreboot latest Temp RAM FSP support is broken right?

Thanks,
Sumo

On Mon, Jun 13, 2022 at 1:38 PM Arthur Heymans  wrote:
> Hi
>
> Can you try selecting "NO_FSP_TEMP_RAM_EXIT" in the denverton_ns Kconfig?
> That way coreboot uses native code to tear down CAR (so without FSP
> TempRamExit)
> INTEL_CAR_NEM_ENHANCED also needs to be select in the FSP_CAR option for
> that to work (so it knows how to tear down CAR).
>
> Kind regards
> Arthur
>
> On Mon, Jun 13, 2022 at 4:46 PM Sumo  wrote:
> > Hi,
> >
> > Update: I found the bad commit:
> >
> > commit 5315e96abfa5b45fcd53149df5ebaa069a830558
> > Author: Arthur Heymans 
> > Date:   Fri May 14 11:22:31 2021 +0200
> > arch/x86/postcar: Use a separate stack for C execution
> >
> > Add a stack in .bss for C execution. This will make it easier to move
> > the setup of MTRRs in C code.
> >
> > Hehehe, this confirmed my suspicion that the problem was in the
> > POSTCAR...should have done a grep for postcar in git log instead of the
> > painful bisect :D
> >
> > So denverton_ns is broken due to this commit, how can we proceed? I guess
> > a git revert is not an option right, since this change looks like a base
> > for future implementations... Please let me know if you need more tests.
> >
> > Kind regards,
> > Sumo
> >
> > On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> > > Thanks Paul, I'll try bisect.
> > >
> > > Do we have any instructions of how to use the normal/fallback coreboot
> > > stage prefixes? During the bisect it will be painful always bricking the
> > > board and having to use a flash programmer for restoring instead of using
> > > flashrom.
> > >
> > > Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then reuse
> > > a prebuilt coreboot.rom with fallback stages included? Anything else needed
> > > besides having a cmos layout?
> > >
> > > Kind regards,
> > > Sumo
> > >
> > > On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel 
> > > wrote:
> > > > Dear Suma,
> > > >
> > > > Am 11.06.22 um 00:16 schrieb Sumo:
> > > > > My denverton system is crashing after the FSM memory init, probably
> > > >
> > > > when
> > > > > jumping to POSTCAR. The following lines are shown before the crash:
> > > > >
> > > > > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > > > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031. filesize:
> > > > > 0x6060 memsize: 0xc358
> > > > > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > > > [DEBUG]  BS: romstage times (exec / console): total (unknown) / 1350
> > > >
> > > > ms
> > > > > Below are my CAR settings:
> > > > > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > > > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > > > CONFIG_ARCH_POSTCAR_X86_32=y
> > > > > CONFIG_POSTCAR_STAGE=y
> > > > > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > > > CONFIG_FSP_CAR=y
> > > > > CONFIG_POSTCAR_CONSOLE=y
> > > > >
> > > > > Everything is fine when using coreboot v4.14.
> > > > >
> > > > > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but
> > > >
> > > > things got
> > > > > even worse - in this case nothing is shown in the console output.
> > > > >
> > > > > Any suggestions?
> > > >
> > > > It’d be great if you could bisect.
> > > >
> > > > Kind regards,
> > > >
> > > > Paul
> > > >
> > > > PS: In your address book please spell coreboot lowercase. ;-)
> >
> > coreboot mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]

[Arthur Heymans]

Hi

Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes the
> issue, which seems pretty odd since I haven't enabled
> INTEL_CAR_NEM_ENHANCED. According to your explanation
> INTEL_CAR_NEM_ENHANCED is required right?

That's right. So FSP-T still sets up the cache as ram but I have no idea
how it does that (it could be similar to INTEL_CAR_NEM or
INTEL_CAR_NEM_ENHANCED).
Selecting INTEL_CAR_NEM_ENHANCED in this case only makes sure that the
enhanced NEM msr are cleared.
Even if FSP-T does not use that, it's fine.

By configuring coreboot this way, the Temp RAM FSP is not used? So for the
> coreboot latest Temp RAM FSP support is broken right?

Actually the native coreboot CAR init is broken... I send a patch to remove
it:
https://review.coreboot.org/c/coreboot/+/55519

On Mon, Jun 13, 2022 at 9:55 PM Sumo  wrote:
> Hi Arthur,
>
> Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> the issue, which seems pretty odd since I haven't enabled
> INTEL_CAR_NEM_ENHANCED. According to your explanation
> INTEL_CAR_NEM_ENHANCED is required right?
>
> But I have also tried adding INTEL_CAR_NEM_ENHANCED which worked as well.
> However when selecting CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED the
> makefile complained about the FSP_CAR dependency which I have then enabled
> in Kconfig also.
> (INTEL_CAR_NEM_ENHANCED is enabled only
> when USE_DENVERTON_NS_CAR_NEM_ENHANCED is set)
>
> diff --git a/src/soc/intel/denverton_ns/Kconfig
> b/src/soc/intel/denverton_ns/Kconfig
> index 92fc065a..cd5e13b8 100644
> --- a/src/soc/intel/denverton_ns/Kconfig
> +++ b/src/soc/intel/denverton_ns/Kconfig
> @@ -20,6 +20,7 @@ config CPU_SPECIFIC_OPTIONS
>         select CPU_INTEL_FIRMWARE_INTERFACE_TABLE
>         select CPU_SUPPORTS_PM_TIMER_EMULATION
>         select DEBUG_GPIO
>   select NO_FSP_TEMP_RAM_EXIT
>   select FSP_M_XIP
>   select FSP_T_XIP if FSP_CAR
>   select HAVE_INTEL_FSP_REPO
>
> @@ -163,6 +164,7 @@ config USE_DENVERTON_NS_CAR_NEM_ENHANCED
>         bool "Enhanced Non-evict mode"
>         select SOC_INTEL_COMMON_BLOCK_CAR
>         select INTEL_CAR_NEM_ENHANCED
>   select FSP_CAR
>   help
>     A current limitation of NEM (Non-Evict mode) is that code and
>
> data sizes
>           are derived from the requirement to not write out any modified
> cache line.
>
> The log output of both tries are almost the same...
> By configuring coreboot this way, the Temp RAM FSP is not used? So for the
> coreboot latest Temp RAM FSP support is broken right?
>
> Thanks,
> Sumo
>
> On Mon, Jun 13, 2022 at 1:38 PM Arthur Heymans 
> wrote:
> > Hi
> >
> > Can you try selecting "NO_FSP_TEMP_RAM_EXIT" in the denverton_ns Kconfig?
> > That way coreboot uses native code to tear down CAR (so without FSP
> > TempRamExit)
> > INTEL_CAR_NEM_ENHANCED also needs to be select in the FSP_CAR option for
> > that to work (so it knows how to tear down CAR).
> >
> > Kind regards
> > Arthur
> >
> > On Mon, Jun 13, 2022 at 4:46 PM Sumo  wrote:
> > > Hi,
> > >
> > > Update: I found the bad commit:
> > >
> > > commit 5315e96abfa5b45fcd53149df5ebaa069a830558
> > > Author: Arthur Heymans 
> > > Date:   Fri May 14 11:22:31 2021 +0200
> > > arch/x86/postcar: Use a separate stack for C execution
> > >
> > > Add a stack in .bss for C execution. This will make it easier to move
> > > the setup of MTRRs in C code.
> > >
> > > Hehehe, this confirmed my suspicion that the problem was in the
> > > POSTCAR...should have done a grep for postcar in git log instead of the
> > > painful bisect :D
> > >
> > > So denverton_ns is broken due to this commit, how can we proceed? I
> > > guess a git revert is not an option right, since this change looks like a
> > > base for future implementations... Please let me know if you need more
> > > tests.
> > >
> > > Kind regards,
> > > Sumo
> > >
> > > On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> > > > Thanks Paul, I'll try bisect.
> > > >
> > > > Do we have any instructions of how to use the normal/fallback coreboot
> > > > stage prefixes? During the bisect it will be painful always bricking the
> > > > board and having to use a flash programmer for restoring instead of using
> > > > flashrom.
> > > >
> > > > Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then reuse
> > > > a prebuilt coreboot.rom with fallback stages included? Anything else needed
> > > > besides having a cmos layout?
> > > >
> > > > Kind regards,
> > > > Sumo
> > > >
> > > > On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel 
> > > > wrote:
> > > > > Dear Suma,
> > > > >
> > > > > Am 11.06.22 um 00:16 schrieb Sumo:
> > > > > > My denverton system is crashing after the FSM memory init, probably
> > > > >
> > > > > when
> > > > > > jumping to POSTCAR. The following lines are shown before the crash:
> > > > > >
> > > > > > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > > > > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031.
> > > > >
> > > > > filesize:
> > > > > > 0x6060 memsize: 0xc358
> > > > > > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > > > > [DEBUG]  BS: romstage times (exec / console): total (unknown) / 1350
> > > > >
> > > > > ms
> > > > > > Below are my CAR settings:
> > > > > > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > > > > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > > > > CONFIG_ARCH_POSTCAR_X86_32=y
> > > > > > CONFIG_POSTCAR_STAGE=y
> > > > > > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > > > > CONFIG_FSP_CAR=y
> > > > > > CONFIG_POSTCAR_CONSOLE=y
> > > > > >
> > > > > > Everything is fine when using coreboot v4.14.
> > > > > >
> > > > > > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but
> > > > >
> > > > > things got
> > > > > > even worse - in this case nothing is shown in the console output.
> > > > > >
> > > > > > Any suggestions?
> > > > >
> > > > > It’d be great if you could bisect.
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Paul
> > > > >
> > > > > PS: In your address book please spell coreboot lowercase. ;-)
> > >
> > > coreboot mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]

[Sumo]

Hi Arthur,

So basically Temp RAM Exit is broken, since we need to use
NO_FSP_TEMP_RAM_EXIT which selects
"./src/soc/intel/common/block/cpu/car/exit_car.S" which is doing nothing
besides disabling MTTRs (as all INTEL_CAR_* defines are unset) and then
returning. Therefore, we have no CAR teardown.

Without NO_FSP_TEMP_RAM_EXIT, we are using
"./src/soc/intel/common/block/cpu/car/exit_car_fsp.S" which it is crashing
without reaching the main() function to invoke the TempRamExit API. This
was working before the commit #5315e96abf - we have the following changes
in exit_cat_fsp.S:

diff --git a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
index 4b906280e6..4d35447a56 100644
--- a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
+++ b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
@@ -10,16 +10,16 @@
  * to tear down the CAR and set up caching which can be overwritten
  * after the API call.  More info can be found in the FSP Integration
  * Guide included with the FSP binary.
  */

.text
 .global chipset_teardown_car
 chipset_teardown_car:

/* Set up new stack. */
-       mov     post_car_stack_top, %esp
+       mov     _estack, %esp
        /* Align the stack. */
        andl    $0xfffffff0, %esp

/* Call C code */
        call    main

The question is why we can't reach main() anymore. Any clues?

Kind regards,
Sumo

On Tue, Jun 14, 2022 at 6:27 AM Arthur Heymans  wrote:
> Hi
>
> Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> > the issue, which seems pretty odd since I haven't enabled
> > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > INTEL_CAR_NEM_ENHANCED is required right?
>
> That's right. So FSP-T still sets up the cache as ram but I have no idea
> how it does that (it could be similar to INTEL_CAR_NEM or
> INTEL_CAR_NEM_ENHANCED).
> Selecting INTEL_CAR_NEM_ENHANCED in this case only makes sure that the
> enhanced NEM msr are cleared.
> Even if FSP-T does not use that, it's fine.
>
> By configuring coreboot this way, the Temp RAM FSP is not used? So for the
> > coreboot latest Temp RAM FSP support is broken right?
>
> Actually the native coreboot CAR init is broken... I send a patch to
> remove it:
> https://review.coreboot.org/c/coreboot/+/55519
>
> On Mon, Jun 13, 2022 at 9:55 PM Sumo  wrote:
> > Hi Arthur,
> >
> > Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> > the issue, which seems pretty odd since I haven't enabled
> > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > INTEL_CAR_NEM_ENHANCED is required right?
> >
> > But I have also tried adding INTEL_CAR_NEM_ENHANCED which worked as well.
> > However when selecting CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED the
> > makefile complained about the FSP_CAR dependency which I have then enabled
> > in Kconfig also.
> > (INTEL_CAR_NEM_ENHANCED is enabled only
> > when USE_DENVERTON_NS_CAR_NEM_ENHANCED is set)
> >
> > diff --git a/src/soc/intel/denverton_ns/Kconfig
> > b/src/soc/intel/denverton_ns/Kconfig
> > index 92fc065a..cd5e13b8 100644
> > --- a/src/soc/intel/denverton_ns/Kconfig
> > +++ b/src/soc/intel/denverton_ns/Kconfig
> > @@ -20,6 +20,7 @@ config CPU_SPECIFIC_OPTIONS
> >         select CPU_INTEL_FIRMWARE_INTERFACE_TABLE
> >         select CPU_SUPPORTS_PM_TIMER_EMULATION
> >         select DEBUG_GPIO
> >   select NO_FSP_TEMP_RAM_EXIT
> >   select FSP_M_XIP
> >   select FSP_T_XIP if FSP_CAR
> >   select HAVE_INTEL_FSP_REPO
> >
> > @@ -163,6 +164,7 @@ config USE_DENVERTON_NS_CAR_NEM_ENHANCED
> >         bool "Enhanced Non-evict mode"
> >         select SOC_INTEL_COMMON_BLOCK_CAR
> >         select INTEL_CAR_NEM_ENHANCED
> >   select FSP_CAR
> >   help
> >     A current limitation of NEM (Non-Evict mode) is that code and
> >
> > data sizes
> >           are derived from the requirement to not write out any modified
> > cache line.
> >
> > The log output of both tries are almost the same...
> > By configuring coreboot this way, the Temp RAM FSP is not used? So for
> > the coreboot latest Temp RAM FSP support is broken right?
> >
> > Thanks,
> > Sumo
> >
> > On Mon, Jun 13, 2022 at 1:38 PM Arthur Heymans 
> > wrote:
> > > Hi
> > >
> > > Can you try selecting "NO_FSP_TEMP_RAM_EXIT" in the denverton_ns Kconfig?
> > > That way coreboot uses native code to tear down CAR (so without FSP
> > > TempRamExit)
> > > INTEL_CAR_NEM_ENHANCED also needs to be select in the FSP_CAR option for
> > > that to work (so it knows how to tear down CAR).
> > >
> > > Kind regards
> > > Arthur
> > >
> > > On Mon, Jun 13, 2022 at 4:46 PM Sumo  wrote:
> > > > Hi,
> > > >
> > > > Update: I found the bad commit:
> > > >
> > > > commit 5315e96abfa5b45fcd53149df5ebaa069a830558
> > > > Author: Arthur Heymans 
> > > > Date:   Fri May 14 11:22:31 2021 +0200
> > > > arch/x86/postcar: Use a separate stack for C execution
> > > >
> > > > Add a stack in .bss for C execution. This will make it easier to
> > > >
> > > > move
> > > >     the setup of MTRRs in C code.
> > > >
> > > > Hehehe, this confirmed my suspicion that the problem was in the
> > > > POSTCAR...should have done a grep for postcar in git log instead of the
> > > > painful bisect :D
> > > >
> > > > So denverton_ns is broken due to this commit, how can we proceed? I
> > > > guess a git revert is not an option right, since this change looks like a
> > > > base for future implementations... Please let me know if you need more
> > > > tests.
> > > >
> > > > Kind regards,
> > > > Sumo
> > > >
> > > > On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> > > > > Thanks Paul, I'll try bisect.
> > > > >
> > > > > Do we have any instructions of how to use the normal/fallback coreboot
> > > > > stage prefixes? During the bisect it will be painful always bricking the
> > > > > board and having to use a flash programmer for restoring instead of using
> > > > > flashrom.
> > > > >
> > > > > Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then
> > > > > reuse a prebuilt coreboot.rom with fallback stages included? Anything else
> > > > > needed besides having a cmos layout?
> > > > >
> > > > > Kind regards,
> > > > > Sumo
> > > > >
> > > > > On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel 
> > > > > wrote:
> > > > > > Dear Suma,
> > > > > >
> > > > > > Am 11.06.22 um 00:16 schrieb Sumo:
> > > > > >
> > > > > > > My denverton system is crashing after the FSM memory init, probably
> > > > > > when
> > > > > > > jumping to POSTCAR. The following lines are shown before the crash:
> > > > > > >
> > > > > > > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > > > > > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031.
> > > > > > filesize:
> > > > > > > 0x6060 memsize: 0xc358
> > > > > > > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > > > > > [DEBUG]  BS: romstage times (exec / console): total (unknown) /
> > > > > > 1350 ms
> > > > > > >
> > > > > > > Below are my CAR settings:
> > > > > > > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > > > > > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > > > > > CONFIG_ARCH_POSTCAR_X86_32=y
> > > > > > > CONFIG_POSTCAR_STAGE=y
> > > > > > > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > > > > > CONFIG_FSP_CAR=y
> > > > > > > CONFIG_POSTCAR_CONSOLE=y
> > > > > > >
> > > > > > > Everything is fine when using coreboot v4.14.
> > > > > > >
> > > > > > > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but
> > > > > > things got
> > > > > > > even worse - in this case nothing is shown in the console output.
> > > > > > >
> > > > > > > Any suggestions?
> > > > > >
> > > > > > It’d be great if you could bisect.
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > > Paul
> > > > > >
> > > > > > PS: In your address book please spell coreboot lowercase. ;-)
> > > >
> > > > coreboot mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]

[Arthur Heymans]

> So basically Temp RAM Exit is broken, since we need to use
> NO_FSP_TEMP_RAM_EXIT which selects
> "./src/soc/intel/common/block/cpu/car/exit_car.S" which is doing nothing
> besides disabling MTTRs (as all INTEL_CAR_* defines are unset) and then
> returning. Therefore, we have no CAR teardown.

That is why I'm recommending to select INTEL_CAR_NEM_ENHANCED as it will do
the right CAR teardown.

I doubt that it's not reaching the main call but rather that something bad
happens when calling the TempRamExit API...
Is it worth investigating if we can make the native coreboot CAR teardown
work?

Arthur

On Tue, Jun 14, 2022 at 8:11 PM Sumo  wrote:
> Hi Arthur,
>
> So basically Temp RAM Exit is broken, since we need to use
> NO_FSP_TEMP_RAM_EXIT which selects
> "./src/soc/intel/common/block/cpu/car/exit_car.S" which is doing nothing
> besides disabling MTTRs (as all INTEL_CAR_* defines are unset) and then
> returning. Therefore, we have no CAR teardown.
>
> Without NO_FSP_TEMP_RAM_EXIT, we are using
> "./src/soc/intel/common/block/cpu/car/exit_car_fsp.S" which it is crashing
> without reaching the main() function to invoke the TempRamExit API. This
> was working before the commit #5315e96abf - we have the following changes
> in exit_cat_fsp.S:
>
> diff --git a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> index 4b906280e6..4d35447a56 100644
> --- a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> +++ b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> @@ -10,16 +10,16 @@
> to tear down the CAR and set up caching which can be overwritten
> after the API call.  More info can be found in the FSP Integration
> Guide included with the FSP binary.
>
> */
>
> .text
>  .global chipset_teardown_car
>  chipset_teardown_car:
>     /* Set up new stack. */
>   mov     post_car_stack_top, %esp
>   mov     _estack, %esp
>   /* Align the stack. */
>   andl    $0xfffffff0, %esp
>
>   /* Call C code */
>   call    main
>
> The question is why we can't reach main() anymore. Any clues?
>
> Kind regards,
> Sumo
>
> On Tue, Jun 14, 2022 at 6:27 AM Arthur Heymans 
> wrote:
> > Hi
> >
> > Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> > > the issue, which seems pretty odd since I haven't enabled
> > > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > > INTEL_CAR_NEM_ENHANCED is required right?
> >
> > That's right. So FSP-T still sets up the cache as ram but I have no idea
> > how it does that (it could be similar to INTEL_CAR_NEM or
> > INTEL_CAR_NEM_ENHANCED).
> > Selecting INTEL_CAR_NEM_ENHANCED in this case only makes sure that the
> > enhanced NEM msr are cleared.
> > Even if FSP-T does not use that, it's fine.
> >
> > By configuring coreboot this way, the Temp RAM FSP is not used? So for
> > > the coreboot latest Temp RAM FSP support is broken right?
> >
> > Actually the native coreboot CAR init is broken... I send a patch to
> > remove it:
> > https://review.coreboot.org/c/coreboot/+/55519
> >
> > On Mon, Jun 13, 2022 at 9:55 PM Sumo  wrote:
> > > Hi Arthur,
> > >
> > > Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> > > the issue, which seems pretty odd since I haven't enabled
> > > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > > INTEL_CAR_NEM_ENHANCED is required right?
> > >
> > > But I have also tried adding INTEL_CAR_NEM_ENHANCED which worked as
> > > well. However when selecting CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED the
> > > makefile complained about the FSP_CAR dependency which I have then enabled
> > > in Kconfig also.
> > > (INTEL_CAR_NEM_ENHANCED is enabled only
> > > when USE_DENVERTON_NS_CAR_NEM_ENHANCED is set)
> > >
> > > diff --git a/src/soc/intel/denverton_ns/Kconfig
> > > b/src/soc/intel/denverton_ns/Kconfig
> > > index 92fc065a..cd5e13b8 100644
> > > --- a/src/soc/intel/denverton_ns/Kconfig
> > > +++ b/src/soc/intel/denverton_ns/Kconfig
> > > @@ -20,6 +20,7 @@ config CPU_SPECIFIC_OPTIONS
> > >         select CPU_INTEL_FIRMWARE_INTERFACE_TABLE
> > >         select CPU_SUPPORTS_PM_TIMER_EMULATION
> > >         select DEBUG_GPIO
> > >   select NO_FSP_TEMP_RAM_EXIT
> > >   select FSP_M_XIP
> > >   select FSP_T_XIP if FSP_CAR
> > >   select HAVE_INTEL_FSP_REPO
> > >
> > > @@ -163,6 +164,7 @@ config USE_DENVERTON_NS_CAR_NEM_ENHANCED
> > >         bool "Enhanced Non-evict mode"
> > >         select SOC_INTEL_COMMON_BLOCK_CAR
> > >         select INTEL_CAR_NEM_ENHANCED
> > >   select FSP_CAR
> > >   help
> > >     A current limitation of NEM (Non-Evict mode) is that code and
> > >
> > > data sizes
> > >           are derived from the requirement to not write out any modified
> > > cache line.
> > >
> > > The log output of both tries are almost the same...
> > > By configuring coreboot this way, the Temp RAM FSP is not used? So for
> > > the coreboot latest Temp RAM FSP support is broken right?
> > >
> > > Thanks,
> > > Sumo
> > >
> > > On Mon, Jun 13, 2022 at 1:38 PM Arthur Heymans 
> > > wrote:
> > > > Hi
> > > >
> > > > Can you try selecting "NO_FSP_TEMP_RAM_EXIT" in the denverton_ns
> > > > Kconfig?
> > > > That way coreboot uses native code to tear down CAR (so without FSP
> > > > TempRamExit)
> > > > INTEL_CAR_NEM_ENHANCED also needs to be select in the FSP_CAR option
> > > > for that to work (so it knows how to tear down CAR).
> > > >
> > > > Kind regards
> > > > Arthur
> > > >
> > > > On Mon, Jun 13, 2022 at 4:46 PM Sumo  wrote:
> > > > > Hi,
> > > > >
> > > > > Update: I found the bad commit:
> > > > >
> > > > > commit 5315e96abfa5b45fcd53149df5ebaa069a830558
> > > > > Author: Arthur Heymans 
> > > > > Date:   Fri May 14 11:22:31 2021 +0200
> > > > > arch/x86/postcar: Use a separate stack for C execution
> > > > >
> > > > > Add a stack in .bss for C execution. This will make it easier to
> > > > >
> > > > > move
> > > > >     the setup of MTRRs in C code.
> > > > >
> > > > > Hehehe, this confirmed my suspicion that the problem was in the
> > > > > POSTCAR...should have done a grep for postcar in git log instead of the
> > > > > painful bisect :D
> > > > >
> > > > > So denverton_ns is broken due to this commit, how can we proceed? I
> > > > > guess a git revert is not an option right, since this change looks like a
> > > > > base for future implementations... Please let me know if you need more
> > > > > tests.
> > > > >
> > > > > Kind regards,
> > > > > Sumo
> > > > >
> > > > > On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> > > > > > Thanks Paul, I'll try bisect.
> > > > > >
> > > > > > Do we have any instructions of how to use the normal/fallback
> > > > > > coreboot stage prefixes? During the bisect it will be painful always
> > > > > > bricking the board and having to use a flash programmer for restoring
> > > > > > instead of using flashrom.
> > > > > >
> > > > > > Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then
> > > > > > reuse a prebuilt coreboot.rom with fallback stages included? Anything else
> > > > > > needed besides having a cmos layout?
> > > > > >
> > > > > > Kind regards,
> > > > > > Sumo
> > > > > >
> > > > > > On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel 
> > > > > > wrote:
> > > > > >
> > > > > > > Dear Suma,
> > > > > > >
> > > > > > >
> > > > > > > Am 11.06.22 um 00:16 schrieb Sumo:
> > > > > > >
> > > > > > > > My denverton system is crashing after the FSM memory init,
> > > > > > > probably when
> > > > > > > > jumping to POSTCAR. The following lines are shown before the crash:
> > > > > > > >
> > > > > > > > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > > > > > > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031.
> > > > > > > filesize:
> > > > > > > > 0x6060 memsize: 0xc358
> > > > > > > > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > > > > > > [DEBUG]  BS: romstage times (exec / console): total (unknown) /
> > > > > > > 1350 ms
> > > > > > > >
> > > > > > > > Below are my CAR settings:
> > > > > > > > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > > > > > > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > > > > > > CONFIG_ARCH_POSTCAR_X86_32=y
> > > > > > > > CONFIG_POSTCAR_STAGE=y
> > > > > > > > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > > > > > > CONFIG_FSP_CAR=y
> > > > > > > > CONFIG_POSTCAR_CONSOLE=y
> > > > > > > >
> > > > > > > > Everything is fine when using coreboot v4.14.
> > > > > > > >
> > > > > > > > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but
> > > > > > > things got
> > > > > > > > even worse - in this case nothing is shown in the console output.
> > > > > > > >
> > > > > > > > Any suggestions?
> > > > > > >
> > > > > > > It’d be great if you could bisect.
> > > > > > >
> > > > > > >
> > > > > > > Kind regards,
> > > > > > >
> > > > > > > Paul
> > > > > > >
> > > > > > >
> > > > > > > PS: In your address book please spell coreboot lowercase. ;-)
> > > > > > >
> > > > > > _______________________________________________
> > > > >
> > > > > coreboot mailing list -- [e-mail]
> > > > > To unsubscribe send an email to [e-mail]

[Sumo]

Hi Arthur,

After adding a '$' sign now we are reaching the main() and then the
TempRamExit is called.

diff --git a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
index 4d35447a56..37644c3e5f 100644
--- a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
+++ b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
@@ -17,7 +17,7 @@
 chipset_teardown_car:

/* Set up new stack. */
-       mov     _estack, %esp
+       mov     $_estack, %esp
        /* Align the stack. */
        andl    $0xfffffff0, %esp

Notice in src/arch/x86/exit_car.S we have a $ sign...
The fsp1_1/exit_car.S module must be fixed as well

Thanks,
Sumo

On Tue, Jun 14, 2022 at 3:25 PM Arthur Heymans  wrote:
> So basically Temp RAM Exit is broken, since we need to use
> > NO_FSP_TEMP_RAM_EXIT which selects
> > "./src/soc/intel/common/block/cpu/car/exit_car.S" which is doing nothing
> > besides disabling MTTRs (as all INTEL_CAR_* defines are unset) and then
> > returning. Therefore, we have no CAR teardown.
>
> That is why I'm recommending to select INTEL_CAR_NEM_ENHANCED as it will
> do the right CAR teardown.
>
> I doubt that it's not reaching the main call but rather that something bad
> happens when calling the TempRamExit API...
> Is it worth investigating if we can make the native coreboot CAR teardown
> work?
>
> Arthur
>
> On Tue, Jun 14, 2022 at 8:11 PM Sumo  wrote:
> > Hi Arthur,
> >
> > So basically Temp RAM Exit is broken, since we need to use
> > NO_FSP_TEMP_RAM_EXIT which selects
> > "./src/soc/intel/common/block/cpu/car/exit_car.S" which is doing nothing
> > besides disabling MTTRs (as all INTEL_CAR_* defines are unset) and then
> > returning. Therefore, we have no CAR teardown.
> >
> > Without NO_FSP_TEMP_RAM_EXIT, we are using
> > "./src/soc/intel/common/block/cpu/car/exit_car_fsp.S" which it is crashing
> > without reaching the main() function to invoke the TempRamExit API. This
> > was working before the commit #5315e96abf - we have the following changes
> > in exit_cat_fsp.S:
> >
> > diff --git a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > index 4b906280e6..4d35447a56 100644
> > --- a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > +++ b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > @@ -10,16 +10,16 @@
> > to tear down the CAR and set up caching which can be overwritten
> > after the API call.  More info can be found in the FSP Integration
> > Guide included with the FSP binary.
> >
> > */
> >
> > .text
> >  .global chipset_teardown_car
> >  chipset_teardown_car:
> >     /* Set up new stack. */
> >   mov     post_car_stack_top, %esp
> >   mov     _estack, %esp
> >   /* Align the stack. */
> >   andl    $0xfffffff0, %esp
> >
> >   /* Call C code */
> >   call    main
> >
> > The question is why we can't reach main() anymore. Any clues?
> >
> > Kind regards,
> > Sumo
> >
> > On Tue, Jun 14, 2022 at 6:27 AM Arthur Heymans 
> > wrote:
> > > Hi
> > >
> > > Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> > > > the issue, which seems pretty odd since I haven't enabled
> > > > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > > > INTEL_CAR_NEM_ENHANCED is required right?
> > >
> > > That's right. So FSP-T still sets up the cache as ram but I have no idea
> > > how it does that (it could be similar to INTEL_CAR_NEM or
> > > INTEL_CAR_NEM_ENHANCED).
> > > Selecting INTEL_CAR_NEM_ENHANCED in this case only makes sure that the
> > > enhanced NEM msr are cleared.
> > > Even if FSP-T does not use that, it's fine.
> > >
> > > By configuring coreboot this way, the Temp RAM FSP is not used? So for
> > > > the coreboot latest Temp RAM FSP support is broken right?
> > >
> > > Actually the native coreboot CAR init is broken... I send a patch to
> > > remove it:
> > > https://review.coreboot.org/c/coreboot/+/55519
> > >
> > > On Mon, Jun 13, 2022 at 9:55 PM Sumo  wrote:
> > > > Hi Arthur,
> > > >
> > > > Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> > > > the issue, which seems pretty odd since I haven't enabled
> > > > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > > > INTEL_CAR_NEM_ENHANCED is required right?
> > > >
> > > > But I have also tried adding INTEL_CAR_NEM_ENHANCED which worked as
> > > > well. However when selecting CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED the
> > > > makefile complained about the FSP_CAR dependency which I have then enabled
> > > > in Kconfig also.
> > > > (INTEL_CAR_NEM_ENHANCED is enabled only
> > > > when USE_DENVERTON_NS_CAR_NEM_ENHANCED is set)
> > > >
> > > > diff --git a/src/soc/intel/denverton_ns/Kconfig
> > > > b/src/soc/intel/denverton_ns/Kconfig
> > > > index 92fc065a..cd5e13b8 100644
> > > > --- a/src/soc/intel/denverton_ns/Kconfig
> > > > +++ b/src/soc/intel/denverton_ns/Kconfig
> > > > @@ -20,6 +20,7 @@ config CPU_SPECIFIC_OPTIONS
> > > >         select CPU_INTEL_FIRMWARE_INTERFACE_TABLE
> > > >         select CPU_SUPPORTS_PM_TIMER_EMULATION
> > > >         select DEBUG_GPIO
> > > >   select NO_FSP_TEMP_RAM_EXIT
> > > >   select FSP_M_XIP
> > > >   select FSP_T_XIP if FSP_CAR
> > > >   select HAVE_INTEL_FSP_REPO
> > > >
> > > > @@ -163,6 +164,7 @@ config USE_DENVERTON_NS_CAR_NEM_ENHANCED
> > > >         bool "Enhanced Non-evict mode"
> > > >         select SOC_INTEL_COMMON_BLOCK_CAR
> > > >         select INTEL_CAR_NEM_ENHANCED
> > > >   select FSP_CAR
> > > >   help
> > > >     A current limitation of NEM (Non-Evict mode) is that code and
> > > >
> > > > data sizes
> > > >           are derived from the requirement to not write out any
> > > > modified cache line.
> > > >
> > > > The log output of both tries are almost the same...
> > > > By configuring coreboot this way, the Temp RAM FSP is not used? So for
> > > > the coreboot latest Temp RAM FSP support is broken right?
> > > >
> > > > Thanks,
> > > > Sumo
> > > >
> > > > On Mon, Jun 13, 2022 at 1:38 PM Arthur Heymans 
> > > > wrote:
> > > > > Hi
> > > > >
> > > > > Can you try selecting "NO_FSP_TEMP_RAM_EXIT" in the denverton_ns
> > > > > Kconfig?
> > > > > That way coreboot uses native code to tear down CAR (so without FSP
> > > > > TempRamExit)
> > > > > INTEL_CAR_NEM_ENHANCED also needs to be select in the FSP_CAR option
> > > > > for that to work (so it knows how to tear down CAR).
> > > > >
> > > > > Kind regards
> > > > > Arthur
> > > > >
> > > > > On Mon, Jun 13, 2022 at 4:46 PM Sumo  wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Update: I found the bad commit:
> > > > > >
> > > > > > commit 5315e96abfa5b45fcd53149df5ebaa069a830558
> > > > > > Author: Arthur Heymans 
> > > > > > Date:   Fri May 14 11:22:31 2021 +0200
> > > > > > arch/x86/postcar: Use a separate stack for C execution
> > > > > >
> > > > > > Add a stack in .bss for C execution. This will make it easier to
> > > > > >
> > > > > > move
> > > > > >     the setup of MTRRs in C code.
> > > > > >
> > > > > > Hehehe, this confirmed my suspicion that the problem was in the
> > > > > > POSTCAR...should have done a grep for postcar in git log instead of the
> > > > > > painful bisect :D
> > > > > >
> > > > > > So denverton_ns is broken due to this commit, how can we proceed? I
> > > > > > guess a git revert is not an option right, since this change looks like a
> > > > > > base for future implementations... Please let me know if you need more
> > > > > > tests.
> > > > > >
> > > > > > Kind regards,
> > > > > > Sumo
> > > > > >
> > > > > > On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> > > > > >
> > > > > > > Thanks Paul, I'll try bisect.
> > > > > > >
> > > > > > > Do we have any instructions of how to use the normal/fallback
> > > > > > > coreboot stage prefixes? During the bisect it will be painful always
> > > > > > > bricking the board and having to use a flash programmer for restoring
> > > > > > > instead of using flashrom.
> > > > > > >
> > > > > > > Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then
> > > > > > > reuse a prebuilt coreboot.rom with fallback stages included? Anything else
> > > > > > > needed besides having a cmos layout?
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > Sumo
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel 
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Dear Suma,
> > > > > > >>
> > > > > > >>
> > > > > > >> Am 11.06.22 um 00:16 schrieb Sumo:
> > > > > > >>
> > > > > > >> > My denverton system is crashing after the FSM memory init,
> > > > > > >> probably when
> > > > > > >> > jumping to POSTCAR. The following lines are shown before the
> > > > > > >> crash:
> > > > > > >> >
> > > > > > >> > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > > > > >> > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031.
> > > > > > >> filesize:
> > > > > > >> > 0x6060 memsize: 0xc358
> > > > > > >> > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > > > > >> > [DEBUG]  BS: romstage times (exec / console): total (unknown) /
> > > > > > >> 1350 ms
> > > > > > >> >
> > > > > > >> > Below are my CAR settings:
> > > > > > >> > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > > > > >> > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > > > > >> > CONFIG_ARCH_POSTCAR_X86_32=y
> > > > > > >> > CONFIG_POSTCAR_STAGE=y
> > > > > > >> > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > > > > >> > CONFIG_FSP_CAR=y
> > > > > > >> > CONFIG_POSTCAR_CONSOLE=y
> > > > > > >> >
> > > > > > >> > Everything is fine when using coreboot v4.14.
> > > > > > >> >
> > > > > > >> > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but
> > > > > > >> things got
> > > > > > >> > even worse - in this case nothing is shown in the console output.
> > > > > > >> >
> > > > > > >> > Any suggestions?
> > > > > > >>
> > > > > > >> It’d be great if you could bisect.
> > > > > > >>
> > > > > > >>
> > > > > > >> Kind regards,
> > > > > > >>
> > > > > > >> Paul
> > > > > > >>
> > > > > > >>
> > > > > > >> PS: In your address book please spell coreboot lowercase. ;-)
> > > > > > >>
> > > > > > > _______________________________________________
> > > > > > coreboot mailing list -- [e-mail]
> > > > > > To unsubscribe send an email to [e-mail]

[Arthur Heymans]

Hi

Nice find!

Kind regards
Arthur

On Tue, 14 Jun 2022, 21:17 Sumo,  wrote:
> Hi Arthur,
>
> After adding a '$' sign now we are reaching the main() and then the
> TempRamExit is called.
>
> diff --git a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> index 4d35447a56..37644c3e5f 100644
> --- a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> +++ b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> @@ -17,7 +17,7 @@
>  chipset_teardown_car:
>     /* Set up new stack. */
>   mov     _estack, %esp
>   mov     $_estack, %esp
>   /* Align the stack. */
>   andl    $0xfffffff0, %esp
>
> Notice in src/arch/x86/exit_car.S we have a $ sign...
> The fsp1_1/exit_car.S module must be fixed as well
>
> Thanks,
> Sumo
>
> On Tue, Jun 14, 2022 at 3:25 PM Arthur Heymans 
> wrote:
> > So basically Temp RAM Exit is broken, since we need to use
> > > NO_FSP_TEMP_RAM_EXIT which selects
> > > "./src/soc/intel/common/block/cpu/car/exit_car.S" which is doing nothing
> > > besides disabling MTTRs (as all INTEL_CAR_* defines are unset) and then
> > > returning. Therefore, we have no CAR teardown.
> >
> > That is why I'm recommending to select INTEL_CAR_NEM_ENHANCED as it will
> > do the right CAR teardown.
> >
> > I doubt that it's not reaching the main call but rather that something
> > bad happens when calling the TempRamExit API...
> > Is it worth investigating if we can make the native coreboot CAR teardown
> > work?
> >
> > Arthur
> >
> > On Tue, Jun 14, 2022 at 8:11 PM Sumo  wrote:
> > > Hi Arthur,
> > >
> > > So basically Temp RAM Exit is broken, since we need to use
> > > NO_FSP_TEMP_RAM_EXIT which selects
> > > "./src/soc/intel/common/block/cpu/car/exit_car.S" which is doing nothing
> > > besides disabling MTTRs (as all INTEL_CAR_* defines are unset) and then
> > > returning. Therefore, we have no CAR teardown.
> > >
> > > Without NO_FSP_TEMP_RAM_EXIT, we are using
> > > "./src/soc/intel/common/block/cpu/car/exit_car_fsp.S" which it is crashing
> > > without reaching the main() function to invoke the TempRamExit API. This
> > > was working before the commit #5315e96abf - we have the following changes
> > > in exit_cat_fsp.S:
> > >
> > > diff --git a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > > b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > > index 4b906280e6..4d35447a56 100644
> > > --- a/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > > +++ b/src/soc/intel/common/block/cpu/car/exit_car_fsp.S
> > > @@ -10,16 +10,16 @@
> > > to tear down the CAR and set up caching which can be overwritten
> > > after the API call.  More info can be found in the FSP Integration
> > > Guide included with the FSP binary.
> > >
> > > */
> > >
> > > .text
> > >  .global chipset_teardown_car
> > >  chipset_teardown_car:
> > >     /* Set up new stack. */
> > >   mov     post_car_stack_top, %esp
> > >   mov     _estack, %esp
> > >   /* Align the stack. */
> > >   andl    $0xfffffff0, %esp
> > >
> > >   /* Call C code */
> > >   call    main
> > >
> > > The question is why we can't reach main() anymore. Any clues?
> > >
> > > Kind regards,
> > > Sumo
> > >
> > > On Tue, Jun 14, 2022 at 6:27 AM Arthur Heymans 
> > > wrote:
> > > > Hi
> > > >
> > > > Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig fixes
> > > > > the issue, which seems pretty odd since I haven't enabled
> > > > > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > > > > INTEL_CAR_NEM_ENHANCED is required right?
> > > >
> > > > That's right. So FSP-T still sets up the cache as ram but I have no
> > > > idea how it does that (it could be similar to INTEL_CAR_NEM or
> > > > INTEL_CAR_NEM_ENHANCED).
> > > > Selecting INTEL_CAR_NEM_ENHANCED in this case only makes sure that the
> > > > enhanced NEM msr are cleared.
> > > > Even if FSP-T does not use that, it's fine.
> > > >
> > > > By configuring coreboot this way, the Temp RAM FSP is not used? So for
> > > > > the coreboot latest Temp RAM FSP support is broken right?
> > > >
> > > > Actually the native coreboot CAR init is broken... I send a patch to
> > > > remove it:
> > > > https://review.coreboot.org/c/coreboot/+/55519
> > > >
> > > > On Mon, Jun 13, 2022 at 9:55 PM Sumo  wrote:
> > > > > Hi Arthur,
> > > > >
> > > > > Adding NO_FSP_TEMP_RAM_EXIT in src/soc/intel/denverton_ns/Kconfig
> > > > > fixes the issue, which seems pretty odd since I haven't enabled
> > > > > INTEL_CAR_NEM_ENHANCED. According to your explanation
> > > > > INTEL_CAR_NEM_ENHANCED is required right?
> > > > >
> > > > > But I have also tried adding INTEL_CAR_NEM_ENHANCED which worked as
> > > > > well. However when selecting CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED the
> > > > > makefile complained about the FSP_CAR dependency which I have then enabled
> > > > > in Kconfig also.
> > > > > (INTEL_CAR_NEM_ENHANCED is enabled only
> > > > > when USE_DENVERTON_NS_CAR_NEM_ENHANCED is set)
> > > > >
> > > > > diff --git a/src/soc/intel/denverton_ns/Kconfig
> > > > > b/src/soc/intel/denverton_ns/Kconfig
> > > > > index 92fc065a..cd5e13b8 100644
> > > > > --- a/src/soc/intel/denverton_ns/Kconfig
> > > > > +++ b/src/soc/intel/denverton_ns/Kconfig
> > > > > @@ -20,6 +20,7 @@ config CPU_SPECIFIC_OPTIONS
> > > > >         select CPU_INTEL_FIRMWARE_INTERFACE_TABLE
> > > > >         select CPU_SUPPORTS_PM_TIMER_EMULATION
> > > > >         select DEBUG_GPIO
> > > > >   select NO_FSP_TEMP_RAM_EXIT
> > > > >   select FSP_M_XIP
> > > > >   select FSP_T_XIP if FSP_CAR
> > > > >   select HAVE_INTEL_FSP_REPO
> > > > >
> > > > > @@ -163,6 +164,7 @@ config USE_DENVERTON_NS_CAR_NEM_ENHANCED
> > > > >         bool "Enhanced Non-evict mode"
> > > > >         select SOC_INTEL_COMMON_BLOCK_CAR
> > > > >         select INTEL_CAR_NEM_ENHANCED
> > > > >   select FSP_CAR
> > > > >   help
> > > > >     A current limitation of NEM (Non-Evict mode) is that code
> > > > >
> > > > > and data sizes
> > > > >           are derived from the requirement to not write out any
> > > > > modified cache line.
> > > > >
> > > > > The log output of both tries are almost the same...
> > > > > By configuring coreboot this way, the Temp RAM FSP is not used? So for
> > > > > the coreboot latest Temp RAM FSP support is broken right?
> > > > >
> > > > > Thanks,
> > > > > Sumo
> > > > >
> > > > > On Mon, Jun 13, 2022 at 1:38 PM Arthur Heymans 
> > > > > wrote:
> > > > > > Hi
> > > > > >
> > > > > > Can you try selecting "NO_FSP_TEMP_RAM_EXIT" in the denverton_ns
> > > > > > Kconfig?
> > > > > > That way coreboot uses native code to tear down CAR (so without FSP
> > > > > > TempRamExit)
> > > > > > INTEL_CAR_NEM_ENHANCED also needs to be select in the FSP_CAR option
> > > > > > for that to work (so it knows how to tear down CAR).
> > > > > >
> > > > > > Kind regards
> > > > > > Arthur
> > > > > >
> > > > > > On Mon, Jun 13, 2022 at 4:46 PM Sumo  wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Update: I found the bad commit:
> > > > > > >
> > > > > > > commit 5315e96abfa5b45fcd53149df5ebaa069a830558
> > > > > > > Author: Arthur Heymans 
> > > > > > > Date:   Fri May 14 11:22:31 2021 +0200
> > > > > > >
> > > > > > >     arch/x86/postcar: Use a separate stack for C execution
> > > > > > >
> > > > > > >     Add a stack in .bss for C execution. This will make it easier to
> > > > > > > move
> > > > > > >     the setup of MTRRs in C code.
> > > > > > >
> > > > > > > Hehehe, this confirmed my suspicion that the problem was in the
> > > > > > > POSTCAR...should have done a grep for postcar in git log instead of the
> > > > > > > painful bisect :D
> > > > > > >
> > > > > > > So denverton_ns is broken due to this commit, how can we proceed? I
> > > > > > > guess a git revert is not an option right, since this change looks like a
> > > > > > > base for future implementations... Please let me know if you need more
> > > > > > > tests.
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > Sumo
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 13, 2022 at 9:37 AM Sumo  wrote:
> > > > > > >
> > > > > > >> Thanks Paul, I'll try bisect.
> > > > > > >>
> > > > > > >> Do we have any instructions of how to use the normal/fallback
> > > > > > >> coreboot stage prefixes? During the bisect it will be painful always
> > > > > > >> bricking the board and having to use a flash programmer for restoring
> > > > > > >> instead of using flashrom.
> > > > > > >>
> > > > > > >> Should I build setting BOOTBLOCK_NORMAL + normal prefix, and then
> > > > > > >> reuse a prebuilt coreboot.rom with fallback stages included? Anything else
> > > > > > >> needed besides having a cmos layout?
> > > > > > >>
> > > > > > >> Kind regards,
> > > > > > >> Sumo
> > > > > > >>
> > > > > > >>
> > > > > > >> On Sat, Jun 11, 2022 at 3:15 AM Paul Menzel 
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Dear Suma,
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Am 11.06.22 um 00:16 schrieb Sumo:
> > > > > > >>>
> > > > > > >>> > My denverton system is crashing after the FSM memory init,
> > > > > > >>> probably when
> > > > > > >>> > jumping to POSTCAR. The following lines are shown before the
> > > > > > >>> crash:
> > > > > > >>> >
> > > > > > >>> > [DEBUG]  TPM: Digest of `CBFS: fallback/postcar` to PCR 2 logged
> > > > > > >>> > [DEBUG]  Loading module at 0x7f7ce000 with entry 0x7f7ce031.
> > > > > > >>> filesize:
> > > > > > >>> > 0x6060 memsize: 0xc358
> > > > > > >>> > [DEBUG]  Processing 246 relocs. Offset value of 0x7d7ce000
> > > > > > >>> > [DEBUG]  BS: romstage times (exec / console): total (unknown) /
> > > > > > >>> 1350 ms
> > > > > > >>> >
> > > > > > >>> > Below are my CAR settings:
> > > > > > >>> > # CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED is not set
> > > > > > >>> > CONFIG_USE_DENVERTON_NS_FSP_CAR=y
> > > > > > >>> > CONFIG_ARCH_POSTCAR_X86_32=y
> > > > > > >>> > CONFIG_POSTCAR_STAGE=y
> > > > > > >>> > CONFIG_CARDBUS_PLUGIN_SUPPORT=y
> > > > > > >>> > CONFIG_FSP_CAR=y
> > > > > > >>> > CONFIG_POSTCAR_CONSOLE=y
> > > > > > >>> >
> > > > > > >>> > Everything is fine when using coreboot v4.14.
> > > > > > >>> >
> > > > > > >>> > I have tried using CONFIG_USE_DENVERTON_NS_CAR_NEM_ENHANCED but
> > > > > > >>> things got
> > > > > > >>> > even worse - in this case nothing is shown in the console output.
> > > > > > >>> >
> > > > > > >>> > Any suggestions?
> > > > > > >>>
> > > > > > >>> It’d be great if you could bisect.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Kind regards,
> > > > > > >>>
> > > > > > >>> Paul
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> PS: In your address book please spell coreboot lowercase. ;-)
> > > > > > >>>
> > > > > > >> _______________________________________________
> > > > > > > coreboot mailing list -- [e-mail]
> > > > > > > To unsubscribe send an email to [e-mail]
> > > > > > >