New API to clear DRAM at boot

[Patrick Rudolph]

Hi coreboot folks,
in order to support TEE like Intel TXT it is necessary to be able to
clear all DRAM at boot on request.

As all of the x86 coreboot code is x86_32, it is necessary to make use
of PAE to clear memory.
Please find the attached patch series which proposes an architecure
independed API, PAE enabled memset on x86, and helper functions to
clear all DRAM on Broadwell DE:


https://review.coreboot.org/#/q/project:coreboot+branch:master+topic:securit...

The code can be easily extended to other platforms.
Please comment, test and improve the patchset.

Regards,
-- 
Patrick Rudolph

9elements Agency GmbH, Kortumstraße 19-21, 44787 Bochum, Germany
Email:  [e-mail]
Phone:  [phone]

Sitz der Gesellschaft: Bochum
Handelsregister: Amtsgericht Bochum, HRB 17519
Geschäftsführung: Sebastian Deutsch, Daniel Hoelzgen

[ron minnich]

On Tue, Feb 26, 2019 at 6:41 AM Patrick Rudolph
<[e-mail]> wrote:
> Hi coreboot folks,
> in order to support TEE like Intel TXT it is necessary to be able to
> clear all DRAM at boot on request.
>
> As all of the x86 coreboot code is x86_32, it is necessary to make use
> of PAE to clear memory.

I would much rather we consider getting into the current century and
having coreboot be able to
run x86_64 :-)

can we do that?

ron

[Nico Huber]

On 26.02.19 20:16, ron minnich wrote:
> On Tue, Feb 26, 2019 at 6:41 AM Patrick Rudolph
> <[e-mail]> wrote:
> > Hi coreboot folks,
> > in order to support TEE like Intel TXT it is necessary to be able to
> > clear all DRAM at boot on request.
> >
> > As all of the x86 coreboot code is x86_32, it is necessary to make use
> > of PAE to clear memory.
>
> I would much rather we consider getting into the current century and
> having coreboot be able to
> run x86_64 :-)>
> can we do that?

Would be a lot easier if coreboot were open source on x86. But it's
still doable, even in the presence of 32-bit blobs.

Though, as it seems, we allow blobs to call into coreboot code now. This
means we wouldn't only have to wrap calls into a 32-bit blob, we'd also
have to wrap everything that may be called by the blob.

Cheers!
Nico

[Rudolf Marek]

Hi,

Dne 26. 02. 19 v 22:58 Nico Huber napsal(a):
> On 26.02.19 20:16, ron minnich wrote:
> > On Tue, Feb 26, 2019 at 6:41 AM Patrick Rudolph
> > <[e-mail]> wrote:
> > > Hi coreboot folks,
> > > in order to support TEE like Intel TXT it is necessary to be able to
> > > clear all DRAM at boot on request.
> > >
> > > As all of the x86 coreboot code is x86_32, it is necessary to make use
> > > of PAE to clear memory.

Why PAE, you can you can use 64-bit paging data structures while still running in 32-bits!
You simply don't leave compatibility mode and thats all.

0) setup 64bit paging structures, set CR3
1) set EFER.LME 
2) enable paging in CR0.PG
3) you are done, because now you run in 32-bits but paging structures are 64-bit...
> > I would much rather we consider getting into the current century and
> > having coreboot be able to
> > run x86_64 :-)>
> > can we do that?
>
> Would be a lot easier if coreboot were open source on x86. But it's
> still doable, even in the presence of 32-bit blobs.
>
> Though, as it seems, we allow blobs to call into coreboot code now. This
> means we wouldn't only have to wrap calls into a 32-bit blob, we'd also
> have to wrap everything that may be called by the blob.

Yes, I can only add that you can jump from/to 64-bit mode using JMP/CALL FAR or equivalent. It is just a matter of changing the CS selector to the desired one.
With that, you can have some wrapper to call some routine from 64-bit to 32-bit and vice versa. The only pain is LP64, because "long" is 64-bit...

Thanks,
Rudolf

[ron minnich]

I looked for my CL from years back in which I showed a way to 64-bit
amd64 ramstage, but I don't see it any more.

But it was easy.

On Tue, Feb 26, 2019 at 2:11 PM Rudolf Marek <[e-mail]> wrote:
> Hi,
>
> Dne 26. 02. 19 v 22:58 Nico Huber napsal(a):
> > On 26.02.19 20:16, ron minnich wrote:
> > > On Tue, Feb 26, 2019 at 6:41 AM Patrick Rudolph
> > > <[e-mail]> wrote:
> > > > Hi coreboot folks,
> > > > in order to support TEE like Intel TXT it is necessary to be able to
> > > > clear all DRAM at boot on request.
> > > >
> > > > As all of the x86 coreboot code is x86_32, it is necessary to make use
> > > > of PAE to clear memory.
>
> Why PAE, you can you can use 64-bit paging data structures while still running in 32-bits!
> You simply don't leave compatibility mode and thats all.
>
> 0) setup 64bit paging structures, set CR3
> 1) set EFER.LME
> 2) enable paging in CR0.PG
> 3) you are done, because now you run in 32-bits but paging structures are 64-bit...
> > > I would much rather we consider getting into the current century and
> > > having coreboot be able to
> > > run x86_64 :-)>
> > > can we do that?
> >
> > Would be a lot easier if coreboot were open source on x86. But it's
> > still doable, even in the presence of 32-bit blobs.
> >
> > Though, as it seems, we allow blobs to call into coreboot code now. This
> > means we wouldn't only have to wrap calls into a 32-bit blob, we'd also
> > have to wrap everything that may be called by the blob.
>
> Yes, I can only add that you can jump from/to 64-bit mode using JMP/CALL FAR or equivalent. It is just a matter of changing the CS selector to the desired one.
> With that, you can have some wrapper to call some routine from 64-bit to 32-bit and vice versa. The only pain is LP64, because "long" is 64-bit...
>
> Thanks,
> Rudolf

[Patrick Rudolph]

On Tue, 2019-02-26 at 11:16 -0800, ron minnich wrote:
> On Tue, Feb 26, 2019 at 6:41 AM Patrick Rudolph
> <[e-mail]> wrote:
> > Hi coreboot folks,
> > in order to support TEE like Intel TXT it is necessary to be able
> > to
> > clear all DRAM at boot on request.
> >
> > As all of the x86 coreboot code is x86_32, it is necessary to make
> > use
> > of PAE to clear memory.
>
> I would much rather we consider getting into the current century and
> having coreboot be able to
> run x86_64 :-)
>
> can we do that?
>
> ron
> Hi Ron,
> that's done for x86 qemu here, but is out of the scope of this patch
> series:

https://review.coreboot.org/#/q/project:coreboot+branch:master+topic:x86_64_...

As we have x86_32 code, the memory clearing code uses PAE for now.
I'm happy to see x86_64 running on real hardware, but that would take
some time.

As already pointed out, the integration into FSP enabled platforms
could become "interresting", that is why PAE is still a "legacy"
option.

This patch series is to be used on server platforms where security is
more important than short boot times. The additional overhead of
clearing memory is expected and can possibly reduced in further
patches.

I'm interrested to see if the proposed long mode paging running in
compability mode gives significant performance improvements.
In the end you often reload page tables as you still can only access
32bit address space.

Regards,
-- 
Patrick Rudolph

9elements Agency GmbH, Kortumstraße 19-21, 44787 Bochum, Germany
Email:  [e-mail]
Phone:  [phone]

Sitz der Gesellschaft: Bochum
Handelsregister: Amtsgericht Bochum, HRB 17519
Geschäftsführung: Sebastian Deutsch, Daniel Hoelzgen

Hi Patrick,

I noticed in your patch you are calling this host CPU based clearing after the return from FSP memory initialization.  If you are calling this in a system with ECC memory, this step is redundant.  Part of the initialization of ECC enabled memory is to write a pattern to all of memory to setup the ECC code.  Additionally the memory initialization is using a HW engine to achieve a bandwidth that is not possible by a SW host CPU based mechanism.  This is an interesting idea and could be useful, however in this specific case you would be adding boot overhead unnecessarily.

Steve Mooney
Senior Consulting Engineer
SysPro Consulting, LLC
https://www.sysproconsulting.com/


-----Original Message-----
From: Patrick Rudolph <[e-mail]> 
Sent: Tuesday, February 26, 2019 7:36 AM
To: Coreboot <[e-mail]>
Subject: [coreboot] New API to clear DRAM at boot

Hi coreboot folks,
in order to support TEE like Intel TXT it is necessary to be able to clear all DRAM at boot on request.

As all of the x86 coreboot code is x86_32, it is necessary to make use of PAE to clear memory.
Please find the attached patch series which proposes an architecure independed API, PAE enabled memset on x86, and helper functions to clear all DRAM on Broadwell DE:


https://review.coreboot.org/#/q/project:coreboot+branch:master+topic:securit...

The code can be easily extended to other platforms.
Please comment, test and improve the patchset.

Regards,
--
Patrick Rudolph

9elements Agency GmbH, Kortumstraße 19-21, 44787 Bochum, Germany
Email:  [e-mail]
Phone:  [phone]

Sitz der Gesellschaft: Bochum
Handelsregister: Amtsgericht Bochum, HRB 17519
Geschäftsführung: Sebastian Deutsch, Daniel Hoelzgen _______________________________________________
coreboot mailing list -- [e-mail] To unsubscribe send an email to [e-mail]

[Patrick Rudolph]

On Wed, 2019-02-27 at 09:19 -0700, [e-mail]
wrote:
> Hi Patrick,
>
> I noticed in your patch you are calling this host CPU based clearing
> after the return from FSP memory initialization.  If you are calling
> this in a system with ECC memory, this step is redundant.  Part of
> the initialization of ECC enabled memory is to write a pattern to all
> of memory to setup the ECC code.  Additionally the memory
> initialization is using a HW engine to achieve a bandwidth that is
> not possible by a SW host CPU based mechanism.  This is an
> interesting idea and could be useful, however in this specific case
> you would be adding boot overhead unnecessarily.

Hi Steve,
yes that's true.
If the CPU supports ECC and the mainboard and the DRAM and FSP is
configured to setup ECC that would be a redundant step.

I'm happy to see a patch from the FSP maintainers to detect this corner
case and skip the slow memory clearing on coreboot site.

Regards,
> Steve Mooney
> Senior Consulting Engineer
> SysPro Consulting, LLC
> https://www.sysproconsulting.com/
>
> -----Original Message-----
> From: Patrick Rudolph <[e-mail]> 
> Sent: Tuesday, February 26, 2019 7:36 AM
> To: Coreboot <[e-mail]>
> Subject: [coreboot] New API to clear DRAM at boot
>
> Hi coreboot folks,
> in order to support TEE like Intel TXT it is necessary to be able to
> clear all DRAM at boot on request.
>
> As all of the x86 coreboot code is x86_32, it is necessary to make
> use of PAE to clear memory.
> Please find the attached patch series which proposes an architecure
> independed API, PAE enabled memset on x86, and helper functions to
> clear all DRAM on Broadwell DE:

https://review.coreboot.org/#/q/project:coreboot+branch:master+topic:securit...
> The code can be easily extended to other platforms.
> Please comment, test and improve the patchset.
>
> Regards,
> --
> Patrick Rudolph
>
> 9elements Agency GmbH, Kortumstraße 19-21, 44787 Bochum, Germany
> Email:  [e-mail]
> Phone:  [phone]
>
> Sitz der Gesellschaft: Bochum
> Handelsregister: Amtsgericht Bochum, HRB 17519
> Geschäftsführung: Sebastian Deutsch, Daniel Hoelzgen
> _______________________________________________
> coreboot mailing list -- [e-mail] To unsubscribe send an
> email to [e-mail]

[ron minnich]

On Wed, Feb 27, 2019 at 8:27 AM <[e-mail]> wrote:
> I noticed in your patch you are calling this host CPU based clearing after the return from FSP memory initialization.  If you are calling this in a system with ECC memory, this step is redundant.  Part of the initialization of ECC enabled memory is to write a pattern to all of memory to setup the ECC code.

This is a bad idea for many people for a simple reason: sometimes you
don't want memory cleared. We deliberately did not clear memory in
coreboot for many years, starting in 1999, following the practice of
unix: memory setup was the kernel's job and could even be done lazily
or on demand. This allows you, even after a power cycle, to do some
post mortem in the event of a failure, or even upload the contents to
a server for later analysis. This is useful in many places, from
supercomputer to data center to laptop, especially in development.
Hardwiring memory clearing behavior as part of ECC setup is a poor
decision IMHO. It's like doing a lobotomy every time you boot, and
after a failure, you REALLY want that memory data. We should at least
have a choice.

The preferred way to do this is to read the memory and write it back,
with ECC traps disabled, to get the ECC bits set up. Memory contents
are preserved and ECC is set up correctly. Especially in this case,
the speed tradeoff is not worth it.

The engine is interesting but it's another case of proprietary
firmware designers (as in FSP) implementing a neat idea without
thinking of user consequences. This is disappointing. Were FSP open
source, we could explore the option space, but as it is, we're just
twiddling the knobs on a black box. The fact that someone as
knowledgeable as Patrick might not have known this tells you how
obscure the information is. That's not good.

ron