Re: [coreboot] coreboot reproducible builds almost there

[Paul Menzel]

Dear coreboot folks,

Am Donnerstag, den 26.02.2015, 17:23 +0200 schrieb Emilian Bold:
> I was trying to duplicate a coreboot build back in November and I noticed I
> couldn't get my ROM file to be identical to the one I found online.
>
> It seems that coreboot doesn't have reproducible builds yet.
>
> Debian has been looking into this for a while
> https://wiki.debian.org/ReproducibleBuilds and I think coreboot should
> adopt this concept.

[…]

Holger Levsen joined #[e-mail] yesterday to report that
he integrated coreboot into the reproducible builds infrastructure [1].

After configuring the used build script [2] to build without a payload,

nice ionice -c 3 \
        	bash util/abuild/abuild --payloads none || true # don't fail the full job just because some targets fail

it looks like most boards are passing the test now [1]. Big thanks to
Alexander (lynxis) for submitting the necessary patches!

The only exceptions are the six boards below.

* a-trend_atc-6220 (256K) is unreproducible.
      * a-trend_atc-6240 (256K) is unreproducible.
      * google_nyan (4096K) is unreproducible.
      * google_nyan_big (4096K) is unreproducible.
      * google_rush (4096K) is unreproducible.
      * google_rush_ryu (8192K) is unreproducible.

Also, as a side node, SeaBIOS also supports to be built reproducible
since commit 624e8127 (build: Support "make VERSION=xyz" to override the
default build version) [3], though not by default.

So the coreboot build system, building the SeaBIOS payload, would need
to be adapted, if a reproducible build with the SeaBIOS payload is
required.

Thanks,

Paul

[1] https://reproducible.debian.net/coreboot/coreboot.html
[2] http://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/tree/bin/reproducib...
[3] http://seabios.org/pipermail/seabios/2015-June/009253.html

coreboot reproducible builds almost there

[Patrick Georgi]

2015-06-11 7:58 GMT+02:00 Paul Menzel :
> Holger Levsen joined #[e-mail] yesterday to report that
> he integrated coreboot into the reproducible builds infrastructure [1].

Thank you for the heads up, and thanks Holger for doing it!
>   * a-trend_atc-6220 (256K) is unreproducible.
>   * a-trend_atc-6240 (256K) is unreproducible.
>   * google_nyan (4096K) is unreproducible.
>   * google_nyan_big (4096K) is unreproducible.
>   * google_rush (4096K) is unreproducible.
>   * google_rush_ryu (8192K) is unreproducible.

The problem is one of collation order configured by LANG (see their
build script). fr_CH.UTF-8 has different rules that sort A_TREND
before AAEON instead of after ASUS, and NYAN before instead of after
NYAN_*.
http://review.coreboot.org/10510 should deal with that.

While at it, http://review.coreboot.org/10512 minimizes the stored
config to make it clearer to users which options were changed.

Patrick

coreboot reproducible builds almost there

[Holger Levsen]

Dear coreboot folks,

adding the Debian reproducible-builds lists to cc: - please either cc: that 
list or me on replies, i'm not subscribed to the coreboot list.

On Donnerstag, 11. Juni 2015, Paul Menzel wrote:
> Holger Levsen joined #[e-mail] yesterday to report that
> he integrated coreboot into the reproducible builds infrastructure [1].

indeed :) Thanks for sharing this here!

I'm now wondering how to proceed further. I *believe* another set of tests, 
this time with payloads, would be useful, if only to prove that the 
combination "reproducible coreboot" plus "reproducible payload" also is 
reproducible.

Things is, _AIUI_, there is no common payload which can be enabled for all 
boards. Is that correct?

If it is, is there another way to sensible group targets by payloads or detect 
sensible ones? Currently I'm building all targets in one pass using abuild, 
but I could also loop through each target and build it individually, as long 
as I dont have to configure 247 different configurations... (rather easy would 
be, all i386 targets use seabios, all mips target $that_one and all arm 
targets $another_one...

And then I'm wondering, "what next"? AIUI you don't ever offer images for 
download and instead expect users to build coreboot themselves. So the whole 
topic of verifying and reproducing the vendors (=your!) binaries is a bit mood 
here, at least atm. Any comments on that?

Last, and probably least: currently the headline of the page says "fast, 
flexible and reproducible Open Source firmware_?_" which I felt was 
appropriate when only a few images with payload were reproducible. Now I've 
been thinking about removing the questionmark, but in a way I like to keep it, 
til you reached 100% (without payloads) - do you think thats "ok"? IOW: do you 
feel offended (or maybe just a bit "hmm") by the questionmark or do you see it 
as motivation (to reach 100%)? ;-)

Oh, and as the page says, it's currently only updated once a month and on 
demand, so please do ping me if you merged some reproducible changes into 
master!

Thanks for your work on coreboot - it's really awesome & amazing! Free the 
hardware! :-)

cheers,
    Holger, happy about any feedback basically. coreboot.html is *your* 
    	page, not mine! :-)

coreboot reproducible builds almost there

[Patrick Georgi]

2015-06-11 10:25 GMT+02:00 Holger Levsen :
> I'm now wondering how to proceed further. I *believe* another set of tests,
> this time with payloads, would be useful, if only to prove that the
> combination "reproducible coreboot" plus "reproducible payload" also is
> reproducible.
>
> Things is, _AIUI_, there is no common payload which can be enabled for all
> boards. Is that correct?

The one that should run everywhere is Google's depthcharge, and that
isn't hooked up to the build system like SeaBIOS is (and even
depthcharge isn't available for emulation/qemu-riscv). So you're
correct, there is no single payload for every board.
> If it is, is there another way to sensible group targets by payloads or detect
> sensible ones? Currently I'm building all targets in one pass using abuild,
> but I could also loop through each target and build it individually, as long
> as I dont have to configure 247 different configurations... (rather easy would
> be, all i386 targets use seabios, all mips target $that_one and all arm
> targets $another_one...

Since we select default payloads in our Kconfig rules (seabios for
x86), I guess we should start doing the same for other architectures.
I don't think that belongs in your test script - things might change
in the future, and that shouldn't require additional effort on your
part.

We'll work on that, and will tell you once there's a plan and/or implementation.
> And then I'm wondering, "what next"? AIUI you don't ever offer images for
> download and instead expect users to build coreboot themselves. So the whole
> topic of verifying and reproducing the vendors (=your!) binaries is a bit mood
> here, at least atm. Any comments on that?

There are distributions, sort of.
Google ships coreboot as part of Chrome OS (which requires a huge
build system even to build just coreboot).
Then there's libreboot (www.libreboot.org), which ships their
coreboot-variant (without blobs and with as few patches against
coreboot as possible) with GRUB2 as payload for a small selection of
boards that they can support.
There are also some other vendors, but they're less visible.

If you want to support coreboot distributions (that ship binaries),
libreboot is currently your best bet.
> Last, and probably least: currently the headline of the page says "fast,
> flexible and reproducible Open Source firmware_?_" which I felt was
> appropriate when only a few images with payload were reproducible.

Oh, I loved it, still do! At some point we should replace the question
mark with an exclamation mark, though :-)
> Oh, and as the page says, it's currently only updated once a month and on
> demand, so please do ping me if you merged some reproducible changes into
> master!

Thanks, will do. Let's start immediately :-)
Since you mentioned 'once we reach 100% (without payloads)' earlier:
the issue of the remaining 7 non-reproducible board should be fixed
since today, in response to your report. I'd appreciate if you could
do another run to verify this.

Regards,
Patrick

coreboot reproducible builds almost there

[Emilian Bold]

Great! So there were no coreboot patches necessary for this? Is it just a
matter of preparing the right build environment? Because when I tried to do
it manually (with SeaBios) it didn't produce the same bytes.

Since SeaBios is reproducible it would be great to make the coreboot +
SeaBios bundle reproducible too.

And if the bundle is reproducible then it is easy to have a script that
*verifies* some external build. Assuming it included the CONFIG values, one
could extract the .config file from the rom (something like grep -a CONFIG_
rom > .config), do a local build and compare the bytes. I guess one new
CONFIG value would be the SeaBios version.

--emi

On Thu, Jun 11, 2015 at 8:58 AM, Paul Menzel <
[e-mail]> wrote:
> Dear coreboot folks,
>
> Am Donnerstag, den 26.02.2015, 17:23 +0200 schrieb Emilian Bold:
> > I was trying to duplicate a coreboot build back in November and I
>
> noticed I
> > couldn't get my ROM file to be identical to the one I found online.
> >
> > It seems that coreboot doesn't have reproducible builds yet.
> >
> > Debian has been looking into this for a while
> > https://wiki.debian.org/ReproducibleBuilds and I think coreboot should
> > adopt this concept.
>
> […]
>
> Holger Levsen joined #[e-mail] yesterday to report that
> he integrated coreboot into the reproducible builds infrastructure [1].
>
> After configuring the used build script [2] to build without a payload,
>     nice ionice -c 3 \
>             bash util/abuild/abuild --payloads none || true # don't
>
> fail the full job just because some targets fail
>
> it looks like most boards are passing the test now [1]. Big thanks to
> Alexander (lynxis) for submitting the necessary patches!
>
> The only exceptions are the six boards below.
>   * a-trend_atc-6220 (256K) is unreproducible.
>   * a-trend_atc-6240 (256K) is unreproducible.
>   * google_nyan (4096K) is unreproducible.
>   * google_nyan_big (4096K) is unreproducible.
>   * google_rush (4096K) is unreproducible.
>   * google_rush_ryu (8192K) is unreproducible.
>
> Also, as a side node, SeaBIOS also supports to be built reproducible
> since commit 624e8127 (build: Support "make VERSION=xyz" to override the
> default build version) [3], though not by default.
>
> So the coreboot build system, building the SeaBIOS payload, would need
> to be adapted, if a reproducible build with the SeaBIOS payload is
> required.
>
> Thanks,
>
> Paul
>
> [1] https://reproducible.debian.net/coreboot/coreboot.html
> [2]
> http://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/tree/bin/reproducib...
> [3] http://seabios.org/pipermail/seabios/2015-June/009253.html