Using Meson to build flashrom

[Richard Hughes]

Hi all,

First, happy new year and I hope you all had a great break. As some of
you know, I've been working with 3mdeb for the last few months, with
the aim of using libflashrom in fwupd to be able to flash firmware
using the LVFS on various embedded systems. This /could/ include
flashrom-compatible consumer hardware in the future, although I've not
worked out all the details on how something like coreboot-on-thinkpad
could be distributed automatically. Anyway, I digress.

The fwupd project has to build in all kinds of odd targets, e.g. for
odd endians, odd instruction sets, and in odd ways, e.g. installing
with a prefix of /app for projects like flatpak. We also have other
"robustness" guarantees and therefore have a comprehensive set of CI
tests which enable a lot of warning flags and run linting and static
analysis code like Coverity. This might explain my ongoing effort to
fix up seemingly benign warnings.

Rather than hack the hell out of Makefile (which I started to do
initially) I ported the codebase to use the meson build-system. Meson
is a(nother) next-generation build system used by a lot of open source
projects ranging from low level libraries to desktop software. As part
of the port, I also copied the CONFIG_ logic from the makefile, e.g.

Option                  Current Value Possible Values Description
  ------                  ------------- --------------- -----------
  config_atahpt           false         [true, false]   Highpoint
(HPT) ATA/RAID controllers
  config_atapromise       false         [true, false]   Promise ATA
controller
  config_atavia           true          [true, false]   VIA VT6421A
LPC memory
...

At the moment I'm using the meson port so I can include flashrom as a
subproject to fwupd (as distros are not yet shipping libflashrom as a
proper shared library) but 3mdeb is some way from being finished with
libflashrom. I wondered if my work would be useful already to anyone
else building flashrom on weird embedded architectures, or of it would
be interesting to upstream with further work. The single commit which
just adds 5 files can be found here:
https://github.com/hughsie/flashrom/tree/wip/hughsie/meson

Comments welcome.

Richard.

[Carl-Daniel Hailfinger]

Hi Hugh,

On 03.01.2019 12:23, Richard Hughes wrote:
> First, happy new year and I hope you all had a great break.

Thanks, you too.
> As some of
> you know, I've been working with 3mdeb for the last few months, with
> the aim of using libflashrom in fwupd to be able to flash firmware
> using the LVFS on various embedded systems. This /could/ include
> flashrom-compatible consumer hardware in the future, although I've not
> worked out all the details on how something like coreboot-on-thinkpad
> could be distributed automatically. Anyway, I digress.
>
> The fwupd project has to build in all kinds of odd targets, e.g. for
> odd endians, odd instruction sets,

Ah yes, that's why we avoided autotools and hacked our own makefile
logic. Autotools break too often even for the trivial case (various
Linux distributions on x86).
> and in odd ways, e.g. installing
> with a prefix of /app for projects like flatpak. We also have other
> "robustness" guarantees and therefore have a comprehensive set of CI
> tests which enable a lot of warning flags and run linting and static
> analysis code like Coverity. This might explain my ongoing effort to
> fix up seemingly benign warnings.

The warning fixes are much appreciated. That's the reason why we worked
pretty early on llvm/clang support for flashrom, and clang found quite a
few bugs in the codebase. There are some places where some gcc versions
are stupid and cause spurious warnings for certain valid compile
options, so eliminating all warnings is not going to be possible with
all gcc versions.
> Rather than hack the hell out of Makefile (which I started to do
> initially) I ported the codebase to use the meson build-system. Meson
> is a(nother) next-generation build system used by a lot of open source
> projects ranging from low level libraries to desktop software. As part

Mh. We investigated CMake quite some time ago and decided against it.
Not because of direct CMake problems (IIRC CMake worked really well),
but because a sufficiently recent CMake was not available on all
supported platforms, most notably old RHEL versions. Back then, forcing
users to install build tools not available on their platform of choice
was frowned upon. Is Meson available by default on the oldest supported
RHEL version? If not, we'll have to recheck if the RHEL versions with a
too ancient CMake have since been retired.
> of the port, I also copied the CONFIG_ logic from the makefile, e.g.
>
> Option                  Current Value Possible Values Description
>
> config_atahpt           false         [true, false]   Highpoint
> (HPT) ATA/RAID controllers
>   config_atapromise       false         [true, false]   Promise ATA
> controller
>   config_atavia           true          [true, false]   VIA VT6421A
> LPC memory
> ...

The Meson build files look really nice. Thanks for adding support for
it. Would you mind pointing me to the part of the logic which takes care
of all the special cases? I didn't see it at first glance, but I
probably looked in the wrong place.

Is the Meson build intended to replace the current Makefile system or is
it intended to handle some currently unsupported special cases only?
> At the moment I'm using the meson port so I can include flashrom as a
> subproject to fwupd (as distros are not yet shipping libflashrom as a
> proper shared library) but 3mdeb is some way from being finished with
> libflashrom. I wondered if my work would be useful already to anyone
> else building flashrom on weird embedded architectures, or of it would
> be interesting to upstream with further work.

I'm always happy if people try to port flashrom to new architectures. In
the past, supporting new architectures with MMIO-based interfaces
usually led to a chase down the rabbit hole until we could find someone
who knew the memory ordering rules and associated Linux kernel
interfaces of the architecture well enough (thanks Segher!).

If you're aware of any architecture or platform or delivery mechanism
where the current Makefile doesn't work, I'd appreciate a pointer so I
can fix the current solution until we have a new solution working on all
platforms.
> The single commit which
> just adds 5 files can be found here:
> https://github.com/hughsie/flashrom/tree/wip/hughsie/meson
>
> Comments welcome.

Correct me if I'm wrong, but it looks like the build logic in your
current tree errors out if someone tries to build the internal
programmer on non-x86 platforms. IIRC flashrom currently builds fine at
least on MIPS (Loongson-2F internal programmer support).

The current Makefile supports the following (sometimes exotic) cases:
- native build on Linux (different libpci versions need different
compile options)
- native build for Debian/KFreeBSD
- native build on Windows
- native build on Mac OS X
- native builds on all the BSDs (Free, Open, Net, Dragonfly)
- standalone build against libpayload
- cross-builds on Linux for DOS
- cross-builds on Linux for Windows
- cross-builds on Linux for Mac OS X
- static builds
- and probably a bunch of others I forgot

Yes, the number of people completely understanding the Makefile is lower
than we'd like. I'm all for cleaning this up if we can find a solution
which works well enough on all supported platforms.

Thanks again for your work.

Regards,
Carl-Daniel

[Richard Hughes]

On Thu, 3 Jan 2019 at 21:05, Carl-Daniel Hailfinger
 wrote:
> Ah yes, that's why we avoided autotools and hacked our own makefile
> logic. Autotools break too often even for the trivial case (various
> Linux distributions on x86).

Right, I never had much luck with autotools either.
> so eliminating all warnings is not going to be possible with
> all gcc versions.

Agreed.
> but because a sufficiently recent CMake was not available on all
> supported platforms, most notably old RHEL versions.

From my point of view (as a Red Hat employee) the earliest RHEL I need
to care about for new features is 8, and in exceptional cases 7.
Unfortunately (or fortunately, depending on how you look at it) fwupd
rebases into RHEL 7 are oft-requested and so I need to make sure the
latest fwupd builds in RHEL 7, which for the 7.6 release meant
preparing a meson package that could work in this environment. But see
below...
> The Meson build files look really nice. Thanks for adding support for
> it. Would you mind pointing me to the part of the logic which takes care
> of all the special cases? I didn't see it at first glance, but I
> probably looked in the wrong place.

I assume you mean stuff like
https://github.com/hughsie/flashrom/commit/b8e57e804d121097f7634b68a8c1d7c5b...
? -- note: I didn't copy all the logic for non-Linux systems, e.g. DOS
or MinGW. Partially because I'd have no way to test the build, but
mostly because I didn't want to spent too much time on it if the
flashrom maintainers thought it was a hugely bad idea and would never
be upstreamed.
> Is the Meson build intended to replace the current Makefile system or is
> it intended to handle some currently unsupported special cases only?

I don't think it has to replace the existing system. The existing
Makefile has years of being battle-tested in all kinds of odd setups
and is probably good enough for most people just installing everything
to /usr. If you're using something like flatpak that's using prefixes
or using flashrom as a subproject to another project then you're going
to massively appreciate meson.

The downside is that adding extra source files means you have to edit
two build files. From someone that's done this for several projects it
does get tiresome, although it does give everyone a large "grace"
period to adjust to the new world order before removing any other
build system.

My gut instinct is that flashrom should continue to have Makefile as
the "primary" build system that blocks CI and let those that want to
use meson send patches to keep the meson.build up to date. It also
means we don't need to support and test every single esoteric OS in
the meson build.
> I'm always happy if people try to port flashrom to new architectures.

I actually ended up disabling flashrom support for fwupd on things
like ppc64le -- on the logic you're probably not going to be using
flashrom to perform updates on such systems...
> Correct me if I'm wrong, but it looks like the build logic in your
> current tree errors out if someone tries to build the internal
> programmer on non-x86 platforms.

That was some debugging that I accidentally left in. If you think it's
worthwhile I can spend some more time on the meson branch and fix up
things like this.
> standalone build against libpayload

I didn't really "get" libpayload -- can somebody describe what it is? Thanks.
> standalone build against libpayload
> cross-builds on Linux for DOS
> cross-builds on Linux for Windows
> cross-builds on Linux for Mac OS X

Cross builds are a lot easier with meson, although I've not actually
tested it with flashrom.
> static builds

This works.

Richard.

[Richard Hughes]

On Fri, 4 Jan 2019 at 10:16, Richard Hughes  wrote:
> My gut instinct is that flashrom should continue to have Makefile as
> the "primary" build system that blocks CI and let those that want to
> use meson send patches to keep the meson.build up to date.

As part of that I've uploaded this patch to Gerrit here:
https://review.coreboot.org/c/flashrom/+/31248
> That was some debugging that I accidentally left in. If you think it's
> worthwhile I can spend some more time on the meson branch and fix up
> things like this.

I've fixed this. I've got a few more trivial fixups I've found when
trying to get a meson flashrom built in all the fwupd CI targets, but
I'll submit those after some initial comments on the gerrit review.
> > standalone build against libpayload
>
> I didn't really "get" libpayload -- can somebody describe what it is? Thanks.

I still don't "get" libpayload. Is the idea a shared libflashrom would
be linked against a static libpayload? How would that work for
something like fwupd? Or is libpayload used when not building the
shared library and only for a minimal flashrom binary?

Richard.

[Nico Huber]

Hi Richard,

Am 06.02.19 um 11:24 schrieb Richard Hughes:
> > > standalone build against libpayload
> >
> > I didn't really "get" libpayload -- can somebody describe what it is? Thanks.
>
> I still don't "get" libpayload. Is the idea a shared libflashrom would
> be linked against a static libpayload? How would that work for
> something like fwupd? Or is libpayload used when not building the
> shared library and only for a minimal flashrom binary?

a payload in this context is a program in flash that runs after
coreboot. libpayload[1] is a collection of library functions (e.g.
basic libc, limited libpci, coreboot related things) and device
drivers, to aid the coreboot-payload development.

If you built libflashrom for the libpayload target, nothing is
linked. libpayload's headers and a gcc wrapper script are used,
that's all. There shouldn't be a libpayload shared library target,
only a static one.

Nico

[1] https://review.coreboot.org/cgit/coreboot.git/tree/payloads/libpayload/