Hi,
As discoursed last dev meeting, we are going to start a working group to adopt meson as single, all cases covering, build system for flashrom.
The first step will be to write down all requirements that we need to meet. Therefor, we'll meet today at 8:30 UTC at https://meet.google.com/sag-dyik-bbr and posting the working draft afterwards on the mailing list. The google doc can be visited at any time. https://docs.google.com/document/d/1TVso_VbrLEbGc7BNrYxqlrP8jMxB23TuMqEI3PnC...
Please feel free to join the meetings, the next ones will be announced much earlier, add comments to the document or reply to this mail with and question or suggestion.
-- Thomas
Sorry, it's 9:30 UTC, wich is 11:30 German time, 19:30 Sydney time
-- Thomas
On Wed, 2022-04-06 at 07:19 +0000, Thomas Heijligen wrote:
Hi,
As discoursed last dev meeting, we are going to start a working group to adopt meson as single, all cases covering, build system for flashrom.
The first step will be to write down all requirements that we need to meet. Therefor, we'll meet today at 8:30 UTC at https://meet.google.com/sag-dyik-bbr%C2%A0and posting the working draft afterwards on the mailing list. The google doc can be visited at any time. https://docs.google.com/document/d/1TVso_VbrLEbGc7BNrYxqlrP8jMxB23TuMqEI3PnC...
Please feel free to join the meetings, the next ones will be announced much earlier, add comments to the document or reply to this mail with and question or suggestion.
-- Thomas _______________________________________________ flashrom mailing list -- flashrom@flashrom.org To unsubscribe send an email to flashrom-leave@flashrom.org
Hi,
Here an overview about the One Build System Project for flashrom. I'm sorry for the extreme late anouncement of the passed meeting. We had some diffent ideas about it. The next one will be anounced in a prper advanced and with a survay to choose a time.
Please feel free to contribute, take over a task, start a discousion, comment in the document or reply to this mail.
https://docs.google.com/document/d/1TVso_VbrLEbGc7BNrYxqlrP8jMxB23TuMqEI3PnC... ----------------------------------------------------------------------- # Flashrom One Build System Working Group [Draft]
## Goal Make meson the only build system for flashrom. Support all currently supported platforms.
## Abstract Modify, extend, rewrite the meson based build system to accommodate all requirements from flashrom. Rearrange the code for a best fit with the meson build system. Adapt the Makefile to these changes to have it working until its end.
## Notify me Add me to all to all build system related patches. (Or only to Make / Meson related one.) * Thomas Heijligen src@posteo.de
## Platforms to support * Systems * Linux (Distros / ChromeOS) * BSD (Free, Net, Open, Dragonfly) * MacOS * Android? * SunOS / Solaris / the current version of it * Debian kFreeBSD * GNU / HURD * Windows (mingw cross / cygwin) * DOS (djgpp cross from Linux) * libpayload (cross from Linux) * Architectures * X86 * ARM * MIPS * POWER x86 supports port I/O (mostly relevant to PCI), others may through MMIO emulation
## Tasks * Update the meson programmer selection to a positive list (see meson snippets) * Clean up the meson file and use best practices for meson * PoC: crosscompile “Hello World” for dos / djgpp with meson (with pci support) * PoC: crosscompile “Hello World” for libpayload with meson (with pci support) * Identify the active programmer in the man page * Add a note for each programmer that is not active in the build * Integrate getopt for dos based builds * Write documentation * Build instructions for all platforms * Design of meson.build * Reason for minimal meson version * How to add programmer * Sync compile options (WIP https://review.coreboot.org/c/flashrom/+/58561) * Create directories for platform abstraction, programmer, include files (https://review.coreboot.org/c/flashrom/+/62899)
## Open questions * System requirements * How long should we support systems / libraries? * As long as they have vendor support * What should be the minimum required version of meson? * The version of the oldest supported system. * MacOS and DirectHW * What is the current status? * What is useful to support? * Can we integrate it into flashrom? * How to test it? * Internal programmer * Should a programmer call / access another programmer? * Can we split the internal programmer into smaller units (x86_bios, mips_longson, xyz_superio, linux_mtd) and use internal as a keyword to select one of those? * Would that make it easier to build and modify? * Build dependencies * Can we fetch dependencies like cmocka? This makes building for some systems easier * Unit tests * Can we build the cmoka test on top of libflashrom.a? * Can this replace self tests? * Meson feature options * Build the shared library as feature * Build the classic_cli as feature * Can be easily deselectet if not needed of not supported (libpayload) * Build the tests as feature * Build man-pages as feature * Build ich_descriptor_tool as feature * Shared libflashrom * How to handle versioning?
## Sequence for switching the primary build system * Meson is feature equivalente with the Makefile * Switch to meson as default build system, update the documentation * Make a release with meson as the primary build system, keep the Makefile for legacy compatible. * Drop the Makefile * Drop dead code from Makefile only based option
## meson snippets - Ideas | option('programmer', type : 'array', choices : ['auto' 'all' 'group_pci' 'group_usb' 'ch341a_spi' ‘drkaiser’]) * auto selects all programmer available on the platform if the dependencies are found * all selects all programmer available on the platform, failing id a dependency is not found * group_* selects all programmer in this group, fails if the system does not support the group or a dependency is not found * <programmer name> selects this programmer, fails if not available on system or missing dependency
| libusb1 = dependency(...) | libusb1_systems = [ ‘linux’ ‘freebsd’ ‘windows’ ‘mac’ … ] | libpci = dependency(...) | libpci_systems = [...] # systems where libpci is available | port_io = declare_dependency(...) # files needed for port I/O support | programmer = { | 'ch341a_spi' { | 'system' = libusb1_systems, | 'srcs' = [ 'ch341a_spi.c' 'usbdev.c' ], | 'flags' = [ 'CONFIG_CH341A_SPI=1'], | 'deps' = [ libusb1], | # 'active' = true / false # selected on runtime | }, | ‘drkaiser’ = { | ‘Systems’ = libpci_systems, | ‘arch’ = [ ‘x86’ ‘amd64’ ], | ‘srcs’ = [ ‘drkaiser.c’ ], | ‘flags’ = [ ‘-DCONFIG_DRKAISER=1’ ], | ‘deps’ = [ ‘libpci’ ‘port_io’ ], | # 'active' = true / false # selected on runtime | }, | } * The programmer struct defines each programmer with its associated things * A for loop iterates through the set and adds ‘active’ = true / false to each programmer depending if it should be build or not * This set can be used to select source files, flags, dependencies, build tests, documentation, … * Use declare_dependency() for internal platform support like x86_msr or serial.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
## Meeting Notes Meeting on Apr 6. 9:30 + 10:30 UTC+0 https://meet.google.com/sag-dyik-bbr Attendees: Peter, Nico, Thomas, Felix, Carl-Daniel, Anastasia 1. Clean up meson so that it doesn’t look like a copy of makefile. Make it easier and more readable. 1. TODO: See meson snippets 2. Internal programmer is special, it has different options for different platform. Hard to implement positive lists 3. Internal can be x86 or linux_mtd, so maybe there can be meta- internal which selects the one which is needed 4. TODO: Internal split? Needs research. Can be done in parallel with prototype of cross-compiling. Ask on dev meeting 5. New feature flag to support port I/O 6. Have proper set of meson options to work with. We have no versioning on libflashrom. API is stable but is not defined as stable. 2. What are the missing features in meson? 1. Make a list of system that we want to support. See systems to support 1. We don’t need to support macOS in the beginning but maybe add later 2. DirectHW ? 3. TODO: Can we ask on mailing list about two above, does it work? Add Stefan Reinauer to the email 2. Check meson can support all of those systems 1. TODO: Prototype for crosscompiling djgpp/dos 2. TODO: Prototype for crosscompiling with libpayload 3. TODO: Sync compile options (WIP https://review.coreboot.org/c/flashrom/+/58561 ) 4. Figure out who can test for each supported system. Not only build , but also test. 1. Collect info who has which hardware and is willing to test 5. Define what testing means. Should we run flashrom or flash something? How do we check that meson supports the system? 1. Build and run unit tests. 2. Run the binary to see that it works 3. Great thing would be to use one (or more :)) programmer. Maybe just dummy? 6. Add meson to documentation 7. Have one release with both meson and makefile, when everything is supported in meson 1. Maybe one beta release so that people can try 2. Have both meson and makefile on the readme 8. After that we can say that meson is the primary build system, and update all the documentation. 9. Start dropping makefile 10. Can the man page show what is enabled in specific build? 1. Can be in parallel 11. For future: Testing at least one hardware access programmer 12. Is libflashrom used by the flashrom binary or is it built into the flashrom binary? How about we build both on user systems? So the flashrom binary would be just the cli application.
Some comments from a packager and experience with switching to cmake, some of it ok, some of it bad. I'm trimming stuff I don't have any comments on as I think you have this mostly right.
Thomas Heijligen src@posteo.de writes:
## Platforms to support
- Systems
- Linux (Distros / ChromeOS)
Long-Term Stable is a real issue. It's not really reasonable for people to expect to build new flashrom with ancient tools, on a system where they have chosen to run old code to get some kind of stability. Yet people routinely demand that they be able to build the latest release of Foo using an LTS with compiler, autoconf, cmake and the like from 5 years ago. They want new Foo, but suggesting they get non-crufty tools and they act like upstream is being unreasonable because their LTS is "supported". My respones is "well, tell RHEL/Ubuntu to do it for you and stop demanding I do it".
* SunOS / Solaris / the current version of it
There are two worlds: Oracle Solaris and illumos (OmniOS, OpenIndiana, and more). Probably doesn't need specific code for both, but good to be aware.
- Architectures
x86 supports port I/O (mostly relevant to PCI), others may through
- X86
- ARM
- MIPS
- POWER
MMIO emulation
I guess that's an interesting question. I'm not clear on the full set of interfaces for reading/writing. But really the code should be portable to any reasonable CPU. x86, arm, mips, and ppc are really pairs of CPU architectures, each of which has a 32-bit and a 64-bit variant. And it's a pretty short list; NetBSD also runs on sparc, sparc64, m68k, and less sensible for flashrom, vax, alpha, ia64, and hppa. And, RISC-V is coming and I expect it to appear whereever arm64 does in a few years.
In requirements, you left out (but I know you are thinking of cross):
cross compilation
Almost entirely, switch code on/off by feature tests, and attempt to drive the number of "#ifdef FooOS" to zero. It's not doable entirely, but autoconf got this right and a lot of things that claim they are better do not.
- Write documentation
- Build instructions for all platforms
Aside from windows, they really should not be different.
## Open questions
- System requirements
- How long should we support systems / libraries?
- As long as they have vendor support
That's very messy. On one hand if vendors (== "os maintainers") drop support early, it's good to go back slightly further. On the other, if someone decides to freeze code in 2020 and maintain it with *security patches only* for 10 years, then I think a reasonable response is "if you want to run code from 2020 with only security patches, good luck - the license gives you the freedom to do that, but it's not our obligation to do work to make that be easy". Note that such people are welcome to keep running the flashrom version from 2020; that is after all what they chose to do.
Yes, I sound cranky, but the crankiness comes from maintaining unison.
* What should be the minimum required version of meson? * The version of the oldest supported system.
Again, same issue.
- Build dependencies
- Can we fetch dependencies like cmocka? This makes building for
some systems easier
No, you can't fetch anything at build time. Packaging systems view that as a bug. It means you can't build on a machine not on the internet. Yes, I have built lots of packages on a disconnected machine after fetching their releaes tarballs and transferring on CDROM, and it almost entirely works.
So options are:
if you want to depend on something, list in in the README and demand that the user have it installed already. If it doesn't already exist as a package in most packaging systems, that's a clue you shouldn't use it.
vendor it in. But you really shouldn't do this, because then when there is a security release of that included code, you really need to have a new micro very fast, and I'll pick 7 days out of a hat.
- Shared libflashrom
- How to handle versioning?
shared library major/minor is pretty well established (major bump on ABI break, minor bump on ABI addition, and try really hard not to break the ABI). I'm not sure what the question is.
## Sequence for switching the primary build system
- Meson is feature equivalente with the Makefile
- Switch to meson as default build system, update the documentation
- Make a release with meson as the primary build system, keep the Makefile for legacy compatible.
- Drop the Makefile
- Drop dead code from Makefile only based option
Understand that packagers will need to put in work to switch, they aren't usually paid 40h/week to package, and they don't think flashrom is the most important thing in the world. So what you said means that they have the time between the release with meson documented as primary and the release w/o Makefile. If meson behaves as well as meson proponents say (which given what I've heard about the plan seems likely), that will be only a few hours or less.
As a nit there should be at least 2 months between "release where meson is primary and no significant meson problems are reported" and "release w/o Makefile".
- Define what testing means. Should we run flashrom or flash
something? How do we check that meson supports the system? 1. Build and run unit tests. 2. Run the binary to see that it works 3. Great thing would be to use one (or more :)) programmer. Maybe just dummy?
I think you need to at least read a ROM and compare, and really flash an update. But that's only when other tests are all happy and less often.
Hope this helps.
pkgsrc has cmocka 1.1.5. Telling the flashrom package that it is needed for running tests, or that it's always needed is one line in the control Makefile. Including it or downloading it is much more painful.
Hi Greg,
thanks for your feedback! I've updated the document with your suggestions.
On Wed, 2022-04-06 at 20:55 -0400, Greg Troxel wrote:
Some comments from a packager and experience with switching to cmake, some of it ok, some of it bad. I'm trimming stuff I don't have any comments on as I think you have this mostly right.
Thomas Heijligen src@posteo.de writes:
## Platforms to support * Systems * Linux (Distros / ChromeOS)
Long-Term Stable is a real issue. It's not really reasonable for people to expect to build new flashrom with ancient tools, on a system where they have chosen to run old code to get some kind of stability. Yet people routinely demand that they be able to build the latest release of Foo using an LTS with compiler, autoconf, cmake and the like from 5 years ago. They want new Foo, but suggesting they get non-crufty tools and they act like upstream is being unreasonable because their LTS is "supported". My respones is "well, tell RHEL/Ubuntu to do it for you and stop demanding I do it".
The current policy with the Makefile is to stay backwards compatible. Since meson is not this long around we have to choose a way to go. Personally I had the problem that my system meson from a rolling release distro was to old to build a project from master branch.
----------------------------------------------------------------------- ## Open questions * System requirements * How long should we support systems / libraries? * The Makefile supports really old systems in theory (not tested) * Our dependencies are quite stable (libpci, libusb1, …) * Should we support systems until their EOL? What's with LTS systems? * Long-Term stable may a issue, as wee need workarounds on our side for older versions * What should be the minimum required version of meson? * How to find a compromise to satisfy all users of flashrom? -----------------------------------------------------------------------
* SunOS / Solaris / the current version of it
There are two worlds: Oracle Solaris and illumos (OmniOS, OpenIndiana, and more). Probably doesn't need specific code for both, but good to be aware.
I didn't know the exact difference sice all systems get build as SunOS ----------------------------------------------------------------------- ## Platforms to support * Systems * SunOS, Solaris / illumos, OmniOS, OpenIndiana -----------------------------------------------------------------------
* Architectures * X86 * ARM * MIPS * POWER x86 supports port I/O (mostly relevant to PCI), others may through MMIO emulation
I guess that's an interesting question. I'm not clear on the full set of interfaces for reading/writing. But really the code should be portable to any reasonable CPU. x86, arm, mips, and ppc are really pairs of CPU architectures, each of which has a 32-bit and a 64-bit variant. And it's a pretty short list; NetBSD also runs on sparc, sparc64, m68k, and less sensible for flashrom, vax, alpha, ia64, and hppa. And, RISC-V is coming and I expect it to appear whereever arm64 does in a few years.
There is code that works only on one architecture, like x86 port I/O or msr access. Some architectures need a sync call fall mmio. When trying flashrom on a unsupported architecture this may result in unwanted behavior.
----------------------------------------------------------------------- ## Platforms to support * Architectures (32 & 64bit sets) * x86 (much exclusive codes paths) * mips (some exclusive mips code paths, processor_enable.c) * ppc (single exclusive code path, hwaccess_physmap.c) * arm * sparc (single exclusive codepath, hwaccess_physmap.c) * alpha * hppa * m68k * riscv * sh * s390 * arc * (cross compiling architectures)
## Tasks * Make sure the mmio sync is implemented for all supported architectures -----------------------------------------------------------------------
In requirements, you left out (but I know you are thinking of cross):
cross compilation
Almost entirely, switch code on/off by feature tests, and attempt to drive the number of "#ifdef FooOS" to zero. It's not doable entirely, but autoconf got this right and a lot of things that claim they are better do not.
----------------------------------------------------------------------- ## Tasks * Reduce the number of #ifdef by moving the decision to the build system -----------------------------------------------------------------------
* Write documentation * Build instructions for all platforms
Aside from windows, they really should not be different.
----------------------------------------------------------------------- ## Tasks * Write documentation * Build instructions for all platforms (especially for cross compiling only targets) -----------------------------------------------------------------------
## Open questions * System requirements * How long should we support systems / libraries? * As long as they have vendor support
That's very messy. On one hand if vendors (== "os maintainers") drop support early, it's good to go back slightly further. On the other, if someone decides to freeze code in 2020 and maintain it with *security patches only* for 10 years, then I think a reasonable response is "if you want to run code from 2020 with only security patches, good luck
the license gives you the freedom to do that, but it's not our obligation to do work to make that be easy". Note that such people are welcome to keep running the flashrom version from 2020; that is after all what they chose to do.
Yes, I sound cranky, but the crankiness comes from maintaining unison.
* What should be the minimum required version of meson? * The version of the oldest supported system.
Again, same issue.
Same as above
* Build dependencies * Can we fetch dependencies like cmocka? This makes building for some systems easier
No, you can't fetch anything at build time. Packaging systems view that as a bug. It means you can't build on a machine not on the internet. Yes, I have built lots of packages on a disconnected machine after fetching their releaes tarballs and transferring on CDROM, and it almost entirely works.
So options are:
if you want to depend on something, list in in the README and demand that the user have it installed already. If it doesn't already exist as a package in most packaging systems, that's a clue you shouldn't use it.
vendor it in. But you really shouldn't do this, because then when there is a security release of that included code, you really need to have a new micro very fast, and I'll pick 7 days out of a hat.
Sorry, that may worded inaccurately. For packaging it is realy bad to fetch things from the internet. For systems without a package manager this may be realy helpfull. Fetching things from the internet should deactivatable and only used as fallback.
----------------------------------------------------------------------- ## Open questions * Build dependencies * Can we have a fallback to fetch dependencies? (like with cmocka currently) * When available use the system version, else fall back to fetch from online * Does this work for all our dependencies? * Nice for systems without package manager (Windows, djgpp-dos- cross) * When packaging you can’t fetch anything from online * Option to enable / disable this -----------------------------------------------------------------------
* Shared libflashrom * How to handle versioning?
shared library major/minor is pretty well established (major bump on ABI break, minor bump on ABI addition, and try really hard not to break the ABI). I'm not sure what the question is.
The API of libflashrom is not stable nor feature complete, but we have users of the shared library with the unstable API.
----------------------------------------------------------------------- ## Open questions * shared libflashrom with no stable API * How to handle versioning of it? * For stable API there is a well established system major.minor.patch * What to do until the API is stable? * When is the API stable enough? * Use major = 0 until it’s stable and increment minor on each change -----------------------------------------------------------------------
## Sequence for switching the primary build system * Meson is feature equivalente with the Makefile * Switch to meson as default build system, update the documentation * Make a release with meson as the primary build system, keep the Makefile for legacy compatible. * Drop the Makefile * Drop dead code from Makefile only based option
Understand that packagers will need to put in work to switch, they aren't usually paid 40h/week to package, and they don't think flashrom is the most important thing in the world. So what you said means that they have the time between the release with meson documented as primary and the release w/o Makefile. If meson behaves as well as meson proponents say (which given what I've heard about the plan seems likely), that will be only a few hours or less.
As a nit there should be at least 2 months between "release where meson is primary and no significant meson problems are reported" and "release w/o Makefile".
The release cycles of flashrom are quite irregular and slow. There should be enough time to adopt. The release brach should also keep the old Makefile. Only the release after that will drop the Makefile.
----------------------------------------------------------------------- ## Sequence for switching the primary build system * Meson is feature equivalente with the Makefile (master branch) * Switch to meson as default build system, update the documentation (master branch) * Beta phase to get feedback (2 month or more) (master release branch) * Make a release with meson as the primary build system, keep the Makefile for legacy compatible. (release branch) * Drop the Makefile (master branch only) * Drop dead code from Makefile only based option (master branch only) -----------------------------------------------------------------------
5. Define what testing means. Should we run flashrom or flash something? How do we check that meson supports the system? 1. Build and run unit tests. 2. Run the binary to see that it works 3. Great thing would be to use one (or more :)) programmer. Maybe just dummy?
I think you need to at least read a ROM and compare, and really flash an update. But that's only when other tests are all happy and less often.
I've moved the testing topic to "Further Projects". This is important, but to much and not quite the scope of this project.
Hope this helps.
This was great help. Thanks a lot!
-- Thomas
flashrom mailing list -- flashrom@flashrom.org To unsubscribe send an email to flashrom-leave@flashrom.org
Thomas Heijligen src@posteo.de writes:
The current policy with the Makefile is to stay backwards compatible.
That's fine because make is stable, even the GNU flavor. There really shouldn't be anything in a makefile that is an issue for 5 year old make. The dependency requirements are a different matter, and not different from make to meson.
Since meson is not this long around we have to choose a way to go. Personally I had the problem that my system meson from a rolling release distro was to old to build a project from master branch.
That's a clue that something is wrong. I would pull a date out of a hat and say that the meson that was the current formal release on 2020-01-01 should be good enough. So that means the build system should work with meson 0.52.1 or higher.
Another view is that Ubuntu 20.04's included meson ought to be good enough. With any luck that's at least 0.52.1.
-
Greg Troxel -
Thomas Heijligen