Hi Anastasia and other flashrom folks,
we had some brainstorming today about GSoC project ideas.
I can't access my Google account right now, so failed to edit the original doc[1]. Maybe somebody can copy the following ideas there, or we just put the document on Gerrit (doesn't have to be submitted) for the moment?
It's a first revision, written in a hurry, so feel free to fix/ adapt anything ;) Also, some of the smaller topics seem too short/boring. Maybe we could combine some into a refactoring project?
Nico
Optimize Erase-Function Selection =================================
To change any 0 to a 1 in the contents of a NOR-flash chip, a full block of data needs to be erased. So to change a single byte, it is sometimes required to erase a whole block, which we'll call erase overhead. Most flash chips support multiple erase-block sizes. Flashrom keeps a list of these sizes and ranges for each supported flash chip and maps them to internal erase functions. Usually these lists are sorted by ascending size.
Traditionally, Flashrom tries all available erase functions for a flash chip during writes. Usually only the first function is used, but if anything goes wrong on the wire, or the programmer rejects a function, it falls back to the next. As the functions are sorted by size, this results in a minimum erase overhead.
However, if big portions of the flash contents are re-written, using the smallest erase-block size results in transaction overhead, and also erasing bigger blocks at once often results in shorter erase times inside the flash chip.
So there is a tradeoff between picking smaller and larger blocks which heavily depends on the amount of data that is changed. With some preparation and a re-write of the loop over the erase-functions, it seems possible to optimize for both cases, the few, scattered changes and long, continuous changes:
* Adapt the programmer API so we can check in advance if a programmer supports an erase-function. * Ensure the erase-function list is sorted, filter it using the new API, and then for each change hunk, pick the function with the smallest erase overhead (favoring functions for bigger blocks if the overhead is equal).
Duration: medium
Requires a skilled C programmer.
Fix Endianness Issues =====================
Over the years, flashrom picked up support for some in-flash partitioning formats (e.g. Intel Flash Descriptor, coreboot's FMAP). These have a (hopefully) defined endianness. But it may differ from the device that runs flashrom.
* Add a unified API to read data endian-safe (a draft exists on Gerrit). * Port the existing code in Flashrom to use this API. * Integrate IFWI (another format by Intel) using the new API (patches exist, but likely need to be updated).
Duration: low/medium
Integration work for CI Containers ==================================
Flashrom supports a massive amount of platforms, both hardware and OS wise. For some time now, we try to get more test-coverage for non-Linux, non-x86 platforms (at least for building Flashrom!). One approach is manibuilder (found under util/manibuilder/ in the source tree): It tries to integrate many Docker containers, and works so far to build-test commits on Gerrit.
However, many things are lacking. For instance,
* Source-code pull only works over IP and is currently hard-coded to `review.coreboot.org`. To test local changes, two ideas exists to get them into the containers: 1. Set up another container with access to the local files that runs an HTTP/Git server. 2. Give every container access to the local files. This could barely be called work for a Linux container, however if we run another OS in QEMU, for instance, file access becomes a lot harder (possible solution: 9pfs).
* To test non-Linux OS's, we currently run QEMU inside the container. This could be integrated much better, for instance the container could be built such that it contains the bare files of the target OS (i.e. not just a disk image with another file system in it). QEMU could then be run transparently so that one can issue normal Docker commands like `run` that automatically target the emulated OS.
* Some popular OS's are currently untested and containers for them would be very welcome, e.g. FreeBSD, OpenBSD, Mac (if that's possible license-wise, might need to be run on Mac hardware).
Duration: open-ended?
Update Meson Integration ========================
We have a `meson.build` but it looks like it only targets Linux on x86. Would be nice to get it fully functional, including cross-compilation.
Duration: ?
Restructure Shutdown Functions ==============================
Flashrom code has an API to register arbitrary functions to be run when flashrom exits. This has led to a rather unstructured code flow that is hard to keep track of. While the API is generally useful, its usage should be restricted. This is something that will likely need further discussion with the whole community. Some things that seem desirable:
* Have a defined shutdown function in the programmer API. Instead of registering a function to be called later, we would call it directly when we are done with a programmer. * There are several wrapper functions to alter hardware state that automatically register a shutdown function to undo what they did (cf. rmmio/rpci/rphysmap/rget_io_perms). They currently rely on global state and as they are not used very often, it might be easier to drop them.
Duration: medium/long
Revise serial/serprog Code ==========================
The serprog code and generally the serial-terminal support code needs an update and testing on non-Linux/non-Windows platforms. For further maintenance it's desirable to extract the per-platform code into separate files. Platforms that don't work currently should be fixed.
Duration: short/medium
Add Mac M1 support ==================
Building flashrom for MacOS is already possible, however the combination of MacOS and ARM is untested. There are generally three classes of programmers to support:
* External programmers connected via serial or USB. (might work OOB) * Internal PCI-based programmers, e.g. targeting the OptionROM of a NIC/graphics card. * The `internal` programmer, targeting flash ROMs on the mainboard.
At least the latter most likely requires a driver, like the unmaintained DirectHW[1] and also specific support for the M1 SoCs.
Duration: flexible, depending on the programmer classes
Requires access to M1 Mac hardware. Potentially experience in low-level programming.
[1] https://review.coreboot.org/plugins/gitiles/directhw/+/refs/heads/master
Nico and everyone who did the brainstorming,
Thank you so much!! This is amazing, and exactly what we need at the moment!
As a first step, I merged everything into our ideas doc https://docs.google.com/document/d/1AxMobB2v8Dv2uUwZPZ_vCmmONYDJuliHcnjfWOs4qIs/edit?usp=sharing, and as a result we have 11 ideas (perfect!). We can refine/rephrase/correct in the doc: I haven't done any changes yet, just copied and formatted the doc. I also linked ideas https://docs.google.com/document/d/1AxMobB2v8Dv2uUwZPZ_vCmmONYDJuliHcnjfWOs4qIs/edit?usp=sharing & proposal template https://docs.google.com/document/d/1DSg1FykuI7Z3JDY1Qtk8C6JjvpsYISMEwyV4932jtBI/edit?usp=sharing into https://www.flashrom.org/GSoC, so now info is available to anyone interested. As a bonus point I added GSoC page into Developers bottom menu, so now it is referenced on the Home page https://www.flashrom.org/Flashrom.
I will re-read list of ideas once again later. And in general, feedback is very welcome! There are 3 more weeks to prepare the application.
Have a nice day, Anastasia.
On Thu, Jan 13, 2022 at 6:18 AM Nico Huber nico.h@gmx.de wrote:
Hi Anastasia and other flashrom folks,
we had some brainstorming today about GSoC project ideas.
I can't access my Google account right now, so failed to edit the original doc[1]. Maybe somebody can copy the following ideas there, or we just put the document on Gerrit (doesn't have to be submitted) for the moment?
It's a first revision, written in a hurry, so feel free to fix/ adapt anything ;) Also, some of the smaller topics seem too short/boring. Maybe we could combine some into a refactoring project?
Nico
Optimize Erase-Function Selection
To change any 0 to a 1 in the contents of a NOR-flash chip, a full block of data needs to be erased. So to change a single byte, it is sometimes required to erase a whole block, which we'll call erase overhead. Most flash chips support multiple erase-block sizes. Flashrom keeps a list of these sizes and ranges for each supported flash chip and maps them to internal erase functions. Usually these lists are sorted by ascending size.
Traditionally, Flashrom tries all available erase functions for a flash chip during writes. Usually only the first function is used, but if anything goes wrong on the wire, or the programmer rejects a function, it falls back to the next. As the functions are sorted by size, this results in a minimum erase overhead.
However, if big portions of the flash contents are re-written, using the smallest erase-block size results in transaction overhead, and also erasing bigger blocks at once often results in shorter erase times inside the flash chip.
So there is a tradeoff between picking smaller and larger blocks which heavily depends on the amount of data that is changed. With some preparation and a re-write of the loop over the erase-functions, it seems possible to optimize for both cases, the few, scattered changes and long, continuous changes:
- Adapt the programmer API so we can check in advance if a programmer supports an erase-function.
- Ensure the erase-function list is sorted, filter it using the new API, and then for each change hunk, pick the function with the smallest erase overhead (favoring functions for bigger blocks if the overhead is equal).
Duration: medium
Requires a skilled C programmer.
Fix Endianness Issues
Over the years, flashrom picked up support for some in-flash partitioning formats (e.g. Intel Flash Descriptor, coreboot's FMAP). These have a (hopefully) defined endianness. But it may differ from the device that runs flashrom.
- Add a unified API to read data endian-safe (a draft exists on Gerrit).
- Port the existing code in Flashrom to use this API.
- Integrate IFWI (another format by Intel) using the new API (patches exist, but likely need to be updated).
Duration: low/medium
Integration work for CI Containers
Flashrom supports a massive amount of platforms, both hardware and OS wise. For some time now, we try to get more test-coverage for non-Linux, non-x86 platforms (at least for building Flashrom!). One approach is manibuilder (found under util/manibuilder/ in the source tree): It tries to integrate many Docker containers, and works so far to build-test commits on Gerrit.
However, many things are lacking. For instance,
Source-code pull only works over IP and is currently hard-coded to `review.coreboot.org`. To test local changes, two ideas exists to get them into the containers:
- Set up another container with access to the local files that runs an HTTP/Git server.
- Give every container access to the local files. This could barely be called work for a Linux container, however if we run another OS in QEMU, for instance, file access becomes a lot harder (possible solution: 9pfs).
To test non-Linux OS's, we currently run QEMU inside the container. This could be integrated much better, for instance the container could be built such that it contains the bare files of the target OS (i.e. not just a disk image with another file system in it). QEMU could then be run transparently so that one can issue normal Docker commands like `run` that automatically target the emulated OS.
Some popular OS's are currently untested and containers for them would be very welcome, e.g. FreeBSD, OpenBSD, Mac (if that's possible license-wise, might need to be run on Mac hardware).
Duration: open-ended?
Update Meson Integration
We have a `meson.build` but it looks like it only targets Linux on x86. Would be nice to get it fully functional, including cross-compilation.
Duration: ?
Restructure Shutdown Functions
Flashrom code has an API to register arbitrary functions to be run when flashrom exits. This has led to a rather unstructured code flow that is hard to keep track of. While the API is generally useful, its usage should be restricted. This is something that will likely need further discussion with the whole community. Some things that seem desirable:
- Have a defined shutdown function in the programmer API. Instead of registering a function to be called later, we would call it directly when we are done with a programmer.
- There are several wrapper functions to alter hardware state that automatically register a shutdown function to undo what they did (cf. rmmio/rpci/rphysmap/rget_io_perms). They currently rely on global state and as they are not used very often, it might be easier to drop them.
Duration: medium/long
Revise serial/serprog Code
The serprog code and generally the serial-terminal support code needs an update and testing on non-Linux/non-Windows platforms. For further maintenance it's desirable to extract the per-platform code into separate files. Platforms that don't work currently should be fixed.
Duration: short/medium
Add Mac M1 support
Building flashrom for MacOS is already possible, however the combination of MacOS and ARM is untested. There are generally three classes of programmers to support:
- External programmers connected via serial or USB. (might work OOB)
- Internal PCI-based programmers, e.g. targeting the OptionROM of a NIC/graphics card.
- The `internal` programmer, targeting flash ROMs on the mainboard.
At least the latter most likely requires a driver, like the unmaintained DirectHW[1] and also specific support for the M1 SoCs.
Duration: flexible, depending on the programmer classes
Requires access to M1 Mac hardware. Potentially experience in low-level programming.
[1] https://review.coreboot.org/plugins/gitiles/directhw/+/refs/heads/master
-
Anastasia Klimchuk -
Nico Huber