[flashrom] Future flashrom development (Stefan's PoV I)

[Stefan Tauner]

Hi everybody,

I think it's best to put the whole story into context by giving a
"brief" history of how the current state emerged (from my PoV).
This first email explains the story up to fall 2016 and what the
problems were that I tried to fix (and then I'll have to run and will
continue later).

In October 2014 there was a coreboot (and flashrom) dev meeting in
Prague where some of the most important stakeholder/devs of flashrom
were present:
 - Carl-Daniel who at the time was already way less involved than up to
   about 2012.
 - David who was one of the main developers of the chromiumos fork of
   flashrom.
 - Nico who had a relatively small number of non-merged changes related
   to his work (at Secunet).
 - Myself who handled large portions of the upstream development the
   last few years.

While there was a bunch of patches that have been piled up back then it
was less of a problem then the increasing divergence between the
chromiumos fork and upstream. Thus we have discussed ways to converge
that (by pulling changes mainly from upstream into chromium but also
vice versa) and also increase the pace of merging stuff into upstream
later. This was still with no intention to switch to git because of
Carl-Daniel's concerns.

My workload at my day job increased quite much over time while
transitioning from being a student to a full-time researcher and
lecturer; Carl-Daniel got two kids, started his PhD and spent less and
less time in front of the (dev) screen... the usual downward spiral of
many FOSS projects I would say.

Carl-Daniel is also big part of the reason why flashrom has not
switched from svn to git earlier because he is very fond of the
monotonic and human-readable revision strings of svn. While I see the
point in that I think git is way too superior in the daily work process
but since git-svn worked so well (I did touch the repo less than a
dozen timer with svn natively) this was not a major issue for myself.
However, his contributions shrunk with time, everybody I spoke with was
very clearly pro git (including random people creating github
repositories with flashrom trees) and infrastructure maintenance
burdens I derived a plan to convince even Carl-Daniel to support the
switch.

By that time it had also become clear that development was quite stuck:
100s of the previous patches (including many of my own) were bitrotting
while waiting for review and new contributors were kinda repelled by
this state. There was also little hope that some newcomer could take
over like I did in 2011. We had a kinda promising GSoC student but that
did not work out either...
Therefore when planing the switch to git I also proposed a change to
the development process.
Till then we had "a very flat and straightforward development model" -
at least from the outside. Indeed there was only a single branch that
got commits after they were (rigorously) reviewed by at least one of the
regular contributors but I have about 100 local branches, there is the
chromiumos fork etc.

The main issue regarding the development process, however, is that
Carl-Daniel and myself have very strict ideas of what code quality we
want to release to ordinary users that often try stupid things with
flashrom (-p internal). However, this requires at least some reviews,
discussion and testing, which took too much time in the past. This
clashes very much with the expectations of the other type of users who
are developers themselves of at least tinkering/recovering/hacking
stuff and mostly use external programmers. They can live with bugs and
bricks because that's most often what they have to begin with. On the
other hand they would rather like to see new features that help with
their respective use cases and tools.

The plan I conceived to fix both was more or less what is written down
in the wiki at the moment:
 - a public staging tree where new patches get pushed if they make
   somewhat sense and go into the right direction. They would ripe
   there, get fixes applied as soon as someone looks closer or bugs are
   noticed while testing/using that branch. Naturally this would be
   what the devs and adventurous users checkout.
 - a stable branch that more or less mimics the svn's trunk branch in
   terms of quality: patches that seem stable enough get cherry-picked/
   squashed with their fixes onto it possibly with some additional small
   fixes as we see fit. This would also be the branch that gets tagged,
   were version branches for backported fixes would be branched off and
   which end up in distribution releases as well.

To convince Carl-Daniel of switching to git and supporting that
development strategy I invented a versioning scheme that worked around
his strongest reservations against git: the completely unreadable
hashes as revision indicators. The basic scheme would be:
<tag>-<# commits to upstream branch>-<upstream branch>-<# commits to commit>-<sha>(-dirty)
resulting in version strings like v0.9.99-3-stable-4-e4f6643-dirty
which would indicate quite clearly where it is situated in the tree (4
commits from stable with some local changes and that stable branch
got 3 commits since the last tag).

Keep in mind that up to that point there were no plans to switch away
from mailing-list based patch submission and reviewing since this was
(and still is) Carl-Daniel's and my favorite way to deal with things.

At that point I did not foresee a major change in the workflow,
responsibilities and people involved. The regulars and especially myself
would be spending most of the time integrating patches into stable
(with the usual discussion/acking if need be) and/or fixing issues on
the staging branch. Regularly merging changes that were applied to
stable but not staging into the latter would keep divergence within
reasonable limits. In some extreme cases changes in staging could also
be completely reverted if they are discovered to be a dead end or make
things worse than expected.

The following picture sums up the process and versioning scheme:
http://buildbot.flashrom.org/dev_tree.png

Carl-Daniel agreed, I wrote it down in the wiki and implemented the
needed changes in the build system (makefile, getrevision script and
user-side git hooks).
Additionally, I also saw the opportunity to rewrite the repository
history to include the earliest of flashrom's commits that were not
integrated in the svn tree (going back to early 2002 instead of late
2003), fix lots of typos and nonsense in old commit messages etc.
This lead to a script that automatically merged these two trees and
fixed the commit messages.
In parallel I worked on a buildbot that is able to at least build test
many of the possible configurations/target architectures and OSes of
flashrom, cf.
http://buildbot.flashrom.org/buildresults/flashrom-000786-HKL/results.html
This was meant to help with finding build-time issues in the staging
branch.
Of course we also need a repository and since there are also
reservations against relying on 3rd party services like github I set up
a cgit repo (with initial help from Patrick) and implemented various
repo-side hooks for mail and IRC announcements, github mirroring and
sanity checks.

At the time this was finished (about late 2016) we were kinda ready to
switch from my point of view. In the next email I'll explain what (not)
happened since then.

-- 
Kind regards/Mit freundlichen Grüßen, Stefan Tauner