[coreboot] release engineering process

[Peter Stuge]

ron minnich wrote:
> I like the idea of releases, and tags,

Let's please stick to a discussion where we use abstract terms to
talk about *what* we want a process to do.

*How* to implement that process needs to be left for later, after
there is agreement on the *what*. Implementation should also not be
discussed in the big round, because it neccessarily requires dealing
with all the implementation details - which are not relevant for
deciding on the *what* of the process.


> feel strongly that the process has to have some teeth

Everybody in coreboot breaks things for someone else, ALL THE TIME.

This happens because developers are ignorant, because they are lazy,
because they are sloppy, or because they just don't have time,
because the deadline has already passed.

No process in coreboot can compensate for that. Especially not a
process where teeth do not bite until 3 months later.

Review is supposed to compensate for it. But it is clear to me that
coreboot developers do not commit to meaningful review. I have seen
everyone submit commits without understanding every bit of them - ie.
submitting without having actually reviewed. I've done it too.

In my humble opinion, that is the problem. People change things they
do not (yet) understand, all the time. This isn't only accepted
behavior, it is even encouraged.

It obviously does not help to have millions of lines of code in tree
which no human in the project at all understands. And it does not
help to have eager developers frantically work on new things without
really careful consideration of all accumulated domain knowledge
already codified throughout the entire source tree.

It's the Linux vs. FreeBSD development models. There are fanboys and
fangirls for either. Actually either one of them would be a great
improvement over the commit landfill that is currently coreboot.git.

As Patrick pointed out, we are unable to test coreboot in a fashion
where an automatic process can truly promise that a given board works
on hardware. Chrome experience shows that even with significant
testing, stressing the code can have everything looking great for a
long time, until there is some weirdo hardware thing which doesn't
show up until after weeks of reboots, and then developers are right
back at the drawing board.


> qualification for being in the release besides "ran 3 years ago."

Generating releases is a trivial mechanical process, no matter what
the criterias for inclusion are.

The difficult question is what those QA criteria (qualification)
shall look like. I honestly don't think that the disparate interests
represented within coreboot can reach agreement on that, but if they
actually can then I think it will probably not happen via email.


When "let's ship it" testing for a commercial product requires weeks
of testing it is obviously infeasible to qualify individual commits.

If individual commits can neither be tested for regressions nor are
carefully reviewed then commits are guaranteed to introduce problems.

What qualification do you want for yourself?

I want for myself that *I* have reviewed every commit, because I
don't feel that the community does a good job. This obviously does
not scale beyond each individual use case.

I'm afraid I don't have any good suggestion. :\


//Peter