[coreboot] Changes to the coreboot Project Structure

[Paul Menzel]

Dear Stefan, dear coreboot folks,


Am Donnerstag, den 20.03.2014, 22:55 +0100 schrieb Stefan Reinauer:
> Changes to the coreboot Project Structure
> 
> We -- the coreboot project -- have succeeded beyond our wildest
> expectations, with every major laptop vendor using our code. Going
> forward, this will require a few structural changes that will ensure
> that the vendors shipping coreboot products can count on a stable,
> reliable code base.
> 
> With that incredible success, there is also a lot of work to be done to
> make sure the project is scaling and keeping up with the requirements of
> the millions of new users that coreboot gets every year. There is a
> large number of new corporate entities investigating their possible
> involvement in coreboot, and we need to make sure that their integration
> is working smoothly while still keeping the project’s interests and
> goals stable and alive.

thank you once again for bringing up this discussion on the list!

I’ll reply in a separate thread about coreboot’s interests and goals.

> Moving forward, we need to make sure that we can welcome these possible
> new contributors without the friction that we have seen in the past.

Could you please name the friction seen in the past?

> The coreboot community currently consists of about 80 contributors with
> varying levels of activity. Among these contributors, three groups can
> be identified as the major stakeholders in the project: Sage Electronic
> Engineering (the largest coreboot IBV),

From a community perspective there is currently not a lot of interaction
between the community and Sage Electronic Engineering. The last commits
in the upstream repository are also a few months back.

> Secunet and Google Inc. (delivering all new Chrome OS devices with
> coreboot based firmware).

Did you count AMD contributions to Sage Electronic Engineering?

> Individuals employed by these three stakeholders make up for the
> majority of code contributions. According to Ohloh, roughly half of the
> coreboot contributions are done by the top ten contributors. Over the
> project life time, over 80% of those have been made by corporate
> contributors.

Do you have an URL? As always such statistics should be taken with a
grain of salt. Especially how AGESA is counted for example and regarding
that a lot of board code is copied over and is not unified.

> While there is a fairly good understanding of the project direction and
> interest between those individual contributors, there are also an
> increasing need for making sure that the coreboot project’s scalability
> and its viability for mobile / consumer products and servers is
> guaranteed.

As written above, to me the direction and interest is not so clear to
me. (See my (future) reply in a separate thread.)

Also the reasons for the need is not so clear. Could you please
elaborate on that?

The only thing I imagine is the long time it takes to upstream a board
for Google because they do not use the upstream infrastructure right
away.

> The goal of this proposal is to enable all of the community
> to have a strong voice and make strong contributions to the project
> while allowing the major stakeholders to steer the project direction and
> pursue ownership of the components they provide.

I thought that is the case currently.

> Traditionally the development model of coreboot was very similar to the
> model of the Linux kernel in many ways, including coding guidelines, the
> requirement of a sign-off process for contributions, and active
> leadership provided by a “benevolent dictator”. Commit rights were
> traditionally given to few of the top contributors of the project.
> With the introduction of git as coreboot’s version control system and
> its multi-branch nature as well as gerrit for code reviews, the
> landscape of the project has slightly shifted towards a more anocratic
> development model in which different interest groups sometimes worked on
> opposing strategies, and thus affecting the project’s overall stability
> and suitability for large scale deployment negatively. The advent of
> these new tools requires the original project founders and major
> stakeholders to provide stronger leadership and strategic direction for
> the project.

Could you please give examples?

My view is, that the problem is simply missing communication. If the
companies develop something in secret without announcing and discussing
their plans and miss what the community does, then this is solved by
improving the communication with the community. Also in the past, my
impression is, that (almost(?)) always the community gave in and
reworked their patches and adapted the “corporate” code and improved
that.

> Measures to improve the coreboot Development Model
> 
> * Require authors to acknowledge changes made in their name (Forge-Author)
>   This change allows contributors to decide when their OWN changes are
>   ready for being integrated into the project, preventing unfinished and
>   experimental work to be submitted accidentally.
> 
>   Status: ACTIVELY ENFORCED BY GERRIT

In my opinion, this change was made due to one incident, which would not
have been happened afterward as people were aware of this problem. But
it is how it is.

> * Define owners for (sets of) mainboards and require these to act as
>   maintainers / gatekeepers, being the controlling instance to submit
>   these mainboard changes.
>   Providing support for a new mainboard / chipset / component in coreboot
>   is a major contribution that requires the contributors to invest dozens
>   / hundreds / thousands of man hours. In the light of this we would like
>   to give those who provide support for one of these components stronger
>   ownership. The submitters of a component need to have the last word
>   about what becomes part of their component and what does not. This will
>   also give component providers a certain freedom of the actual
>   implementation of the component, that might, in some rare cases, be
>   different from the implementational structure of the framework and
>   generic code.

Where has this been a problem in the past? Please give specific
examples.

Why is it necessary to divert? If it is needed and has advantages people
won’t oppose anyhow even currently. But there have to be reason for
diverting from the implementational structure of the framework and
generic code. Otherwise the question is if this is only be happening in
rare cases or going to be the norm.

(Also the maintainers/gatekeepers should (of course) have to adhere to
current policies.)

How is that going to change anything anyhow? Do you believe the title is
going to make the community developers spent more time on coreboot. My
impression currently is they spent as much of their free time with
coreboot already.

>   Status: TO BE ENFORCED BY POLICY
> 
> * Build a MAINTAINERS file for common code, and encourage people to keep
>   subsystem maintainers in the loop for changes
>   Aiming for top notch code quality, the coreboot project is generally
>   trying to provide a solid and scalable framework for development as well
>   as a number of generic cross-chipset components. Changes to these parts
>   of the coreboot code affect a large number of supported systems. Hence
>   it is important to have gatekeepers in place to guarantee they stay
>   operational.

I claim it is currently done the exact same way without having/needing a
MAINTAINERS file.

>   Status: TO BE ENFORCED BY POLICY
> 
> * Look into making gerrit automatically add reviewers based on maintainer
>   lists
>   In order to streamline the code review process in a project that on
>   average now has over 200 contributions per month, maintainers / owners /
>   gatekeepers should be added to the list of code reviewers automatically.
> 
>   Status: TO BE INVESTIGATED

That sounds good. But I had the feeling people know which people to add
as reviewers.

> * Import previous code reviews and results
>   The tree major stakeholders in the project all own internal code review
>   systems, some of which are public (e.g. the Google Chrome OS repository)
>   and have code reviews that include representatives outside of Google to
>   ensure general code quality and making sure the improvements made for
>   products don’t negatively impact upstreamability. For those cases it
>   would be extremely helpful to honor the code reviews already done in the
>   upstream repository
> 
>   Status: TO BE INVESTIGATED

As Kyösti pointed out, there is unfortunately the need to do the extra
review, as it is apparent that for example the Google patches are mostly
(of course) only developed and tested with their boards so it has to be
made sure they do not decrease the overall quality, do not affect other
boards/components and do not increase the maintenance burden for the
community. Currently it is the way, that Google only upstreams after the
code already ships and then goes on with the next product, while not
maintaining the old boards anymore.

> * Significantly reduce number of submitters
>   To ensure consistency, scalability and conformity with the general
>   coreboot strategy, we need to define a clear committer structure that
>   defines the original project structure of having a benevolent dictator
>   aka project leader (Stefan Reinauer) and gatekeepers in place. The
>   suggested structure will need to value both community interests and
>   corporate interests equally and hence a good setup would be to have a
>   final amount of six developers with submit rights, three community
>   representatives and three corporate representatives (on from each major
>   stakeholder):

I also do not see, why this is necessary and the current state has to be
changed. Ron stated in the past that this is due to commits being pushed
breaking things. But that was not the case in the last two or three
years. And if it was, it were experienced submitters breaking the tree,
which of course is more likely to happen when you do more reviews/write
more code.

As Kyösti already asked, please provide the commit hashes of the commits
that broke the tree and making you come up with the proposal.

Regarding the dictator, in my opinion the “dictator” main job is
communication and not writing code as Linus Torvalds is doing it for
Linux for example. In that regard, you were pretty silent (at least on
the list, which is the medium for communication) especially since you
stared working for Google. Ron is a little more active on the list, but
also a little biased towards your employer, which is to be expected.

I guess the position for the dictator is not up for a vote, as you own
the infrastructure anyhow I guess.

Patrick is very active in the IRC channel #coreboot (on
<irc.freenode.net>) and a coreboot veteran.

In the end I did not miss a dictator the last three years and thought
that a solution could always be found.

>   Current suggestions:
> 
>   Patrick Georgi (Secunet)
>   Marc Jones (Sage)
>   Stefan Reinauer (Google)

In my opinion, Aaron Durban did a great job in the past year(?) keeping
in touch with the community and addressing the concerns.

>   Peter Stuge, Kyösti Mälkki, N.N. (non-commercial part of the
>   coreboot community)

The community has very capable people for this kind of job in my
opinion. Vladimir and Alexandru come to mind.

>   Status: TO BE IMPLEMENTED

I agree with others that the problems should be elaborated more and then
solutions for them have to be thought of. From the current messages it
is not clear what lead to the proposals. If the analysis of the problems
is incorrect the solutions/proposals probably won’t solve the real
problems or improve the situation.

So to summarize, in my opinion, there are the issues of upstreaming
patches from Google which costs Google and the community a lot of time,
money and developer’s nerves and missing communication/interaction
between the community and companies. Other than that my impression is
that the current process works fine.

For example it was never announced when Google added new tags to commit
messages. It is a problem for community developers to look at changes in
the Chrome OS repository and it is wasting their time to jump between
different Web sites.

Maybe the infrastructure can be adapted too and also more automatic
tests (tests of more Kconfig options) and checks be implemented. This
would give developers more time on reviewing the actual code than
looking at the trivial policy violations.

It should also be though of making some of the internal Google policies
an upstream policy too like having to put that TEST tag in there.

Some solutions would be for Google (and Sage, AMD) (and probably cheaper
than the spent developer time) to give out devices for free to
developers so that these are maintained and tested even when the company
went on to the next device. (See the board-status repository how many
Google devices have been tested although Ron pushed for that
board-status repository.)


Thanks,

Paul
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://www.coreboot.org/pipermail/coreboot/attachments/20140324/aef1352b/attachment-0001.sig>