Dear Stefan, dear coreboot folks,
Am Donnerstag, den 20.03.2014, 22:55 +0100 schrieb Stefan Reinauer:
Changes to the coreboot Project Structure
We -- the coreboot project -- have succeeded beyond our wildest expectations, with every major laptop vendor using our code. Going forward, this will require a few structural changes that will ensure that the vendors shipping coreboot products can count on a stable, reliable code base.
With that incredible success, there is also a lot of work to be done to make sure the project is scaling and keeping up with the requirements of the millions of new users that coreboot gets every year. There is a large number of new corporate entities investigating their possible involvement in coreboot, and we need to make sure that their integration is working smoothly while still keeping the project’s interests and goals stable and alive.
thank you once again for bringing up this discussion on the list!
I’ll reply in a separate thread about coreboot’s interests and goals.
Moving forward, we need to make sure that we can welcome these possible new contributors without the friction that we have seen in the past.
Could you please name the friction seen in the past?
The coreboot community currently consists of about 80 contributors with varying levels of activity. Among these contributors, three groups can be identified as the major stakeholders in the project: Sage Electronic Engineering (the largest coreboot IBV),
From a community perspective there is currently not a lot of interaction between the community and Sage Electronic Engineering. The last commits in the upstream repository are also a few months back.
Secunet and Google Inc. (delivering all new Chrome OS devices with coreboot based firmware).
Did you count AMD contributions to Sage Electronic Engineering?
Individuals employed by these three stakeholders make up for the majority of code contributions. According to Ohloh, roughly half of the coreboot contributions are done by the top ten contributors. Over the project life time, over 80% of those have been made by corporate contributors.
Do you have an URL? As always such statistics should be taken with a grain of salt. Especially how AGESA is counted for example and regarding that a lot of board code is copied over and is not unified.
While there is a fairly good understanding of the project direction and interest between those individual contributors, there are also an increasing need for making sure that the coreboot project’s scalability and its viability for mobile / consumer products and servers is guaranteed.
As written above, to me the direction and interest is not so clear to me. (See my (future) reply in a separate thread.)
Also the reasons for the need is not so clear. Could you please elaborate on that?
The only thing I imagine is the long time it takes to upstream a board for Google because they do not use the upstream infrastructure right away.
The goal of this proposal is to enable all of the community to have a strong voice and make strong contributions to the project while allowing the major stakeholders to steer the project direction and pursue ownership of the components they provide.
I thought that is the case currently.
Traditionally the development model of coreboot was very similar to the model of the Linux kernel in many ways, including coding guidelines, the requirement of a sign-off process for contributions, and active leadership provided by a “benevolent dictator”. Commit rights were traditionally given to few of the top contributors of the project. With the introduction of git as coreboot’s version control system and its multi-branch nature as well as gerrit for code reviews, the landscape of the project has slightly shifted towards a more anocratic development model in which different interest groups sometimes worked on opposing strategies, and thus affecting the project’s overall stability and suitability for large scale deployment negatively. The advent of these new tools requires the original project founders and major stakeholders to provide stronger leadership and strategic direction for the project.
Could you please give examples?
My view is, that the problem is simply missing communication. If the companies develop something in secret without announcing and discussing their plans and miss what the community does, then this is solved by improving the communication with the community. Also in the past, my impression is, that (almost(?)) always the community gave in and reworked their patches and adapted the “corporate” code and improved that.
Measures to improve the coreboot Development Model
Require authors to acknowledge changes made in their name (Forge-Author) This change allows contributors to decide when their OWN changes are ready for being integrated into the project, preventing unfinished and experimental work to be submitted accidentally.
Status: ACTIVELY ENFORCED BY GERRIT
In my opinion, this change was made due to one incident, which would not have been happened afterward as people were aware of this problem. But it is how it is.
- Define owners for (sets of) mainboards and require these to act as maintainers / gatekeepers, being the controlling instance to submit these mainboard changes. Providing support for a new mainboard / chipset / component in coreboot is a major contribution that requires the contributors to invest dozens / hundreds / thousands of man hours. In the light of this we would like to give those who provide support for one of these components stronger ownership. The submitters of a component need to have the last word about what becomes part of their component and what does not. This will also give component providers a certain freedom of the actual implementation of the component, that might, in some rare cases, be different from the implementational structure of the framework and generic code.
Where has this been a problem in the past? Please give specific examples.
Why is it necessary to divert? If it is needed and has advantages people won’t oppose anyhow even currently. But there have to be reason for diverting from the implementational structure of the framework and generic code. Otherwise the question is if this is only be happening in rare cases or going to be the norm.
(Also the maintainers/gatekeepers should (of course) have to adhere to current policies.)
How is that going to change anything anyhow? Do you believe the title is going to make the community developers spent more time on coreboot. My impression currently is they spent as much of their free time with coreboot already.
Status: TO BE ENFORCED BY POLICY
- Build a MAINTAINERS file for common code, and encourage people to keep subsystem maintainers in the loop for changes Aiming for top notch code quality, the coreboot project is generally trying to provide a solid and scalable framework for development as well as a number of generic cross-chipset components. Changes to these parts of the coreboot code affect a large number of supported systems. Hence it is important to have gatekeepers in place to guarantee they stay operational.
I claim it is currently done the exact same way without having/needing a MAINTAINERS file.
Status: TO BE ENFORCED BY POLICY
Look into making gerrit automatically add reviewers based on maintainer lists In order to streamline the code review process in a project that on average now has over 200 contributions per month, maintainers / owners / gatekeepers should be added to the list of code reviewers automatically.
Status: TO BE INVESTIGATED
That sounds good. But I had the feeling people know which people to add as reviewers.
Import previous code reviews and results The tree major stakeholders in the project all own internal code review systems, some of which are public (e.g. the Google Chrome OS repository) and have code reviews that include representatives outside of Google to ensure general code quality and making sure the improvements made for products don’t negatively impact upstreamability. For those cases it would be extremely helpful to honor the code reviews already done in the upstream repository
Status: TO BE INVESTIGATED
As Kyösti pointed out, there is unfortunately the need to do the extra review, as it is apparent that for example the Google patches are mostly (of course) only developed and tested with their boards so it has to be made sure they do not decrease the overall quality, do not affect other boards/components and do not increase the maintenance burden for the community. Currently it is the way, that Google only upstreams after the code already ships and then goes on with the next product, while not maintaining the old boards anymore.
- Significantly reduce number of submitters To ensure consistency, scalability and conformity with the general coreboot strategy, we need to define a clear committer structure that defines the original project structure of having a benevolent dictator aka project leader (Stefan Reinauer) and gatekeepers in place. The suggested structure will need to value both community interests and corporate interests equally and hence a good setup would be to have a final amount of six developers with submit rights, three community representatives and three corporate representatives (on from each major stakeholder):
I also do not see, why this is necessary and the current state has to be changed. Ron stated in the past that this is due to commits being pushed breaking things. But that was not the case in the last two or three years. And if it was, it were experienced submitters breaking the tree, which of course is more likely to happen when you do more reviews/write more code.
As Kyösti already asked, please provide the commit hashes of the commits that broke the tree and making you come up with the proposal.
Regarding the dictator, in my opinion the “dictator” main job is communication and not writing code as Linus Torvalds is doing it for Linux for example. In that regard, you were pretty silent (at least on the list, which is the medium for communication) especially since you stared working for Google. Ron is a little more active on the list, but also a little biased towards your employer, which is to be expected.
I guess the position for the dictator is not up for a vote, as you own the infrastructure anyhow I guess.
Patrick is very active in the IRC channel #coreboot (on <irc.freenode.net>) and a coreboot veteran.
In the end I did not miss a dictator the last three years and thought that a solution could always be found.
Current suggestions:
Patrick Georgi (Secunet) Marc Jones (Sage) Stefan Reinauer (Google)
In my opinion, Aaron Durban did a great job in the past year(?) keeping in touch with the community and addressing the concerns.
Peter Stuge, Kyösti Mälkki, N.N. (non-commercial part of the coreboot community)
The community has very capable people for this kind of job in my opinion. Vladimir and Alexandru come to mind.
Status: TO BE IMPLEMENTED
I agree with others that the problems should be elaborated more and then solutions for them have to be thought of. From the current messages it is not clear what lead to the proposals. If the analysis of the problems is incorrect the solutions/proposals probably won’t solve the real problems or improve the situation.
So to summarize, in my opinion, there are the issues of upstreaming patches from Google which costs Google and the community a lot of time, money and developer’s nerves and missing communication/interaction between the community and companies. Other than that my impression is that the current process works fine.
For example it was never announced when Google added new tags to commit messages. It is a problem for community developers to look at changes in the Chrome OS repository and it is wasting their time to jump between different Web sites.
Maybe the infrastructure can be adapted too and also more automatic tests (tests of more Kconfig options) and checks be implemented. This would give developers more time on reviewing the actual code than looking at the trivial policy violations.
It should also be though of making some of the internal Google policies an upstream policy too like having to put that TEST tag in there.
Some solutions would be for Google (and Sage, AMD) (and probably cheaper than the spent developer time) to give out devices for free to developers so that these are maintained and tested even when the company went on to the next device. (See the board-status repository how many Google devices have been tested although Ron pushed for that board-status repository.)
Thanks,
Paul