release engineering process

[Patrick Georgi]

Hi all,

those walls of text recently certainly gave a lot of information to digest. I'd like to discuss one aspect that got indirect coverage only, which is the desire to have some boards be "stable" (for some definition of stable). It comes up every now and then and this time it was in the form of "please leave boards X, Y, Z alone".

We have a pretty chaotic development process that allows for lots of velocity all over the place. Unfortunately that makes the only useful stable name in our repository ("master") quite unreliable for users. The board-status script and wiki page is a nice approach, but given that every board is tested on a different commit id, I wouldn't trust a "green" box there to say anything more than that _exactly_ the tester's configuration is going to work. For all I know, a theoretically suspend-capable board tested without that feature enabled (never-mind that board-status doesn't say anything about wake-up in the first place), may not even build with suspend enabled because that part of the tree was broken in just that revision.

With that in mind, I want to throw out some ideas:

* Let's have regular release cycles. maybe quarterly, eg in the middle of each quarter (that is, feb 15, may 15, aug 15, nov 15) to avoid certain vacation-heavy periods.

* A number of weeks (2? 4?) before the official release date, we branch and only accept bug fixes into the branches.

* for each release, we tag from the branch (so we have a _single_ commit id for everything), do a nice write-up on which boards we know are functional, known issues, and what notable changes there are compared to the previous release.

* board-status gets extended to track branches and tags properly (we need that for the classic branch already, since some boards are on the way out on master)

IMHO this gives us a couple of advantages over the current non-process:
It gives us the opportunity for exposure to the world since releases tend to draw some media attention.
And it may encourage some more focussed testing (on board-status) than the current approach, which may increase participation - having your port listed as working on the 2015.08 release notes is a bigger incentive than some green box on some obscure wiki-page.
We can point people to certain revisions with more confidence (and with a nicer name than a SHA1 of a random tree), which will make vendors' lives easier (and in combination with that media exposure, also community support on #coreboot and the list when people ask what revision to use).

This is definitely written for discussion, not a blueprint I intend to implement ASAP.
I'm not even sure if I still like the proposal tomorrow.
Any specifics are just there to give the concept more meat, I have no idea if they work out.
Maybe we don't even need anything like this?
Consider it a conversation starter and add your comments :-)

Thanks,
Patrick

[ron minnich]

the last 15 years shows that the number of people who want to build is
a tiny fraction of those who just want an image they can burn. If we
focus on creating a "library" of burnable images, that can be the
release cycle.

So, thinking as I type: John Lewis is Da Man in terms of maintaining a
system that lets people pull an image, flash it automagically,
recover, and so on, all without having to build it.

So how about we forget about release cycles for the tree and think in
terms of the problem of distribution of good images for people? Does
that make things easier? I don't know.

ron

[Kevin O'Connor]

On Thu, Nov 06, 2014 at 07:57:23PM +0100, Patrick Georgi wrote:
[...]
> With that in mind, I want to throw out some ideas:
> Let's have regular release cycles. maybe quarterly, eg in the
>
> middle of each quarter (that is, feb 15, may 15, aug 15, nov 15) to
> avoid certain vacation-heavy periods.

Just my $.02, but I think having regular coreboot releases would be a
very good thing.

[...]
> IMHO this gives us a couple of advantages over the current non-process:
> It gives us the opportunity for exposure to the world since releases tend to draw some media attention.
> And it may encourage some more focussed testing (on board-status) than the current approach, which may increase participation - having your port listed as working on the 2015.08 release notes is a bigger incentive than some green box on some obscure wiki-page.
> We can point people to certain revisions with more confidence (and with a nicer name than a SHA1 of a random tree), which will make vendors' lives easier (and in combination with that media exposure, also community support on #coreboot and the list when people ask what revision to use).

Very well said.

-Kevin

[ron minnich]

I'm fine with regular releases with one proviso: if we announce a
release date, and a version, and a test, then we drop all boards that
are not working as of that date. It's git, we can bring them back. But
a release to me means that everything I find in src/mainboard/* is
working for that release. If we don't achieve that goal then a release
(to me) has no meaning.

Or else we add a level to mainboard:
src/RELEASE/mainboard
or we drop the mainboard as a name, and just have src/<release>/vendor/etc.
and you can easily see where a mainboard is for a given release.
src/2015.1/google/peppy tells you that the peppy was known to work as
of the 2015 Q1 release. Bringing a mainboard forward works like this:
git mv src/2015.1/google/pepppy src/2015.2/google

It does not move forward unless we have a passing test. We need to
define a passing test. The ubuntu one looks as good as any.

But we need a process for killing old junk, and a release process is
as good as any.

ron

[Marc Jones]

Hi Patrick,

Thanks for starting this discussion.

On Thu Nov 06 2014 at 11:58:03 AM Patrick Georgi 
wrote:
> Hi all,
>
> those walls of text recently certainly gave a lot of information to
> digest. I'd like to discuss one aspect that got indirect coverage only,
> which is the desire to have some boards be "stable" (for some definition of
> stable). It comes up every now and then and this time it was in the form of
> "please leave boards X, Y, Z alone".
>
> We have a pretty chaotic development process that allows for lots of
> velocity all over the place. Unfortunately that makes the only useful
> stable name in our repository ("master") quite unreliable for users. The
> board-status script and wiki page is a nice approach, but given that every
> board is tested on a different commit id, I wouldn't trust a "green" box
> there to say anything more than that _exactly_ the tester's configuration
> is going to work. For all I know, a theoretically suspend-capable board
> tested without that feature enabled (never-mind that board-status doesn't
> say anything about wake-up in the first place), may not even build with
> suspend enabled because that part of the tree was broken in just that
> revision.
>
> With that in mind, I want to throw out some ideas:
> Let's have regular release cycles. maybe quarterly, eg in the middle of
>
> each quarter (that is, feb 15, may 15, aug 15, nov 15) to avoid certain
> vacation-heavy periods.

I suspect that quarterly might be too frequent. May 2 or 3 times a year.
That would give more time to test and stabilize.
> A number of weeks (2? 4?) before the official release date, we branch
>
> and only accept bug fixes into the branches.
>
> I am concerned about branching too soon and trying to maintain both master

and branch. Personally, I would like to see master go through a bugfix
period before branching. I think it would keep master healthier and
minimize the merge back effort after release.
> for each release, we tag from the branch (so we have a _single_ commit
>
> id for everything), do a nice write-up on which boards we know are
> functional, known issues, and what notable changes there are compared to
> the previous release.

We could update the coreboot KERNELVERSION as part of the branch process.
That will be helpful for for derivative works.

* board-status gets extended to track branches and tags properly (we need
> that for the classic branch already, since some boards are on the way out
> on master)

IMHO this gives us a couple of advantages over the current non-process:
> It gives us the opportunity for exposure to the world since releases tend
> to draw some media attention.
> And it may encourage some more focussed testing (on board-status) than the
> current approach, which may increase participation - having your port
> listed as working on the 2015.08 release notes is a bigger incentive than
> some green box on some obscure wiki-page.
> We can point people to certain revisions with more confidence (and with a
> nicer name than a SHA1 of a random tree), which will make vendors' lives
> easier (and in combination with that media exposure, also community support
> on #coreboot and the list when people ask what revision to use).

These are all very valid reasons and I support any improvement was can make
in releasing coreboot source.  Thanks for Starting the discussion

Marc

This is definitely written for discussion, not a blueprint I intend to
> implement ASAP.
> I'm not even sure if I still like the proposal tomorrow.
> Any specifics are just there to give the concept more meat, I have no idea
> if they work out.
> Maybe we don't even need anything like this?
> Consider it a conversation starter and add your comments :-)
>
> Thanks,
> Patrick