Re: [coreboot] [RFC] Board status script

[Paul Menzel]

Am Montag, den 04.11.2013, 08:05 -0800 schrieb ron minnich:
> well, now that this has come up here, I might as well put the full plan out.

Thanks. Very much appreciated.
> The mainboard status repo needs repair.

I did not know that we had one. Or do you mean the status page in the
coreboot Wiki?
> We want to make a script that produces concise results that make it
> easy for a potential OEM to make a go/no-go decision. Right now, they
> can't do this, and we've been asked to fix that problem.
>
> The vendors want to know two things:
> can I build it
> will it boot [my os] -- where that may not be linux.
>
> So we need a simple script that produces this information. It needs to
> be easy to run, have very few dependencies, and communicate the key
> information without drowning the user in a bunch of extraneous data.
>
> How do we answer those questions?
>
> We need to present results that are current. So, starting Nov. 15, the
> static mainboard status page will be more dynamic. The table will
> become a waterfall of tables, updated each week. Older results will be
> found in older tables.
>
> So the 'Can I build it' will be answered by looking at the table.
> How will be answered by seeing what date a build was reported, getting
> the hash and config, and then trying to build it.

Doesn’t our build bot Jenkins already answer this question? Every board
builds with the default configuration.
> Will it boot can only be answered by trying to boot it, but the log
> might be helpful.

Booting an operating system still does not give you the information, if
all devices are supported and work. Especially suspend and resume, which
does not work on the ASRock E350M1 for example.
> The huge amount of detail Paul is proposing doesn't help that much
> with these questions. This is not about regressions. This is about
> whether a board worked for someone. The mainboard status page is full
> of errors and if I could remove it today, I would.

If you mean the Wiki, then in my experience it is pretty accurate.
> And, again, the value of dmesg is quite overstated. Linux dmesg is so
> variable that for the most part it's one bit of information: 'got a
> dmesg/did not get a dmesg'.

For me as a non-professional having the Linux messages for comparisons
helped me several times to improve things or figure out problems.
> What this means is we're going to make work for our community. If you
> have a working board, YOU need to run that script and push the
> results. The pressure to run the script will increase each week, as
> your results fall lower down on the waterfall :-)

What I have seen so far, I have no incentive to run that script just to
help some vendor, whoever that is. I am looking forward to November 15th
and try to stay open minded, but from the non-corporate community, I am
again disappointed to be left out and just have probably a non-optimal
end product.

Also, you pushed David’s patch with the errors and leave the boring,
stupid work to clean it up to the community. Uwe Hermann did a lot of
clean up work too in the past and moved to a different project.

I think committing the patch was rushed again. The same with the Lenovo
X60 native VGA init patch, which you pushed [2].

Thanks,

Paul

[2] http://review.coreboot.org/2998

[RFC] Board status script

[ron minnich]

On Mon, Nov 4, 2013 at 10:36 AM, Paul Menzel
 wrote:
> Doesn’t our build bot Jenkins already answer this question? Every board
> builds with the default configuration.

And it's clearly insufficient. If it answered the question accurately
do you think we'd be going to all this trouble?
> If you mean the Wiki, then in my experience it is pretty accurate.

You might be the only person with that experience. Again, we are doing
this driven by a clearly stated request from a number of people. The
current page is a problem.
> Also, you pushed David’s patch with the errors and leave the boring,
> stupid work to clean it up to the community.

To be honest, I pushed what I knew was an incomplete patch in need of
work and yep, it's up to us to clean it up. At some point, the
bikeshedding halts forward momentum and I don't want that to happen
(again). We need this status page, and even a bad one that is accurate
is better than what we have now.

ron

[RFC] Board status script

[David Hendricks]

On Mon, Nov 4, 2013 at 10:36 AM, Paul Menzel <
[e-mail]> wrote:
> Am Montag, den 04.11.2013, 08:05 -0800 schrieb ron minnich:
> > well, now that this has come up here, I might as well put the full plan
>
> out.
>
> Thanks. Very much appreciated.
> > The mainboard status repo needs repair.
>
> I did not know that we had one. Or do you mean the status page in the
> coreboot Wiki?

Yes, the wiki.
> > We want to make a script that produces concise results that make it
> > easy for a potential OEM to make a go/no-go decision. Right now, they
> > can't do this, and we've been asked to fix that problem.
> >
> > The vendors want to know two things:
> > can I build it
> > will it boot [my os] -- where that may not be linux.
> >
> > So we need a simple script that produces this information. It needs to
> > be easy to run, have very few dependencies, and communicate the key
> > information without drowning the user in a bunch of extraneous data.
> >
> > How do we answer those questions?
> >
> > We need to present results that are current. So, starting Nov. 15, the
> > static mainboard status page will be more dynamic. The table will
> > become a waterfall of tables, updated each week. Older results will be
> > found in older tables.
> >
> > So the 'Can I build it' will be answered by looking at the table.
> > How will be answered by seeing what date a build was reported, getting
> > the hash and config, and then trying to build it.
>
> Doesn’t our build bot Jenkins already answer this question? Every board
> builds with the default configuration.

It answers the question about building, but not booting.
> Will it boot can only be answered by trying to boot it, but the log
> > might be helpful.
>
> Booting an operating system still does not give you the information, if
> all devices are supported and work. Especially suspend and resume, which
> does not work on the ASRock E350M1 for example.

True. But booting an OS alone is a very high level of assurance in itself.
It's a simple pass/fail metric.

It would be nice to eventually upload test results the same way as Ron
proposes to catch regressions in things like suspend/resume. That will come
with time.
> > The huge amount of detail Paul is proposing doesn't help that much
> > with these questions. This is not about regressions. This is about
> > whether a board worked for someone. The mainboard status page is full
> > of errors and if I could remove it today, I would.
>
> If you mean the Wiki, then in my experience it is pretty accurate.

We will need to agree to disagree here. Ron and Stefan have gotten many
inquiries and complaints off-list from vendors and developers who were
confused due to obsolete entries on the wiki. Right now it's causing more
problems than it's solving. We want to replace it with something more
automated, scalable, and accurate. The alternative will likely be to remove
it entirely from coreboot.org :-(
> And, again, the value of dmesg is quite overstated. Linux dmesg is so
> > variable that for the most part it's one bit of information: 'got a
> > dmesg/did not get a dmesg'.
>
> For me as a non-professional having the Linux messages for comparisons
> helped me several times to improve things or figure out problems.

Agreed, and we can work this in as we go along.
> > What this means is we're going to make work for our community. If you
> > have a working board, YOU need to run that script and push the
> > results. The pressure to run the script will increase each week, as
> > your results fall lower down on the waterfall :-)
>
> What I have seen so far, I have no incentive to run that script just to
> help some vendor, whoever that is.

Then don't. The vendor should be responsible for that. We're trying to make
it easier for them to do so.
> I am looking forward to November 15th
> and try to stay open minded, but from the non-corporate community, I am
> again disappointed to be left out and just have probably a non-optimal
> end product.

Your input has been very helpful. We're really trying hard to improve the
situation for vendors and independent developers alike.
> Also, you pushed David’s patch with the errors and leave the boring,
> stupid work to clean it up to the community.

Yeah, that was premature :-/ A +2 would have sufficed so I'd know to just
fix up the minor issues before submitting. Bad Ron.

FWIW I suspect the script will be torn apart and re-written by people who
are much better than I at writing scripts (like the getrevision.sh for
flashrom). So I don't think it matters that the script was committed with a
few flaws. At least now it's there so others can more easily hack on it.
> Uwe Hermann did a lot of
> clean up work too in the past and moved to a different project.
>
> I think committing the patch was rushed again. The same with the Lenovo
> X60 native VGA init patch, which you pushed [2].
>
> Thanks,
>
> Paul
>
> [2] http://review.coreboot.org/2998