[coreboot] Automated test system: Nominations wanted

[Timothy Pearson]

On 03/18/2015 02:49 AM, Carl-Daniel Hailfinger wrote:
> [off-list]
>
> Hi Timothy,
>
> On 16.03.2015 18:15, Timothy Pearson wrote:
>> On 03/16/2015 08:17 AM, Alexander Couzens wrote:
>>> On Mon, 16 Mar 2015 01:20:17 -0500
>>> Timothy Pearson<tpearson at raptorengineeringinc.com>   wrote:
>>>
>>>> Just wanted to mention that Raptor Engineering now has an automated
>>>> test
>>>> stand for the ASUS KFSN4-DRE board, run nightly with automatic bricking
>>>> recovery.  It has two Opteron 2431 (AMD Family 10h, 6 core @
>>>> 2.4GHz)CPUs
>>>> and 6GB of DDR2-667 memory installed on Node 0.
>>>>
>>>> Each successful test result is recorded to the board-status repository,
>>>> and each failure is reported to this list.
>>>
>>> can you please write down (wiki?) how your system is setted up?
>>> How do you do automatic bricking recovery?
>>
>> Bricking recovery uses the fallback mechanism.  There is a supervisory
>> board attached to the target; it controls physical power on/power
>> off/CMOS reset and also sends build/flash/test commands to the target
>> as needed.
>>
>> The exact code/details for the controller are not public at this time,
>
> Understood. What is needed for you to make this public and/or provide
> more information about it? Money? Time? PR?

Some combination of the above, primarily the first two.  Right now the 
system is functional but its code is not exactly up to our normal FOSS 
configurability / maintainability standards.  I could clean it up, make 
it easy to configure for other test systems, and generate documentation, 
but that would require time + proper motivation (e.g. put the work under 
contract with my company).

I also want to let the test stand run for a month or two to make sure 
some unforeseen corner case doesn't take it out--this delay can be 
decreased significantly by throwing random Gerrit changesets at it 
(including failures to build, NVRAM layout changes, etc.) until I'm 
confident it won't break under normal use.

One of my main concerns is that unless the test rig is 1.) inexpensive 
and 2.) almost fully plug + play people won't bother to set it up / make 
it available for use.

Right now the test stand requires an external NFS server and SMTP 
access, along with a 24/7 Internet connection.  It doesn't use Java at 
all; only needing around 500 lines of Python.  The test coreboot image 
is built on the target itself, which is only powered on for compilation 
and test, thus saving energy.

> My main motivation is that I'd like to avoid reinventing this from
> scratch now that I've started to order parts for my automated test system.

Understood.

> Stefan Reinauer had a test system running in the past, but he dismantled
> it because nobody used it. That was during a time when we didn't have
> Jenkins and hooking up tests was impossible: Manual submission and
> manual reaction to the result was the only way. I think he published how
> he set up his test system back then, but your test system looks like it
> is fully automated and thus way easier to use and integrate.

This is one of my main concerns ATM.  If this is something the coreboot 
community wants badly enough I can provide a professional open-source 
solution.  If not, I'll continue to run the test stand here as both a 
courtesy to the community and a double-check on bootability for the 
systems in use here.

-- 
Timothy Pearson
Raptor Engineering
+1 (415) 727-8645
http://www.raptorengineeringinc.com