Automated test system: Nominations wanted

[Carl-Daniel Hailfinger]

Hi,

I am currently planning to set up a test system with 5 (later up to 10)
machines boot testing each new coreboot commit. This test system will be
serviced (i.e. recovery from bricking) Mo-Fr during CET/CEST office hours.

Current goals for every commit:
- Check if coreboot + SeaBIOS are able to boot Linux to a point where
network is up and running

Current goals for every work day:
- Check if screen, keyboard and touchpad/mouse work
- Check if USB works and has the expected transfer speed (i.e. if USB
High and Super Speed both work)

Future goals for every work day:
- Run memtest86+ (short run)
- Run GRUB2 and boot Linux
- Check if USB works (see above)

Once any test running once per work day can be automated with reasonable
effort (i.e. not requiring robots or human interaction), it can be moved
to a per-commit goal.

The selection of target systems should include:
1. at least one x86 laptop without an active ME (present but inactive
would be OK)
2. at least one x86 laptop which can boot x86-blob-free (except microcode)
3. at least one x86 board or laptop which needs neither blobs (except
microcode) nor ME
4. at least one x86 board with reasonable (past or present) business
market penetration
5. if the first two laptops use the same CPU vendor, a board using a
different x86 CPU vendor

1+2 are designed to partially remove the potential for nasty surprises,
3 should remove nasty surprises completely, 4 is the one I need to show
that the test system is actually relevant for our goals, 5 should give
us better coverage outside the most common systems used by coreboot
developers.


Please nominate machines (e.g. "Thinkpad T60 with Intel graphics") and
tell me to which category they belong. If a system fits into multiple
categories, please specify that as well.
I will try to consolidate the recommendations and buy those machines.
Once the system is up and running (hopefully in May), I will add more
machines suggested by the community as time permits.

The time window for suggestions will close at the end of February.

Fire away!

Regards,
Carl-Daniel

[Idwer Vollering]

2015-02-19 0:14 GMT+01:00 Carl-Daniel Hailfinger
<[e-mail]>:
> Hi,
>
> I am currently planning to set up a test system with 5 (later up to 10)
> machines boot testing each new coreboot commit. This test system will be
> serviced (i.e. recovery from bricking) Mo-Fr during CET/CEST office hours.

(..)
> Please nominate machines (e.g. "Thinkpad T60 with Intel graphics") and
> tell me to which category they belong. If a system fits into multiple
> categories, please specify that as well.

asus f2a85-m, desktop board
tyan s8226 and/or supermicro h8scm and/or supermicro h8qgi, server boards
> I will try to consolidate the recommendations and buy those machines.
> Once the system is up and running (hopefully in May), I will add more
> machines suggested by the community as time permits.
>
> The time window for suggestions will close at the end of February.
>
> Fire away!
>
> Regards,
> Carl-Daniel

[Patrick Georgi]

2015-02-19 0:14 GMT+01:00 Carl-Daniel Hailfinger
<[e-mail]>:
> I am currently planning to set up a test system with 5 (later up to 10)
> machines boot testing each new coreboot commit. This test system will be
> serviced (i.e. recovery from bricking) Mo-Fr during CET/CEST office hours.
>
> Current goals for every commit:
> - Check if coreboot + SeaBIOS are able to boot Linux to a point where
> network is up and running
>
> Current goals for every work day:
> - Check if screen, keyboard and touchpad/mouse work
> - Check if USB works and has the expected transfer speed (i.e. if USB
> High and Super Speed both work)
> http://blogs.coreboot.org/blog/author/ayushsagar/ documents last
> year's GSoC project to implement some of those - incl. a screen test
> using the display present signals.

Through external flashing, there's also no need to handle unbricking manually.


Patrick
-- 
Google Germany GmbH
ABC-Str. 19
20354 Hamburg

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores

[Myles Watson]

On Wed, Feb 18, 2015 at 3:14 PM, Carl-Daniel Hailfinger <
[e-mail]> wrote:
> Hi,
>
> I am currently planning to set up a test system with 5 (later up to 10)
> machines boot testing each new coreboot commit. This test system will be
> serviced (i.e. recovery from bricking) Mo-Fr during CET/CEST office hours.
>
> Current goals for every commit:
> - Check if coreboot + SeaBIOS are able to boot Linux to a point where
> network is up and running
>
> Current goals for every work day:
> - Check if screen, keyboard and touchpad/mouse work
> - Check if USB works and has the expected transfer speed (i.e. if USB
> High and Super Speed both work)
>
> Future goals for every work day:
> - Run memtest86+ (short run)
> - Run GRUB2 and boot Linux
> - Check if USB works (see above)
>
> Once any test running once per work day can be automated with reasonable
> effort (i.e. not requiring robots or human interaction), it can be moved
> to a per-commit goal.
>
> The selection of target systems should include:
> 1. at least one x86 laptop without an active ME (present but inactive
> would be OK)
> 2. at least one x86 laptop which can boot x86-blob-free (except microcode)
> 3. at least one x86 board or laptop which needs neither blobs (except
> microcode) nor ME
> 4. at least one x86 board with reasonable (past or present) business
> market penetration
> 5. if the first two laptops use the same CPU vendor, a board using a
> different x86 CPU vendor

Hi Carl-Daniel,

Thanks for setting this up.  I think it would be great to have Qemu and a
couple of the boards supported by SimNOW.  They don't really fit into your
categories, but it is important to keep them working.

Thanks,
Myles
> 1+2 are designed to partially remove the potential for nasty surprises,
> 3 should remove nasty surprises completely, 4 is the one I need to show
> that the test system is actually relevant for our goals, 5 should give
> us better coverage outside the most common systems used by coreboot
> developers.
>
> Please nominate machines (e.g. "Thinkpad T60 with Intel graphics") and
> tell me to which category they belong. If a system fits into multiple
> categories, please specify that as well.
> I will try to consolidate the recommendations and buy those machines.
> Once the system is up and running (hopefully in May), I will add more
> machines suggested by the community as time permits.
>
> The time window for suggestions will close at the end of February.
>
> Fire away!
>
> Regards,
> Carl-Daniel

[Carl-Daniel Hailfinger]

This is a friendly reminder. 2 days left for nominations of systems.

On 19.02.2015 00:14, Carl-Daniel Hailfinger wrote:
> The selection of target systems should include:
> 1. at least one x86 laptop without an active ME (present but inactive
> would be OK)
> 2. at least one x86 laptop which can boot x86-blob-free (except microcode)
> 3. at least one x86 board or laptop which needs neither blobs (except
> microcode) nor ME
> 4. at least one x86 board with reasonable (past or present) business
> market penetration
> 5. if the first two laptops use the same CPU vendor, a board using a
> different x86 CPU vendor
> [...]
> Please nominate machines (e.g. "Thinkpad T60 with Intel graphics") and
> tell me to which category they belong. If a system fits into multiple
> categories, please specify that as well.
> I will try to consolidate the recommendations and buy those machines.
> Once the system is up and running (hopefully in May), I will add more
> machines suggested by the community as time permits.
>
> The time window for suggestions will close at the end of February.

So far I got the following nominations:
- asus f2a85-m (desktop board)
- tyan s8226 and/or supermicro h8scm and/or supermicro h8qgi (server board)
- Qemu and a couple of the boards supported by SimNOW (no category)
- Thinkpad T60 with ATI graphics (no ME, but with graphics blob)

I can't find the suggestions on IRC anymore (searching in the scrollback
buffer is harder than I thought), so if you still remember which
boards/machines were proposed on IRC, please send an email listing them.

Thanks,
Carl-Daniel

[Paul Menzel]

Am Freitag, den 27.02.2015, 12:20 +0100 schrieb Carl-Daniel Hailfinger:
> On 19.02.2015 00:14, Carl-Daniel Hailfinger wrote:
> > The selection of target systems should include:
> > 1. at least one x86 laptop without an active ME (present but inactive
> > would be OK)
> > 2. at least one x86 laptop which can boot x86-blob-free (except microcode)
> > 3. at least one x86 board or laptop which needs neither blobs (except
> > microcode) nor ME
> > 4. at least one x86 board with reasonable (past or present) business
> > market penetration
> > 5. if the first two laptops use the same CPU vendor, a board using a
> > different x86 CPU vendor
> > [...]
> > Please nominate machines (e.g. "Thinkpad T60 with Intel graphics") and
> > tell me to which category they belong. If a system fits into multiple
> > categories, please specify that as well.
> > I will try to consolidate the recommendations and buy those machines.
> > Once the system is up and running (hopefully in May), I will add more
> > machines suggested by the community as time permits.
> >
> > The time window for suggestions will close at the end of February.
>
> So far I got the following nominations:
> - asus f2a85-m (desktop board)
> - tyan s8226 and/or supermicro h8scm and/or supermicro h8qgi (server board)
> - Qemu and a couple of the boards supported by SimNOW (no category)
> - Thinkpad T60 with ATI graphics (no ME, but with graphics blob)

Lenovo ThinkPad T60/X60 with Intel graphics also exist.

If you could get reference boards from AMD that might be also good.

Adlink cExpress-GFR and CM2-GF (lippert/toucan-af and
lippert/frontrunner-af), jetway/nf81-t56n-lf might also be interesting.

Lenovo G505s as a “recent” AMD laptop should also be tested frequently.

[…]


Thanks,

Paul

[Timothy Pearson]

On 02/18/2015 05:14 PM, Carl-Daniel Hailfinger wrote:
> Hi,
>
> I am currently planning to set up a test system with 5 (later up to 10)
> machines boot testing each new coreboot commit. This test system will be
> serviced (i.e. recovery from bricking) Mo-Fr during CET/CEST office hours.

Just wanted to mention that Raptor Engineering now has an automated test 
stand for the ASUS KFSN4-DRE board, run nightly with automatic bricking 
recovery.  It has two Opteron 2431 (AMD Family 10h, 6 core @ 2.4GHz)CPUs 
and 6GB of DDR2-667 memory installed on Node 0.

Each successful test result is recorded to the board-status repository, 
and each failure is reported to this list.

Enjoy!

-- 
Timothy Pearson
Raptor Engineering
[phone]
http://www.raptorengineeringinc.com

[Alexander Couzens]

On Mon, 16 Mar 2015 01:20:17 -0500
Timothy Pearson <[e-mail]> wrote:
> Just wanted to mention that Raptor Engineering now has an automated test 
> stand for the ASUS KFSN4-DRE board, run nightly with automatic bricking 
> recovery.  It has two Opteron 2431 (AMD Family 10h, 6 core @ 2.4GHz)CPUs 
> and 6GB of DDR2-667 memory installed on Node 0.
>
> Each successful test result is recorded to the board-status repository, 
> and each failure is reported to this list.

Hi Timothy,

can you please write down (wiki?) how your system is setted up?
How do you do automatic bricking recovery?

Best,
lynxis
-- 
Alexander Couzens

mail: [e-mail]
jabber: [e-mail]
mobile: [phone]

[Timothy Pearson]

On 03/16/2015 08:17 AM, Alexander Couzens wrote:
> On Mon, 16 Mar 2015 01:20:17 -0500
> Timothy Pearson<[e-mail]>  wrote:
> > Just wanted to mention that Raptor Engineering now has an automated test
> > stand for the ASUS KFSN4-DRE board, run nightly with automatic bricking
> > recovery.  It has two Opteron 2431 (AMD Family 10h, 6 core @ 2.4GHz)CPUs
> > and 6GB of DDR2-667 memory installed on Node 0.
> >
> > Each successful test result is recorded to the board-status repository,
> > and each failure is reported to this list.
>
> Hi Timothy,
>
> can you please write down (wiki?) how your system is setted up?
> How do you do automatic bricking recovery?
>
> Best,
> lynxis

Bricking recovery uses the fallback mechanism.  There is a supervisory 
board attached to the target; it controls physical power on/power 
off/CMOS reset and also sends build/flash/test commands to the target as 
needed.

The exact code/details for the controller are not public at this time, 
though I would be happy to provide additional information on the target 
and/or entertain target hardware configuration requests (add on cards, etc.)

-- 
Timothy Pearson
Raptor Engineering
[phone]
http://www.raptorengineeringinc.com