Running QEMU targets in Jenkins

[Julius Werner]

I'm just curious... have we ever considered booting some of our QEMU
targets as part of the Jenkins CI? I know they don't do a lot but they
do cover some stuff (e.g. CBFS). I randomly happened to boot one of my
in-flight patch trains on qemu-i440fx recently and discovered that I
accidentally broke rmodule loading. Would be nice if Jenkins could
just do that for you automatically.

Just wanted to know whether this had been discussed before and people
have come up with good reasons not to do it, or if it's just a matter
of nobody had time to implement it yet. (And if someone wanted to
implement it, what would be the best hook point? Put it into abuild?)

[Patrick Georgi]

Hi Julius,

https://qa.coreboot.org/job/coreboot-boot-test/ sends off ToT builds to
9esec's Lava system where they are run on some virtual and real devices.
See for example the comments to
https://review.coreboot.org/c/coreboot/+/51189 where Lava reports passing
on 5 qemu configs and 3 real devices.

Regards,
Patrick

Am Do., 4. März 2021 um 02:42 Uhr schrieb Julius Werner <
[e-mail]>:
> I'm just curious... have we ever considered booting some of our QEMU
> targets as part of the Jenkins CI? I know they don't do a lot but they
> do cover some stuff (e.g. CBFS). I randomly happened to boot one of my
> in-flight patch trains on qemu-i440fx recently and discovered that I
> accidentally broke rmodule loading. Would be nice if Jenkins could
> just do that for you automatically.
>
> Just wanted to know whether this had been discussed before and people
> have come up with good reasons not to do it, or if it's just a matter
> of nobody had time to implement it yet. (And if someone wanted to
> implement it, what would be the best hook point? Put it into abuild?)

[Christian Walter]

Hi Julius,

you can also check this out: https://lava.9esec.io/scheduler/alljobs

It currently runs on:
* Various QEmu Targets
* HP Z220
* HP8200
* Lenovo T500

we are about to add another target (Prodrive Hermes / CFL).

If you have more ideas what could be added with little effort - feel 
free to ping us. :)

Best,

Chris

On 3/4/21 11:14 AM, Patrick Georgi via coreboot wrote:
> Hi Julius,
>
> https://qa.coreboot.org/job/coreboot-boot-test/ 
> https://qa.coreboot.org/job/coreboot-boot-test/ sends off ToT builds 
> to 9esec's Lava system where they are run on some virtual and real 
> devices. See for example the comments to 
> https://review.coreboot.org/c/coreboot/+/51189 
> https://review.coreboot.org/c/coreboot/+/51189 where Lava reports 
> passing on 5 qemu configs and 3 real devices.
>
> Regards,
> Patrick
>
> Am Do., 4. März 2021 um 02:42 Uhr schrieb Julius Werner 
> <[e-mail] >:
> I'm just curious... have we ever considered booting some of our QEMU
> targets as part of the Jenkins CI? I know they don't do a lot but they
> do cover some stuff (e.g. CBFS). I randomly happened to boot one of my
> in-flight patch trains on qemu-i440fx recently and discovered that I
> accidentally broke rmodule loading. Would be nice if Jenkins could
> just do that for you automatically.
>
> Just wanted to know whether this had been discussed before and people
> have come up with good reasons not to do it, or if it's just a matter
> of nobody had time to implement it yet. (And if someone wanted to
> implement it, what would be the best hook point? Put it into abuild?)
>
>
>
> coreboot mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Julius Werner]

> https://qa.coreboot.org/job/coreboot-boot-test/ sends off ToT builds to
> 9esec's Lava system where they are run on some virtual and real devices.
> See for example the comments to
> https://review.coreboot.org/c/coreboot/+/51189 where Lava reports passing
> on 5 qemu configs and 3 real devices.
>
> https://9elements.com/privacy

Right, but that's different... those tests only run on patches after they
are merged (and IIRC they do get broken occasionally and I don't think many
people pay attention to them. When I see those broken on one of my patches
I tend to just assume there was an existing breakage in the tree already).

I understand that there may be good reasons why we can't run the real
hardware tests on every patch set and make them block Verified+1. But for
QEMU that shouldn't be too hard? We already build images for the emulation
boards anyway, running QEMU to boot them just takes a second, and it
shouldn't require anything that isn't already available on the machines
that do all the compiling. Why can't we do that as part of that step
already?

[Paul Menzel]

Dear Julius,

Am 04.03.21 um 23:20 schrieb Julius Werner:
> > https://qa.coreboot.org/job/coreboot-boot-test/ sends off ToT builds to
> > 9esec's Lava system where they are run on some virtual and real devices.
> > See for example the comments to
> > https://review.coreboot.org/c/coreboot/+/51189 where Lava reports passing
> > on 5 qemu configs and 3 real devices.
> Right, but that's different... those tests only run on patches after they
> are merged (and IIRC they do get broken occasionally and I don't think many
> people pay attention to them. When I see those broken on one of my patches
> I tend to just assume there was an existing breakage in the tree already).
>
> I understand that there may be good reasons why we can't run the real
> hardware tests on every patch set and make them block Verified+1. But for
> QEMU that shouldn't be too hard? We already build images for the emulation
> boards anyway, running QEMU to boot them just takes a second, and it
> shouldn't require anything that isn't already available on the machines
> that do all the compiling. Why can't we do that as part of that step
> already?

I pinged you on #[e-mail], but looks like you didn’t 
see it.

I think the reason is, nobody has finished setting it up. Martin’s 
change-set CB:20877 (WIP: Add extended tests) [1] from 2017 adds basic 
QEMU tests, but wasn’t finished.

Kind regards,

Paul

[1]: https://review.coreboot.org/c/coreboot/+/20877