I think there are two issues here. First, we can't get it all right at once. But, second, we have to think in terms of eventual scale. The system will need to make the status of hundreds of motherboards accessible.
The Google doc is wonderful for a deep dive, but a bit overwhelming as the first thing one sees.
I'm not sure the testportal scales that well either. The only report system I've seen that approaches our needs is the chromeos build waterfall, but even that may not be enough.
I think all the efforts to date were very good, and we need to learn what we can from them. I also feel they are not quite right.
On Sat, Jul 17, 2021 at 2:45 AM Daniel Kulesz via coreboot coreboot@coreboot.org wrote:
Hi Patrick,
Looking at the software you described it seems a wonderful tool for humans to create, execute test steps and analyse test results entered manually by a human.
Actually, this was the primary goal - we wanted to support testing of systems that are close to hardware (such as coreboot). Especially with coreboot, personally, I found too often that I bought a board that was officially "supported" just to find out that some things were actually broken while they seemed to have been working in past versions well (happened to me lately with a Thinkpad T410, see my previous postings to the list about this). The idea in SystemTestPortal is to support testing for such regressions - but this of course requires the availability of humans that execute these tests.
What we are looking for is only the "data store" and visual representation of such, where automated tests are run by robots. Those (self-hosted or propietary) robots need to publish their test results somewhere using a web API.
I see. Well, there is a different project named "ReportPortal.io" that (imho) does exactly that. We had a joint stand at FOSDEM 2019 together with them and KiwiTCMS. It might be worth looking into that.
Does SystemTestPortal support input from robots over for example REST API?
Not at the moment but it is a feature request we received multiple times so we will eventually add this in the future.
Does it support the idea of different products/configurations?
Yes, it does. We have a two-dimensional concept of a products and variants, but I think coreboot would need three dimensions or even more, right? So for example you could have:
- coreboot 4.14 ("clean" without patches)
- on a Thinkpad T410
- built with config options X and Y enabled but Z disabled
In addition, there could be also different configurations of the target machine, different OSs on which you would test etc. - the key challenge here is to decide what to put into the test cases themselves and what to put in the products/variants/configuration metadata. Maybe you could try to describe a data model that would be ideal from the coreboot perspective?
Cheers, Daniel _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org