Hi Patrick,
Looking at the software you described it seems a wonderful tool for humans to create, execute test steps and analyse test results entered manually by a human.
Actually, this was the primary goal - we wanted to support testing of systems that are close to hardware (such as coreboot). Especially with coreboot, personally, I found too often that I bought a board that was officially "supported" just to find out that some things were actually broken while they seemed to have been working in past versions well (happened to me lately with a Thinkpad T410, see my previous postings to the list about this). The idea in SystemTestPortal is to support testing for such regressions - but this of course requires the availability of humans that execute these tests.
What we are looking for is only the "data store" and visual representation of such, where automated tests are run by robots. Those (self-hosted or propietary) robots need to publish their test results somewhere using a web API.
I see. Well, there is a different project named "ReportPortal.io" that (imho) does exactly that. We had a joint stand at FOSDEM 2019 together with them and KiwiTCMS. It might be worth looking into that.
Does SystemTestPortal support input from robots over for example REST API?
Not at the moment but it is a feature request we received multiple times so we will eventually add this in the future.
Does it support the idea of different products/configurations?
Yes, it does. We have a two-dimensional concept of a products and variants, but I think coreboot would need three dimensions or even more, right? So for example you could have:
- coreboot 4.14 ("clean" without patches) - on a Thinkpad T410 - built with config options X and Y enabled but Z disabled
In addition, there could be also different configurations of the target machine, different OSs on which you would test etc. - the key challenge here is to decide what to put into the test cases themselves and what to put in the products/variants/configuration metadata. Maybe you could try to describe a data model that would be ideal from the coreboot perspective?
Cheers, Daniel