Hi Piotr, Patrick et al.,
actually, there is dedicated software to achieve exactly what you are talking about. I initiated the SystemTestPortal project [1] a while ago to provide an open source solution that focuses on simplicity and low entry barriers - you can watch my 2019 talk at FrOsCon [2] to get a first overview.
Currently, the project is stalled and, thus, the software is ready for production due to the lack of maintenance. However I plan tp resume the project soon. There are also other open source projects that provide similar solutions in this space. Yet, I must confess that to date I had a really hard time convincing developers to give our tool a serious try. While this might be due to bugs in the tool itself, I think it is mainly because many developers and project leaders don't buy into such a "community testing" approach as they think that the only right way to test software is through test automation.
I think that coreboot is a perfect example of a project that could benefit from such a tool (not necessarily our tool, but one from this category).
Cheers, Daniel
[1] https://www.systemtestportal.org/ [2] https://media.ccc.de/v/froscon2019-2359-testing_software_yes_please
Hi Daniel, Looking at the software you described it seems a wonderful tool for humans to create, execute test steps and analyse test results entered manually by a human.
What we are looking for is only the "data store" and visual representation of such, where automated tests are run by robots. Those (self-hosted or propietary) robots need to publish their test results somewhere using a web API.
Does SystemTestPortal support input from robots over for example REST API? Does it support the idea of different products/configurations?
Regards, Patrick
On Friday, July 16, 2021, Daniel Kulesz via coreboot coreboot@coreboot.org wrote:
Hi Piotr, Patrick et al.,
actually, there is dedicated software to achieve exactly what you are talking about. I initiated the SystemTestPortal project [1] a while ago to provide an open source solution that focuses on simplicity and low entry barriers - you can watch my 2019 talk at FrOsCon [2] to get a first overview.
Currently, the project is stalled and, thus, the software is ready for production due to the lack of maintenance. However I plan tp resume the project soon. There are also other open source projects that provide similar solutions in this space. Yet, I must confess that to date I had a really hard time convincing developers to give our tool a serious try. While this might be due to bugs in the tool itself, I think it is mainly because many developers and project leaders don't buy into such a "community testing" approach as they think that the only right way to test software is through test automation.
I think that coreboot is a perfect example of a project that could benefit from such a tool (not necessarily our tool, but one from this category).
Cheers, Daniel
[1] https://www.systemtestportal.org/ [2] https://media.ccc.de/v/froscon2019-2359-testing_software_yes_please _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Hi Patrick,
Looking at the software you described it seems a wonderful tool for humans to create, execute test steps and analyse test results entered manually by a human.
Actually, this was the primary goal - we wanted to support testing of systems that are close to hardware (such as coreboot). Especially with coreboot, personally, I found too often that I bought a board that was officially "supported" just to find out that some things were actually broken while they seemed to have been working in past versions well (happened to me lately with a Thinkpad T410, see my previous postings to the list about this). The idea in SystemTestPortal is to support testing for such regressions - but this of course requires the availability of humans that execute these tests.
What we are looking for is only the "data store" and visual representation of such, where automated tests are run by robots. Those (self-hosted or propietary) robots need to publish their test results somewhere using a web API.
I see. Well, there is a different project named "ReportPortal.io" that (imho) does exactly that. We had a joint stand at FOSDEM 2019 together with them and KiwiTCMS. It might be worth looking into that.
Does SystemTestPortal support input from robots over for example REST API?
Not at the moment but it is a feature request we received multiple times so we will eventually add this in the future.
Does it support the idea of different products/configurations?
Yes, it does. We have a two-dimensional concept of a products and variants, but I think coreboot would need three dimensions or even more, right? So for example you could have:
- coreboot 4.14 ("clean" without patches) - on a Thinkpad T410 - built with config options X and Y enabled but Z disabled
In addition, there could be also different configurations of the target machine, different OSs on which you would test etc. - the key challenge here is to decide what to put into the test cases themselves and what to put in the products/variants/configuration metadata. Maybe you could try to describe a data model that would be ideal from the coreboot perspective?
Cheers, Daniel
I think there are two issues here. First, we can't get it all right at once. But, second, we have to think in terms of eventual scale. The system will need to make the status of hundreds of motherboards accessible.
The Google doc is wonderful for a deep dive, but a bit overwhelming as the first thing one sees.
I'm not sure the testportal scales that well either. The only report system I've seen that approaches our needs is the chromeos build waterfall, but even that may not be enough.
I think all the efforts to date were very good, and we need to learn what we can from them. I also feel they are not quite right.
On Sat, Jul 17, 2021 at 2:45 AM Daniel Kulesz via coreboot coreboot@coreboot.org wrote:
Hi Patrick,
Looking at the software you described it seems a wonderful tool for humans to create, execute test steps and analyse test results entered manually by a human.
Actually, this was the primary goal - we wanted to support testing of systems that are close to hardware (such as coreboot). Especially with coreboot, personally, I found too often that I bought a board that was officially "supported" just to find out that some things were actually broken while they seemed to have been working in past versions well (happened to me lately with a Thinkpad T410, see my previous postings to the list about this). The idea in SystemTestPortal is to support testing for such regressions - but this of course requires the availability of humans that execute these tests.
What we are looking for is only the "data store" and visual representation of such, where automated tests are run by robots. Those (self-hosted or propietary) robots need to publish their test results somewhere using a web API.
I see. Well, there is a different project named "ReportPortal.io" that (imho) does exactly that. We had a joint stand at FOSDEM 2019 together with them and KiwiTCMS. It might be worth looking into that.
Does SystemTestPortal support input from robots over for example REST API?
Not at the moment but it is a feature request we received multiple times so we will eventually add this in the future.
Does it support the idea of different products/configurations?
Yes, it does. We have a two-dimensional concept of a products and variants, but I think coreboot would need three dimensions or even more, right? So for example you could have:
- coreboot 4.14 ("clean" without patches)
- on a Thinkpad T410
- built with config options X and Y enabled but Z disabled
In addition, there could be also different configurations of the target machine, different OSs on which you would test etc. - the key challenge here is to decide what to put into the test cases themselves and what to put in the products/variants/configuration metadata. Maybe you could try to describe a data model that would be ideal from the coreboot perspective?
Cheers, Daniel _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-
Daniel Kulesz -
Patrick Rudolph -
ron minnich