Right now, when you have a CL that builds fine but fails unit tests, Jenkins with Verified -1 it with a failure page like this https://qa.coreboot.org/job/coreboot-gerrit/215770/ , which just says "Test Result (no failures)". If you know your way around Jenkins you can click on the "Console Output" link on the left and scroll down to see the actual unit test errors, but most people who see this for the first time will probably be very confused and have a hard chance finding that on their own.
Is there any way we could make that more obvious and display the unit tests as a real element among the test results (or at least put some sort of "click Console Output to see errors" message on there)? Unfortunately I have no idea how anything on that page is configured or generated -- Patrick or Martin, I hope one of you guys would know more?
Am 02.09.2022 um 03:38 schrieb Julius Werner:
Right now, when you have a CL that builds fine but fails unit tests, Jenkins with Verified -1 it with a failure page like this https://qa.coreboot.org/job/coreboot-gerrit/215770/ , which just says "Test Result (no failures)". If you know your way around Jenkins you can click on the "Console Output" link on the left and scroll down to see the actual unit test errors, but most people who see this for the first time will probably be very confused and have a hard chance finding that on their own.
This isn't "the unit test fails", it's "the unit test failed to compile".
Is there any way we could make that more obvious and display the unit tests as a real element among the test results (or at least put some sort of "click Console Output to see errors" message on there)? Unfortunately I have no idea how anything on that page is configured or generated -- Patrick or Martin, I hope one of you guys would know more?
We'd have to wrap the build process for the unit tests to report into junit on error.
tests/Makefile.inc already looks for JUNIT_OUTPUT. The target in tests/Makefile.common for $($(1)-bin) would have to be extended to create a report on build failure.
The junit.xml target in payloads/libpayload/Makefile.inc might serve as inspiration.
Patrick
We'd have to wrap the build process for the unit tests to report into junit on error.
tests/Makefile.inc already looks for JUNIT_OUTPUT. The target in tests/Makefile.common for $($(1)-bin) would have to be extended to create a report on build failure.
It will not be enough. $($(1)-bin) target is responsible for linking test binary. $($(1)-objs) would also have to be extended to detect whether a particular file failed to compile.
The junit.xml target in payloads/libpayload/Makefile.inc might serve as inspiration.
Thanks for the tip. It will come in handy. I will try to find some time to extend unit-tests framework to support tests compilation failures reports in coreboot and libpayload makefiles.
-- Jakub
Implementation [1] of unit-tests build failures reporting is now available for review. With this change failing unit-tests builds are visible [2] in the Jenkins job status in the Test Results section.
[1] https://review.coreboot.org/c/coreboot/+/67372 [2]* https://qa.coreboot.org/job/coreboot-gerrit/215975/
* Link [2] might stop working soon as Jenkins is not keeping everything forever
-
Jakub Czapiga -
Julius Werner -
Patrick Georgi