Can we make unit test failures more obvious in Gerrit?

[Julius Werner]

Right now, when you have a CL that builds fine but fails unit tests,
Jenkins with Verified -1 it with a failure page like this
https://qa.coreboot.org/job/coreboot-gerrit/215770/ , which just says
"Test Result (no failures)". If you know your way around Jenkins you
can click on the "Console Output" link on the left and scroll down to
see the actual unit test errors, but most people who see this for the
first time will probably be very confused and have a hard chance
finding that on their own.

Is there any way we could make that more obvious and display the unit
tests as a real element among the test results (or at least put some
sort of "click Console Output to see errors" message on there)?
Unfortunately I have no idea how anything on that page is configured
or generated -- Patrick or Martin, I hope one of you guys would know
more?

[Patrick Georgi]

Am 02.09.2022 um 03:38 schrieb Julius Werner:
> Right now, when you have a CL that builds fine but fails unit tests,
> Jenkins with Verified -1 it with a failure page like this
> https://qa.coreboot.org/job/coreboot-gerrit/215770/ , which just says
> "Test Result (no failures)". If you know your way around Jenkins you
> can click on the "Console Output" link on the left and scroll down to
> see the actual unit test errors, but most people who see this for the
> first time will probably be very confused and have a hard chance
> finding that on their own.

This isn't "the unit test fails", it's "the unit test failed to compile".
> Is there any way we could make that more obvious and display the unit
> tests as a real element among the test results (or at least put some
> sort of "click Console Output to see errors" message on there)?
> Unfortunately I have no idea how anything on that page is configured
> or generated -- Patrick or Martin, I hope one of you guys would know
> more?

We'd have to wrap the build process for the unit tests to report into 
junit on error.

tests/Makefile.inc already looks for JUNIT_OUTPUT. The target in 
tests/Makefile.common for $($(1)-bin) would have to be extended to 
create a report on build failure.

The junit.xml target in payloads/libpayload/Makefile.inc might serve as 
inspiration.


Patrick

[Jakub Czapiga]

> We'd have to wrap the build process for the unit tests to report into
> junit on error.
>
> tests/Makefile.inc already looks for JUNIT_OUTPUT. The target in
> tests/Makefile.common for $($(1)-bin) would have to be extended to
> create a report on build failure.

It will not be enough. $($(1)-bin) target is responsible for linking test
binary. $($(1)-objs) would also have to be extended to detect whether
a particular file failed to compile.
> The junit.xml target in payloads/libpayload/Makefile.inc might serve as
> inspiration.

Thanks for the tip. It will come in handy.
I will try to find some time to extend unit-tests framework to support
tests compilation failures reports in coreboot and libpayload makefiles.

--
Jakub

[Jakub Czapiga]

Implementation [1] of unit-tests build failures reporting is now
available for review. With this change failing unit-tests builds are visible [2]
in the Jenkins job status in the Test Results section.

[1] https://review.coreboot.org/c/coreboot/+/67372
[2]* https://qa.coreboot.org/job/coreboot-gerrit/215975/

* Link [2] might stop working soon as Jenkins is not keeping everything forever

-- 
Jakub

On Mon, Sep 5, 2022 at 5:33 PM Jakub Czapiga <[e-mail]> wrote:
> > We'd have to wrap the build process for the unit tests to report into
> > junit on error.
> >
> > tests/Makefile.inc already looks for JUNIT_OUTPUT. The target in
> > tests/Makefile.common for $($(1)-bin) would have to be extended to
> > create a report on build failure.
>
> It will not be enough. $($(1)-bin) target is responsible for linking test
> binary. $($(1)-objs) would also have to be extended to detect whether
> a particular file failed to compile.
> > The junit.xml target in payloads/libpayload/Makefile.inc might serve as
> > inspiration.
>
> Thanks for the tip. It will come in handy.
> I will try to find some time to extend unit-tests framework to support
> tests compilation failures reports in coreboot and libpayload makefiles.