Hi Julius, I've fixed the issue you were seeing. Thanks for bringing it to our attention.
There were 2 things that were contributing to jenkins not seeing the libpayload files: 1) They were deeper in the tree than we were looking for junit*.xml files. These were only getting picked up two levels deep into the payloads directory: payloads/*/*/junit*.xml 2) We were also looking for junit.xml files anywhere in the tree, but the output files only match junit*.xml files.
Here's the list of files we needed to find: ./payloads/libpayload/build/coverage/junit-libpayload-tests-build.xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-has-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-no-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-has-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-no-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-has-fallback-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-no-fallback-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libc_fmap_locate_area-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_drivers_speaker-test(tests).xml
I've updated the general pattern to find junit*.xml files anywhere in the tree, and now they're seen by the jenkins log collector.
In the output of https://qa.coreboot.org/job/coreboot-gerrit/253059/%C2%A0you can see that we now get 4 errors: Test Result (4 failures / +4) tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned
Thanks again for helping to fix this problem.
Martin
Feb 28, 2024, 16:22 by jwerner@chromium.org:
Hmm... well, I just uploaded a CL that clearly breaks one of the tests, and Jenkins still gave me a Verified+1: https://review.coreboot.org/80788
I also don't see any info about the tests running on the test report (https://qa.coreboot.org/job/coreboot-gerrit/252941/testReport/(root)/). So if the tests are running (and the code you pointed out strongly looks like they should be), it's probably a problem with reading back the test results? How does Jenkins collect those XML files?
On Tue, Feb 27, 2024 at 7:04 PM Martin Roth gaumless@tutanota.com wrote:
Hey Julius, Here's what we running: https://review.coreboot.org/plugins/gitiles/coreboot/+/refs/heads/main/util/...
Martin Feb 27, 2024 at 18:31 by jwerner@chromium.org:
Hi,
We've had unit test support in libpayload for over two years now, but I just noticed that it doesn't look like we're actually running them in the CI? E.g. if I look through a test run like this https://qa.coreboot.org/job/coreboot-gerrit/252761/testReport/(root)/ I can't find them anywhere (most of them have confusingly similar names with coreboot unit tests, but "fmap_locate_area-test" is a unique one).
Is that intended or an oversight? Would it be easy to fix? (I'm not super familiar with how the tests are integrated into Jenkins, but the libpayload framework is pretty much a direct copy of the coreboot framework, so if we can run the coreboot ones hopefully it should be easy to run the libpayload ones as well?) _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Great, thanks a lot for tracking this down and getting it fixed!
On Sun, Mar 3, 2024 at 12:50 PM Martin Roth gaumless@tutanota.com wrote:
Hi Julius, I've fixed the issue you were seeing. Thanks for bringing it to our attention.
There were 2 things that were contributing to jenkins not seeing the libpayload files:
- They were deeper in the tree than we were looking for junit*.xml files. These were only getting picked up two levels deep into the payloads directory: payloads/*/*/junit*.xml
- We were also looking for junit.xml files anywhere in the tree, but the output files only match junit*.xml files.
Here's the list of files we needed to find: ./payloads/libpayload/build/coverage/junit-libpayload-tests-build.xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-has-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-no-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-has-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-no-sha512-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-has-fallback-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-no-fallback-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_libc_fmap_locate_area-test(tests).xml ./payloads/libpayload/build/coverage/junit-libpayload-tests_drivers_speaker-test(tests).xml
I've updated the general pattern to find junit*.xml files anywhere in the tree, and now they're seen by the jenkins log collector.
In the output of https://qa.coreboot.org/job/coreboot-gerrit/253059/ you can see that we now get 4 errors: Test Result (4 failures / +4) tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned
Thanks again for helping to fix this problem.
Martin
Feb 28, 2024, 16:22 by jwerner@chromium.org:
Hmm... well, I just uploaded a CL that clearly breaks one of the tests, and Jenkins still gave me a Verified+1: https://review.coreboot.org/80788
I also don't see any info about the tests running on the test report (https://qa.coreboot.org/job/coreboot-gerrit/252941/testReport/(root)/). So if the tests are running (and the code you pointed out strongly looks like they should be), it's probably a problem with reading back the test results? How does Jenkins collect those XML files?
On Tue, Feb 27, 2024 at 7:04 PM Martin Roth gaumless@tutanota.com wrote:
Hey Julius, Here's what we running: https://review.coreboot.org/plugins/gitiles/coreboot/+/refs/heads/main/util/...
Martin Feb 27, 2024 at 18:31 by jwerner@chromium.org:
Hi,
We've had unit test support in libpayload for over two years now, but I just noticed that it doesn't look like we're actually running them in the CI? E.g. if I look through a test run like this https://qa.coreboot.org/job/coreboot-gerrit/252761/testReport/(root)/ I can't find them anywhere (most of them have confusingly similar names with coreboot unit tests, but "fmap_locate_area-test" is a unique one).
Is that intended or an oversight? Would it be easy to fix? (I'm not super familiar with how the tests are integrated into Jenkins, but the libpayload framework is pretty much a direct copy of the coreboot framework, so if we can run the coreboot ones hopefully it should be easy to run the libpayload ones as well?) _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-
Julius Werner -
Martin Roth