Hi folks, this is an attempt to improve firmware testing by using the infrastructure and knowledge of the kernelci community. If you think this is not the right place, please point me in the right direction.
I'm a coreboot[1] developer trying to make sure that the master branch[2] doesn't regress. Currently there's no public firmware testing, only internal validation suites used by some companies that lack direct and automated feedback before a commit is actually merged. As this isn't a coreboot only topic, but applies to all open source "bios vendors", I added the u-boot project in CC as well.
For me firmware testing looks pretty similar to kernel testing: * flash firmware to test * boot a known good linux kernel * run tests in userspace and verify hardware/software works as expected
On the hardware side we have boards in our lab that allow remote power cycling and firmware flashing. It is attached to self hosted stock LAVA2018. But as we are firmware engineers, we don't want to deal with the administration of servers.
Here are a few questions for you: * Would it make sense to also cover open source firmware tests on kernelci? * Do you build the linux images yourself? * Would you accept firmware images generated by a third party? * Can anybody get an account for the LAVA server to run firmware test? * What communication channels do you recommend? * Will there be meetings or conferences to get in contact with the community to talk about this?
[1]: https://coreboot.org/ [2]: https://review.coreboot.org/
Kind Regards, Patrick Rudolph
9elements GmbH, Kortumstraße 19-21, 44787 Bochum, Germany Email: patrick.rudolph@9elements.com Phone: +49 234 / 68 94 188
Sitz der Gesellschaft: Bochum Handelsregister: Amtsgericht Bochum, HRB 13207 Geschäftsführung: Eray Basar, Sebastian Deutsch
On Fri, Nov 08, 2019 at 10:35:10AM +0100, Patrick Rudolph wrote:
Hi folks, this is an attempt to improve firmware testing by using the infrastructure and knowledge of the kernelci community. If you think this is not the right place, please point me in the right direction.
I'm a coreboot[1] developer trying to make sure that the master branch[2] doesn't regress. Currently there's no public firmware testing, only internal validation suites used by some companies that lack direct and automated feedback before a commit is actually merged. As this isn't a coreboot only topic, but applies to all open source "bios vendors", I added the u-boot project in CC as well.
For me firmware testing looks pretty similar to kernel testing:
- flash firmware to test
- boot a known good linux kernel
- run tests in userspace and verify hardware/software works as expected
On the hardware side we have boards in our lab that allow remote power cycling and firmware flashing. It is attached to self hosted stock LAVA2018. But as we are firmware engineers, we don't want to deal with the administration of servers.
Here are a few questions for you:
- Would it make sense to also cover open source firmware tests on kernelci?
- Do you build the linux images yourself?
- Would you accept firmware images generated by a third party?
- Can anybody get an account for the LAVA server to run firmware test?
- What communication channels do you recommend?
- Will there be meetings or conferences to get in contact with the
community to talk about this?
Over in U-Boot, yes, we have some test suites that we run on real HW as well as QEMU-based hardware. And historically I've talked with Kevin once or twice about testing a current U-Boot on various easy-to-recover boards rather than just what it shipped with, but things never moved past the talking about it stage.
"Patrick Rudolph" patrick.rudolph@9elements.com writes:
Hi folks, this is an attempt to improve firmware testing by using the infrastructure and knowledge of the kernelci community. If you think this is not the right place, please point me in the right direction.
I'm a coreboot[1] developer trying to make sure that the master branch[2] doesn't regress. Currently there's no public firmware testing, only internal validation suites used by some companies that lack direct and automated feedback before a commit is actually merged. As this isn't a coreboot only topic, but applies to all open source "bios vendors", I added the u-boot project in CC as well.
For me firmware testing looks pretty similar to kernel testing:
- flash firmware to test
- boot a known good linux kernel
- run tests in userspace and verify hardware/software works as expected
On the hardware side we have boards in our lab that allow remote power cycling and firmware flashing. It is attached to self hosted stock LAVA2018. But as we are firmware engineers, we don't want to deal with the administration of servers.
Here are a few questions for you:
- Would it make sense to also cover open source firmware tests on kernelci?
Yes. We've talked about this a few times over the years, and all the infra and tooling we have built for kernelCI would be well suited for firmware testing.
The catch is mostly in time/resources invested. For now, we are primarily focused on kernel testing, but if there are contributors that want to extend this to firmware, we are an open-source project and welcome the contributions.
I would say the primary challenge here is that each SoC family, and sometimes even each board has unique challenges to (re)flashing the firmware as well as the ability to recover from non-working firmware.
This presents a problem not necessariuly for KernelCI but for the lab admins for that hardware, and especially those writing the LAVA device-types for those platforms.
I've added the main LAVA developer Remi Durrafort to Cc since he's been doing a lot of work in LAVA to make firmware updates and recovery mode usable from LAVA. Recent versions of LAVA seem to be growing better support for this: https://docs.lavasoftware.org/lava/bootloaders.html
- Do you build the linux images yourself?
We build all the linux kernel images, but rely on the lab admins to have build/flashed working firmware and bootloaders.
- Would you accept firmware images generated by a third party?
As long as the source is available and can be (re)built, that's typically fine.
- Can anybody get an account for the LAVA server to run firmware test?
That's up to each LAVA lab administrator. Today, KernelCI uses a distributed set of (mostly LAVA) labs. Each lab admin has given priveleges to KernelCI to send jobs.
- What communication channels do you recommend?
This mailing list is a good place to start, but to be completely honest, firmware testing is relatively low on our current list of priorites. But as I said above, it's feature we will welcome with open arms, even if it's not a feature that the current set of developers will be very active on.
I would say your first step is probably to get some support for the devices you care about into upstream LAVA. Once that is working, we would work on how KernelCI could start generating jobs for those devices.
- Will there be meetings or conferences to get in contact with the
community to talk about this?
We just had an automated-testing summit connected to Embedded Linux Conference Europe last month, where many of us were gathered.
Remi even gave a talk on bootloader testing with LAVA, maybe he can share the slides for that.
Kevin
-
Kevin Hilman -
Patrick Rudolph -
Tom Rini