Patrick Georgi has uploaded this change for review. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 176 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/1
diff --git a/Documentation/technotes/2019-12-firmware-testing.md b/Documentation/technotes/2019-12-firmware-testing.md new file mode 100644 index 0000000..64838a7 --- /dev/null +++ b/Documentation/technotes/2019-12-firmware-testing.md @@ -0,0 +1,175 @@ +# Draft design of a firmware-aware integration test system + +## Problem statement + +Most full system test systems have assumptions baked in that make them +unsuitable for firmware level testing. For example, both Chrome OS test +runners assume that the DUT (Device Under Test) has an operational +kernel + network + sshd. That's a useful assumption when covering +userland testing, but not for firmware. Lava has its downsides as well +(TODO: as reported by Philipp, need details). + +This document outlines a system that, when implemented, helps +automatically test properties of firmware level code. It's _likely_ +also suitable for later stages (kernel and userland), but they're +not the main concern: there's enough tooling out there for them. + +## Collecting requirements + +This project doesn't consider a test system running only on the DUT +so it will require a test host that remains in control. Apart from +that, the system should be useful at all ends of the problem space: +hobbyists with little money to spare just the same as test labs +running automated continuous testing. + +Beyond DUT and test host, the system requires some way for them to +communicate, so we'll expect a bidirectional serial console connection +between them (although we don't care much about how this connection +looks like in practice). + +Such a bare-bone operation might require manual intervention directed +by the test host (e.g. the host asking the user to reboot the DUT when +it got stuck). It may also be augmented by additional circuitry that +allows automating these steps. + +These constraints ensure that a test suite can be executed "one-off" +by a developer with minimal hardware investment while also allowing +automated systems to execute the tests without user intervention. + +So these are + +*Requirement 1*: The system must be usable with minimal investment for +manual operation: a host, a DUT and a bidirectional serial connection +between them. + +and + +*Requirement 2*: The system must be able to drive additional controls +to support fully automated operation. + +There will be lots of different types of DUTs, and not all tests +apply to all of them: There's no battery on a desktop board, so no +need to test charging. A test for charging should be automatically +skipped on such a DUT without counting as failure. + +*Requirement 3*: The system must allow to specify DUTs and their +properties, as well as the properties that a test requires to be +meaningful. + +Tests should be self-contained. This means that they should start +from a flash cycle (that a tool like flashrom would optimize away to +a no-op if flash is still in the expected state) and power-on. + +*Requirement 4*: Tests always start with the required firmware image +to write and explicit power-on. + +Tests must be able to state that other tests must have succeeded before +them: A test that exercises boot loader options doesn't fail when the +boot process never gets to the options screen, it just shouldn't run. + +*Requirement 5*: Tests must be able to declare tests they depend on. + +State transitions should be well-defined: The DUT can move from the +power-off state to the power-on state. It can also fail to do so and +remain powered-off. These things need to be spelled out, together +with the consequences that arise from each transition: it's expected +that most transitions would be failures, but that should be noted down. + +And yet, this verbosity shouldn't mean that the tests become +unreadable. Ideally they could be used by a human operator as "regular +text" (with odd characters interspersed) outlining steps to execute +manually, without a test runner. + +*Requirement 6*: It must be simple to state "Initiate action, success +is W, X while failure is Y, Z". + +During the test, the host monitors the serial console and extracts +events from it. Such events could be, for example, "coreboot enters +ramstage". A test can expect events to appear on the stream and fail if +unexpected events appear or no events appear within a given deadline. + +The observed events are reported to the user. + +*Requirement 7*: There's an event parser looking for things that +happen on the DUT. + +The host can send characters to the DUT, for example in a boot loader +or on a terminal provided by an OS that eventually boots. + +*Requirement 8*: Test must be able to run an expect-like language to +drive complex interactions on the serial console. + +Some tests may be better run on the target system (e.g. to parse +out tables in memory, or otherwise inspect boot states). They should +have a minimal set of dependencies so they don't fail because the base +system changed slightly, be portable across architectures and operating +systems. The language should be popular and the implementation robust. + +*Requirement 9*: Implementation language must be popular, robust, +portable and operate with minimal dependencies. + +## Implementation proposal + +The test system will be implemented in Go: it's reasonably popular +(unlike Forth), robust (unlike shell scripts), its statically linked +binaries have minimal dependencies on the host environment (unlike +Python) and it supports multiple architectures and operating systems. + +The build creates a host side test runner (native OS/arch) and a +variable number of DUT test runners (for all supported DUT OS/arch +pairs) that are provided to the DUT on some storage medium (e.g. USB +stick). In a fully automated environment this might require switching +USB ports between host and DUT. + +There's a [liberally licensed implementation of expect][goexpect] that +could be integrated for handling interactive consoles. It supports +parallel execution which is useful to have a listener on the serial +console (when not driven by expect). + +[goexpect]: https://github.com/google/goexpect + +The test runner is called with the names of a set of tests to execute +(individually or as suites that tests can be assigned to) and the +type of DUT to work with. + +From this, it knows how to drive the DUT. The "manual operation" mode +would be just a special type of button driver that asks the user to +conduct some operation and potentially to press Enter. + +Tests are defined in go packages, with per-package set up and tear +down routines. The runner optimizes operations by telling the tear down +routine if the next test will be from the same package (so the amount +of tear down could be reduced). Setup isn't told about the previous +state but should assume a random state. It's only allowed to assume +that the tests it requires to execute earlier have passed. It can +still optimize based on things it measures such as flashrom not writing +unchanged blocks. + +### Flow + +``` + DUT name + Tests to execute +--------------+ + + +---->+Console driver| + | | +--------------+ + | +---->+Button driver | + | | +--------------+ + v v + +-----------+ +----------+ + |Test runner+--------+DUT object+<+ + +-----------+ +----------+ | + | executes |generates | + | v | + | sends+------------+ | control + +<-----+event stream| +----+ + | +------------+ | | + | | | + | +----------+ | + +------------->+Host tests| | + | +----------+ | + | | + | +--------------+ call +------------------+ + +------------->+Target tests +----------+Target test runner| + +--------------+ +------------------+ +``` + diff --git a/Documentation/technotes/index.md b/Documentation/technotes/index.md index 7c231fc..0df2a1b 100644 --- a/Documentation/technotes/index.md +++ b/Documentation/technotes/index.md @@ -2,3 +2,4 @@
* [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md) * [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md) +* [firmware test runner](2019-12-firmware-testing.md)