Patrick Georgi has uploaded this change for review.
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06
Signed-off-by: Patrick Georgi <pgeorgi@google.com>
---
A Documentation/technotes/2019-12-firmware-testing.md
M Documentation/technotes/index.md
2 files changed, 176 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/1
diff --git a/Documentation/technotes/2019-12-firmware-testing.md b/Documentation/technotes/2019-12-firmware-testing.md
new file mode 100644
index 0000000..64838a7
--- /dev/null
+++ b/Documentation/technotes/2019-12-firmware-testing.md
@@ -0,0 +1,175 @@
+# Draft design of a firmware-aware integration test system
+
+## Problem statement
+
+Most full system test systems have assumptions baked in that make them
+unsuitable for firmware level testing. For example, both Chrome OS test
+runners assume that the DUT (Device Under Test) has an operational
+kernel + network + sshd. That's a useful assumption when covering
+userland testing, but not for firmware. Lava has its downsides as well
+(TODO: as reported by Philipp, need details).
+
+This document outlines a system that, when implemented, helps
+automatically test properties of firmware level code. It's _likely_
+also suitable for later stages (kernel and userland), but they're
+not the main concern: there's enough tooling out there for them.
+
+## Collecting requirements
+
+This project doesn't consider a test system running only on the DUT
+so it will require a test host that remains in control. Apart from
+that, the system should be useful at all ends of the problem space:
+hobbyists with little money to spare just the same as test labs
+running automated continuous testing.
+
+Beyond DUT and test host, the system requires some way for them to
+communicate, so we'll expect a bidirectional serial console connection
+between them (although we don't care much about how this connection
+looks like in practice).
+
+Such a bare-bone operation might require manual intervention directed
+by the test host (e.g. the host asking the user to reboot the DUT when
+it got stuck). It may also be augmented by additional circuitry that
+allows automating these steps.
+
+These constraints ensure that a test suite can be executed "one-off"
+by a developer with minimal hardware investment while also allowing
+automated systems to execute the tests without user intervention.
+
+So these are
+
+*Requirement 1*: The system must be usable with minimal investment for
+manual operation: a host, a DUT and a bidirectional serial connection
+between them.
+
+and
+
+*Requirement 2*: The system must be able to drive additional controls
+to support fully automated operation.
+
+There will be lots of different types of DUTs, and not all tests
+apply to all of them: There's no battery on a desktop board, so no
+need to test charging. A test for charging should be automatically
+skipped on such a DUT without counting as failure.
+
+*Requirement 3*: The system must allow to specify DUTs and their
+properties, as well as the properties that a test requires to be
+meaningful.
+
+Tests should be self-contained. This means that they should start
+from a flash cycle (that a tool like flashrom would optimize away to
+a no-op if flash is still in the expected state) and power-on.
+
+*Requirement 4*: Tests always start with the required firmware image
+to write and explicit power-on.
+
+Tests must be able to state that other tests must have succeeded before
+them: A test that exercises boot loader options doesn't fail when the
+boot process never gets to the options screen, it just shouldn't run.
+
+*Requirement 5*: Tests must be able to declare tests they depend on.
+
+State transitions should be well-defined: The DUT can move from the
+power-off state to the power-on state. It can also fail to do so and
+remain powered-off. These things need to be spelled out, together
+with the consequences that arise from each transition: it's expected
+that most transitions would be failures, but that should be noted down.
+
+And yet, this verbosity shouldn't mean that the tests become
+unreadable. Ideally they could be used by a human operator as "regular
+text" (with odd characters interspersed) outlining steps to execute
+manually, without a test runner.
+
+*Requirement 6*: It must be simple to state "Initiate action, success
+is W, X while failure is Y, Z".
+
+During the test, the host monitors the serial console and extracts
+events from it. Such events could be, for example, "coreboot enters
+ramstage". A test can expect events to appear on the stream and fail if
+unexpected events appear or no events appear within a given deadline.
+
+The observed events are reported to the user.
+
+*Requirement 7*: There's an event parser looking for things that
+happen on the DUT.
+
+The host can send characters to the DUT, for example in a boot loader
+or on a terminal provided by an OS that eventually boots.
+
+*Requirement 8*: Test must be able to run an expect-like language to
+drive complex interactions on the serial console.
+
+Some tests may be better run on the target system (e.g. to parse
+out tables in memory, or otherwise inspect boot states). They should
+have a minimal set of dependencies so they don't fail because the base
+system changed slightly, be portable across architectures and operating
+systems. The language should be popular and the implementation robust.
+
+*Requirement 9*: Implementation language must be popular, robust,
+portable and operate with minimal dependencies.
+
+## Implementation proposal
+
+The test system will be implemented in Go: it's reasonably popular
+(unlike Forth), robust (unlike shell scripts), its statically linked
+binaries have minimal dependencies on the host environment (unlike
+Python) and it supports multiple architectures and operating systems.
+
+The build creates a host side test runner (native OS/arch) and a
+variable number of DUT test runners (for all supported DUT OS/arch
+pairs) that are provided to the DUT on some storage medium (e.g. USB
+stick). In a fully automated environment this might require switching
+USB ports between host and DUT.
+
+There's a [liberally licensed implementation of expect][goexpect] that
+could be integrated for handling interactive consoles. It supports
+parallel execution which is useful to have a listener on the serial
+console (when not driven by expect).
+
+[goexpect]: https://github.com/google/goexpect
+
+The test runner is called with the names of a set of tests to execute
+(individually or as suites that tests can be assigned to) and the
+type of DUT to work with.
+
+From this, it knows how to drive the DUT. The "manual operation" mode
+would be just a special type of button driver that asks the user to
+conduct some operation and potentially to press Enter.
+
+Tests are defined in go packages, with per-package set up and tear
+down routines. The runner optimizes operations by telling the tear down
+routine if the next test will be from the same package (so the amount
+of tear down could be reduced). Setup isn't told about the previous
+state but should assume a random state. It's only allowed to assume
+that the tests it requires to execute earlier have passed. It can
+still optimize based on things it measures such as flashrom not writing
+unchanged blocks.
+
+### Flow
+
+```
+ DUT name
+ Tests to execute +--------------+
+ + +---->+Console driver|
+ | | +--------------+
+ | +---->+Button driver |
+ | | +--------------+
+ v v
+ +-----------+ +----------+
+ |Test runner+--------+DUT object+<+
+ +-----------+ +----------+ |
+ | executes |generates |
+ | v |
+ | sends+------------+ | control
+ +<-----+event stream| +----+
+ | +------------+ | |
+ | | |
+ | +----------+ |
+ +------------->+Host tests| |
+ | +----------+ |
+ | |
+ | +--------------+ call +------------------+
+ +------------->+Target tests +----------+Target test runner|
+ +--------------+ +------------------+
+```
+
diff --git a/Documentation/technotes/index.md b/Documentation/technotes/index.md
index 7c231fc..0df2a1b 100644
--- a/Documentation/technotes/index.md
+++ b/Documentation/technotes/index.md
@@ -2,3 +2,4 @@
* [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md)
* [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md)
+* [firmware test runner](2019-12-firmware-testing.md)
To view, visit change 37643. To unsubscribe, or for help writing mail filters, visit settings.