Patrick Georgi has uploaded this change for review. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 176 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/1
diff --git a/Documentation/technotes/2019-12-firmware-testing.md b/Documentation/technotes/2019-12-firmware-testing.md new file mode 100644 index 0000000..64838a7 --- /dev/null +++ b/Documentation/technotes/2019-12-firmware-testing.md @@ -0,0 +1,175 @@ +# Draft design of a firmware-aware integration test system + +## Problem statement + +Most full system test systems have assumptions baked in that make them +unsuitable for firmware level testing. For example, both Chrome OS test +runners assume that the DUT (Device Under Test) has an operational +kernel + network + sshd. That's a useful assumption when covering +userland testing, but not for firmware. Lava has its downsides as well +(TODO: as reported by Philipp, need details). + +This document outlines a system that, when implemented, helps +automatically test properties of firmware level code. It's _likely_ +also suitable for later stages (kernel and userland), but they're +not the main concern: there's enough tooling out there for them. + +## Collecting requirements + +This project doesn't consider a test system running only on the DUT +so it will require a test host that remains in control. Apart from +that, the system should be useful at all ends of the problem space: +hobbyists with little money to spare just the same as test labs +running automated continuous testing. + +Beyond DUT and test host, the system requires some way for them to +communicate, so we'll expect a bidirectional serial console connection +between them (although we don't care much about how this connection +looks like in practice). + +Such a bare-bone operation might require manual intervention directed +by the test host (e.g. the host asking the user to reboot the DUT when +it got stuck). It may also be augmented by additional circuitry that +allows automating these steps. + +These constraints ensure that a test suite can be executed "one-off" +by a developer with minimal hardware investment while also allowing +automated systems to execute the tests without user intervention. + +So these are + +*Requirement 1*: The system must be usable with minimal investment for +manual operation: a host, a DUT and a bidirectional serial connection +between them. + +and + +*Requirement 2*: The system must be able to drive additional controls +to support fully automated operation. + +There will be lots of different types of DUTs, and not all tests +apply to all of them: There's no battery on a desktop board, so no +need to test charging. A test for charging should be automatically +skipped on such a DUT without counting as failure. + +*Requirement 3*: The system must allow to specify DUTs and their +properties, as well as the properties that a test requires to be +meaningful. + +Tests should be self-contained. This means that they should start +from a flash cycle (that a tool like flashrom would optimize away to +a no-op if flash is still in the expected state) and power-on. + +*Requirement 4*: Tests always start with the required firmware image +to write and explicit power-on. + +Tests must be able to state that other tests must have succeeded before +them: A test that exercises boot loader options doesn't fail when the +boot process never gets to the options screen, it just shouldn't run. + +*Requirement 5*: Tests must be able to declare tests they depend on. + +State transitions should be well-defined: The DUT can move from the +power-off state to the power-on state. It can also fail to do so and +remain powered-off. These things need to be spelled out, together +with the consequences that arise from each transition: it's expected +that most transitions would be failures, but that should be noted down. + +And yet, this verbosity shouldn't mean that the tests become +unreadable. Ideally they could be used by a human operator as "regular +text" (with odd characters interspersed) outlining steps to execute +manually, without a test runner. + +*Requirement 6*: It must be simple to state "Initiate action, success +is W, X while failure is Y, Z". + +During the test, the host monitors the serial console and extracts +events from it. Such events could be, for example, "coreboot enters +ramstage". A test can expect events to appear on the stream and fail if +unexpected events appear or no events appear within a given deadline. + +The observed events are reported to the user. + +*Requirement 7*: There's an event parser looking for things that +happen on the DUT. + +The host can send characters to the DUT, for example in a boot loader +or on a terminal provided by an OS that eventually boots. + +*Requirement 8*: Test must be able to run an expect-like language to +drive complex interactions on the serial console. + +Some tests may be better run on the target system (e.g. to parse +out tables in memory, or otherwise inspect boot states). They should +have a minimal set of dependencies so they don't fail because the base +system changed slightly, be portable across architectures and operating +systems. The language should be popular and the implementation robust. + +*Requirement 9*: Implementation language must be popular, robust, +portable and operate with minimal dependencies. + +## Implementation proposal + +The test system will be implemented in Go: it's reasonably popular +(unlike Forth), robust (unlike shell scripts), its statically linked +binaries have minimal dependencies on the host environment (unlike +Python) and it supports multiple architectures and operating systems. + +The build creates a host side test runner (native OS/arch) and a +variable number of DUT test runners (for all supported DUT OS/arch +pairs) that are provided to the DUT on some storage medium (e.g. USB +stick). In a fully automated environment this might require switching +USB ports between host and DUT. + +There's a [liberally licensed implementation of expect][goexpect] that +could be integrated for handling interactive consoles. It supports +parallel execution which is useful to have a listener on the serial +console (when not driven by expect). + +[goexpect]: https://github.com/google/goexpect + +The test runner is called with the names of a set of tests to execute +(individually or as suites that tests can be assigned to) and the +type of DUT to work with. + +From this, it knows how to drive the DUT. The "manual operation" mode +would be just a special type of button driver that asks the user to +conduct some operation and potentially to press Enter. + +Tests are defined in go packages, with per-package set up and tear +down routines. The runner optimizes operations by telling the tear down +routine if the next test will be from the same package (so the amount +of tear down could be reduced). Setup isn't told about the previous +state but should assume a random state. It's only allowed to assume +that the tests it requires to execute earlier have passed. It can +still optimize based on things it measures such as flashrom not writing +unchanged blocks. + +### Flow + +``` + DUT name + Tests to execute +--------------+ + + +---->+Console driver| + | | +--------------+ + | +---->+Button driver | + | | +--------------+ + v v + +-----------+ +----------+ + |Test runner+--------+DUT object+<+ + +-----------+ +----------+ | + | executes |generates | + | v | + | sends+------------+ | control + +<-----+event stream| +----+ + | +------------+ | | + | | | + | +----------+ | + +------------->+Host tests| | + | +----------+ | + | | + | +--------------+ call +------------------+ + +------------->+Target tests +----------+Target test runner| + +--------------+ +------------------+ +``` + diff --git a/Documentation/technotes/index.md b/Documentation/technotes/index.md index 7c231fc..0df2a1b 100644 --- a/Documentation/technotes/index.md +++ b/Documentation/technotes/index.md @@ -2,3 +2,4 @@
* [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md) * [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md) +* [firmware test runner](2019-12-firmware-testing.md)
Hello Philipp Deppenwiese, build bot (Jenkins),
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#2).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 185 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/2
Hello Philipp Deppenwiese, build bot (Jenkins),
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#3).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 186 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/3
Paul Menzel has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 3:
(2 comments)
The format of the test results data for parsing and collection is still missing or outside the scope?
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 106: The language should be popular and the implementation robust. Required already below?
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 147: set up set-up/setup?
Angel Pons has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 3:
(5 comments)
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 9: Lava This lava? https://minecraft.gamepedia.com/Lava
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 20: test host Would be nice if the test host could be in control of more than one DUT. This is rather easy to do if the interfaces with the DUT are via USB devices. After booting, the network is also useful to have.
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 32: It may also be augmented by additional circuitry that : allows automating these steps. That's exactly the only thing I've done of my automated test setup. With just a few transistors and resistors, I managed to make my Raspberry Pi able to control my Asrock G41M-S3. I used three GPIOs: one of them controls the power button, another monitors the power LED state (to know if the board is on or off) and the last one switches between two flash chips using the HOLD# pin.
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 39: So these are Seems like something went missing here?
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 45: and ?
Hello Angel Pons, Philipp Deppenwiese, build bot (Jenkins),
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#4).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 191 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/4
Patrick Georgi has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 3:
(7 comments)
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 9: Lava
This lava? https://minecraft.gamepedia. […]
no, added the link.
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 20: test host
Would be nice if the test host could be in control of more than one DUT. […]
Ack on driving multiple DUTs (added). As for network, I consider that outside the scope of the test runner itself. I think that relying on network for test communication only adds complexity (which side failed configuring its network _this_ time?). The idea is to tailor this around the barest minimum of components and, incidentally, cut down on log sizes as well (the logs we get from Chromium OS' test infra are gigantic, and I can't claim that this improves anything).
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 32: It may also be augmented by additional circuitry that : allows automating these steps.
That's exactly the only thing I've done of my automated test setup. […]
so my plan will be to provide (compile-time) pluggable modules for that, and driving two kinds myself: the manual operator UI (on some random DUT I have here) and integration with Chrome OS' servo tools.
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 39: So these are
Seems like something went missing here?
"So these are ... Requirement 1 ... and ... Requirement 2", maybe I got overboard with being fancy here? ;-)
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 45: and
?
see above.
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 106: The language should be popular and the implementation robust.
Required already below?
I styled the doc in having prose explaining the properties and with requirements extracted from them interspersed. You're right that the direct repetition isn't helpful though. Hmm...
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 147: set up
set-up/setup?
Done
Patrick Rudolph has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 4:
(2 comments)
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 63: Requirement Does it include things like IFD or ME? How are they stored? How do they end up in the final image?
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 70: Requirement I don't see that being useful. It only adds complexity in the dispatcher.
Patrick Rudolph has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 4:
(1 comment)
https://review.coreboot.org/c/coreboot/+/37643/4/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/4/Documentation/technotes/201... PS4, Line 10: ( Issues with LAVA: * lava-slave and lava-master must use same version. Undefined behavior if it's not the case. * Qemu on debian9 is broken, must use lava docker 2019.12 at least * Lava older than 2019.10 required KVM to be enabled for the qemu target * No build-in job groups * No error message on connection failures or invalid config * No support for parsing text, expects a serial shell to run commands * UEFI/U-Boot is the lowest supported firmware (as it has a shell)
Hello Angel Pons, Philipp Deppenwiese, build bot (Jenkins),
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#5).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 223 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/5
Patrick Georgi has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 5:
(5 comments)
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 39: So these are
"So these are ... Requirement 1 ... and ... […]
Done
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 63: Requirement
Does it include things like IFD or ME? […]
That's really orthogonal to the test tool itself. Clarified that getting an image is out of scope of testing.
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 70: Requirement
I don't see that being useful. It only adds complexity in the dispatcher.
It's mostly "if test X didn't succeed, don't bother with tests Y and Z (because they're bound to fail)". Rephrased it.
https://review.coreboot.org/c/coreboot/+/37643/3/Documentation/technotes/201... PS3, Line 106: The language should be popular and the implementation robust.
I styled the doc in having prose explaining the properties and with requirements extracted from them […]
Rephrased it here to provide some more motivation _why_ requirement 9 is desirable.
https://review.coreboot.org/c/coreboot/+/37643/4/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/4/Documentation/technotes/201... PS4, Line 10: (
Issues with LAVA: […]
Thanks, I added this.
Hello Angel Pons, Philipp Deppenwiese, build bot (Jenkins),
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#6).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 224 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/6
Paul Menzel has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 6:
(1 comment)
https://review.coreboot.org/c/coreboot/+/37643/6/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/6/Documentation/technotes/201... PS6, Line 213: setup set up
Patrick Georgi has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 6:
(1 comment)
https://review.coreboot.org/c/coreboot/+/37643/6/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/6/Documentation/technotes/201... PS6, Line 213: setup
set up
Thanks, done.
Hello Angel Pons, Philipp Deppenwiese, build bot (Jenkins),
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#7).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 224 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/7
Jett Rink has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 7:
(7 comments)
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 46: * nit: can we make this **bold** instead of *italic*; I think it would be easier to read. Thoughts?
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 58: must allow to specify DUTs and nit: must allow DUTs to specify
(minor wording issue)
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 134: The test system will be implemented in Go: Are we also going to adopt Golang's table driven test methodology? https://github.com/golang/go/wiki/TableDrivenTests
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 139: The build creates a host side test runner (native OS/arch) and a : variable number of DUT test runners (for all supported DUT OS/arch : pairs) that are provided to the DUT on some storage medium (e.g. USB : stick). In a fully automated environment this might require switching : USB ports between host and DUT. are we going to use https://github.com/mndrix/tap-go for golang TAP?
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 173: dita is this supposed to render as something in gitiles?
What do you think about making this in lucidcharts or something?
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 203: no not
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 203: solution soultions
Kevin Shelton has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 7:
(1 comment)
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 6: both Chrome OS test : runners assume that the DUT (Device Under Test) has an operational : kernel + network + sshd The tast design principles (https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/desig...) do mention communication happening over a single persistent ssh connection, but I suspect changing that wouldn't be overly controversial. Have you considered trying to remove the assumption from tast?
Patrick Georgi has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 7:
(1 comment)
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 6: both Chrome OS test : runners assume that the DUT (Device Under Test) has an operational : kernel + network + sshd
The tast design principles (https://chromium.googlesource. […]
It's not just moving the connection somewhere else. The test system also (for the most part) assumes that the interaction over that connection is "run a program on the DUT" which we also don't do.
I guess we could get rid of that as well, but at what point are we just stuffing two completely different systems into a single binary?
Kevin Shelton has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 7:
(1 comment)
Patch Set 7:
(1 comment)
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 6: both Chrome OS test : runners assume that the DUT (Device Under Test) has an operational : kernel + network + sshd
It's not just moving the connection somewhere else. […]
I see, a dedicated test runner makes sense to me in that context (avoiding two different systems in a single binary).
Hello Philipp Deppenwiese, build bot (Jenkins), Angel Pons,
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#8).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 228 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/8
Patrick Georgi has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 7:
(7 comments)
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... File Documentation/technotes/2019-12-firmware-testing.md:
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 46: *
nit: can we make this **bold** instead of *italic*; I think it would be easier to read. […]
Done
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 58: must allow to specify DUTs and
nit: must allow DUTs to specify […]
that... depends... I was thinking of having the management side know about the DUTs' properties. While the DUTs could report that for themselves, such an approach requires the DUTs to be functional enough to state that. So if anything I'd see this as a confirmation thing (serialtest claims $set_of_properties, DUT disagrees -> failure).
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 134: The test system will be implemented in Go:
Are we also going to adopt Golang's table driven test methodology? https://github. […]
That seems to be rather limited and mostly suitable for unit tests (where you want to check that a=f(b) holds true for a set of a,b pairs)
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 139: The build creates a host side test runner (native OS/arch) and a : variable number of DUT test runners (for all supported DUT OS/arch : pairs) that are provided to the DUT on some storage medium (e.g. USB : stick). In a fully automated environment this might require switching : USB ports between host and DUT.
are we going to use https://github. […]
I was looking into option for TAP and I think this one came up, yes. Added after requirement 10.
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 173: dita
is this supposed to render as something in gitiles? […]
These documents are rendered to doc.coreboot.org which has support for ditaa. The advantage of this is that people without rendering get a useful approximation in the form of ascii art.
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 203: no
not
Done
https://review.coreboot.org/c/coreboot/+/37643/7/Documentation/technotes/201... PS7, Line 203: solution
soultions
Done
Hello Philipp Deppenwiese, build bot (Jenkins), Angel Pons,
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/37643
to look at the new patch set (#9).
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Documentation: Add proposal for firmware testing
Change-Id: Icb9380050f8ff1aa13ecbb501079e2556e43ca06 Signed-off-by: Patrick Georgi pgeorgi@google.com --- A Documentation/technotes/2019-12-firmware-testing.md M Documentation/technotes/index.md 2 files changed, 228 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/43/37643/9
Angel Pons has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Patch Set 9: Code-Review+1
Patrick Georgi has abandoned this change. ( https://review.coreboot.org/c/coreboot/+/37643 )
Change subject: Documentation: Add proposal for firmware testing ......................................................................
Abandoned
-
Angel Pons (Code Review) -
Jett Rink (Code Review)
-
Kevin Shelton (Code Review)
-
Patrick Georgi (Code Review)
-
Patrick Rudolph (Code Review)
-
Paul Menzel (Code Review)