Name of user not set #1002873 has uploaded this change for review.
Documentation: Add proposal for firmware unit testing
Signed-off-by: Jan Dabros <jsd@semihalf.com>
Change-Id: I552d6c3373219978b8e5fd4304f993d920425431
---
A Documentation/technotes/2020-03-unit-testing-coreboot.md
M Documentation/technotes/index.md
2 files changed, 257 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/93/39893/1
diff --git a/Documentation/technotes/2020-03-unit-testing-coreboot.md b/Documentation/technotes/2020-03-unit-testing-coreboot.md
new file mode 100644
index 0000000..fcc9b0c
--- /dev/null
+++ b/Documentation/technotes/2020-03-unit-testing-coreboot.md
@@ -0,0 +1,256 @@
+# Unit testing coreboot
+
+## Preface
+First part of this document, Introduction, comprises disambiguation for what
+unit testing is and what is not. This definition will be a basis for the whole
+paper.
+
+Next, Rationale, explains why to use unit testing and how coreboot specifically
+may benefit from it.
+
+This is followed by evaluation of different available free C unit test
+frameworks. Firstly, collection of requirements is provided. Secondly, there is
+a description of a few selected candidates. Finally, requirements are applied to
+candidates to see if they might be a good fit.
+
+Fourth part is a summary of evaluation, with proposal of unit test frameworks
+for coreboot to be used. This may be a good starting point for a discussion
+within community.
+
+Finally, Implementation proposal paragraph touches how build system and coreboot
+codebase in general should be organized, in order to support unit testing. This
+comprises couple of design considerations which need to be addressed.
+
+## Introduction
+A unit test is supposed to test a single unit of code in isolation. In C
+language (in contrary to OOP) unit usually means a function. One may also
+consider unit under test to be a single compilation unit which exposes some
+API (set of functions). A function, talking to some external component can be
+tested if this component can be mocked out.
+
+In other words (looking from C compilation angle), there should be no extra
+dependencies (executables) required beside unit under test and test harness in
+order to compile unit test binary. Test harness, beside code examining a
+routines, may comprise test framework implementation.
+
+Unit testing is not an integration testing and it doesn't replace it. First of
+all, integration tests cover larger set of components and interactions between
+them. Positive integration test result gives more confidence than a positive
+unit test does. Furthermore, unit tests are running on the build machine, while
+integration tests usually are executed on the target (or simulator).
+
+## Rationale
+Considering above, what is the benefit of unit testing, especially keeping in
+mind that coreboot is low-level firmware? Unit tests should be quick, thus may
+be executed frequently during development process. It is much easier to build
+and run a unit test on a build machine, than any integration test. This in turn
+may be used by dev to gather extra confidence early during code development
+process. Actually developer may even write unit tests earlier than the code -
+see [TDD](https://en.wikipedia.org/wiki/Test-driven_development) concept.
+
+That being said, unit testing embedded C code is a difficult task, due to
+significant amount of dependencies on underlying hardware. This dependencies
+may be broken using mocking concept, to some degree. However when mocks need to
+be too complex, then such a unit test cannot address an idea of being quick and
+simple.
+
+Writing unit tests for a code (both new and currently existing) may be favorable
+for the code quality. It is not only about finding bugs, but in general - easily
+testable code is a good code.
+
+coreboot benefits the most from testing common libraries (lib/, commonlib/,
+payloads/libpayload), coreboot infrastructure (console/, device/, security/) and
+to some extent hardware-related code.
+
+## Evaluation of unit testing frameworks
+
+### Requirements
+Requirements for unit testing frameworks:
+
+* Easy to use
+* Few dependencies
+
+ Standard C library is all we should need
+
+* Isolation between tests
+* Support for mocking
+* Easy to integrate with build system/build tools
+
+ JUnit-like XML output format for Jenkins
+
+* Compiler similarity
+
+ Ideally the same compiler should be used for building firmware executables
+ and test binaries, however this is very unlikely (even if the ARCH for build
+ machine and target is the same). At least the compiler on build machine should
+ support the same dialect as the compiler for target executables.
+
+* Popularity is a plus
+
+ The bigger community the better
+
+* Language similarity
+
+ Assumption that all firmware devs know C
+
+* Extra features may be a plus
+* Proper license
+
+ This should not be a blocker, since test binaries are not distributed.
+ However ideally GPL.
+
+### Candidates
+There is a lot of frameworks which allow unit testing C code
+([list](https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C) from
+Wikipedia). While not all of them were evaluated for obvious reasons, couple
+of them were selected based on the good opinions among C devs, popularity and
+fitting above criteria.
+
+* [SputUnit](https://www.use-strict.de/sput-unit-testing/)
+* [GoogleTest](https://github.com/google/googletest)
+* [Cmocka](https://cmocka.org/)
+* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling)
+
+Three more frameworks should be mentioned here, however they weren't tried
+within coreboot:
+ * [Check](https://libcheck.github.io/check/)
+ * [Criterion](https://github.com/Snaipe/Criterion/)
+
+They offer similar functionality as Unity, while at the same time don’t seem
+to have similarly active communities.
+
+ * [CUnit](http://cunit.sourceforge.net/)
+
+Another one which is rather widely used, however it doesn't have good support
+for mocking.
+
+If anyone has good experience with some framework which is not listed above, it
+will be highly appreciated to give a note about this to the author. In such
+cases, it may be necessary to investigate such tools deeply and then reconsider
+the decision.
+
+### Evaluation
+* [SputUnit](https://www.use-strict.de/sput-unit-testing/)
+ * Pros
+ * No dependencies, one header file to include - that’s all
+ * Pure C
+ * Very easy to use
+ * Cons
+ * Main repo doesn’t have support for generating JUnit XML reports for
+ Jenkins to consume - this feature is available only on the fork from
+ SputUnit called “Sput_report”. It makes it niche in a niche, so there are
+ some reservations whether support for this will be satisfying
+ * No support for mocks
+ * Not too popular
+ * No automatic test registration
+ * BSD license
+* [GoogleTest](https://github.com/google/googletest)
+ * Pros
+ * Automatic test registration
+ * Support for different output formats (including XML for Jenkins)
+ * Good support, widely used, the biggest and the most active community ouf
+ of all frameworks that were investigated
+ * Available as a package in the most common distributions
+ * Test fixtures easily available
+ * Well documented
+ * Easy to integrate with an IDE
+ * Cons
+ * Requires C++11 compiler
+ * To make most out of it (use GMock) C++ knowledge is required
+ * BSD license
+* [Cmocka](https://cmocka.org/)
+ * Pros
+ * Self-contained, autonomous framework
+ * Pure C
+ * API is well documented
+ * Multiple output formats (including XML for Jenkins)
+ * Available as a package in the most common distributions
+ * Used in some popular open source projects (libssh, OpenVPN, Samba)
+ * Test fixtures available
+ * Support for exception handling
+ * Cons
+ * No automatic test registration
+ * It will require some effort to make it works from within an IDE
+ * Apache 2.0 license (not compatible with GPLv2)
+* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling)
+ * Pros
+ * Pure C (Unity testing framework itself, not test runner)
+ * Support for different output formats (including XML for Jenkins)
+ * There are some (rather easy) hints how to use this from an IDE (e.g. Eclipse)
+ * Cons
+ * Test runner (Ceedling) is not written in C - uses Ruby
+ * Mocking/Exception handling functionalities are actually separate tools
+ * No automatic test registration
+ * Not too popular
+
+### Summary & framework proposal
+After research, proposal is to go with Cmocka unit test framework. It fulfills
+all evaluation criteria we have. It is rather easy to use, doesn’t have
+extra dependencies, written fully in C, allows for tests fixtures and some
+popular open source projects already are using it. Cmocka also include support
+for mocks.
+
+There are some cons, like for example lack of automatic test registration,
+however this is minor issue, just requires minimal additional work from
+developer. In the same time, it may be worth to propose improvement to Cmocka
+community or simply apply some extra wrapper with demanded functionality.
+
+That being said, Unity and GoogleTest frameworks are also very good. However
+GoogleTest requires C++ familiarity and compiler, and Unity is rather not too
+popular framework (which doesn’t mean worse). It is worth to mention, that such
+evaluation will always be impacted by author subjective point of view.
+
+## Implementation proposal
+
+### Framework as a submodule or external package
+Unit test frameworks may be either compiled from source (form a git submodule
+under 3rdparty/) or pre-compiled as a package. The second option seems to be
+easier to maintain, while in the same time may bring some unwanted consequences
+(different version across distributions, frequent changes in API). It make sense
+to initially experiment with packages and check how it works. If this will
+brings any issues, then it is always possible to switch to submodule approach.
+
+### Build configuration
+While building unit under test object file, it is necessary to apply some
+configuration (config) just like when building whole firmware. For simplicity,
+one config.h may be pre-built and then used for building all unit tests. In the
+same time, if integrated with Jenkins, it may be preferred to use every
+available config for periodic builds.
+
+My proposal is to go with `qemu_x86_i440fx` config for usual developer builds.
+
+### Integration with build system
+To get the most out of unit testing framework, it should be integrated with
+Jenkins automation server. Verification of all unit tests for new changes may
+improve code reliability to some extent.
+
+### Directory structure
+Tests should be kept separate from the code, while in the same time it must be
+easy to match code with test harness.
+
+My proposal is to create new directory for test files and every test file name
+should have added a prefix "-test" to the basename of a corresponding unit
+under test module. Below example shows how this may be organized:
+
+```bash
+├── build
+│ ├── tests <-test binaries
+│
+├── src
+│ ├── lib
+│ │ ├── malloc.c <- unit under test
+│ │ ├── string.c <- unit under test
+│ ├── commonlib
+│ ├── sort.c <- unit under test
+│
+├── tests
+│ ├── include
+│ │ ├── config.h <- pre-built config used for tests' builds
+│ ├── Makefile.inc
+│ ├── lib
+│ │ ├── malloc-test.c <- test code for src/lib/malloc.c
+│ │ ├── string-test.c <- test code for src/lib/string.c
+│ ├── commonlib
+│ ├── sort-test.c <- test code for src/commonlib/sort.c
+├──
+```
diff --git a/Documentation/technotes/index.md b/Documentation/technotes/index.md
index 7c231fc..5367e69 100644
--- a/Documentation/technotes/index.md
+++ b/Documentation/technotes/index.md
@@ -2,3 +2,4 @@
* [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md)
* [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md)
+* [Unit testing coreboot](2020-03-unit-testing-coreboot.md)
To view, visit change 39893. To unsubscribe, or for help writing mail filters, visit settings.