Name of user not set #1002873 has uploaded this change for review. ( https://review.coreboot.org/c/coreboot/+/39893 )
Change subject: Documentation: Add proposal for firmware unit testing ......................................................................
Documentation: Add proposal for firmware unit testing
Signed-off-by: Jan Dabros jsd@semihalf.com Change-Id: I552d6c3373219978b8e5fd4304f993d920425431 --- A Documentation/technotes/2020-03-unit-testing-coreboot.md M Documentation/technotes/index.md 2 files changed, 257 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/93/39893/1
diff --git a/Documentation/technotes/2020-03-unit-testing-coreboot.md b/Documentation/technotes/2020-03-unit-testing-coreboot.md new file mode 100644 index 0000000..fcc9b0c --- /dev/null +++ b/Documentation/technotes/2020-03-unit-testing-coreboot.md @@ -0,0 +1,256 @@ +# Unit testing coreboot + +## Preface +First part of this document, Introduction, comprises disambiguation for what +unit testing is and what is not. This definition will be a basis for the whole +paper. + +Next, Rationale, explains why to use unit testing and how coreboot specifically +may benefit from it. + +This is followed by evaluation of different available free C unit test +frameworks. Firstly, collection of requirements is provided. Secondly, there is +a description of a few selected candidates. Finally, requirements are applied to +candidates to see if they might be a good fit. + +Fourth part is a summary of evaluation, with proposal of unit test frameworks +for coreboot to be used. This may be a good starting point for a discussion +within community. + +Finally, Implementation proposal paragraph touches how build system and coreboot +codebase in general should be organized, in order to support unit testing. This +comprises couple of design considerations which need to be addressed. + +## Introduction +A unit test is supposed to test a single unit of code in isolation. In C +language (in contrary to OOP) unit usually means a function. One may also +consider unit under test to be a single compilation unit which exposes some +API (set of functions). A function, talking to some external component can be +tested if this component can be mocked out. + +In other words (looking from C compilation angle), there should be no extra +dependencies (executables) required beside unit under test and test harness in +order to compile unit test binary. Test harness, beside code examining a +routines, may comprise test framework implementation. + +Unit testing is not an integration testing and it doesn't replace it. First of +all, integration tests cover larger set of components and interactions between +them. Positive integration test result gives more confidence than a positive +unit test does. Furthermore, unit tests are running on the build machine, while +integration tests usually are executed on the target (or simulator). + +## Rationale +Considering above, what is the benefit of unit testing, especially keeping in +mind that coreboot is low-level firmware? Unit tests should be quick, thus may +be executed frequently during development process. It is much easier to build +and run a unit test on a build machine, than any integration test. This in turn +may be used by dev to gather extra confidence early during code development +process. Actually developer may even write unit tests earlier than the code - +see [TDD](https://en.wikipedia.org/wiki/Test-driven_development) concept. + +That being said, unit testing embedded C code is a difficult task, due to +significant amount of dependencies on underlying hardware. This dependencies +may be broken using mocking concept, to some degree. However when mocks need to +be too complex, then such a unit test cannot address an idea of being quick and +simple. + +Writing unit tests for a code (both new and currently existing) may be favorable +for the code quality. It is not only about finding bugs, but in general - easily +testable code is a good code. + +coreboot benefits the most from testing common libraries (lib/, commonlib/, +payloads/libpayload), coreboot infrastructure (console/, device/, security/) and +to some extent hardware-related code. + +## Evaluation of unit testing frameworks + +### Requirements +Requirements for unit testing frameworks: + +* Easy to use +* Few dependencies + + Standard C library is all we should need + +* Isolation between tests +* Support for mocking +* Easy to integrate with build system/build tools + + JUnit-like XML output format for Jenkins + +* Compiler similarity + + Ideally the same compiler should be used for building firmware executables + and test binaries, however this is very unlikely (even if the ARCH for build + machine and target is the same). At least the compiler on build machine should + support the same dialect as the compiler for target executables. + +* Popularity is a plus + + The bigger community the better + +* Language similarity + + Assumption that all firmware devs know C + +* Extra features may be a plus +* Proper license + + This should not be a blocker, since test binaries are not distributed. + However ideally GPL. + +### Candidates +There is a lot of frameworks which allow unit testing C code +([list](https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C) from +Wikipedia). While not all of them were evaluated for obvious reasons, couple +of them were selected based on the good opinions among C devs, popularity and +fitting above criteria. + +* [SputUnit](https://www.use-strict.de/sput-unit-testing/) +* [GoogleTest](https://github.com/google/googletest) +* [Cmocka](https://cmocka.org/) +* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling) + +Three more frameworks should be mentioned here, however they weren't tried +within coreboot: + * [Check](https://libcheck.github.io/check/) + * [Criterion](https://github.com/Snaipe/Criterion/) + +They offer similar functionality as Unity, while at the same time don’t seem +to have similarly active communities. + + * [CUnit](http://cunit.sourceforge.net/) + +Another one which is rather widely used, however it doesn't have good support +for mocking. + +If anyone has good experience with some framework which is not listed above, it +will be highly appreciated to give a note about this to the author. In such +cases, it may be necessary to investigate such tools deeply and then reconsider +the decision. + +### Evaluation +* [SputUnit](https://www.use-strict.de/sput-unit-testing/) + * Pros + * No dependencies, one header file to include - that’s all + * Pure C + * Very easy to use + * Cons + * Main repo doesn’t have support for generating JUnit XML reports for + Jenkins to consume - this feature is available only on the fork from + SputUnit called “Sput_report”. It makes it niche in a niche, so there are + some reservations whether support for this will be satisfying + * No support for mocks + * Not too popular + * No automatic test registration + * BSD license +* [GoogleTest](https://github.com/google/googletest) + * Pros + * Automatic test registration + * Support for different output formats (including XML for Jenkins) + * Good support, widely used, the biggest and the most active community ouf + of all frameworks that were investigated + * Available as a package in the most common distributions + * Test fixtures easily available + * Well documented + * Easy to integrate with an IDE + * Cons + * Requires C++11 compiler + * To make most out of it (use GMock) C++ knowledge is required + * BSD license +* [Cmocka](https://cmocka.org/) + * Pros + * Self-contained, autonomous framework + * Pure C + * API is well documented + * Multiple output formats (including XML for Jenkins) + * Available as a package in the most common distributions + * Used in some popular open source projects (libssh, OpenVPN, Samba) + * Test fixtures available + * Support for exception handling + * Cons + * No automatic test registration + * It will require some effort to make it works from within an IDE + * Apache 2.0 license (not compatible with GPLv2) +* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling) + * Pros + * Pure C (Unity testing framework itself, not test runner) + * Support for different output formats (including XML for Jenkins) + * There are some (rather easy) hints how to use this from an IDE (e.g. Eclipse) + * Cons + * Test runner (Ceedling) is not written in C - uses Ruby + * Mocking/Exception handling functionalities are actually separate tools + * No automatic test registration + * Not too popular + +### Summary & framework proposal +After research, proposal is to go with Cmocka unit test framework. It fulfills +all evaluation criteria we have. It is rather easy to use, doesn’t have +extra dependencies, written fully in C, allows for tests fixtures and some +popular open source projects already are using it. Cmocka also include support +for mocks. + +There are some cons, like for example lack of automatic test registration, +however this is minor issue, just requires minimal additional work from +developer. In the same time, it may be worth to propose improvement to Cmocka +community or simply apply some extra wrapper with demanded functionality. + +That being said, Unity and GoogleTest frameworks are also very good. However +GoogleTest requires C++ familiarity and compiler, and Unity is rather not too +popular framework (which doesn’t mean worse). It is worth to mention, that such +evaluation will always be impacted by author subjective point of view. + +## Implementation proposal + +### Framework as a submodule or external package +Unit test frameworks may be either compiled from source (form a git submodule +under 3rdparty/) or pre-compiled as a package. The second option seems to be +easier to maintain, while in the same time may bring some unwanted consequences +(different version across distributions, frequent changes in API). It make sense +to initially experiment with packages and check how it works. If this will +brings any issues, then it is always possible to switch to submodule approach. + +### Build configuration +While building unit under test object file, it is necessary to apply some +configuration (config) just like when building whole firmware. For simplicity, +one config.h may be pre-built and then used for building all unit tests. In the +same time, if integrated with Jenkins, it may be preferred to use every +available config for periodic builds. + +My proposal is to go with `qemu_x86_i440fx` config for usual developer builds. + +### Integration with build system +To get the most out of unit testing framework, it should be integrated with +Jenkins automation server. Verification of all unit tests for new changes may +improve code reliability to some extent. + +### Directory structure +Tests should be kept separate from the code, while in the same time it must be +easy to match code with test harness. + +My proposal is to create new directory for test files and every test file name +should have added a prefix "-test" to the basename of a corresponding unit +under test module. Below example shows how this may be organized: + +```bash +├── build +│ ├── tests <-test binaries +│ +├── src +│ ├── lib +│ │ ├── malloc.c <- unit under test +│ │ ├── string.c <- unit under test +│ ├── commonlib +│ ├── sort.c <- unit under test +│ +├── tests +│ ├── include +│ │ ├── config.h <- pre-built config used for tests' builds +│ ├── Makefile.inc +│ ├── lib +│ │ ├── malloc-test.c <- test code for src/lib/malloc.c +│ │ ├── string-test.c <- test code for src/lib/string.c +│ ├── commonlib +│ ├── sort-test.c <- test code for src/commonlib/sort.c +├── +``` diff --git a/Documentation/technotes/index.md b/Documentation/technotes/index.md index 7c231fc..5367e69 100644 --- a/Documentation/technotes/index.md +++ b/Documentation/technotes/index.md @@ -2,3 +2,4 @@
* [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md) * [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md) +* [Unit testing coreboot](2020-03-unit-testing-coreboot.md)