Anastasia Klimchuk has uploaded this change for review. ( https://review.coreboot.org/c/flashrom/+/80340?usp=email )
Change subject: doc: Add doc how to add unit tests ......................................................................
doc: Add doc how to add unit tests
Change-Id: I73f6add194a531c4f88b822d92c31ec67c23d2dc Signed-off-by: Anastasia Klimchuk aklm@flashrom.org --- A doc/contrib_howtos/how_to_add_unit_test.rst M doc/contrib_howtos/index.rst M doc/dev_guide/building_from_source.rst 3 files changed, 103 insertions(+), 0 deletions(-)
git pull ssh://review.coreboot.org:29418/flashrom refs/changes/40/80340/1
diff --git a/doc/contrib_howtos/how_to_add_unit_test.rst b/doc/contrib_howtos/how_to_add_unit_test.rst new file mode 100644 index 0000000..cb2aa37 --- /dev/null +++ b/doc/contrib_howtos/how_to_add_unit_test.rst @@ -0,0 +1,101 @@ +====================== +How to add a unit test +====================== + +Unit tests help to maintain higher bar for code quality. A new unit test which adds coverage to the code is always useful, +no matter how small or large it is! Unit test is a valuable contribution, moreover it makes a good starter project, since +you don't need specific hardware (apart from you development host machine). + +For more details on how to run unit tests and measure coverage, check the dev guide: :ref:`unit tests`. + +To see the examples of existing unit tests, check the ``/tests`` directory in the source tree. If it helps, you can also look +at git history for examples of previous commits that add new tests. + +When is a good time to add a unit test? Any time is a good time. Test can go in its own separate patch, and also it can go +together in a patch which introduces a new code. + +Unit tests are using `cmocka https://cmocka.org/`_ as a mocking framework, but we also have flashrom mock framework +on the top of that. + +Mocking +========= + +Unit tests mock all interactions with hardware, interactions with filesystem, syscalls, 3rd party libraries calls +(e.g. libusb, libpci) etc. You can think of a flashrom unit test as a mini-emulator. The goal is to cover as much as possible +flashrom code, but you don't need to go outside of that. + +See the list of all current mocks (which are called wraps in cmocka terminology) in ``/tests/wraps.h``. These might be enough for +your new test, or you might need to add more wraps as a part of new test. + +New wrap needs to be added to ``/tests/wraps.h``, ``/tests/tests.c``, ``/tests/meson.build``. If it's fine for new wrap to +do nothing, log invokation and return success, all good. + +If a wrap need to behave in a specific way for a test, and the behaviour can be different from one test to another, you need to +extend the wrap into ``/tests/io_mock.h`` framework. + +Add corresponding member (a function pointer) to ``struct io_mock`` +and redirect calls from a wrap function in ``/tests/tests.c`` into a member of ``io_mock``. The exact implementation +of the member function needs to be defined in your new test. At the beginning of a test scenario, define function pointers that your +test needs in your own ``struct io_mock`` and then register by calling ``io_mock_register``. At the end of a test, clean up +by calling ``io_mock_register(NULL)``. + +Note that ``io_mock`` can support state (if needed). State is a piece of custom data which is carried around for the duration +of the test scenario and is available in all functions in ``io_mock``. + +Adding a new test to a framework +================================ + +To add new test you will either add a new test function in existing .c file, or add new .c file and new function(s) there. + +If you add new file, you need to add it into the list of test source files in ``/tests/meson.build``. + +Each new test function needs to be added into ``/tests/tests.h`` and ``/tests/tests.c``. Follow existing entries as examples +how to do it. + +Types of tests +============== + +Programmers tests +----------------- + +The concept of a unit test for flashrom programmer is based on a programmer lifecycle. The options supported by the flashrom +test framework are the following (but you are very welcome to try implement new ideas). + +The smallest possible lifecycle is called basic and it does initialisation -> shutdown of a programmer (nothing else). +Another option is probing lifecycle, which does initialisation -> probing a chip -> shutdown. +These two expect successful scenarious, the whole scenario is expected to run until the end with no errors and +success return codes. + +One more option is to test expected failure for programmer initialisation. This is useful to test known invalid +(and potentially dangerous) combination of programmer parameters, when such combination should be detected at init time. +If invalid combination of parameters is detected, initialisation expected to fail early and programmer must not continue, +and not send any opcodes to the chip. + +For more details, source code is in ``/tests/lifecycle.h`` and ``/tests/lifecycle.c``. + +If you want to add new test(s) for a programmer, first you look whether that programmer has any tests already, or none at all. +Test source file has the same name as a programmer itself, for example programmer ``dummyflasher.c`` has its dedicated tests in +``/tests/dummyflasher.c`` file. Either add your tests to an existing file, or create new file for a programmer that had no tests +so far. The latter is more challenging, but it is very useful and highly appreciated. + +For programmers tests, the test scenario most likely won't be long: most likely it is one of the options to run lifecycle with +given combination of programmer params as an input. Most time and effort is typically spent on mocking (see above), and this +type of tests will indeed look like a mini-emulator. + +Chip operations tests +--------------------- + +These tests are based on dummyflasher programmer and they are running operations of a chip: read, write, erase, verify. +The test defines mock chip(s) with given properties, and all the operations of the chip are redirected to mock functions. +Mock chip has its own mock memory (an array of bytes) and all operations are performed on this array of bytes. +As all the others, these tests are completely independent of hardware, and are focused on testing core flashrom logic +for read, write, erase, verify. + +Examples of chip operation tests are: ``tests/chip.c``, ``/tests/chip_wp.c`` (focused on write-protection logic), +``/tests/erase_func_algo.c`` (focused on erasing and writing logic, especially the choice of erase blocks for given +layout regions). + +Misc +---- + +All other tests. You choose a function that you want to test, call it with given arguments, assert the results are as expected. diff --git a/doc/contrib_howtos/index.rst b/doc/contrib_howtos/index.rst index 0a4fd0e..59326cd 100644 --- a/doc/contrib_howtos/index.rst +++ b/doc/contrib_howtos/index.rst @@ -6,3 +6,4 @@
how_to_add_new_chip how_to_mark_chip_tested + how_to_add_unit_test diff --git a/doc/dev_guide/building_from_source.rst b/doc/dev_guide/building_from_source.rst index 4e50753..c997832 100644 --- a/doc/dev_guide/building_from_source.rst +++ b/doc/dev_guide/building_from_source.rst @@ -224,6 +224,7 @@
meson configure [updated builtin options] [updated flashrom options] <builddir>
+.. _unit tests:
Unit Tests ----------
-
Anastasia Klimchuk (Code Review)