[flashrom] RFC: Unit testing framework

[Stefan Reinauer]

Hi Carl-Daniel,

On Tue, Nov 23, 2010 at 11:50 AM, Carl-Daniel Hailfinger
<c-d.hailfinger.devel.2006 at gmx.net> wrote:
> Agreed. Here is an interesting question: Given that the initial backup
> and the final restore are done with the system flashrom binary,
> shouldn't we try to restore the backup directly after making the backup?
> If that fails, we can't restore the image at the end, and we're better
> off detecting this problem before we mess with the chip.

Good point! I think there's no big point in restoring it right away,
because if that fails we have an inconsistent state anyways, so we
won't mess much up when running the tests. But we could try to restore
it again with the flashrom under test binary if a restore with the
system binary fails. It might be that the system binary is old enough
and the tested binary is at least able to get the flash part back into
a non-bricked state (even if some of the tests might fail)

> In this case we should warn the user that each of the tests has the
> potential to brick his device.

Absolutely! Maybe the do_test script should ask for manual
confirmation if not run with a --non-interactive parameter.

> AFAICS the partial write torture test
> can/will result in a hanging EC if the EC code lives in the bottom 128
> kB of the flash chip.

Is there some way we can detect this? Or have a black list with
devices that have this issue?

>> arguments. The invocation: FLASHROM="../flashrom" FLASHROM_PARAM="-p
>> internal:bus=spi" sh do_tests.sh partial_writes_x86_bios.sh chip_size.sh
>> wp-toggle.sh wp-range.sh
>
> Why the "sh" after FLASHROM_PARAM="..."?

Assuming this is because util/ or . is not in the users path, I guess
./do_tests.sh would do the job, too.

> The big question for the size and protection tests is what you actually
> intend to test and whether the test will work with any chips besides the
> one you wrote the tests for. This is not a complaint directed at the
> RFC, it is rather a hint that writing good tests is hard (but you
> already know that).

The test should _work_ with any flash chip unless there is a bug in
flashrom. That does not necessarily imply that it tests something
useful though.

> Overall, I think that more tests are good.

Do you have suggestions what kinds of tests would be needed?

Does it make sense to add instrumentation (code coverage) support so
we can figure out the test depth? gcc in user space makes this kind of
trivial.

-- 
Stefan Reinauer
Google Inc.