[flashrom] RFC: Unit testing framework

[Stefan Reinauer]

On Tue, Nov 23, 2010 at 2:29 PM, David Hendricks <dhendrix at google.com> wrote:
> On Tue, Nov 23, 2010 at 11:50 AM, Carl-Daniel Hailfinger
> <c-d.hailfinger.devel.2006 at gmx.net> wrote:
> Ah yes... Flashrom messages have been a pain point for us. The upstream
> messages are too verbose, annoy developers, and confuse scripts. The
> messages we have now were designed to work around issues as we encountered
> them. I hope to work on a long-term fix soon...

They're also a pain point for users. While "Error: WIP bit after WRSR
never cleared" is nice for everyone who knows the code, I can imagine
quite some people would find "Could not disable Write Protection" more
useful.

>> Agreed. Here is an interesting question: Given that the initial backup
>> and the final restore are done with the system flashrom binary,
>> shouldn't we try to restore the backup directly after making the backup?
>> If that fails, we can't restore the image at the end, and we're better
>> off detecting this problem before we mess with the chip.
>
> Good point! However, before we implement that, we'll need to modify flashrom
> to force rewrite all blocks. Otherwise, all blocks will simply be skipped.

By the time we figured out that we can not restore, the chip is
already messed up, no?

>> In this case we should warn the user that each of the tests has the
>> potential to brick his device. AFAICS the partial write torture test
>> can/will result in a hanging EC if the EC code lives in the bottom 128
>> kB of the flash chip.
>
> An interesting thought, but I think people get used to ignoring warnings.

Hence asking them "Do you want to proceed" and expecting them to enter
"I consciously do" or some such. That will get their attention (or
keep everyone from running tests)

> What if we created two (or more) sub-directories to categorize tests and
> keep dangerous ones separate? For starters, let's call them default_tests/
> and unsafe_tests/.

Are not all tests kind of unsafe? (except just reading the chip or
looking at it's size, but that's unsafe too if we "recover" the image
at the end.)


> Yes, indeed! There are numerous assumptions made about the size and
> capabilities of the chips that need to be abstracted better. I think your
> SPI chip emulator patch will help tremendously here.

Hm.. can we use the chip emulator with some kind of error injection
emulation and see how well flashrom detects certain errors?

Stefan


-- 
Stefan Reinauer
Google Inc.