[flashrom] RFC: New testing script with remote test capabilities and region awareness

[David Hendricks]

On Tue, Jul 5, 2016 at 12:11 AM, Stefan Tauner
<stefan.tauner at alumni.tuwien.ac.at> wrote:
>
> I don't have time to look at the code right now

No rush - It's still in flux and the script itself is pretty monstrous
so I don't expect a thorough review any time soon. So far I've been
the only person to use it, but that will change soon and we'll shake
out more bugs by virtue of having more users to report issues.

If you have a chance to run it even without reviewing, any feedback /
bug reports will be appreciated (make sure to use the latest patch on
gerrit (https://chromium-review.googlesource.com/#/c/353912/).

> question: the primary/secondary programmer scheme is used to verify one
> while the other is used as kind of golden sample, right?

I actually tried to get away from the "golden sample" approach. The
reason it was important to have the secondary programmer supported by
this script is so that it can follow-along with what the primary
programmer is doing at each step, not just the end result.

In region_partial_write_test(), for example, the primary programmer
writes blobs that are generated to exercise different aspects of
partial writes. There are 8 generated blobs in this test. Each time
the new flashrom + primary programmer is used to write one, the old
flashrom + primary programmer and, if enabled, the old flashrom +
secondary programmer are used to verify.

I think this makes it easier to vigorously test partial write cases
(or other cases) since we have all the filenames, variables, etc.
contained in one place.

> It might make sense to not limit this to two programmers but have one
> golden sample and an unlimited programmers to actually test.
> (The hardware i was talking about above would be a PCB that can be
> remotely cross-switched between any of the attached programmers and
> multiple flash chips (e.g. one at45db, several ordinary *25*, some
> newer *25* with 4B addressing etc.)

In this case I recommend invoking the script multiple times using the
attached programmer which is selected via the switch as the "primary
programmer" in each case. Have some other script select the programmer
you want to test, then invoke the test script by tweaking the primary
programmer args.

I added a diagram and example invocation for testing with only an
external programmer as the first use case in the doc since that's
probably going to be popular with developers.