Re: [flashrom] RFC: New testing script with remote test capabilities and region awareness
On Mon, 4 Jul 2016 22:22:29 -0700 David Hendricks dhendrix@chromium.org wrote:
Hi folks, As part of an effort underway to get chromium.org's flashrom sync'd with upstream (for real), I have written an entirely new test script to help get thru regression testing. It's still rather fugly and will likely go thru several more revisions, but is currently usable for all major use cases I could think of and would benefit from some early feedback.
I wrote up a document with usage examples here: https://goo.gl/3jNoL7 . Long-term the plan is to migrate this to the flashrom wiki.
There were a few "must haves" for this script in the near-term:
- Region awareness. The script understands how to use layout files and
flashmaps (for Chromebooks). Descriptor awareness is on the TODO list for Intel platforms.
- Can control flashrom on a remote host.
- Can utilize an external programmer in addition to the native programmer
on the target machine. This allows us to verify changes made to programmer code since one programmer can be considered to be trusted while the other can have code that was modified.
Cool, something like that is on my long-term todo list... (with some additional test stand hardware). I don't have time to look at the code right now so just some quick question: the primary/secondary programmer scheme is used to verify one while the other is used as kind of golden sample, right? It might make sense to not limit this to two programmers but have one golden sample and an unlimited programmers to actually test. (The hardware i was talking about above would be a PCB that can be remotely cross-switched between any of the attached programmers and multiple flash chips (e.g. one at45db, several ordinary *25*, some newer *25* with 4B addressing etc.)
On Tue, Jul 5, 2016 at 12:11 AM, Stefan Tauner stefan.tauner@alumni.tuwien.ac.at wrote:
I don't have time to look at the code right now
No rush - It's still in flux and the script itself is pretty monstrous so I don't expect a thorough review any time soon. So far I've been the only person to use it, but that will change soon and we'll shake out more bugs by virtue of having more users to report issues.
If you have a chance to run it even without reviewing, any feedback / bug reports will be appreciated (make sure to use the latest patch on gerrit (https://chromium-review.googlesource.com/#/c/353912/).
question: the primary/secondary programmer scheme is used to verify one while the other is used as kind of golden sample, right?
I actually tried to get away from the "golden sample" approach. The reason it was important to have the secondary programmer supported by this script is so that it can follow-along with what the primary programmer is doing at each step, not just the end result.
In region_partial_write_test(), for example, the primary programmer writes blobs that are generated to exercise different aspects of partial writes. There are 8 generated blobs in this test. Each time the new flashrom + primary programmer is used to write one, the old flashrom + primary programmer and, if enabled, the old flashrom + secondary programmer are used to verify.
I think this makes it easier to vigorously test partial write cases (or other cases) since we have all the filenames, variables, etc. contained in one place.
It might make sense to not limit this to two programmers but have one golden sample and an unlimited programmers to actually test. (The hardware i was talking about above would be a PCB that can be remotely cross-switched between any of the attached programmers and multiple flash chips (e.g. one at45db, several ordinary *25*, some newer *25* with 4B addressing etc.)
In this case I recommend invoking the script multiple times using the attached programmer which is selected via the switch as the "primary programmer" in each case. Have some other script select the programmer you want to test, then invoke the test script by tweaking the primary programmer args.
I added a diagram and example invocation for testing with only an external programmer as the first use case in the doc since that's probably going to be popular with developers.
-
David Hendricks -
Stefan Tauner