flashrom as programmer slow because it reads the whole SPI Flash contents before writing anything

[R. Diez]

Hi all:

I am trying to use flashrom with a fast generic FTDI FT2232H adapter as 
an SPI Flash programmer in a small production line.

The SPI Flash chip can hold 8 MiB of data, but we are only using the 
first 512 KiB of it.

I found flashrom to be surprisingly slow. I started digging and the step 
"Reading old flash chip contents..." is taking minutes. It looks like 
flashrom is reading the whole contents before writing anything at all.

I am using 7.5 MHz at the moment. I cannot pump up the frequency too 
much, as I am still using flywires to connect the SPI flash chip to the 
programmer.

I know that the flash chips are empty (0xFF), so flashrom does not need 
to read anything beforehand. If it is going to read anything, the first 
512 KiB would suffice, because I am not touching anything else.

I found the following in the source code:

int read_all_first = 1; /* FIXME: Make this configurable. */

I also found the following message about it:

http://www.flashrom.org/pipermail/flashrom/2011-December/008440.html

I guess the corresponding command-line argument is still missing.

I cannot tell how hard it would be to code and test the missing parts. 
The "Development Guidelines" are several pages long. There are very many 
unmerged patches in the pipeline, and some of them are years old. The 
repository is in Subversion. In summary, not very encouraging.

Does anybody know of some fork or alternative that does not read 
everything before writing any data at all?

Please copy me on any answers because I am not subscribed to the mailing 
list.

Regards,
   rdiez

[Stefan Tauner]

On Mon, 11 Jan 2016 14:58:12 +0100
"R. Diez"  wrote:
> Hi all:
>
> I am trying to use flashrom with a fast generic FTDI FT2232H adapter as 
> an SPI Flash programmer in a small production line.
>
> The SPI Flash chip can hold 8 MiB of data, but we are only using the 
> first 512 KiB of it.
>
> I found flashrom to be surprisingly slow. I started digging and the step 
> "Reading old flash chip contents..." is taking minutes. It looks like 
> flashrom is reading the whole contents before writing anything at all.
>
> I am using 7.5 MHz at the moment. I cannot pump up the frequency too 
> much, as I am still using flywires to connect the SPI flash chip to the 
> programmer.
>
> I know that the flash chips are empty (0xFF), so flashrom does not need 
> to read anything beforehand. If it is going to read anything, the first 
> 512 KiB would suffice, because I am not touching anything else.
>
> I found the following in the source code:
>
> int read_all_first = 1; /* FIXME: Make this configurable. */
>
> I also found the following message about it:
>
> http://www.flashrom.org/pipermail/flashrom/2011-December/008440.html
>
> I guess the corresponding command-line argument is still missing.
>
> I cannot tell how hard it would be to code and test the missing parts. 
> The "Development Guidelines" are several pages long. There are very many 
> unmerged patches in the pipeline, and some of them are years old. The 
> repository is in Subversion. In summary, not very encouraging.
>
> Does anybody know of some fork or alternative that does not read 
> everything before writing any data at all?
>
> Please copy me on any answers because I am not subscribed to the mailing 
> list.

The reading serves a purpose (more than one actually)... one that does
not give an advantage in your case though. flashrom can decide which
blocks need to be rewritten and which can be left alone after reading
the initial contents. This actually speeds up the process dramatically
in many use cases.
flashrom also reads the whole chip after writing to verify everything
is fine which would slow you down even further. That's the way it is
and it won't change anytime soon but we are aware of it.

[Nico Huber]

Hi rdiez,

On 11.01.2016 14:58, R. Diez wrote:
> Hi all:
>
> I am trying to use flashrom with a fast generic FTDI FT2232H adapter as
> an SPI Flash programmer in a small production line.

Nice to here.
> The SPI Flash chip can hold 8 MiB of data, but we are only using the
> first 512 KiB of it.

I guess, a very simple patch would be to change the size of the chip in
flashchips.c.
> I found flashrom to be surprisingly slow. I started digging and the step
> "Reading old flash chip contents..." is taking minutes. It looks like
> flashrom is reading the whole contents before writing anything at all.

flashrom was designed with very low-latency internal programmers in
mind. What makes it slow is the overhead for synchronous USB reads.
This would help to reduce overall read times:
http://www.flashrom.org/pipermail/flashrom/2015-August/013823.html
(Needs to be enabled per chip in flashchips.c)
> I am using 7.5 MHz at the moment. I cannot pump up the frequency too
> much, as I am still using flywires to connect the SPI flash chip to the
> programmer.

With very short wires, I got my FT2232H up to the full 30MHz once...
Also, sometimes I forget that 6 is a valid divisor between 8 and 4 :)
> I know that the flash chips are empty (0xFF), so flashrom does not need
> to read anything beforehand. If it is going to read anything, the first
> 512 KiB would suffice, because I am not touching anything else.

There are some layout patches floating around that would do the trick.
Yeah, more unmerged patches...
> I found the following in the source code:
>
> int read_all_first = 1; /* FIXME: Make this configurable. */
>
> I also found the following message about it:
>
> http://www.flashrom.org/pipermail/flashrom/2011-December/008440.html
>
> I guess the corresponding command-line argument is still missing.

There is:
http://www.flashrom.org/pipermail/flashrom/2015-December/014034.html

Regards,
Nico

[Carl-Daniel Hailfinger]

Hi Ruben,

On 11.01.2016 14:58, R. Diez wrote:
> I am trying to use flashrom with a fast generic FTDI FT2232H adapter
> as an SPI Flash programmer in a small production line.
>
> The SPI Flash chip can hold 8 MiB of data, but we are only using the
> first 512 KiB of it.
>
> I found flashrom to be surprisingly slow. I started digging and the
> step "Reading old flash chip contents..." is taking minutes.

If you can switch your production line to use flashrom on a Raspberry Pi
or a Beaglebone Black for SPI programming with the SPI pins on those
boards, reading a flash chip will only take seconds and writes will be
really fast as well because the per-command latency is almost zero.
With a SPI frequency of 8 MHz and a flash chip size of 8 MiB, reading
the whole chip on a RPi or BBB takes
roughly 9 seconds (to be precise, 8388608/256*(4+256)*8/8000000 seconds).
For FT2232 based programmers, you get a lot of USB overhead (mostly
latency) which makes reading slow, but writing is even worse.

Regards,
Carl-Daniel