[RFC] [flashrom] "accelerated" high-level external programmer functions and serial external programmer protocol
Hi all,
I have a device that i would like to make a flashrom supported external programmer, an atmega168 with 3 shift registers as address bus and a DIL32 socket connected via RS232. Currently it runs it's own special software and i have special software on the PC to use it - but this combination supports (at the moment) only AT29C020 and W29C020C, so using flashrom's great database of flash chips would be great. Ok, so i would just implement a protocol for passing read and write cycles though the serial bus, sounds easy but there's a caveeat: eg. for the W29C020C there's a timeout between the bytes in the page load algo, 200us. For serial only i might be able to do that, but as my device isnt really attached to my computer's serial port (it's transmitter is busted..) but instead an serial<->ethernet adapter, there's no way one can do a "ping" in 200us, same problem applies for the USB FTDI chips i think. Even if it would work, the performance would be way too low.
So, i suggest a way for external programmers to provide alternatives to the normal flash chip functions like probe_jedec, read_memmapped, write_jedec, etc. In the external programmer init function the code could/would "attach" a table to a global pointer that defines re-mappings of chip functions, and before calling a chip function flashrom would check if the pointer is non-null and then check if there's an alternative to the function that is to be used, and would call the alternative instead. The alternative function would then use it's special protocol to actually cause the function to be "executed" on the programmer device, requiring much less bandwidth and making those timing requirements easy to meet. In case there's no re-mapping of some flash chip function (i wouldnt want to implement all the flash chip functions on my external programmer at once), the external programmer would ofcourse also implement the basic read and write cycles and the original flash chip access function would try to operate using those - whether that works is a different issue.
That's the first part of this "RFC" ... comments please.
Then i suggest that we would try to make some kind of a "standard" for an external serial programmer protocol - it could run over the normal serial port, an USB<->serial converter chip or over an TCP socket. This would allow use to reuse a simple "serialprog" external programmer module for any serial programming devices which some of us might make. That would allow eg. to support my device, but because that is propably an one-off device (it's cheap only because the base board was free for me) make the protocol usable with other possible devices too.
My suggestion would be a simple protocol in wich "calls" are the sending of a lenght word + function byte + parameters. The device would return with support byte + optional return value. eg in theory a read cycle: PC: 0x04 0x00 0x00 0xAA 0xBB 0xCC ---- 16-bit little endian lenght (of bytes after it) + function number 0 (read byte) + 24-bit address then device would return eg: DEV: 0xAA 0xBB 0xAA = I support this function, 0xBB = data returned or DEV: 0x55 = I dont support this function.
The values i picked are pretty much arbitary, but the idea is that different devices could support different sets of functions and could skip over not-supported functions in the data stream, just reporting that they dont support it. The scheme would allow upto 256 different functions with arbitary parameters (per-function). I chose 24-bit addresses as i havent heard of a bios flash chip that's even near 16MiB, and because we are tight on space in the protocol.
I might be into implementing this all, but it will take some time, and i would like to get it right so that it's of best possible use for all. Any input is appreciated.
What about introducing a per-programmer delay function pointer instead? Your programmer could buffer everything (delays, writes) until the first read, then send them as a batch.
AFAICS this would solve almost all issues and be a good first step forward. The only remaining issue would be fast read, but for that it's easy to replace chip_readl (32bits) with chip_readn (n bytes).
Regards, Carl-Daniel
On 05.06.2009 13:39 Uhr, Carl-Daniel Hailfinger wrote:
What about introducing a per-programmer delay function pointer instead? Your programmer could buffer everything (delays, writes) until the first read, then send them as a batch.
Which one is like that?
Stefan
Hi Urja,
Urja Rannikko wrote:
a way for external programmers to provide alternatives to the normal flash chip functions like probe_jedec, read_memmapped, write_jedec, etc.
..
That's the first part of this "RFC" ... comments please.
Then i suggest that we would try to make some kind of a "standard" for an external serial programmer protocol
I think these two issues go hand in hand. You have the right idea about how this should work to be efficient. A similar scheme was discussed around the Windows flashrom GSoC project, where the "external serial programmer" would have been a Windows kernel driver.
The conclusion back then was that while it would be a great solution it was maybe not feasible for the summer project, so another solution was implemented.
Some of our ramblings back then are probably still good, and some are no longer useful. Two links to the archive for some of the discussion but there might be more:
http://www.coreboot.org/pipermail/coreboot/2007-April/020206.html http://www.coreboot.org/pipermail/coreboot/2007-May/020488.html
I might be into implementing this all, but it will take some time, and i would like to get it right so that it's of best possible use for all. Any input is appreciated.
I think there are two parts to it. One is about the flashrom data model, which has been expanded a little lately but really needs to be very different. The timing properties you mention are required for some chips and other chip types have other required property sets.
The challenge here is to create a complete data model.
The second part is to communicate that data from flashrom to satellite entities which all implement the same state machine.
The challenge here is to create a state machine which works for abstracting all the different ways we want to reach flash chips.
//Peter
I'm essentially replying to two mails at once now ...
On Fri, Jun 5, 2009 at 15:17, Carl-Daniel Hailfingerc-d.hailfinger.devel.2006@gmx.net wrote:
The buffer management would be performed by the programmer driver. So if your programmer can execute (or buffer) 1024 bytes of commands/address/data, make sure flashrom accumulates no more than 1024 bytes in its internal buffer before flushing its buffer to the device.
Actually this got me to thinking of the protocol and buffer storage format again. There would be commands to: - query buffer size (so different devices can support different amounts of buffer) - initialize/clear buffer - entering a write command to the buffer (with data and address) - appending to a previous write command with a byte - entering a delay command to the buffer (how long could it be? eg. is 16-bit udelay enough?) - execute buffer
A write command with data and address would be if it's a single byte, 6 bytes on buffer - 1 byte op, 1 byte len, 3 byte addr, 1 byte data, but continuation with append data to buffer would be the minimalist 1 byte per byte, unless it's already 256 bytes of data inserted (encoded as 0-byte write) - then it would create a new write command. A delay would take 3 bytes on the buffer, 1 byte op, 2 byte delay value(?).
This would allow for the programmer driver to feed the data to the device during flashrom execution, and upon the first read perform the execute buffer command. I think that with this format the AVR's SRAM is enough for this to work with all the parallel chips with 256 byte or less page size, propably even 512. (1 byte op, 1 byte len (0), 3 byte addr, 256 byte data = 261 bytes)
And here i'm a bit worried about flow control - or actually the lack of it in my hardware. Making each command immediately return an "OK" byte - and waiting for it makes it way too slow, so I'm thinking of by default making those "add write/delay command to buffer" commands return nothing and adding a NOP command that returns a byte and a command that returns the size of the serial buffer on the device, so that the programmer driver can send n bytes of commands in a stream, then send NOP, then wait for answer, then again send n bytes.
(Peter wrote:) (i lost the header when editing...)
I think these two issues go hand in hand.
That's propably why i brought them both up in the same mail.
I think there are two parts to it. One is about the flashrom data model, which has been expanded a little lately but really needs to be very different. The timing properties you mention are required for some chips and other chip types have other required property sets.
The challenge here is to create a complete data model.
The second part is to communicate that data from flashrom to satellite entities which all implement the same state machine.
Hmm programming a satellite remotely :P (j/k)
The challenge here is to create a state machine which works for abstracting all the different ways we want to reach flash chips.
You think of this a bit more abstract than i usually do, but basically i agree.
Continuing on the protocol - it would need (ofcourse) also an n-byte read, I'm thinking that it could even be with 24-bit n so that the device could be instructed to read the entire chip in a single operation.
Then maybe some command to return information about programmer functionality would be good - eg. - what bus type(s) it supports - if parallel, how many address lines it has attached to the chip socket - actually offtopic, but i think it would be good for external programmers to have the ability to report the size of chip they support (eg 128k for the 3C905B) and for flashrom to issue a warning in case a bigger chip is detected (dont bail - it's still of some use for some purposes where you dont for example need the lower half of an 256k chip). I think that LPC and FWH can be operated over the same read/write/delay protocol as parallel, but i dont know how different SPI chips are and what extensions (if any) using them would need.
And in reality my programmer supports only parallel (at the moment), so that will be my priority - but i hope i can make a protocol that suits any chip type well.
On 05.06.2009 15:44, Urja Rannikko wrote:
I'm essentially replying to two mails at once now ...
On Fri, Jun 5, 2009 at 15:17, Carl-Daniel Hailfingerc-d.hailfinger.devel.2006@gmx.net wrote:
The buffer management would be performed by the programmer driver. So if your programmer can execute (or buffer) 1024 bytes of commands/address/data, make sure flashrom accumulates no more than 1024 bytes in its internal buffer before flushing its buffer to the device.
One of the most important design considerations is to make buffer management optional. On-board programming doesn't need it (and would be slowed down by it), lots of external programmers don't need it either.
Actually this got me to thinking of the protocol and buffer storage format again. There would be commands to:
- query buffer size (so different devices can support different
amounts of buffer)
OK.
- initialize/clear buffer
Maybe two separate functions for this.
- entering a write command to the buffer (with data and address)
OK.
- appending to a previous write command with a byte
Special case, should not end up in the interface. Either the programmer driver performs auto-erge of write commands or it simply enters a new write command.
- entering a delay command to the buffer (how long could it be? eg. is
16-bit udelay enough?)
Some chips have delays of more than 60 seconds (yes, not ms or us). 32bit udelay is the way to go forward. If your programmer can't handle a given delay length, the driver should either split it into multiple delays upon downloading it to the device or reject enqueuing such a delay.
- execute buffer
OK.
- read byte is missing from your design.
For SPI, you also need select_chip() and deselect_chip().
Please make sure that the flashrom buffer format has no dependencies on the programmer buffer format. Since hardware is more difficult to change than software, we definitely want to keep the flashrom buffer design changeable without breaking hardware. 16-bit vs. 32-bit delays could be one such difference.
Some external programmers (Cheetah, some FT2232 designs) also have their own buffer management. Any buffer management design will have to be compatible with them.
A write command with data and address would be if it's a single byte, 6 bytes on buffer - 1 byte op, 1 byte len, 3 byte addr, 1 byte data, but continuation with append data to buffer would be the minimalist 1 byte per byte, unless it's already 256 bytes of data inserted (encoded as 0-byte write) - then it would create a new write command. A delay would take 3 bytes on the buffer, 1 byte op, 2 byte delay value(?).
This would allow for the programmer driver to feed the data to the device during flashrom execution, and upon the first read perform the execute buffer command. I think that with this format the AVR's SRAM is enough for this to work with all the parallel chips with 256 byte or less page size, propably even 512. (1 byte op, 1 byte len (0), 3 byte addr, 256 byte data = 261 bytes)
And here i'm a bit worried about flow control - or actually the lack of it in my hardware. Making each command immediately return an "OK" byte - and waiting for it makes it way too slow, so I'm thinking of by default making those "add write/delay command to buffer" commands return nothing and adding a NOP command that returns a byte and a command that returns the size of the serial buffer on the device, so that the programmer driver can send n bytes of commands in a stream, then send NOP, then wait for answer, then again send n bytes.
What you want is an explicit execute_buffer() function. Writes and delays don't return anything by default. The programmer driver decides when to issue execute_buffer().
Peter wrote:
I think there are two parts to it. One is about the flashrom data model, which has been expanded a little lately but really needs to be very different.
The big problem with a totally different data model is that we can't know for sure if it will work for everything in the future. Our current model was extended when it didn't fit anymore and each time it seemed that the model would be good enough after the change. There's no guarantee that we won't fall into the same trap with a complete redesign.
The timing properties you mention are required for some chips and other chip types have other required property sets.
The challenge here is to create a complete data model.
Agreed. However, the data model needs to be understandable as well.
The second part is to communicate that data from flashrom to satellite entities which all implement the same state machine.
We have two state machines: One in the external flasher driver and one in the external flasher. The only thing we need is translation capability.
The challenge here is to create a state machine which works for abstracting all the different ways we want to reach flash chips.
You think of this a bit more abstract than i usually do, but basically i agree.
Continuing on the protocol - it would need (ofcourse) also an n-byte read, I'm thinking that it could even be with 24-bit n so that the device could be instructed to read the entire chip in a single operation.
That's what I wrote about the chip_readn function.
Then maybe some command to return information about programmer functionality would be good - eg.
- what bus type(s) it supports
The code already does that.
- if parallel, how many address lines it has attached to the chip socket
- actually offtopic, but i think it would be good for external
programmers to have the ability to report the size of chip they support (eg 128k for the 3C905B) and for flashrom to issue a warning in case a bigger chip is detected (dont bail - it's still of some use for some purposes where you dont for example need the lower half of an 256k chip).
Let's add programmer_can_access(struct flashchip *flash) which performs size checking etc so that probe_* can indeed return success even for too big chips. Some SPI controllers need that badly.
I think that LPC and FWH can be operated over the same read/write/delay protocol as parallel, but i dont know how different SPI chips are and what extensions (if any) using them would need.
And in reality my programmer supports only parallel (at the moment), so that will be my priority - but i hope i can make a protocol that suits any chip type well.
For SPI, the design is necessarily different, but I'll submit a driver for the TotalPhase Cheetah SPI programmer soon.
Regards, Carl-Daniel
On Fri, Jun 5, 2009 at 17:18, Carl-Daniel Hailfingerc-d.hailfinger.devel.2006@gmx.net wrote:
- appending to a previous write command with a byte
Special case, should not end up in the interface. Either the programmer driver performs auto-erge of write commands or it simply enters a new write command.
The point is here that the buffer management isnt visible outside of the programmer driver. Eg. when the programmers chip_writeb is called, it will check if the address is previous_address+1 and then send the combine.
- entering a delay command to the buffer (how long could it be? eg. is
16-bit udelay enough?)
Some chips have delays of more than 60 seconds (yes, not ms or us). 32bit udelay is the way to go forward. If your programmer can't handle a given delay length, the driver should either split it into multiple delays upon downloading it to the device or reject enqueuing such a delay.
You mean delays or timeouts? I (@work) only had access to a bit old flashrom source (a webview or "LXR" would be nice, is there?), but there were no calls to myusec_delay bigger than the 10ms used for AT29C020 detect.
- execute buffer
OK.
- read byte
is missing from your design.
I was listing only commands related to the buffer handling (at the serial protocol level, between the device and the flashrom external programmer driver)
For SPI, you also need select_chip() and deselect_chip().
I'll have to read more on SPI flash chips someday.
Please make sure that the flashrom buffer format has no dependencies on the programmer buffer format. Since hardware is more difficult to change than software, we definitely want to keep the flashrom buffer design changeable without breaking hardware. 16-bit vs. 32-bit delays could be one such difference.
The point in this was that there would be no buffer on flashrom's side - only on the device memory - although that buffer would be handled by the in-flashrom external programmer driver. I did consider skipping that write-combining so that the device would compress the stream, but then i thought that when using real serial, that would slow down writing manyfold to pass all the addresses to the device (AVR).
Some external programmers (Cheetah, some FT2232 designs) also have their own buffer management. Any buffer management design will have to be compatible with them.
What you want is an explicit execute_buffer() function. Writes and delays don't return anything by default. The programmer driver decides when to issue execute_buffer().
There will be execute buffer command on the AVR, but if (eg.) the AVR (for any reason) cannot process serial data as fast as it can receive serial data to it's serial buffer, then sending more bytes of commands than it's serial buffer to it could cause it to lose data. Eg. you need to be able to ask it that it has processed all the commands given.
Peter wrote:
I think there are two parts to it. One is about the flashrom data model, which has been expanded a little lately but really needs to be very different.
The big problem with a totally different data model is that we can't know for sure if it will work for everything in the future. Our current model was extended when it didn't fit anymore and each time it seemed that the model would be good enough after the change. There's no guarantee that we won't fall into the same trap with a complete redesign.
The timing properties you mention are required for some chips and other chip types have other required property sets.
The challenge here is to create a complete data model.
Agreed. However, the data model needs to be understandable as well.
The second part is to communicate that data from flashrom to satellite entities which all implement the same state machine.
We have two state machines: One in the external flasher driver and one in the external flasher. The only thing we need is translation capability.
The challenge here is to create a state machine which works for abstracting all the different ways we want to reach flash chips.
I wouldnt go and change a lot of things just to find out that the new design has some big flaw...
Then maybe some command to return information about programmer functionality would be good - eg.
- what bus type(s) it supports
The code already does that.
I'm talking about the serial protocol - i knew that flashrom already has support for this information.
Let's add programmer_can_access(struct flashchip *flash) which performs size checking etc so that probe_* can indeed return success even for too big chips. Some SPI controllers need that badly.
Ok.
On 05.06.2009 16:43, Urja Rannikko wrote:
On Fri, Jun 5, 2009 at 17:18, Carl-Daniel Hailfinger wrote:
Then maybe some command to return information about programmer functionality would be good - eg.
- what bus type(s) it supports
The code already does that.
I'm talking about the serial protocol - i knew that flashrom already has support for this information.
Sorry, can you elaborate? Which serial protocol are we talking about?
Regards, Carl-Daniel
On 05.06.2009 16:43, Urja Rannikko wrote:
On Fri, Jun 5, 2009 at 17:18, Carl-Daniel Hailfingerc-d.hailfinger.devel.2006@gmx.net wrote:
- appending to a previous write command with a byte
Special case, should not end up in the interface. Either the programmer driver performs auto-erge of write commands or it simply enters a new write command.
The point is here that the buffer management isnt visible outside of the programmer driver. Eg. when the programmers chip_writeb is called, it will check if the address is previous_address+1 and then send the combine.
Looks nice. Where's the problem? Sorry if I am being dense.
- entering a delay command to the buffer (how long could it be? eg. is
16-bit udelay enough?)
Some chips have delays of more than 60 seconds (yes, not ms or us). 32bit udelay is the way to go forward. If your programmer can't handle a given delay length, the driver should either split it into multiple delays upon downloading it to the device or reject enqueuing such a delay.
You mean delays or timeouts? I (@work) only had access to a bit old flashrom source (a webview or "LXR" would be nice, is there?), but
Needs to be done for the new flashrom tree. I'll ping Stefan or Patrick.
there were no calls to myusec_delay bigger than the 10ms used for AT29C020 detect.
We have sleep(1) in some places.
- execute buffer
OK.
- read byte
is missing from your design.
I was listing only commands related to the buffer handling (at the serial protocol level, between the device and the flashrom external programmer driver)
The Cheetah wants read commands to be added to the device buffer...
Anyway, I just sent a patch which adds multi-byte read and write to the external programmer interface. Can you take a look at it?
Regards, Carl-Daniel
On 05.06.2009 17:41, Carl-Daniel Hailfinger wrote:
Anyway, I just sent a patch which adds multi-byte read and write to the external programmer interface. Can you take a look at it?
I also sent a patch to handle delays in the external programmer interface.
Regards, Carl-Daniel
Looks nice. Where's the problem? Sorry if I am being dense.
Was just clarifying and checking if somebody thought that there's a problem. I'll propably get writing a real protocol spec, and then i'll have to write code for the ATmega168 to implement the device part, and then i'll do the flashrom part. Now that there's also that delay interface, flashrom in general will be pretty much ready for it. Thanks for the idea of buffering the operations.
Regards, Carl-Daniel
On 05.06.2009 19:37, Urja Rannikko wrote:
Was just clarifying and checking if somebody thought that there's a problem. I'll propably get writing a real protocol spec, and then i'll have to write code for the ATmega168 to implement the device part, and then i'll do the flashrom part.
Neat.
Now that there's also that delay interface, flashrom in general will be pretty much ready for it. Thanks for the idea of buffering the operations.
You're welcome.
I'm very interested in the buffer management code. Once you have an early prototype, please send it. The earlier the prototype is, the easier it is to change the design where you see problems.
If you need anything else from flashrom, please speak up.
Regards, Carl-Daniel
On 05.06.2009 16:43, Urja Rannikko wrote:
On Fri, Jun 5, 2009 at 17:18, Carl-Daniel Hailfinger wrote:
The point is here that the buffer management isnt visible outside of the programmer driver. Eg. when the programmers chip_writeb is called, it will check if the address is previous_address+1 and then send the combine.
Some chips have delays of more than 60 seconds (yes, not ms or us). 32bit udelay is the way to go forward. If your programmer can't handle a given delay length, the driver should either split it into multiple delays upon downloading it to the device or reject enqueuing such a delay.
You mean delays or timeouts?
Usually, short delays are indeed timing-critical delays whereas longer delays are simple timeouts.
Please make sure that the flashrom buffer format has no dependencies on the programmer buffer format. Since hardware is more difficult to change than software, we definitely want to keep the flashrom buffer design changeable without breaking hardware. 16-bit vs. 32-bit delays could be one such difference.
The point in this was that there would be no buffer on flashrom's side
- only on the device memory - although that buffer would be handled by
the in-flashrom external programmer driver. I did consider skipping that write-combining so that the device would compress the stream, but then i thought that when using real serial, that would slow down writing manyfold to pass all the addresses to the device (AVR).
OK, that works as well. My idea was to have a buffer on the flashrom side which can then be compacted or optimized and translated by the programmer driver before sending it off to the device. Such a buffer inside flashrom is optional, though.
What you want is an explicit execute_buffer() function. Writes and delays don't return anything by default. The programmer driver decides when to issue execute_buffer()
There will be execute buffer command on the AVR, but if (eg.) the AVR (for any reason) cannot process serial data as fast as it can receive serial data to it's serial buffer, then sending more bytes of commands than it's serial buffer to it could cause it to lose data. Eg. you need to be able to ask it that it has processed all the commands given.
Yes, for that you need flow control. For example, you you have the programmer return the amount of bytes free in the buffer after each command stream you send to the device.
Regards, Carl-Daniel
Yes, for that you need flow control. For example, you you have the programmer return the amount of bytes free in the buffer after each command stream you send to the device.
For a moment i was adding a "return bytes free in buffer" command to the protocol, but then i realized that if the code sent that command as the last in command stream, and waited for the answer, the answer would always be that the buffer is empty, because it just had to go though all the data in the buffer to get the command. So i'm sticking with the - query buffer size in the beginning - when doing opbuf writes, count bytes sent - if about to have sent too many bytes, instead send NOP and wait for the ACK and clear the counter and continue
I have the protocol specification ready and the AVR code compiles already, i'm just moving to flashrom code. I'll attach the .txt to this mail, it's better viewed with proper tabs and fixed-width font (leafpad fullscreen 1280x1024 is nice :P).
On 05.06.2009 23:12, Urja Rannikko wrote:
Yes, for that you need flow control. For example, you you have the programmer return the amount of bytes free in the buffer after each command stream you send to the device.
For a moment i was adding a "return bytes free in buffer" command to the protocol, but then i realized that if the code sent that command as the last in command stream, and waited for the answer, the answer would always be that the buffer is empty, because it just had to go though all the data in the buffer to get the command. So i'm sticking with the
- query buffer size in the beginning
- when doing opbuf writes, count bytes sent
- if about to have sent too many bytes, instead send NOP and wait for
the ACK and clear the counter and continue
Good idea.
I have the protocol specification ready and the AVR code compiles already, i'm just moving to flashrom code. I'll attach the .txt to this mail, it's better viewed with proper tabs and fixed-width font (leafpad fullscreen 1280x1024 is nice :P).
I like the protocol specification. A few minor design notes, though.
Serial Flasher Protocol Specification
Command And Answer Sequence - not all commands give an answer PC: LENGHT(8bit) COMMAND(8bit) <LENGHT PARAMETER BYTES> DEV: ACK/NAK(8bit) <OPTIONAL RETURN BYTES (only if ACK)> or nothing
ACK = 0xAA NAK = 0x55
Since you comunicate over serial, it may happen that one bit is lost/inserted. Since you specified ACK = NAK <<1 this could result in spurious acks if there are synchronization problems. I have seen bitstream sync problems too often with SPI communication, so I'm extra careful. What about ACK = 0x10 (Is Ok) NAK = 0xBA (BAd)
Your choice, though. Anything else would be just as good if one value is not a shift of the other.
All multibyte values are little-endian.
COMMAND Description Parameters Return Value 0x00 NOP none ACK
Maybe call it NOPACK.
0x01 Query serial buffer size none ACK + 16bit size / NAK 0x02 Query supported bustypes none ACK + 8-bit flags (as per flashrom) / NAK: bit 0: PARALLEL bit 1: LPC bit 2: FWH bit 3: SPI if ever supported 0x03 Query supported chip size none ACK + 8bit power of two / NAK 0x04 Query operation buffer size none ACK + 16bit size / NAK 0x05 Read byte 24-bit addr ACK + BYTE / NAK 0x06 Read n bytes 24-bit addr + 24-bit lenght ACK + lenght bytes / NAK
lenght -> length
0x07 Initialize operation buffer none ACK / NAK 0x08 Write to opbuf: Write byte 24-bit addr + 8-bit byte nothing / NAK (NOTE: takes 6 bytes in opbuf) 0x09 Write to opbuf: Write byte seq 8-bit byte nothing / NAK (NOTE: takes 1 bytes in opbuf)
Hm. I don't understand the opbuf lengths here. By the way, having a single-byte write command (like you have) and a multi-byte write command would probably be a more future-proof design. 0x09 Write to opbuf: Write byte seq 24-bit addr + 24-bit length + n bytes data ACK/NAK (Note: takes 7+n bytes in opbuf)
0x0A Write to opbuf: delay 32-bit usecs nothing / NAK (NOTE: takes 5 bytes in opbuf)
I'd like all commands to send a return code. All those write opcodes could be returning ACK even before they are sent.
0x0B Execute operation buffer none ACK / NAK - Execute operation buffer will also clear it 0x?? unimplemented command any NAK
0x0C Query programmer interface version 4 bytes "pRoG" ACK + 2 bytes version (>0) 0x0D Query programmer name none ACK + 16 bytes string (NULL padding)
I'd insert interface version and programmer name as 0x00 and 0x01, moving everything else down. That way, we can change the interface later and query it easily. The only thing we need to keep constant in the interface is opcode 0x00. Everything else can be changed as needed.
Regards, Carl-Daniel
On Sat, Jun 6, 2009 at 00:37, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
On 05.06.2009 23:12, Urja Rannikko wrote:
Yes, for that you need flow control. For example, you you have the programmer return the amount of bytes free in the buffer after each command stream you send to the device.
For a moment i was adding a "return bytes free in buffer" command to the protocol, but then i realized that if the code sent that command as the last in command stream, and waited for the answer, the answer would always be that the buffer is empty, because it just had to go though all the data in the buffer to get the command. So i'm sticking with the
- query buffer size in the beginning
- when doing opbuf writes, count bytes sent
- if about to have sent too many bytes, instead send NOP and wait for
the ACK and clear the counter and continue
Good idea.
I have the protocol specification ready and the AVR code compiles already, i'm just moving to flashrom code. I'll attach the .txt to this mail, it's better viewed with proper tabs and fixed-width font (leafpad fullscreen 1280x1024 is nice :P).
I like the protocol specification. A few minor design notes, though.
Serial Flasher Protocol Specification
Command And Answer Sequence - not all commands give an answer PC: LENGHT(8bit) COMMAND(8bit) <LENGHT PARAMETER BYTES> DEV: ACK/NAK(8bit) <OPTIONAL RETURN BYTES (only if ACK)> or nothing
ACK = 0xAA NAK = 0x55
Since you comunicate over serial, it may happen that one bit is lost/inserted. Since you specified ACK = NAK <<1 this could result in spurious acks if there are synchronization problems. I have seen bitstream sync problems too often with SPI communication, so I'm extra careful. What about ACK = 0x10 (Is Ok) NAK = 0xBA (BAd)
Your choice, though. Anything else would be just as good if one value is not a shift of the other.
0x10 and 0xBA are ok.
All multibyte values are little-endian.
COMMAND Description Parameters Return Value 0x00 NOP none ACK
Maybe call it NOPACK.
0x01 Query serial buffer size none ACK + 16bit size / NAK 0x02 Query supported bustypes none ACK + 8-bit flags (as per flashrom) / NAK: bit 0: PARALLEL bit 1: LPC bit 2: FWH bit 3: SPI if ever supported 0x03 Query supported chip size none ACK + 8bit power of two / NAK 0x04 Query operation buffer size none ACK + 16bit size / NAK 0x05 Read byte 24-bit addr ACK + BYTE / NAK 0x06 Read n bytes 24-bit addr + 24-bit lenght ACK + lenght bytes / NAK
lenght -> length
0x07 Initialize operation buffer none ACK / NAK 0x08 Write to opbuf: Write byte 24-bit addr + 8-bit byte nothing / NAK (NOTE: takes 6 bytes in opbuf) 0x09 Write to opbuf: Write byte seq 8-bit byte nothing / NAK (NOTE: takes 1 bytes in opbuf)
Hm. I don't understand the opbuf lengths here. By the way, having a single-byte write command (like you have) and a multi-byte write command would probably be a more future-proof design. 0x09 Write to opbuf: Write byte seq 24-bit addr + 24-bit length + n bytes data ACK/NAK (Note: takes 7+n bytes in opbuf)
Firstly, realize that the above command would have a second length parameter of different size than the first in parameters - a parser without knowledge of the commands parameters would surely fail.
the opbuf lenghts are because the driver must take note not to buffer too much operations in the AVR before executing them. A write command format in the AVR memory is essentially 0x00 (writeop) 0xnn (lenght) 0xaa 0xbb 0cc (begin address) + n bytes of data. the 0x08 operation creates the initial 1-lenght write op that takes 6 bytes (1 op, 1 lenght, 3 addr, 1 data) and each 0x09 operation adds one to lenght and appends the data. In case the length is already 0 (meaning 256), a new write operation with modified address (+256) is created, etc. So a continouous write of 256 bytes takes 261 bytes in the buffer. I considered adding a write n function, but realized that it would make the parameter lenght variable - now i can check that it is correct with a simple op2len table (a double-check that the parameters are correct). I dont see it worth the hassle, and it could code at a maximum 252 bytes of data with the current opcode lenght format - and the AVR's memory would surely run out if i asked it to take in some
1k operation anyways.
The read n operation is in many ways different - it can have the length in parameters because that is the return data length (not command parameter length), and the AVR doesnt need to buffer the data it sends in a reply.
0x0A Write to opbuf: delay 32-bit usecs nothing / NAK (NOTE: takes 5 bytes in opbuf)
I'd like all commands to send a return code. All those write opcodes could be returning ACK even before they are sent.
Huh? (My AVR doesnt know the future... And the computer would surely be baffled about an ACK before sending the command :P)
0x0B Execute operation buffer none ACK / NAK - Execute operation buffer will also clear it 0x?? unimplemented command any NAK
0x0C Query programmer interface version 4 bytes "pRoG" ACK + 2 bytes version (>0) 0x0D Query programmer name none ACK + 16 bytes string (NULL padding)
I'd insert interface version and programmer name as 0x00 and 0x01, moving everything else down. That way, we can change the interface later and query it easily. The only thing we need to keep constant in the interface is opcode 0x00. Everything else can be changed as needed.
Implementing...
On 06.06.2009 00:08, Urja Rannikko wrote:
On Sat, Jun 6, 2009 at 00:37, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
On 05.06.2009 23:12, Urja Rannikko wrote:
Serial Flasher Protocol Specification
All multibyte values are little-endian.
COMMAND Description Parameters Return Value
0x07 Initialize operation buffer none ACK / NAK 0x08 Write to opbuf: Write byte 24-bit addr + 8-bit byte nothing / NAK (NOTE: takes 6 bytes in opbuf) 0x09 Write to opbuf: Write byte seq 8-bit byte nothing / NAK (NOTE: takes 1 bytes in opbuf)
Hm. I don't understand the opbuf lengths here. By the way, having a single-byte write command (like you have) and a multi-byte write command would probably be a more future-proof design. 0x09 Write to opbuf: Write byte seq 24-bit addr + 24-bit length + n bytes data ACK/NAK (Note: takes 7+n bytes in opbuf)
Firstly, realize that the above command would have a second length parameter of different size than the first in parameters - a parser without knowledge of the commands parameters would surely fail.
We're misunderstanding each other here. I propose that flashrom does all the merging internally (up to a maximum write size). Thus the n-byte read command and the n-byte write command would be very similar.
the opbuf lenghts are because the driver must take note not to buffer too much operations in the AVR before executing them. A write command format in the AVR memory is essentially 0x00 (writeop) 0xnn (lenght) 0xaa 0xbb 0cc (begin address) + n bytes of data. the 0x08 operation creates the initial 1-lenght write op that takes 6 bytes (1 op, 1 lenght, 3 addr, 1 data) and each 0x09 operation adds one to lenght and appends the data. In case the length is already 0 (meaning 256), a new write operation with modified address (+256) is created, etc. So a continouous write of 256 bytes takes 261 bytes in the buffer.
Unless there is some device limitation, I think it is easier to already send a byte stream the AVR uses internally than having the AVR translate it in a complicated way. OTOH, your AVR internal representation is designed for compactness.
I considered adding a write n function, but realized that it would make the parameter lenght variable - now i can check that it is correct with a simple op2len table (a double-check that the parameters are correct). I dont see it worth the hassle, and it could code at a maximum 252 bytes of data with the current opcode lenght format - and the AVR's memory would surely run out if i asked it to take in some
1k operation anyways.
So you check the commands the AVR receives from the host, but not the commands in the internal AVR buffer? Because according to your description, the internal commands can have variable size.
The read n operation is in many ways different - it can have the length in parameters because that is the return data length (not command parameter length), and the AVR doesnt need to buffer the data it sends in a reply.
If I understand this correctly, your command-from-host parser in the AVR is easier to write if the commands from the host have per-opcode constant length? If that is the case, I propose to add an additional opcode 0x02 which returns a 32-bit field of supported opcodes. Then your device could mark the write-n as unsupported and the write-next as supported.
I'd like all commands to send a return code. All those write opcodes could be returning ACK even before they are sent.
Huh? (My AVR doesnt know the future... And the computer would surely be baffled about an ACK before sending the command :P)
Bad wording on my side. A write opcode should be acked after it is received by the programmer. No need to wait until the write reaches the flash chip.
From your proposal it is not clear if flashrom receives an ACK/NAK after
sending the command to the programmer or after successful execution of the command. Could you elaborate?
Regards, Carl-Daniel
On Sat, Jun 6, 2009 at 01:36, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
On 06.06.2009 00:08, Urja Rannikko wrote:
On Sat, Jun 6, 2009 at 00:37, Carl-Daniel Hailfinger c-d.hailfinger.devel.2006@gmx.net wrote:
On 05.06.2009 23:12, Urja Rannikko wrote:
Serial Flasher Protocol Specification
All multibyte values are little-endian.
COMMAND Description Parameters Return Value
0x07 Initialize operation buffer none ACK / NAK 0x08 Write to opbuf: Write byte 24-bit addr + 8-bit byte nothing / NAK (NOTE: takes 6 bytes in opbuf) 0x09 Write to opbuf: Write byte seq 8-bit byte nothing / NAK (NOTE: takes 1 bytes in opbuf)
Hm. I don't understand the opbuf lengths here. By the way, having a single-byte write command (like you have) and a multi-byte write command would probably be a more future-proof design. 0x09 Write to opbuf: Write byte seq 24-bit addr + 24-bit length + n bytes data ACK/NAK (Note: takes 7+n bytes in opbuf)
Firstly, realize that the above command would have a second length parameter of different size than the first in parameters - a parser without knowledge of the commands parameters would surely fail.
We're misunderstanding each other here. I propose that flashrom does all the merging internally (up to a maximum write size). Thus the n-byte read command and the n-byte write command would be very similar.
I mean that take a look of the start of file: PC: LENGHT(8bit) COMMAND(8bit) <LENGHT PARAMETER BYTES> This allows the AVR to skip commands for which it doesnt know the purpose of the parameters and just NAK them. "24-bit addr + 24-bit length + n bytes data" would cause "lenght" + "command" + "addr" + "bigger lenght" + data to be sent to the AVR --- if after the command there's a different amount of bytes than specified in the first lenght ... boom
Unless there is some device limitation, I think it is easier to already send a byte stream the AVR uses internally than having the AVR translate it in a complicated way. OTOH, your AVR internal representation is designed for compactness.
I considered adding a write n function, but realized that it would make the parameter lenght variable - now i can check that it is correct with a simple op2len table (a double-check that the parameters are correct). I dont see it worth the hassle, and it could code at a maximum 252 bytes of data with the current opcode lenght format - and the AVR's memory would surely run out if i asked it to take in some >1k operation anyways.
So you check the commands the AVR receives from the host, but not the commands in the internal AVR buffer? Because according to your description, the internal commands can have variable size.
Well the commands in the internal AVR buffer are generated by the checked commands, so they are "sure" to be correct - no need to check them, though the code does basic checking on that stream too during execution, for example if a command that doesnt exist is found in the stream or the stream ends mid-command, a NAK is returned.
The read n operation is in many ways different - it can have the length in parameters because that is the return data length (not command parameter length), and the AVR doesnt need to buffer the data it sends in a reply.
If I understand this correctly, your command-from-host parser in the AVR is easier to write if the commands from the host have per-opcode constant length? If that is the case, I propose to add an additional opcode 0x02 which returns a 32-bit field of supported opcodes. Then your device could mark the write-n as unsupported and the write-next as supported.
A write-n with n max 252 would be easy to implement as compatible with the protocol, and actually even the AVR could handle it. The point in the ACK/NAK system was in the beginning to allow a programmer to not implement any opcode (exept NOPACK - now also query interface version) and return NAK when trying to do that (the flashrom driver could issue write-n, get NAK, try with singular/"AAI" writes as a fallback). I chose not to have 24-bit lenght header in every opcode because of the limits on serial speeds.
I'd like all commands to send a return code. All those write opcodes could be returning ACK even before they are sent.
Bad wording on my side. A write opcode should be acked after it is received by the programmer. No need to wait until the write reaches the flash chip. From your proposal it is not clear if flashrom receives an ACK/NAK after sending the command to the programmer or after successful execution of the command. Could you elaborate?
Hmm... the only ACK from a write on the buffer is got when executing the buffer ("the whole buffer was executed") as a reply to execute buffer, this is in purpose not to have PC's doing "ping-pong" with the AVR during writing of the buffer - the host can do non-blocking I/O to check for a NAK after streaming the write commands. (Or send an NOPACK and read until you get an ACK - then there were no NAK's when writing). Of course it would also be possible to have them all return ACK and then do write lots + count all the ACK's on the host side, but i dont see that as necessary.
Hmm... the only ACK from a write on the buffer is got when executing the buffer ("the whole buffer was executed") as a reply to execute buffer, this is in purpose not to have PC's doing "ping-pong" with the AVR during writing of the buffer - the host can do non-blocking I/O to check for a NAK after streaming the write commands. (Or send an NOPACK and read until you get an ACK - then there were no NAK's when writing). Of course it would also be possible to have them all return ACK and then do write lots + count all the ACK's on the host side, but i dont see that as necessary.
Ok scratch that :P, i changed those commands to return ACK too, because that allows me to get by with less job on the flow control, eg. i can wait for all the ACK's to "come home" - then the serial buffer is empty. Simpler, and it allows to use a generic ping-style command function on any command (in case this streaming performance is not needed).
I'm still pondering on the n-byte write ---
Anyways, i attach (just for getting an idea on the protocol, dunno) the main AVR parser source and the newest version of the protocol specification. The AVR source isnt stand alone as in the project has many other .c files which arent needed to understand the protocol, but i dont mean you to compile it, just to be looked at (it does compile though (with all the support files)). It also lacks license header. (It'll be GPLv2+ when tested as working - eg. when i get the flashrom part done).
Now to sleep...
The newest version of the specification is attached.
A more compact protocol specification attached - i'll try and clean the flashrom driver code a bit and send a patch next - but for now a small log clip (with jedec.c doing erase checking etc (eg. as is in svn now) - on a real serial port at 115200): (and yeah i did a ncftp "mget *" on the flashrom directory to move the stuff over to the another computer - so it didnt copy the svn data - so no revision on that) --- root /home/urjaman/flashrom # time ./flashrom -p serprog=/dev/ttyS0:115200 -w randomdata.bin flashrom v0.9.0-r serprog: Programmer name "ATMega168 SFR" Calibrating delay loop... OK. Found chip "Winbond W29C020C" (256 KB) at physical address 0xfffc0000. Flash image seems to be a legacy BIOS. Disabling checks. Programming page: 2047 at address: 0x0003ff80
real 1m57.793s user 0m0.773s sys 0m0.003s ---
Is this serial flashing protocol currently supported in flashrom? If so, why isn't it shown in the -L list of supported flashing methods?
Thanks, wt
On 2009-06-07, Urja Rannikko urjaman@gmail.com wrote:
A more compact protocol specification attached - i'll try and clean the flashrom driver code a bit and send a patch next - but for now a small log clip (with jedec.c doing erase checking etc (eg. as is in svn now) - on a real serial port at 115200): (and yeah i did a ncftp "mget *" on the flashrom directory to move the stuff over to the another computer - so it didnt copy the svn data - so no revision on that)
root /home/urjaman/flashrom # time ./flashrom -p serprog=/dev/ttyS0:115200 -w randomdata.bin flashrom v0.9.0-r serprog: Programmer name "ATMega168 SFR" Calibrating delay loop... OK. Found chip "Winbond W29C020C" (256 KB) at physical address 0xfffc0000. Flash image seems to be a legacy BIOS. Disabling checks. Programming page: 2047 at address: 0x0003ff80
real 1m57.793s user 0m0.773s sys 0m0.003s
-- urjaman
On Mon, Jun 8, 2009 at 07:05, Warren Turkalwt@penguintechs.org wrote:
Is this serial flashing protocol currently supported in flashrom? If so, why isn't it shown in the -L list of supported flashing methods?
It's only supported in my flashrom, yet :P aka it's in development and i'll most likely send a patch tonight (Finland time).
-
Carl-Daniel Hailfinger -
Peter Stuge -
Stefan Reinauer -
Urja Rannikko -
Warren Turkal