Hi Richard, David,
On 16.08.2017 23:02, David Hendricks wrote:
On Tue, Aug 15, 2017 at 4:55 AM, Richard Hughes hughsient@gmail.com wrote:
Hi all,
I'm the maintainer of fwupd, which is a daemon for doing firmware updates in Linux. Using fwupd about 200,000 people update firmware every month. At the moment I have an out-of-tree patch to use the flashrom CLI for updating coreboot on Purism laptops, which exec's a flashrom binary with the correct arguments and then screen scrapes the output. This is less than ideal. I saw the appearance of a libflashrom in the staging branch and got very excited. I'm assuming the long term goal here is to install a shared with stable API/ABI library and a pkg-config file. If so, please keep reading.
I'm only using it as a static library, kept reading anyway :) The cur- rent API is to be considered unstable. We'd like to have it stabilized before the next release (whenever that may happen), but it's not a top priority.
One thing that is really important for fwupd is progress completion; as flashing the firmware is an inherently scary thing to do users do like to see a progress bar move from 0% to 100% in a gradual way as it's very re-assuring. This is even more important for the GUI, where people that just see a spinner don't know if the process is going to take 10 seconds, or 10 minutes.
Agreed! Especially since a lot of users have large chips, slow programmers, or sometimes both. Simple read operations can take a worryingly long time on some common hardware these days.
Would it be possible to have a process callback for flashrom_image_read(), flashrom_image_write() and flashrom_image_verify()? It'd need to have some kind of callback userdata too, for instance:
You already helped to improve our API: I forgot that one might need a userdata pointer for the log callback. Will fix that...
int flashrom_image_verify(struct flashrom_flashctx *, const void *buffer, size_t buffer_len, flashrom_progress_callback callback_fn, void *callback_user_data);
and also:
typedef void(flashrom_progress_callback)(uint64 completed, unit64 total, void *user_data)
I guess this would suffice for read and verify, which are simple, continuous operations. Write, however, is comprised of one, two or three steps (depending on options and the data): Read (needed to know which bytes have to change), (comparing to the data to be written) write (if necessary) and optionally verify. So there is no straight forward way to tell during read, for instance, if we are at 100% after the read or not. We could compare the data early, but that might result in an awk- ward implementation. My idea (not yet convinced if it's any good) would be to add a `stage` parameter to the callback:
enum flashrom_progress_stage { FLASHROM_PROGRESS_READ, FLASHROM_PROGRESS_WRITE, FLASHROM_PROGRESS_VERIFY, };
The short answer is yes. Nico also briefly mentioned an idea on IRC to use a mechanism similar to flashrom_set_log_callback() to avoid complicating function signatures for the read, write, and verify functions.
Correct, I would prefer a method to set the current callback and private pointer over adding more parameters. Though, we could have another set of functions ofc, with more parameters.
Richard, please let us know, what you'd prefer. After all, the interface should be designed to be most usable (and not driven by internals). e.g. do you expect the callback or the userdata pointer to change between calls?
I'm not familiar at all with flashrom internals, but could try to produce a patch if required -- although to propagate the progress through operations would mean patching a lot of chip->read() callbacks which probably requires a more experienced hand. Ideas welcome, thanks.
I'd be glad to help implementing it. But let's discuss the interface first.
Nico
On Thu, 17 Aug 2017 00:53:36 +0200 Nico Huber nico.h@gmx.de wrote:
I guess this would suffice for read and verify, which are simple, continuous operations. Write, however, is comprised of one, two or three steps (depending on options and the data): Read (needed to know which bytes have to change), (comparing to the data to be written) write (if necessary) and optionally verify. So there is no straight forward way to tell during read, for instance, if we are at 100% after the read or not. We could compare the data early, but that might result in an awk- ward implementation. My idea (not yet convinced if it's any good) would be to add a `stage` parameter to the callback:
enum flashrom_progress_stage { FLASHROM_PROGRESS_READ, FLASHROM_PROGRESS_WRITE, FLASHROM_PROGRESS_VERIFY, };
Is there really a good reason to leak these much details to the UI? Of course it would allow it to draw a very detailed and realistic picture (and a dedicated flashrom UI might want to do that (possibly just for the lols though)) but for a generic UI where libflashrom is just one of multiple targets I don't think this is helpful at all but complicates things.
Richard, we cannot easily determine how many net (as in netto) bytes we have to transfer (in and out) because it very much depends on the circumstances as Nico explained. However, we can make pretty good guesses that get updated during the operation and could simply pass a percentage. The progress would not be constant in speed (e.g. because reading is usually much faster than writing), and it would sometimes even be non-monotonic (if a write fails, we re-read the whole chip to make absolutely sure we get everything correct thus increasing the total number of bytes to transfer... or after the first read we might notice that we can skip writing 90% of the blocks because only a small part is updated) but this would drastically reduce the complexity of the interface and would still update often (spinner-ish) and usually relatively consistently and monotonic. I think this is the best compromise between doing nothing and getting insane by trying to get all information to the caller that it needs to get a complete picture (because the latter is by definition impossible since we don't know it either).
I personally don't like to lie to users even if they beg for it. If flashrom does not exit it does useful stuff (of course there are some corner cases, e.g., if hardware is broken but these cases should a) be fixed by timeouts within flashrom, and b) be covered by timeouts in the callers and NOT by the user manually killing anything...). So IMHO a message to not worry + a spinner as long as your UI is working correctly (+ a timeout that really is the last resort) is way better and more honest than a made up progress indicator that can never tell the full story... but that's my subjective opinion. flashrom as of yet does not even have the spinner (and that's bad!) and to add that we need the same infrastructure as a full progressbar anyway so we of course have considered implementing this a few times but certainly we will add some way to the API eventually and your input is very welcome...:
The short answer is yes. Nico also briefly mentioned an idea on IRC to use a mechanism similar to flashrom_set_log_callback() to avoid complicating function signatures for the read, write, and verify functions.
Correct, I would prefer a method to set the current callback and private pointer over adding more parameters. Though, we could have another set of functions ofc, with more parameters.
Richard, please let us know, what you'd prefer. After all, the interface should be designed to be most usable (and not driven by internals). e.g. do you expect the callback or the userdata pointer to change between calls?
2017-08-17 1:51 GMT+02:00 Stefan Tauner stefan.tauner@gmx.at:
I personally don't like to lie to users even if they beg for it. If flashrom does not exit it does useful stuff (of course there are some corner cases, e.g., if hardware is broken but these cases should a) be fixed by timeouts within flashrom, and b) be covered by timeouts in the callers and NOT by the user manually killing anything...). So IMHO a message to not worry + a spinner as long as your UI is working correctly (+ a timeout that really is the last resort) is way better and more honest than a made up progress indicator that can never tell the full story...
The problem with a spinner is that (unlike those in older CLI tools) they run in the UI thread. There are systems where everything but the UI system can be killed and dead, but the spinner is happily chugging along. People came to doubt spinners. Those things still spin, but is anything still going on really?
So where's the lie in this?
step 1/3: Reading flash for comparison 25% ... step 2/3: Writing to flash 75% (this may either be non-monotonous in speed, in which case 75% means "75% of flash matches input" or monotonous in speed after discounting the blocks that needn't be written) ... Error during write phase: checking flash state. Please wait and *KEEP THE SYSTEM RUNNING*...
That's more complex than a simple progress bar, but today's progress bars usually allow showing some inline text.
Patrick
On 16 August 2017 at 23:53, Nico Huber nico.h@gmx.de wrote: > I'm only using it as a static library, kept reading anyway :) The cur-
rent API is to be considered unstable. We'd like to have it stabilized before the next release (whenever that may happen), but it's not a top priority.
Right, for fwupd the Purism plugin isn't even in master yet, so I'm fine for an unstable library for months if required. I think at this stage keeping it API unstable is probably best, or maybe installing it but making it clear that the soname is going to be bumped and that you're not going to feel bad for doing so!
You already helped to improve our API: I forgot that one might need a userdata pointer for the log callback. Will fix that...
Great, thanks already!
So there is no straight forward way to tell during read, for instance, if we are at 100% after the read or not. We could compare the data early, but that might result in an awk- ward implementation. My idea (not yet convinced if it's any good) would be to add a `stage` parameter to the callback:
enum flashrom_progress_stage { FLASHROM_PROGRESS_READ, FLASHROM_PROGRESS_WRITE, FLASHROM_PROGRESS_VERIFY, };
That would actually work for me, in fwupd we have something very similar: https://github.com/hughsie/fwupd/blob/master/libfwupd/fwupd-enums.h#L34
Richard, please let us know, what you'd prefer. After all, the interface should be designed to be most usable (and not driven by internals).
Being honest, I don't mind much. For the callback I'd need to setup a helper struct to pass around the FuPlugin and FuDevice objects to the callback, so it's really not much more work to include anything else. Just getting the state and the percentage would be a huuge leap forward to the screen scraping and guessing we have now.
do you expect the callback or the userdata pointer to change between calls?
In my case, no.
On 17 August 2017 at 00:51, Stefan Tauner stefan.tauner@gmx.at wrote:
enum flashrom_progress_stage {
Is there really a good reason to leak these much details to the UI?
Yes, even non-technical users have been shown to "trust" the update process more if they know what's happening. Erase, Write, Verify are used in lots of other places, although I admit I don't show erase in the GUI and just lump it together with write to avoid the use thinking that the disk is being erased. In the end-user case I'd probably use text like "preparing for write".
However, we can make pretty good guesses that get updated during the operation and could simply pass a percentage.
That's actually all I bubble up to the GUI, although be aware a uint8 for the percentage might not be good enough; if the GUI is 2000px wide, then each 1% increase is 20px wide which makes it look "jerky". This is doubly true if the update is going to take some time, for instance a 5 minute update (read, erase, write, read) only gives the UI an update every 3 seconds, which again looks pretty bad. I can interpolate this over time, but it's much better if the underlying library gives us bytes or something like centi-percent instead.
The progress would not be constant in speed (e.g. because reading is usually much faster than writing),
I think that's fine.
and it would sometimes even be non-monotonic (if a write fails, we re-read the whole chip to make absolutely sure we get everything correct thus increasing the total number of bytes to transfer...
If the percentage does go backwards then we need to give the user a really good reason of why, otherwise users *will* file issues and bugs about it.
Thanks to all so far,
Richard