flashrom March 2024

flashrom@flashrom.org

10 participants
8 discussions

RFC: remove the calibrated delay loop
by Peter Marheine 12 Apr '24

12 Apr '24

Proposed changes in https://review.coreboot.org/c/flashrom/+/80806 have gotten me thinking about the problems with flashrom's current delay implementation, following which I've got a few ideas about how it can be improved. If other developers agree that these changes are reasonable, I think I'll go ahead and work on them. Although I've measured some data, I haven't yet attempted to quantify the overall performance effect to flashrom from these changes because that would require implementing them. I do expect that I would gather that data for an actual implementation. ## Proposals 1. Remove support for delaying inside flashrom using a calibrated busy-loop. Delete `myusec_delay` and `myusec_calibrate_delay`. 2. Remove probing for OS timer resolution. Delete `clock_check_res` and `measure_os_delay_resolution`. Always use `clock_gettime`-based timing for busy-loops, or `gettimeofday` if `clock_gettime` is unavailable. 3. Make the threshold in `default_delay` to choose between sleep or busy-looping configurable at build time. Reduce the default from 100ms to 100μs. 4. Automatically attempt to run flashrom at realtime priority when supported by the system, to minimize timing error. ## Rationale Each of these proposed changes fits in with the others to simplify how flashrom delays while waiting for work to complete, but their reasons are largely independent. ### Calibrated busy-loop removal Flashrom's calibrated busy-loop is as old as flashrom itself, with its principle of operation first being established before 2003; the first commit to the flashrom repository included a version of `myusec_calibrate_delay` which attempted to count the number of iterations of an empty loop that the CPU could execute in 250ms. That initial version searched by doubling its delay time for each attempt, so delays could have been up to twice the length specified. On modern systems, it is a grave mistake to assume that any CPU-based calibration process can accurately measure real time by counting the number of instructions executed (or the number of loop iterations completed). Runtime frequency changes have become much more common in the intervening 20 years, and if CPU frequency changes while flashrom is running then calibration becomes wrong. On systems where all processors have the same maximum frequency, delay loop calibration tends to be fairly accurate because the current implementation calibrates for 1 second and this is at least as long as most systems take to reach their maximum clock speed under load. Short delays could be much longer than expected if the CPU clocked back down after calibration, but the calibrated delay would generally be close to correct at maximum speed so it would be unlikely to delay for less time than expected. Today's CPUs are even more heterogenous, however. ARM machines have used heterogenous multiprocessors since about 2011, and recent x86 systems from both Intel and AMD can also have different maximum clock speeds for different cores. On these machines, a delay loop calibrated on a CPU with lower maximum frequency could delay for too short a time when run on a different CPU with higher maximum speed. Even SMT systems that have dynamic thread scheduling policy for each physical core might exhibit higher performance if a sibling thread is idle! Although it may be possible to compensate for heterogenous multiprocessors (for example by calibrating delays for each CPU in the system or pinning flashrom to a single CPU using OS APIs), doing so would either increase runtime costs further or add to the maintenance burden of the flashrom maintainers. Because it is difficult to ensure a calibrated delay loop is actually correct, we should never use one. ### Ignoring OS-reported timer resolution On Linux, `clock_getres` is documented to lie sometimes: a "tickless" kernel can generally report times with nanosecond resolution, but `clock_getres` reports resolution of around 1 millisecond. This confounds flashrom, causing it to use the calibrated delay loop instead of OS timing. Because we cannot assume a CPU busy-loop to ever be truly calibrated on modern machines without significant effort, it is useless to try to use reported timer resolution as a signal to use a busy-loop instead (because there is no busy-loop to fall back to). We should instead use whatever OS timing is available. `gettimeofday` is specified as early as System V r4, but is deprecated in POSIX.1-2008. `clock_gettime` is mandatory since POSIX.1-2008. Some systems that we wish to support may lack `clock_gettime`, which is preferred because it offers nanosecond resolution to `gettimeofday`'s microsecond resolution and offers clock options that will never be discontinuous (for example due to daylight savings or leap seconds). Between these two functions, all platforms that flashrom wishes to support should be supportable. ### Reduced sleep thresholds On modern machines, the current 100 millisecond maximum busy-wait time is an eternity (and until recently it was 1 second, which is even more ridiculous). We can save a lot of power by giving up the CPU more readily, but choosing a threshold is somewhat challenging. Modern x86 systems have typical context switch latency (from suspending one process, deferring to the kernel, and restarting another process) of around 2 microseconds, so this is a reasonable starting point for the maximum time to busy-wait for: it's unlikely that another process can do anything useful in less that 2 microseconds, or that the OS can suspend the CPU to save energy. From surveying delay calls within flashrom, they span a large range from 1 microsecond up to a whole second. By inspection, 10 microseconds seems to be a common delay used when polling for completion of some operation. A few delay users describe how long it's usually expected for a polled operation to take, but most don't so it's difficult to tell how accurate the delays actually should be in order to minimize wasted time. #### Wakeup latency measurements To evaluate how much timing slop might be introduced by sleeping instead of busy-looping for short delays (less than 1 millisecond), I measured the latency of several operations on a selection of various Linux machines to evaluate how much longer it takes a program to actually resume from `nanosleep` versus the requested wait time. The systems were: * Mediatek Kompanio 500 (MT8173). Four ARMv7 cores running at up to 2 GHz. * Intel Xeon Gold 6154 (Xeon). Two sockets, each with 18 physical cores running 36 threads. * Intel Celeron N3350 (APL). Two x86 cores running at 1.1 GHz, up to 2.4 GHz. * AMD A4-9120C (AMD). Two x86 cores at 1.6 GHz, up to 2.4 GHz. * Intel Core Ultra 5 115U (MTL). Eight x86 cores in three performance tiers at 0.7-1.5 GHz, up to 2.1-4.2 GHz. I ran four tests each 16k times, getting the time elapsed for each iteration by calling `clock_gettime` immediately before and after each test. The minimum, maximum and mean time taken for each test was recorded. On all systems the reported timer resolution via `clock_getres` was 1 nanosecond. * nop: do nothing * sched_yield: call sched_yield() to attempt to run a different task without specifying a wait time * nanosleep 1u: call nanosleep() with a requested wait time of 1 microsecond * nanosleep 100u: call nanosleep() with a requested wait time of 100 microseconds The nop case provides baseline information on how long it takes to measure the time it takes an operation to complete. sched_yield approximates syscall and scheduling overhead, under the assumption that the OS will run its scheduler to see if another process wants the CPU. If it does, the delay time could be greatly increased because another process may get the CPU. The two nanosleep cases allow us to see how long a process that sleeps actually suspends for, and looking at two different delays is useful in case the OS tends to get sloppier if we request longer delays. In the following table, measured times are in microseconds, listed as mean / min / max. | Machine / Test | nop | sched_yield | nanosleep 1u | nanosleep 100u | |----------------|---------------------|-------------------|--------------------|-----------------------| | AMD | 0.1 / 0.06 / 38.3 | 0.6 / 0.6 / 100.1 | 64.5 / 6.4 / 2151 | 106.3 / 104.2 / 9619 | | MT8173 | 0.1 / 0.07 / 16.7 | 1.8 / 1.5 / 251.9 | 82.4 / 7.8 / 3051 | 195.9 / 109.3 / 5616 | | APL | 0.05 / 0.04 / 14.3 | 0.7 / 0.6 / 51.3 | 57.2 / 7.1 / 1236 | 217.6 / 108.4 / 10430 | | Xeon | 0.02 / 0.02 / 0.07 | 1.7 / 1.4 / 73.2 | 57.8 / 11.3 / 618 | 157.8 / 106.9 / 1299 | | MTL | 0.05 / 0.03 / 101.9 | 0.4 / 0.4 / 7.1 | 53.0 / 4.6 / 139 | 151.7 / 101.5 / 245 | Latency in excess of the requested wait time is fairly consistent, on average about 60 microseconds and within as little as 10% of the requested time for 100 microsecond delays. Very short delays tend to be worse, and the average case does notably better on systems with more CPU cores- probably because the CPU sometimes gets given to another process and causes long delays while that other process runs or the OS is doing some uninterruptible operation. These numbers are similar to the ones reported in https://review.coreboot.org/c/flashrom/+/80808, which lends credibility to this measurement method. High-resolution timers there improved minimum latency slightly, but as we'll see next, different scheduling improves the situation more. --- I also ran the tests with realtime scheduling policy (as in `sched_setscheduler(SCHED_FIFO)`) using `chrt --fifo 1` to run at the minimum priority within the FIFO real-time class. This is expected to reduce latency both by giving this process priority over all other processes on the system, and by allowing the OS to use a simpler scheduling algorithm when ultimately choosing to restart the process under test. In this table the nop and sched_yield tests are omitted because they are not as informative as the actual nanosleep numbers. | Machine / Test | nanosleep 1u | nanosleep 100u | |----------------|----------------------|------------------------| | AMD | 9.2 / 4.8 / 18.6 | 129.1 / 104.7 / 512.2 | | MT8173 | 13.5 / 6.8 / 1505 | 117.2 / 106.7 / 1022 | | APL | 5.4 / 4.8 / 45.2 | 106.3 / 104.2 / 452.3 | | Xeon | 4.4 / 4.2 / 60.2 | 104.9 / 103.0 / 164.9 | | MTL | 2.4 / 2.1 / 74.2 | 102.4 / 101.2 / 123.0 | At real-time priority, 1-microsecond sleeps look like they're basically only paying the cost of a context switch and 100-microsecond sleeps are more consistent than they were at normal priority: the average case is now accurate to within better than 10% on the faster machines with the slower ones not doing much worse. #### Practical sleep thresholds The existing code in flashrom that uses `clock_getres` rejects OS timers as a timing source if the resolution is less than 100 nanoseconds. Since the delay functions operate with microsecond precision, this suggests that it targets 10% accuracy for short waits and significantly better than that for long ones. Because we see that 10% accuracy is feasible on most systems for 100 microsecond sleeps, this seems like a reasonable minimum time to sleep for. Even on systems where accuracy is worse than 10%, use of `sleep`-style APIs guarantees that delays will never be less than the specified time: this approach will never malfunction, but unexpectedly long delays could make flash programming take longer than it would otherwise. Not all users or distributors will agree with this decision. Some very slow or old machines might achieve much worse precision such that it's considered worthwhile to busy-wait more aggressively (in order to complete programming more quickly), while some users might prefer to take every opportunity to save power by always sleeping. As a result, I believe this threshold should be made a compile-time option defaulting to 100 microseconds (although the final choice for the default can easily be adjusted). #### Use of real-time scheduling The accuracy of `sleep` APIs can be improved by using real-time scheduling classes as specified by POSIX, but enabling that for a given process usually requires special privileges. Many users run flashrom as the root user so it has the ability to make itself realtime, but there are also many applications where it need not run with root privileges. Because realtime priority is not required for correct operation, flashrom should attempt to make itself realtime but continue even if unable to do so. An informational message should be emitted if that fails so users can be aware that granting flashrom additional privileges may yield better performance. In libflashrom, realtime scheduling should not be enabled by default because it is very difficult to reason about the possible effects of changing a host process' scheduling class. Programs using flashrom as a library should manage priority themselves.

5 10

RDID Values Constantly Changing
by Ryne Everett 03 Apr '24

03 Apr '24

Hello group, I'm attempting to flash a W25Q64FW with a CH341A. Despite this chip's RDID being supported (under "W25Q64.W"), flashrom identifies it as an "unknown SPI chip (RDID)" which I presume explains the "status NOT WORKING for operations: PROBE READ ERASE WRITE". Upon further inspection with '-VV' I see that id1 and id2 are not only wrong -- they're different every time I run flashrom! What could cause this? Thanks, Ryne

2 3

adding part #
by Vlim 01 Apr '24

01 Apr '24

Hi, FlashRom team, I am the FAE director in Gigadevice and I would like to add some part #s to the flashchip.c and flashchip.h. Can you please provide some guidance? Regards, Victor Lim FAE Director GigaDevice Semiconductor 4088833856

4 11

Stripping down flashrom
by bzt 31 Mar '24

31 Mar '24

Hi, I've a question. I'm writing a software that installs a specific Linux distro on the target computer automatically. It also should flash coreboot as part of the install process, and that's where I'm stuck. From what I've gathered, for that I would need a stripped down version of libflashrom that only has the internal SPI driver, nothing else (without laptop-checks, Paralell, LPC, NICs, external USB programmers etc.) What is the best way to achieve this? I've already set up my test environment: I've configured and compiled qemu with "-enable-m25p80", which emulates lots of flash chips, among others the one that my target machine has. Then I can run qemu with "-device m25p80" and start a minimal Linux VM with the flashrom utility that I've compiled from source. I've also cleaned up flashrom.c to only list the chips with the BUS_SPI interface (I need to support one specific chip only, but my understanding is, all of these use the same driver so it shouldn't be a problem having all these chips in this list). Now I need to remove files and functions that are not related to internal SPI, and I'm not sure how to proceed. Which would be the easiest and suggested way to have a stripped down libflashrom with just the internal SPI driver? ps: unfortunately using flashrom as-is is out of question because it has way too many dependencies that the installer image simply does not have. For example, there's no libpci, libusb and no /dev/spidev device either, so I must restrict libflashrom to use its own driver, and also my installer can't call system(), so I must link it with libflashrom. Can somebody help me with this?

4 4

Report on "Could not determine chipset generation".
by Konstantinos Sk 19 Mar '24

19 Mar '24

Hello, I disabled secure boot so I can run flashrom as root in my openSUSE Tumbleweed machine. It reported the following: > flashrom -p internal --wp-status flashrom v1.3.0 on Linux 6.7.9-1-default (x86_64) flashrom is free software, get the source code at https://flashrom.org Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns). Found chipset "AMD FP4". Enabling flash write... Disabling read write protection of flash addresses from 0xfc000000 to 0xfc05ffff failed. Disabling read write protection of flash addresses from 0xfd000000 to 0xfd05ffff failed. Disabling read write protection of flash addresses from 0xfe000000 to 0xfe05ffff failed. Disabling read write protection of flash addresses from 0xff000000 to 0xff05ffff failed. FCH device found but SMBus revision 0x51 does not match known values. Please report this to flashrom(a)flashrom.org and include this log and the output of lspci -nnvx, thanks!. Could not determine chipset generation.PROBLEMS, continuing anyway No EEPROM/flash device found. Note: flashrom can never write if the flash chip isn't found automatically. I also attached the output of `lspci -nnvx` as requested. Best regards,

1 0

Macronix flash chip
by Krasen Tsonevski 15 Mar '24

15 Mar '24

Hello, I'm trying to flash a chip: MX25L25635F Error from flashrom: :~# /usr/local/sbin/flashrom -p ch341a_spi -w REDFISH_X10_391_20200806_unsigned\ 1.bin flashrom 1.4.0-devel (git:v1.2-1407-g54b053e6) on Linux 5.4.0-173-generic (x86_64) flashrom is free software, get the source code at https://flashrom.org Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns). Found Macronix flash chip "MX25L25635F/MX25L25645G" (32768 kB, SPI) on ch341a_spi. === This flash part has status UNTESTED for operations: WP The test status of this chip may have been updated in the latest development version of flashrom. If you are running the latest development version, please email a report to flashrom(a)flashrom.org if any of the above operations work correctly for you with this flash chip. Please include the flashrom log file for all operations you tested (see the man page for details), and mention which mainboard or programmer you tested in the subject line. Thanks for your help! Reading old flash chip contents... done. Erase/write done from 0 to 1ffffff Verifying flash... FAILED at 0x01873880! Expected=0x02, Found=0x05, failed byte count from 0x00000000-0x01ffffff: 0x775 Your flash chip is in an unknown state. Please report this to the mailing list at flashrom(a)flashrom.org or on IRC (see https://www.flashrom.org/Contact for details), thanks! Can you help me with this? Best regards, --- Krasen Tsonevski IT Manager office: +359 2 975 16 16 [ http://www.neterra.net/ | www.neterra.net ] [ https://neterra.net/network-access/dia-dedicated-internet-access ]

1 0

FAILED: UPN-ADLN97-A10
by Stefan Kent 10 Mar '24

10 Mar '24

Hello, I tried to flash an Intel board using flashrom and it seems to have failed. The image was supplied by the manufacturer. The manufacturer page for the board is here: https://www.aaeon.com/en/p/up-board-up-squared-pro-7000 I took a backup of the factory image before I started but flashrom fails to restore the default image with the same error. Are you able to assist in getting the default flash restored so I can once again reboot this board? I've attached a copy of the verbose log for a failed flash. -Stefan

2 1

Verification after write vs erase operations
by Anastasia Klimchuk 03 Mar '24

03 Mar '24

Hello everyone, I want to bring into discussion a topic of verification of chip content which is performed after write or erase operations. Also for context prior discussion in this ticket: https://ticket.coreboot.org/issues/520 Currently we have a flag `-n` or `dont_verify_it` which is false by default. What the flag is doing is deciding whether verification needs to be performed after writing. The flag is false by default, so by default verification is happening after writing (which makes sense). Now the flag is only controlling write operation, and has no effect on erase operation. So for erase, the flag is ignored and verification is performed always. Just for context, it has always been like this. So for write operation we have a power option to say "ignore verification", and since it's false by default, the idea is, if you set it true then you know what you are doing. There is no such option for erase which seems a bit unfair. The two potential options to introduce equivalent option for erase are: 1) Add new flag `dont_verify_erase` (or some similar name) As a support for this option, it does not affect any existing use cases/scripts, and can be added any time. 2) Add coverage for erase operation under existing flag `-n` or `dont_verify_it` As a support for this option, erase operation can be seen as a variant of writing, since it does modify content of the chip, and so the fact it's not covered by the existing flag can be treated as a bug. What do people think about it? -- Anastasia.

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

flashrom March 2024