On 17.03.22 15:35, Subrata Banik wrote:
On Thu, Mar 17, 2022 at 6:08 PM Nico Huber nico.h@gmx.de wrote:
On 16.03.22 17:19, Subrata Banik wrote:
The reason for this refactoring of HW sequencing SPI driver code are:
- We (Chrome OS team) recently ran into a firmware update issue on
dogfooders (test device utilise by real users and share feedback) systems, where the attempt to perform SPI flash update operation from host-cpu (using underlying flashrom utility) is silently failing. Furthermore, we debug the issue with help of Intel and figure out the problem is related to multiple master is accessing the SPI flash and operation triggered by host-cpu side using flashrom (write and erase operation) is getting timed out (due to given lesser timeout boundary) due to underlying SPI bus is occupied by other master.
Does the issue persist with the following commit applied to libflashrom, activated and integrated into futility?
commit 330088d77 [CHROMIUM]: libflashrom: Also use USE_BIG_LOCK
This would give us a hint if the issue is indeed due to multiple masters or actually due to multiple programs trying to control the same master.
[Subrata] Yes, we are seeing failure even with USE_BIG_LOCK code changes being integrated. The only code change that helps us to W/A this issue is here https://chromium-review.googlesource.com/c/chromiumos/third_party/flashrom/+... (increase timeout in flashrom operations). Also, Intel disable the CSE runtime access to SPI using a custom tool and that helped to fix this update failure as well. These are the evidence towards multi master theory in AU failure issue.
I see. I was probably too distracted by the refactoring patches earlier. CB:62867 is indeed necessary.
We already assumed a 5s timeout is needed for erasing a block. Worst case seems to be that all masters try to do this at once and then one master would have to wait (NM-1)*5s before its transaction even starts (NM being the number of masters).
We could make it depend on the number of masters, or just choose some- thing safe like 30s. WDYT?
This is the special case, starting from Tiger Lake Chrome Platform, where
we have enable the PVAP (Protected Audio Video Path) which request CSE to perform some erase and write operation as needed.
We used flashrom on PAVP enabled systems for more than a decade without issues. Did something change in that area for Tiger Lake in particular?
[Subrata] I'm not sure about other platforms, but from TGL onwards, on Chrome devices we have enabled PAVP use case and Intel had tried to replicate this issue on older platforms as well and informed us it's replicated the issue on TGL platform as well where PAVP first being enabled and not on JSL (which doesn't enable PAVP). Right now the recommendation that is coming out from Intel is that, on latest Chrome OS devices, there might be concurrent SPI accesses between host CPU and CSE. But I will ask this question about the older generation platform as you have highlighted which has PAVP enabled. Can you please let me know those platform names ? Also, do you know on those devices if we are trying to update the FW using flashrom tool as well or not?
Thanks for looking into it. As our timeouts were way below the worst case, I assume enabling PAVP just makes the problem more visible and is not directly related. It could also be any other code running on the CSE that changed between JSL and TGL (unless you know, ofc., that PAVP is the only code that writes to flash).
Flashrom basically runs on everything so the platform names would be all platforms that support PAVP. Probably not worth looking further into it unless there are still issues with a reasonable timeout.
- Maintain the code symmetry between coreboot and flashrom SPI hardware
sequencing operations.
Flashrom has tighter requirements compared to coreboot because we try to support all compatible chipset generations with the same code. If you want to align the two code-bases, please use flashrom as reference and patch coreboot.
[Subrata] Let me understand this a little better, are you suggesting not to increase the SPI operation timing in flashrom upstream master ? knowing that there is a hidden issue acknowledged by Intel with their validation data, also, promised to update the SPI programming guide to capture the recommendation of timeout in multi master scenarios.
No I'm not suggesting any such thing. What I wrote was related to your 2. point as quoted above.
Apart from timeout increasing CLs, other code changes are to increase the code modularity (like using macro and helper function), are you suggesting to not touch the flashrom code? can't we all contribute towards validating the concerned target platforms. At Least in the coreboot community we all help each other to validate the platform if we are out of required devices.
We can have a look at the individual changes if there is a reason to change the code, i.e. to make it better. Only your reasoning "maintain the code symmetry" is not enough to justify changes.
I'm sorry if this affects your work, but we have very bad experience with unnecessary code changes by CrOS developers. Currently `sb600spi` (the equivalent to `ichspi` for AMD platforms) is broken since months and only CrOS developers could clarify why the code was changed as it was. We need to prevent that this happens to `ichspi` too, right?
Nico