On Thu, Mar 17, 2022 at 10:35 AM Nico Huber nico.h@gmx.de wrote:
On 17.03.22 15:35, Subrata Banik wrote:
On Thu, Mar 17, 2022 at 6:08 PM Nico Huber nico.h@gmx.de wrote:
On 16.03.22 17:19, Subrata Banik wrote:
The reason for this refactoring of HW sequencing SPI driver code are:
- We (Chrome OS team) recently ran into a firmware update issue on
dogfooders (test device utilise by real users and share feedback) systems, where the attempt to perform SPI flash update operation from host-cpu (using underlying flashrom utility) is silently failing. Furthermore,
we
debug the issue with help of Intel and figure out the problem is
related
to multiple master is accessing the SPI flash and operation triggered by host-cpu side using flashrom (write and erase operation) is getting
timed
out (due to given lesser timeout boundary) due to underlying SPI bus is occupied by other master.
Does the issue persist with the following commit applied to libflashrom, activated and integrated into futility?
commit 330088d77 [CHROMIUM]: libflashrom: Also use USE_BIG_LOCK
This would give us a hint if the issue is indeed due to multiple masters or actually due to multiple programs trying to control the same master.
[Subrata] Yes, we are seeing failure even with USE_BIG_LOCK code changes being integrated. The only code change that helps us to W/A this issue is here
https://chromium-review.googlesource.com/c/chromiumos/third_party/flashrom/+...
(increase timeout in flashrom operations). Also, Intel disable the CSE runtime
access
to SPI using a custom tool and that helped to fix this update failure as well. These are the evidence towards multi master theory in AU failure issue.
I see. I was probably too distracted by the refactoring patches earlier. CB:62867 is indeed necessary.
We seem to have actually figured out what the actual root cause of our problem was, and it indeed does seem to be the multi-master problem and flashrom not waiting long enough in these cases. tl;dr widevine provisioning was failing, and repeating endlessly in a loop (~each second or so) and each attempt actually causes the CSE to write to its data area in flash. Unfortunately, we do not have fantastic visibility into the processes that are running during AU, and so this was very hard to track down. We have fixed the issues with widevine provisioning and also added a max 5 attempts to provision a device instead of writing endlessly to the flash in the failure case.
We already assumed a 5s timeout is needed for erasing a block. Worst case seems to be that all masters try to do this at once and then one master would have to wait (NM-1)*5s before its transaction even starts (NM being the number of masters).
We could make it depend on the number of masters, or just choose some- thing safe like 30s. WDYT?
I think the longer the timeout (until it gets ridiculous), probably the better. Unless there is buggy hardware out there, there is no good reason for these operations to timeout; there could for sure be SPI flash chips out there that perhaps aren't always performing to-spec, who knows.
But in this case, since do we know the maximum number of masters, and 5s per erase seems reasonable too, then 30 seconds makes sense to me.
This is the special case, starting from Tiger Lake Chrome Platform,
where
we have enable the PVAP (Protected Audio Video Path) which request CSE
to
perform some erase and write operation as needed.
We used flashrom on PAVP enabled systems for more than a decade without issues. Did something change in that area for Tiger Lake in particular?
[Subrata] I'm not sure about other platforms, but from TGL onwards, on Chrome devices we have enabled PAVP use case and Intel had tried to replicate this issue on older platforms as well and informed us it's replicated the issue on TGL platform as well where PAVP first being
enabled
and not on JSL (which doesn't enable PAVP). Right now the recommendation that is coming out from Intel is that, on latest Chrome OS devices, there might be concurrent SPI accesses between host CPU and CSE. But I will ask this question about the older generation platform as you have highlighted which has PAVP enabled. Can you please let me know those platform names ? Also, do you know on those devices if we are trying to update the FW
using
flashrom tool as well or not?
Thanks for looking into it. As our timeouts were way below the worst case, I assume enabling PAVP just makes the problem more visible and is not directly related. It could also be any other code running on the CSE that changed between JSL and TGL (unless you know, ofc., that PAVP is the only code that writes to flash).
PAVP is the only thing we know of ATM that writes to the flash aside from perhaps after the first flash of the ME region.
The reason this became visible on brya only now I explained just above :)
Flashrom basically runs on everything so the platform names would be all platforms that support PAVP. Probably not worth looking further into it unless there are still issues with a reasonable timeout.
- Maintain the code symmetry between coreboot and flashrom SPI
hardware
sequencing operations.
Flashrom has tighter requirements compared to coreboot because we try to support all compatible chipset generations with the same code. If you want to align the two code-bases, please use flashrom as reference and patch coreboot.
[Subrata] Let me understand this a little better, are you suggesting not
to
increase the SPI operation timing in flashrom upstream master ? knowing that there is a hidden issue acknowledged by Intel with their validation data, also, promised to update the SPI programming guide to capture the recommendation of timeout in multi master scenarios.
No I'm not suggesting any such thing. What I wrote was related to your 2. point as quoted above.
Apart from timeout increasing CLs, other code changes are to increase the code modularity (like using macro and helper function), are you
suggesting
to not touch the flashrom code? can't we all contribute towards
validating
the concerned target platforms. At Least in the coreboot community we all help each other to validate the platform if we are out of required
devices.
We can have a look at the individual changes if there is a reason to change the code, i.e. to make it better. Only your reasoning "maintain the code symmetry" is not enough to justify changes.
I'm sorry if this affects your work, but we have very bad experience with unnecessary code changes by CrOS developers. Currently `sb600spi` (the equivalent to `ichspi` for AMD platforms) is broken since months and only CrOS developers could clarify why the code was changed as it was. We need to prevent that this happens to `ichspi` too, right?
Nico
We are certainly not intending to break ichspi or make it worse, we are trying to make it better, perhaps we were a little hasty in trying to get through solutions that were not perhaps the root of the problem, but were discovered along the way.
We are also working with Intel to get some documentation out about the expected behaviors with regard to the multi-master scenarios, hopefully that will help clarify things some in this regard.
-Tim