Hi,
I've been looking into S3 resume on GM45 mainboards, which often fails in rather interesting ways.
Many X200 units resume on every try (barring any panics in Linux, of course). Some only resume about half of the time. And some/most T400 units fail to resume all of the time.
These systems fail to resume in one of the following ways:
* S3 resume (indicated by the SLP_TYP bit) is detected, SLP_TYP is cleared, DRAM receive-enable calibration fails with a timing under/overflow, the system resets, and coreboot boots normally into the payload (with the sleep LED still on) because SLP_TYP is now unset. See x200-resume-fail-receive-enable-calibration.log and t400-resume-fail-receive-enable-calibration.log. * S3 resume is detected, SLP_TYP is cleared, raminit and the rest of romstage completes without error, but then something between the southbridge's smm_init() and cpu_initialize() hangs (maybe the system is stuck in SMM). See x200-resume-fail-smm-hang.log and t400-resume-fail-smm-hang.log. * S3 resume is detected, SLP_TYP is cleared, romstage completes, but something within smm_init() hangs before dumping (possibly while clearing [1]) TCO1_STS bits. See t400-resume-fail-tco-hang.log
There are a couple of other ways in which I've seen S3 resume fail, but these are the most common.
I thought of working around the first issue (clearing SLP_TYP, resetting due to a raminit error, then booting into the payload) by clearing SLP_TYP near the end of the romstage main() (after raminit). So I tried the following patch:
--- diff --git a/src/mainboard/lenovo/x200/romstage.c b/src/mainboard/lenovo/x200/romstage.c index 86a973f..915baf2 100644 --- a/src/mainboard/lenovo/x200/romstage.c +++ b/src/mainboard/lenovo/x200/romstage.c @@ -103,10 +103,6 @@ void main(unsigned long bist) #if CONFIG_HAVE_ACPI_RESUME printk(BIOS_DEBUG, "Resume from S3 detected.\n"); s3resume = 1; - /* Clear SLP_TYPE. This will break stage2 but - * we care for that when we get there. - */ - outl(pm1_cnt & ~(7 << 10), DEFAULT_PMBASE + 0x04); #else printk(BIOS_DEBUG, "Resume from S3 detected, but disabled.\n"); #endif @@ -190,6 +186,11 @@ void main(unsigned long bist)
/* Magic for S3 resume */ pci_write_config32(PCI_DEV(0, 0, 0), D0F0_SKPD, SKPAD_ACPI_S3_MAGIC); + + /* Clear SLP_TYPE. This will break stage2 but + * we care for that when we get there. + */ + outl(pm1_cnt & ~(7 << 10), DEFAULT_PMBASE + 0x04); } else { /* Magic for S3 resume */ pci_write_config32(PCI_DEV(0, 0, 0), D0F0_SKPD, SKPAD_NORMAL_BOOT_MAGIC); ---
But that just made these errors even more frequent. Trying to resume from S3 put the system into a reset loop with receive-enable calibration errors (see x200-patched-resume-fail-receive-enable-loop.log). So instead of rebooting into the payload or hanging, the system just resets forever.
So, it seems that whenever the system boots with SLP_TYP set, something is very likely to go wrong in coreboot and cause either a reset or a hang.
Does anyone have any ideas why this might be?
[1]: Tangentially, I noticed that the i82801ix reset_tco_status() says "Don't clear BOOT_STS before SECOND_TO_STS" when it clears BOOT_STS. In the next two lines it clears BOOT_STS if set. It never clears SECOND_TO_STS. Is this a bug? However, according to the ICH9 datasheet, there is no SECOND_TO_STS bit in TCO1_STS (the high bits of that register are reserved).
Thanks,
Hi,
On 11.11.2015 00:49, Patrick 'P. J.' McDermott wrote:
I've been looking into S3 resume on GM45 mainboards, which often fails in rather interesting ways.
Well, the S3 support wasn't really tested during GM45 development. Maybe it's just plainly broken. My development system at work (roda/rk9) doesn't resume because of another problem (but didn't fail raminit on the resume path in 3 of 3 tries). So it will need some work before I can test this.
Thanks for taking the time to test on different systems and looking into this. Can you try to find out which processors (model, stepping) and northbridge stepping was used? Also did you use the latest processor microcode?
[1]: Tangentially, I noticed that the i82801ix reset_tco_status() says "Don't clear BOOT_STS before SECOND_TO_STS" when it clears BOOT_STS. In the next two lines it clears BOOT_STS if set. It never clears SECOND_TO_STS. Is this a bug? However, according to the ICH9 datasheet, there is no SECOND_TO_STS bit in TCO1_STS (the high bits of that register are reserved).
Those are R/WC read/write-clear bits. The bits get cleared by writing a '1'. That's for convenience as you can clear them all by writing the value you just read. If there is a SECOND_TO_STS bit and it's set, it gets automagically cleared by the first write (by writing the value we read before). Clearing BOOT_STS gets deferred to the second write.
Ah, just had a closer look at the datasheet: The bits are defined in the next register. I'm not sure if it's valid to read/write both registers at once with a 32-bit access, though. But it seems to work for other chipsets (e.g. i82801gx, bd82x6x).
Nico
On 2015-11-11 16:50, Nico Huber wrote:
Hi,
On 11.11.2015 00:49, Patrick 'P. J.' McDermott wrote:
I've been looking into S3 resume on GM45 mainboards, which often fails in rather interesting ways.
Well, the S3 support wasn't really tested during GM45 development. Maybe it's just plainly broken. My development system at work (roda/rk9) doesn't resume because of another problem (but didn't fail raminit on the resume path in 3 of 3 tries). So it will need some work before I can test this.
Ah, OK. What kind of other problem? In addition to the raminit reset, I've seen resume fail by an SMM hang and in other ways.
Thanks for taking the time to test on different systems and looking into this. Can you try to find out which processors (model, stepping) and northbridge stepping was used? Also did you use the latest processor microcode?
I've seen S3 resume problems occur on X200 units with CPUID 10676 (microcode revision 0x6, stepping M0). I've also seen reports of problems on X200 units with CPUID 1067A (microcode revision 0xA, stepping R0, the latest available stepping/microcode).
I have a T400 with CPUID 1067A (again, latest available microcode) that fails to resume, but I've also seen a report of a T400 on which S3 resume works.
On all tested systems, the northbridge stepping is B3, the only available stepping for this northbridge.
The logs in my previous message are from a couple of X200 units with 10676 and a T400 with 1067A.
Hi,
had a look at your logs:
On 11.11.2015 00:49, Patrick 'P. J.' McDermott wrote:
These systems fail to resume in one of the following ways:
- S3 resume (indicated by the SLP_TYP bit) is detected, SLP_TYP is cleared, DRAM receive-enable calibration fails with a timing under/overflow, the system resets, and coreboot boots normally into the payload (with the sleep LED still on) because SLP_TYP is now unset. See x200-resume-fail-receive-enable-calibration.log and t400-resume-fail-receive-enable-calibration.log.
- S3 resume is detected, SLP_TYP is cleared, raminit and the rest of romstage completes without error, but then something between the southbridge's smm_init() and cpu_initialize() hangs (maybe the system is stuck in SMM). See x200-resume-fail-smm-hang.log and t400-resume-fail-smm-hang.log.
I have yet no idea about the SMM hang.
- S3 resume is detected, SLP_TYP is cleared, romstage completes, but something within smm_init() hangs before dumping (possibly while clearing [1]) TCO1_STS bits. See t400-resume-fail-tco-hang.log
The logs are all a little garbled. It looks to me like this is exactly the same hang as in *-resume-fail-smm-hang.log.
There are a couple of other ways in which I've seen S3 resume fail, but these are the most common.
I thought of working around the first issue (clearing SLP_TYP, resetting due to a raminit error, then booting into the payload) by clearing SLP_TYP near the end of the romstage main() (after raminit). So I tried the following patch:
diff --git a/src/mainboard/lenovo/x200/romstage.c b/src/mainboard/lenovo/x200/romstage.c index 86a973f..915baf2 100644 --- a/src/mainboard/lenovo/x200/romstage.c +++ b/src/mainboard/lenovo/x200/romstage.c @@ -103,10 +103,6 @@ void main(unsigned long bist) #if CONFIG_HAVE_ACPI_RESUME printk(BIOS_DEBUG, "Resume from S3 detected.\n"); s3resume = 1;
/* Clear SLP_TYPE. This will break stage2 but* we care for that when we get there.*/outl(pm1_cnt & ~(7 << 10), DEFAULT_PMBASE + 0x04);#else printk(BIOS_DEBUG, "Resume from S3 detected, but disabled.\n"); #endif @@ -190,6 +186,11 @@ void main(unsigned long bist)
/* Magic for S3 resume */ pci_write_config32(PCI_DEV(0, 0, 0), D0F0_SKPD, SKPAD_ACPI_S3_MAGIC);
/* Clear SLP_TYPE. This will break stage2 but* we care for that when we get there.*/outl(pm1_cnt & ~(7 << 10), DEFAULT_PMBASE + 0x04);
But that just made these errors even more frequent. Trying to resume from S3 put the system into a reset loop with receive-enable calibration errors (see x200-patched-resume-fail-receive-enable-loop.log). So instead of rebooting into the payload or hanging, the system just resets forever.
This reset loop is very interesting. Did it end sometime? It could mean the worst, i.e. the RAM lost its configuration (self refresh failed). I suspect that's the case as there is not much difference in the normal vs. the resume path until receive-enable calibration.
Nico
On 12.11.2015 04:37, Patrick 'P. J.' McDermott wrote:
On 2015-11-11 16:50, Nico Huber wrote:
Hi,
On 11.11.2015 00:49, Patrick 'P. J.' McDermott wrote:
I've been looking into S3 resume on GM45 mainboards, which often fails in rather interesting ways.
Well, the S3 support wasn't really tested during GM45 development. Maybe it's just plainly broken. My development system at work (roda/rk9) doesn't resume because of another problem (but didn't fail raminit on the resume path in 3 of 3 tries). So it will need some work before I can test this.
Ah, OK. What kind of other problem? In addition to the raminit reset, I've seen resume fail by an SMM hang and in other ways.
There was a minor flaw in the mainboard code: http://review.coreboot.org/#/q/topic:rk9-resume With patches applied, it works halfway reliable: about 30 good suspend- resume cycles before it fails. Only eye-catching thing in dmesg was a warning about the backlight being already enabled. I haven't yet a serial log for a failed resume.
One more thing that came in mind: The reset after a failed receive- enable calibration is kind of wanted. IIRC, we left the watchdog enable for a case of failing raminit. But I don't remember what the exact fai- lure was. It wasn't very unlikely to occur (> 1/1000). If we run into this on the resume path too, it might never work reliable :-/
Nico
On 2015-11-12 13:55, Nico Huber wrote:
Hi,
had a look at your logs:
On 11.11.2015 00:49, Patrick 'P. J.' McDermott wrote:
These systems fail to resume in one of the following ways:
- S3 resume (indicated by the SLP_TYP bit) is detected, SLP_TYP is cleared, DRAM receive-enable calibration fails with a timing under/overflow, the system resets, and coreboot boots normally into the payload (with the sleep LED still on) because SLP_TYP is now unset. See x200-resume-fail-receive-enable-calibration.log and t400-resume-fail-receive-enable-calibration.log.
- S3 resume is detected, SLP_TYP is cleared, raminit and the rest of romstage completes without error, but then something between the southbridge's smm_init() and cpu_initialize() hangs (maybe the system is stuck in SMM). See x200-resume-fail-smm-hang.log and t400-resume-fail-smm-hang.log.
I have yet no idea about the SMM hang.
- S3 resume is detected, SLP_TYP is cleared, romstage completes, but something within smm_init() hangs before dumping (possibly while clearing [1]) TCO1_STS bits. See t400-resume-fail-tco-hang.log
The logs are all a little garbled. It looks to me like this is exactly the same hang as in *-resume-fail-smm-hang.log.
There are a couple of other ways in which I've seen S3 resume fail, but these are the most common.
I thought of working around the first issue (clearing SLP_TYP, resetting due to a raminit error, then booting into the payload) by clearing SLP_TYP near the end of the romstage main() (after raminit). So I tried the following patch:
diff --git a/src/mainboard/lenovo/x200/romstage.c b/src/mainboard/lenovo/x200/romstage.c index 86a973f..915baf2 100644 --- a/src/mainboard/lenovo/x200/romstage.c +++ b/src/mainboard/lenovo/x200/romstage.c @@ -103,10 +103,6 @@ void main(unsigned long bist) #if CONFIG_HAVE_ACPI_RESUME printk(BIOS_DEBUG, "Resume from S3 detected.\n"); s3resume = 1;
/* Clear SLP_TYPE. This will break stage2 but* we care for that when we get there.*/outl(pm1_cnt & ~(7 << 10), DEFAULT_PMBASE + 0x04);#else printk(BIOS_DEBUG, "Resume from S3 detected, but disabled.\n"); #endif @@ -190,6 +186,11 @@ void main(unsigned long bist)
/* Magic for S3 resume */ pci_write_config32(PCI_DEV(0, 0, 0), D0F0_SKPD, SKPAD_ACPI_S3_MAGIC);
/* Clear SLP_TYPE. This will break stage2 but* we care for that when we get there.*/outl(pm1_cnt & ~(7 << 10), DEFAULT_PMBASE + 0x04);
But that just made these errors even more frequent. Trying to resume from S3 put the system into a reset loop with receive-enable calibration errors (see x200-patched-resume-fail-receive-enable-loop.log). So instead of rebooting into the payload or hanging, the system just resets forever.
This reset loop is very interesting. Did it end sometime? It could mean the worst, i.e. the RAM lost its configuration (self refresh failed). I suspect that's the case as there is not much difference in the normal vs. the resume path until receive-enable calibration.
No, it didn't end. I once left it running for probably at least 10 or 20 minutes, so it must have gone through hundreds of raminit/reset cycles.
As shown in *-resume-fail-receive-enable-calibration.log, that kind of raminit failure happens with an unpatched coreboot as well (more commonly on the X200 than the SMM hang does). So it makes some sense that patching romstage main() in the way that I did would cause that error to happen in a loop. Basically, that patch fixed the problem of losing SLP_TYP after reset but worsened the problem of raminit failing. (It's a little odd that the loop /never/ ended, while sometimes an unpatched coreboot would get past receive-enable calibration in the resume path.)
What doesn't make sense is that receive-enable calibration only fails when SLP_TYP is set. But it always works when SLP_TYP is unset (as in a normal boot or with an unpatched coreboot after S3 resume is detected, raminit fails, and the system resets).
[Resending after accidentally replying off-list]
On 2015-11-12 14:32, Nico Huber wrote:
On 12.11.2015 04:37, Patrick 'P. J.' McDermott wrote:
On 2015-11-11 16:50, Nico Huber wrote:
Hi,
On 11.11.2015 00:49, Patrick 'P. J.' McDermott wrote:
I've been looking into S3 resume on GM45 mainboards, which often fails in rather interesting ways.
Well, the S3 support wasn't really tested during GM45 development. Maybe it's just plainly broken. My development system at work (roda/rk9) doesn't resume because of another problem (but didn't fail raminit on the resume path in 3 of 3 tries). So it will need some work before I can test this.
Ah, OK. What kind of other problem? In addition to the raminit reset, I've seen resume fail by an SMM hang and in other ways.
There was a minor flaw in the mainboard code: http://review.coreboot.org/#/q/topic:rk9-resume
Ah, I see. Yeah, X200 and T400 romstages don't have that flaw.
With patches applied, it works halfway reliable: about 30 good suspend- resume cycles before it fails. Only eye-catching thing in dmesg was a warning about the backlight being already enabled. I haven't yet a serial log for a failed resume.
How does it fail?
One more thing that came in mind: The reset after a failed receive- enable calibration is kind of wanted. IIRC, we left the watchdog enable for a case of failing raminit. But I don't remember what the exact fai- lure was. It wasn't very unlikely to occur (> 1/1000). If we run into this on the resume path too, it might never work reliable :-/
-
Nico Huber -
Patrick 'P. J.' McDermott