Hi,
I was hoping to get some advice/guidance on how to disable the automatic refresh of the DRAM during the ROM stage of coreboot.
I am booting an ASUS F2A85-M motherboard with a AMD A6-5400k processor (Trinity core) using coreboot. The processor is initialized via the AGESA bootstrap protocol.
I would like to use the AGESA softwarehttps://github.com/coreboot/coreboot/tree/master/src/vendorcode/amd/agesa/f15tn in order to measure the retention time of DRAM cells (i.e. the time a DRAM cell holds its value with refresh shut off). To do this, I must (1) run my code in cache-as-RAM mode because DRAM becomes unreliable without refresh and (2) have a way to switch DRAM refresh on/off from software. (1) is accomplished by running my code in the cache-as-RAM phase of coreboot (i.e. the ROM stage). I am, however, unclear on how to accomplish (2) and I was hoping to receive some pointers/guidance on how to do this.
The AGESA protocol seems to handle memory initialization in the AmdInitPosthttps://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Common/AmdInitPost.c#L223 phase, specifically in AmdMemAutohttps://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mmflow.c#L135 In reading this code I discovered a call with which one could disable/enable the auto-refresh feature of the DRAM. Specifically,
MemNSetBitFieldNbhttps://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/NB/mnreg.c#L218 (NBPtr, BFDisAutoRefresh, 1);
where NBPtr is a pointer to a data structure representing the northbridge configuration and BFDisAutoRefreshhttps://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/mm.h#L365 is a bit field which can be set to 1 in order to disable auto-refresh of the DRAM (see AMD BKDGhttp://support.amd.com/TechDocs/42300_15h_Mod_10h-1Fh_BKDG.pdf, page 344 on DisAutoRefresh).
With this interface, I thought I could (1) write a value to memory (using movnti instruction to bypass the cache), (2) disable auto-refresh using MemNSetBitFieldNb, (3) wait some number of seconds, (4) enable auto-refresh using MemNSetBitFieldNb, and (5) check whether the value has persisted over the given period of time. Unfortunately, I discovered that even after waiting 3 minutes, the value in DRAM persisted (contrary to what seems to be expected by the literaturehttp://www.pdl.cmu.edu/PDL-FTP/NVM/dram-retention_isca13.pdf). Note that I perform this procedure inside the AmdMemAutohttps://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mmflow.c#L285 procedure of the AGESA code once all memory initialization tasks have completed and before teardown occurs.
Worried that the processor may be caching the value, I decided to flush data at the address before disabling auto-refresh with the following interface provided by the AGESA codebase:
MemUFlushPatternhttps://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mu.c#L219 (Addr, 1);
where 1 simply represents the number of cache lines to flush from the address Addr. This, however, proved not to solve my issue.
I am curious if anyone has dug into the AGESA vendor code and has any experience advice for how to approach the functionality I am trying to achieve.
Thank you for all your help! Berj
Berj K Chilingirian wrote:
I was hoping to get some advice/guidance on how to disable the automatic refresh of the DRAM during the ROM stage of coreboot.
Reset the memory controller, or the entire platform.
Or, if you find no software solution, maybe you can use a fast FET or logic gate on chip select or another key signal, and use your choice of microcontroller to block refreshes. I don't know if DRAM needs to reply to refreshes, in that case the idea will probably /not/ work, the memory controller will probably get upset without replies.
If you do try it, remember to gate the gate such that it never switches in the middle of transactions/commands.
Regardless of how you succeed, I'm interested in your findings.
//Peter
On Fri, Mar 24, 2017 at 2:15 AM, Berj K Chilingirian berjc@mit.edu wrote:
Hi,
I was hoping to get some advice/guidance on how to disable the automatic refresh of the DRAM during the ROM stage of coreboot.
I am booting an ASUS F2A85-M motherboard with a AMD A6-5400k processor (Trinity core) using coreboot. The processor is initialized via the AGESA bootstrap protocol.
I would like to use the AGESA software https://github.com/coreboot/coreboot/tree/master/src/vendorcode/amd/agesa/f15tn in order to measure the retention time of DRAM cells (i.e. the time a DRAM cell holds its value with refresh shut off). To do this, I must (1) run my code in cache-as-RAM mode because DRAM becomes unreliable without refresh and (2) have a way to switch DRAM refresh on/off from software. (1) is accomplished by running my code in the cache-as-RAM phase of coreboot (i.e. the ROM stage). I am, however, unclear on how to accomplish (2) and I was hoping to receive some pointers/guidance on how to do this.
The AGESA protocol seems to handle memory initialization in the AmdInitPost https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Common/AmdInitPost.c#L223 phase, specifically in AmdMemAuto https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mmflow.c#L135 In reading this code I discovered a call with which one could disable/enable the auto-refresh feature of the DRAM. Specifically,
MemNSetBitFieldNb https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/NB/mnreg.c#L218 (NBPtr, BFDisAutoRefresh, 1);
where NBPtr is a pointer to a data structure representing the northbridge configuration and BFDisAutoRefresh https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/mm.h#L365 is a bit field which can be set to 1 in order to disable auto-refresh of the DRAM (see AMD BKDG http://support.amd.com/TechDocs/42300_15h_Mod_10h-1Fh_BKDG.pdf, page 344 on DisAutoRefresh).
With this interface, I thought I could (1) write a value to memory (using movnti instruction to bypass the cache), (2) disable auto-refresh using MemNSetBitFieldNb, (3) wait some number of seconds, (4) enable auto-refresh using MemNSetBitFieldNb, and (5) check whether the value has persisted over the given period of time. Unfortunately, I discovered that even after waiting 3 minutes, the value in DRAM persisted (contrary to what seems to be expected by the literature http://www.pdl.cmu.edu/PDL-FTP/NVM/dram-retention_isca13.pdf). Note that I perform this procedure inside the AmdMemAuto https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mmflow.c#L285 procedure of the AGESA code once all memory initialization tasks have completed and before teardown occurs.
Worried that the processor may be caching the value, I decided to flush data at the address before disabling auto-refresh with the following interface provided by the AGESA codebase:
Patch against the source is easier to diagnose for possible mistakes than verbal expression of what you believe you are doing. Eg. the address you write to makes a difference.
Kyösti
I would like to use the AGESA software
https://github.com/coreboot/coreboot/tree/master/src/vendorcode/amd/agesa/f15tn in order to measure the retention time of DRAM cells
(i.e. the time a DRAM cell holds its value with refresh shut off).
I am not sure what are you really trying to do, and, mostly WHY you are trying to do what you are trying to do?! :bang:
Since I found the following: *Typically, manufacturers specify that each row must be refreshed every 64 ms or less, as defined by the JEDEC https://en.wikipedia.org/wiki/JEDEC standard.*
*Some systems refresh every row in a burst of activity involving all rows every 64 ms. Other systems refresh one row at a time staggered throughout the 64 ms interval. For example, a system with 213 = 8,192 rows would require a staggered refresh rate https://en.wikipedia.org/wiki/Refresh_rate of one row every 7.8 µs which is 64 ms divided by 8,192 rows.*
Now, since this is how the DRAM cell design typically looks like: https://upload.wikimedia.org/wikipedia/commons/b/bd/DRAM_Cell_ Structure_%28Model_of_Single_Circuit_Cell%29.PNG
How you can ensure that individual C is the same value, or similar values for the entire silicon? Nope, you can not, so your determined timings will be mostly out of order. Since these individual Cs can differ even 20%, even more in values. As well as channel wide and length of each individual FET you see there.
In other words, reliability and controlability of your measurements are in question/doubt?!
Manufacturer of DRAM ensures that C is no less that some minimum value, where (my best guess) each manufacturer is off around minimal 20% less timing, to ensure the refresh will be done completely correctly/safely/reliable.
My advice to you will be to follow manufacturer's DRAM Data Sheet, which will give you the most reliable refresh time available to you. ;-)
If you need to save some time tuning IMC, I would advice to you to pay for the faster CPU and faster (DDR3 -> DDR4) memories.
Zoran
On Fri, Mar 24, 2017 at 1:15 AM, Berj K Chilingirian berjc@mit.edu wrote:
Hi,
I was hoping to get some advice/guidance on how to disable the automatic refresh of the DRAM during the ROM stage of coreboot.
I am booting an ASUS F2A85-M motherboard with a AMD A6-5400k processor (Trinity core) using coreboot. The processor is initialized via the AGESA bootstrap protocol.
I would like to use the AGESA software https://github.com/coreboot/coreboot/tree/master/src/vendorcode/amd/agesa/f15tn in order to measure the retention time of DRAM cells (i.e. the time a DRAM cell holds its value with refresh shut off). To do this, I must (1) run my code in cache-as-RAM mode because DRAM becomes unreliable without refresh and (2) have a way to switch DRAM refresh on/off from software. (1) is accomplished by running my code in the cache-as-RAM phase of coreboot (i.e. the ROM stage). I am, however, unclear on how to accomplish (2) and I was hoping to receive some pointers/guidance on how to do this.
The AGESA protocol seems to handle memory initialization in the AmdInitPost https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Common/AmdInitPost.c#L223 phase, specifically in AmdMemAuto https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mmflow.c#L135 In reading this code I discovered a call with which one could disable/enable the auto-refresh feature of the DRAM. Specifically,
MemNSetBitFieldNb https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/NB/mnreg.c#L218 (NBPtr, BFDisAutoRefresh, 1);
where NBPtr is a pointer to a data structure representing the northbridge configuration and BFDisAutoRefresh https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/mm.h#L365 is a bit field which can be set to 1 in order to disable auto-refresh of the DRAM (see AMD BKDG http://support.amd.com/TechDocs/42300_15h_Mod_10h-1Fh_BKDG.pdf, page 344 on DisAutoRefresh).
With this interface, I thought I could (1) write a value to memory (using movnti instruction to bypass the cache), (2) disable auto-refresh using MemNSetBitFieldNb, (3) wait some number of seconds, (4) enable auto-refresh using MemNSetBitFieldNb, and (5) check whether the value has persisted over the given period of time. Unfortunately, I discovered that even after waiting 3 minutes, the value in DRAM persisted (contrary to what seems to be expected by the literature http://www.pdl.cmu.edu/PDL-FTP/NVM/dram-retention_isca13.pdf). Note that I perform this procedure inside the AmdMemAuto https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mmflow.c#L285 procedure of the AGESA code once all memory initialization tasks have completed and before teardown occurs.
Worried that the processor may be caching the value, I decided to flush data at the address before disabling auto-refresh with the following interface provided by the AGESA codebase:
MemUFlushPattern https://github.com/coreboot/coreboot/blob/master/src/vendorcode/amd/agesa/f15tn/Proc/Mem/Main/mu.c#L219 (Addr, 1);
where 1 simply represents the number of cache lines to flush from the address Addr. This, however, proved not to solve my issue.
I am curious if anyone has dug into the AGESA vendor code and has any experience advice for how to approach the functionality I am trying to achieve.
Thank you for all your help! Berj
-- coreboot mailing list: coreboot@coreboot.org https://www.coreboot.org/mailman/listinfo/coreboot
-
Berj K Chilingirian -
Kyösti Mälkki -
Peter Stuge -
Zoran Stojsavljevic