My VX800 bug of "boot linux with 2G mem >10 minutes" has been proved to be the problem of MTRR setting. Thanks for Carldani, Rudolf and Bari. 1 The root cause is: 1) The free memory I report to coreboot through ram_resource is [0, 1983M=2G-64M(Framebuffer)-1M(TOP 1M SM Memory)]. If I report [0, 1984M=2G-64M(Framebuffer)], then all is OK. 2) The src/cpu/x86/mtrr.c setting result of [0,1983M] is: reg00: base=0x00000000 ( 0MB), size=1024MB: write-back, count=1 reg01: base=0x40000000 (1024MB), size= 512MB: write-back, count=1 reg02: base=0x60000000 (1536MB), size= 256MB: write-back, count=1 reg03: base=0x70000000 (1792MB), size= 128MB: write-back, count=1 reg04: base=0x78000000 (1920MB), size= 32MB: write-back, count=1 reg05: base=0x7a000000 (1952MB), size= 16MB: write-back, count=1 reg06: base=0x7b000000 (1968MB), size= 8MB: write-back, count=1 reg07: base=0x7b800000 (1976MB), size= 4MB: write-back, count=1 This setting make the last 3MB is uncached due to the lack of MTRR registers. And It seems that the last 3MB is frequently used by linux with kernel before(2.6.25.19), so the OS become slow. However XP and the linux with newest kernel (2.6.26.6-49.fc8) are not disturbed by the uncache of last 3MB.
The mtrr.c setting of [0,1984M] is: reg00: base=0x00000000 ( 0MB), size=1024MB: write-back, count=1 reg01: base=0x40000000 (1024MB), size= 512MB: write-back, count=1 reg02: base=0x60000000 (1536MB), size= 256MB: write-back, count=1 reg03: base=0x70000000 (1792MB), size= 128MB: write-back, count=1 reg04: base=0x78000000 (1920MB), size= 64MB: write-back, count=1 all of memory area is cached, and all OS run at a good speed.
2 So we need improve mtrr.c to make it can treat with such input[0,1983M]. The legacybios handle my [0,1983] like this Cache [0,2048M] Unchace [1983M, 1984M] size=1M Uncache [1984M,2048M] size =64M. I guess the legacybios compare the number of '1' and '0' in 1983 to decide how to use minimum number of MTRR.
Carldaniel, you said on irc that mtrr.c also have the ability like legacybios(cache 2G,then uncache some area of 2G). But I didn't find any logic like that in mtrr.c.
3 And Stefan, your cx700 code will meet the same problem. Since the cx700's northbridge.c report the memory size in the same way of [0,xxxx-1M(TOP 1M SM Memory)]. The last 1M will also confuse the mtrr.c.
-jasonzhao
-----Original Message----- From: Carl-Daniel Hailfinger [mailto:c-d.hailfinger.devel.2006@gmx.net] Sent: Saturday, January 24, 2009 7:53 PM To: Jason Zhao Cc: r.marek@assembler.cz Subject: Re: [coreboot] Two test result(linux+2G-RAM>10minutes and install ubuntu8.10) of vx800 patch
Hi Jason,
thanks for the dumps.
On 24.01.2009 08:08, JasonZhao@viatech.com.cn wrote:
Carl and ruik: Here is some more result of acpidump, /proc/mtrr and /proc/meminfo. Some explanation of my memory using in coreboot: [topmem-64,tompmem] as framebuffer, [topmem-64M-64k,topmem-64M] as coreboot-table (In tables.c:rom_table_start = (*mem_top) - 64*1024;).
It is a bit strange that the DSDTs are identical for both memory configurations. The following parts of the code are suspicious:
Method (_CRS, 0, NotSerialized) { Name (BUF0, ResourceTemplate () {[...] DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, Cacheable, ReadWrite, 0x00000000, // Granularity 0x80000000, // Range Minimum 0xBFFFFFFF, // Range Maximum 0x00000000, // Translation Offset 0x40000000, // Length ,, _Y00, AddressRangeMemory, TypeStatic) })
That address range is always the same, even if you only have 512M memory. Why? Will it cause problems if you have 3G memory?
And our MTRR setup is incomplete for the 2G case. The last 4 MB are uncached because we run out of MTRR registers. That will cause an extreme slowdown during boot.
I think the MTRR setup explains almost everything.
Regards, Carl-Daniel
On Thu, Feb 05, 2009 at 11:33:02AM +0800, JasonZhao@viatech.com.cn wrote:
My VX800 bug of "boot linux with 2G mem >10 minutes" has been proved to be the problem of MTRR setting. Thanks for Carldani, Rudolf and Bari. 1 The root cause is:
- The free memory I report to coreboot through ram_resource is [0,
1983M=2G-64M(Framebuffer)-1M(TOP 1M SM Memory)]. If I report [0, 1984M=2G-64M(Framebuffer)], then all is OK.
Is "SM Memory" the memory used by System Management Mode? Shouldn't this memory be cached anyway?
-Kevin
Kevin O'Connor wrote:
On Thu, Feb 05, 2009 at 11:33:02AM +0800, JasonZhao@viatech.com.cn wrote:
My VX800 bug of "boot linux with 2G mem >10 minutes" has been proved to be the problem of MTRR setting. Thanks for Carldani, Rudolf and Bari. 1 The root cause is:
- The free memory I report to coreboot through ram_resource is [0,
1983M=2G-64M(Framebuffer)-1M(TOP 1M SM Memory)]. If I report [0, 1984M=2G-64M(Framebuffer)], then all is OK.
Is "SM Memory" the memory used by System Management Mode? Shouldn't this memory be cached anyway?
This depends on CPU imho.
R.
-Kevin
-- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
-jasonzhao
-----Original Message----- From: Kevin O'Connor [mailto:kevin@koconnor.net] Sent: Thursday, February 05, 2009 9:37 PM To: Jason Zhao Cc: coreboot@coreboot.org Subject: Re: [coreboot] FW: Two test result(linux+2G-RAM>10minutes
andinstall
ubuntu8.10) of vx800 patch
On Thu, Feb 05, 2009 at 11:33:02AM +0800, JasonZhao@viatech.com.cn
wrote:
My VX800 bug of "boot linux with 2G mem >10 minutes" has been proved
to
be the problem of MTRR setting. Thanks for Carldani, Rudolf and
Bari.
1 The root cause is:
- The free memory I report to coreboot through ram_resource is [0,
1983M=2G-64M(Framebuffer)-1M(TOP 1M SM Memory)]. If I report [0, 1984M=2G-64M(Framebuffer)], then all is OK.
Is "SM Memory" the memory used by System Management Mode? Shouldn't
I think it is. (I do not use System Management Mode in my vx800/coreboot-v2)
this memory be cached anyway?
I ask my colleague who work for EFI. He said in his EFI project, SMM area should not be cached, and he had tried to cache that area, but cause system crashed. I don't know if he is right, since I can not find any reason why SMM area should not be cached.
-Kevin
On Thu, Feb 5, 2009 at 5:43 PM, JasonZhao@viatech.com.cn wrote:
I ask my colleague who work for EFI. He said in his EFI project, SMM area should not be cached, and he had tried to cache that area, but cause system crashed. I don't know if he is right, since I can not find any reason why SMM area should not be cached.
Stepan can tell us, since he has working smm on kontron, but I can not believe smm can not be cached.
I am glad you found the problem, now can know that fixing mtrr code will be very helpful.
thanks
ron
On 06.02.2009 02:52, ron minnich wrote:
On Thu, Feb 5, 2009 at 5:43 PM, JasonZhao@viatech.com.cn wrote:
I ask my colleague who work for EFI. He said in his EFI project, SMM area should not be cached, and he had tried to cache that area, but cause system crashed. I don't know if he is right, since I can not find any reason why SMM area should not be cached.
Stepan can tell us, since he has working smm on kontron, but I can not believe smm can not be cached.
Maybe if you have SMM memory hoisted to the same place as other memory. In that case, the CPU may get confused and accidentially write SMM memory contents to the normal RAM or video RAM at the same location.
Regards, Carl-Daniel
ron minnich wrote:
On Thu, Feb 5, 2009 at 5:43 PM, JasonZhao@viatech.com.cn wrote:
I ask my colleague who work for EFI. He said in his EFI project, SMM area should not be cached, and he had tried to cache that area, but cause system crashed. I don't know if he is right, since I can not find any reason why SMM area should not be cached.
Stepan can tell us, since he has working smm on kontron, but I can not believe smm can not be cached.
Occasionally a CPU might do speculative readahead on the SMRAM memory while not in SMM. The chipset will generate master aborts on the PCI bus, so the cached data is incorrect (0xff) and upon SMM entry the CPU goes to nirvana. The SMM area in ASEG is always uncached. The upper SMRAM areas can be cached, but they must only be cached while in SMM. A way to do this could be to set up an MTRR at the beginning of the SMI handler and restoring it upon exit. When not in in the SMI handler, TSEG or HSEG (high SMRAM and static address high SMRAM) must be uncached.
Stefan
On Fri, Feb 06, 2009 at 11:50:26PM +0100, Stefan Reinauer wrote:
ron minnich wrote:
Stepan can tell us, since he has working smm on kontron, but I can not believe smm can not be cached.
Occasionally a CPU might do speculative readahead on the SMRAM memory while not in SMM. The chipset will generate master aborts on the PCI bus, so the cached data is incorrect (0xff) and upon SMM entry the CPU goes to nirvana. The SMM area in ASEG is always uncached. The upper SMRAM areas can be cached, but they must only be cached while in SMM.
If there is memory only accessible from SMM mode, then I agree it would need to be uncached. I thought that Jason was just reserving memory for use by SMM - memory that could also be read/written in non-SMM mode. In that case, I did not think caching needed to be disabled. Indeed, the intel system programming guide recommends using some cached memory for SMM.
-Kevin
Random thought - I wonder if the OS could "break into" SMM mode by turning on caching for the SMM area and then manipulating the cache contents so that an SMI used icache/dcache contents set by the OS.
On 07.02.2009 01:39, Kevin O'Connor wrote:
On Fri, Feb 06, 2009 at 11:50:26PM +0100, Stefan Reinauer wrote:
ron minnich wrote:
Stepan can tell us, since he has working smm on kontron, but I can not believe smm can not be cached.
Occasionally a CPU might do speculative readahead on the SMRAM memory while not in SMM. The chipset will generate master aborts on the PCI bus, so the cached data is incorrect (0xff) and upon SMM entry the CPU goes to nirvana. The SMM area in ASEG is always uncached. The upper SMRAM areas can be cached, but they must only be cached while in SMM.
If there is memory only accessible from SMM mode, then I agree it would need to be uncached. I thought that Jason was just reserving memory for use by SMM - memory that could also be read/written in non-SMM mode. In that case, I did not think caching needed to be disabled. Indeed, the intel system programming guide recommends using some cached memory for SMM.
-Kevin
Random thought - I wonder if the OS could "break into" SMM mode by turning on caching for the SMM area and then manipulating the cache contents so that an SMI used icache/dcache contents set by the OS.
AFAICS it should work. CAR would be ideal for that purpose. No idea what happens if the SMM code plays with the cache regs as well, though.
Regards, Carl-Daniel
Kevin O'Connor wrote:
On Fri, Feb 06, 2009 at 11:50:26PM +0100, Stefan Reinauer wrote:
Occasionally a CPU might do speculative readahead on the SMRAM memory while not in SMM. The chipset will generate master aborts on the PCI bus, so the cached data is incorrect (0xff) and upon SMM entry the CPU goes to nirvana. The SMM area in ASEG is always uncached. The upper SMRAM areas can be cached, but they must only be cached while in SMM.
If there is memory only accessible from SMM mode, then I agree it would need to be uncached. I thought that Jason was just reserving memory for use by SMM - memory that could also be read/written in non-SMM mode. In that case, I did not think caching needed to be disabled. Indeed, the intel system programming guide recommends using some cached memory for SMM.
I'm not involved in the details - But you are correct: The issue I described can only occur when it is "locked" SMM memory, not just normal RAM that happens to be accessed from SMM as well.
Random thought - I wonder if the OS could "break into" SMM mode by turning on caching for the SMM area and then manipulating the cache contents so that an SMI used icache/dcache contents set by the OS
The attack scenario is realistic - the OS might gain control over the cache contents through setting up MTRRs in some cases. The questions come up whether this is exploitable and whether it's possible to make an SMM handler safe against this kind of attack. I don't think you can reasonably use the prefetchers to put exploit code in the cache by a CAR approach. On some CPUs it is even explicitly suggested that the prefetchers are disabled during CAR mode. This leaves the OS dependent on some PCI device to answer the bus requests instead leaving the chipset to generate a Master Abort.
The simplest way to avoid this is by putting the main SMI handler at 0xa0000, so it has control over the cache before it accesses the high SMM memory. Right now the SMI handler I wrote does not use the high SMM memory at all, so there's no immediate risk.
Stefan
On Sat, Feb 07, 2009 at 01:35:40PM +0100, Stefan Reinauer wrote:
Kevin O'Connor wrote:
Random thought - I wonder if the OS could "break into" SMM mode by turning on caching for the SMM area and then manipulating the cache contents so that an SMI used icache/dcache contents set by the OS
[...]
The simplest way to avoid this is by putting the main SMI handler at 0xa0000
We're probably getting off topic. But here is what I was thinking:
* OS uses mtrr to cache memory at 0xa0000
* OS loads a "jmp 0x10000" insn into L2 cache at 0xa0000
* OS invokes an SMI (acpi defines a way to do this)
If the SMI handler is set to run code at 0xa0000 (ie, it has an SMBASE of 0x98000), then it would see the 'jmp' insn and start running OS code at 0x10000.
so it has control over the cache before it accesses the high SMM memory. Right now the SMI handler I wrote does not use the high SMM memory at all, so there's no immediate risk.
I don't think there is much risk anyway - if someone has full access to the machine to change cache settings, then they don't need SMM mode.
-Kevin
On 07.02.2009 17:11, Kevin O'Connor wrote:
On Sat, Feb 07, 2009 at 01:35:40PM +0100, Stefan Reinauer wrote:
Kevin O'Connor wrote:
Random thought - I wonder if the OS could "break into" SMM mode by turning on caching for the SMM area and then manipulating the cache contents so that an SMI used icache/dcache contents set by the OS
[...]
The simplest way to avoid this is by putting the main SMI handler at 0xa0000
We're probably getting off topic. But here is what I was thinking:
OS uses mtrr to cache memory at 0xa0000
OS loads a "jmp 0x10000" insn into L2 cache at 0xa0000
OS invokes an SMI (acpi defines a way to do this)
If the SMI handler is set to run code at 0xa0000 (ie, it has an SMBASE of 0x98000), then it would see the 'jmp' insn and start running OS code at 0x10000.
This needs a unified cache, though.
so it has control over the cache before it accesses the high SMM memory. Right now the SMI handler I wrote does not use the high SMM memory at all, so there's no immediate risk.
I don't think there is much risk anyway - if someone has full access to the machine to change cache settings, then they don't need SMM mode.
Well, some chipsets allow you to reflash the ROM only in SMM if the flash interface is locked.
Regards, Carl-Daniel
On Sat, Feb 07, 2009 at 05:31:37PM +0100, Carl-Daniel Hailfinger wrote:
If the SMI handler is set to run code at 0xa0000 (ie, it has an SMBASE of 0x98000), then it would see the 'jmp' insn and start running OS code at 0x10000.
This needs a unified cache, though.
If you get the jmp insn into the L2 cache, the icache should pull from L2. It will be a little tricky to get the insn into L2 (and not just L1 dcache), but I can think of several ways to try.
-Kevin
-
Carl-Daniel Hailfinger -
JasonZhao＠viatech.com.cn -
Kevin O'Connor -
ron minnich -
Rudolf Marek -
Stefan Reinauer