[LinuxBIOS] Opteron caching of device memory

Marc Jones Marc.Jones at AMD.com
Thu Jun 14 22:39:14 CEST 2007

Roman Kononov wrote:
> On 06/13/2007 09:39 AM, Myles Watson wrote:
>> I’m using LinuxBIOS on my Tyan s2892.  I have a device that maps a lot 
>> of the memory space, but I’m struggling trying to get the Opteron to 
>> read and write to my device in larger blocks.  I have set the variable 
>> MTRRs in the device driver to writeback (witnessed by /proc/mtrr), but I 
>> still get 64-bit accesses instead of 64-byte (cache line).
> Some time ago I officially asked an AMD's rep about this. Someone from AMD 
> unofficially said that MemIO can be only Write-Combined. And I gave up with 
> caching.
> Below is what I asked. Maybe you can spot what I missed. Please let me know 
> if you succeed.
> Roman
> --------------------------------------------------------------------
> I have a difficulty related to cacheability of Memory-Mapped I/O memory in 
> Opteron. Can you please assist me?
> I have an Opteron-based system, where the Opteron and an FPGA are connected 
> using a HyperTransport link. As a target of HT traffic, the FPGA responses 
> to certain range of physical addresses. I am trying to configure the CPU and 
> the North Bridge to treat the FPGA as Writeback memory. I cannot do this.
> I've been able to make the FPGA Uncacheable, Write-Combining, Write-Protect 
> and Writethrough. When I try to make it Writeback it behaves as if it were 
> Writethrough.
> Is it possible to make the FPGA Writeback? If yes, what should I change?
> I have observability of the HT traffic from the FPGA point of view, so I 
> know what is written and what is requested by the CPU.
> Here is how I program the Opteron:
> The processor is running in the Long Mode with paging enabled.
> A pair of NB PCI Function 1, Memory-Mapped I/O Base and Limit registers is 
> programmed with the FPGA physical address range, proper link and node IDs, 
> posted.
> A pair of the variable MTRR Base and Mask (MSR 0x200-0x20f) registers is 
> programmed with the FPGA physical address range, valid bit and the desired 
> caching method. Another pair of MTRR registers describes the DRAM, all 
> others are disabled.
> The PAT register (MSR 0x277) is 0x0606060606060606ull.
> The Page Table Entry for the virtual address mapped to the FPGA has PAT, PCD 
> and PWT bits cleared.
> Bit CD in the CR0 register is cleared.
I am not an expert but these are my thoughts....

Now that I think about it, it makes sense that you can't set writeback 
to noncoherent (nonsystem) memory space. What if another device wants to 
write to that memory. There is no way for the cache to snoop that it was 
written. You would need a coherent HT link to to your FPGA to get the 
cache snoop messaging.


Marc Jones
Senior Software Engineer
(970) 226-9684 Office
mailto:Marc.Jones at amd.com

More information about the coreboot mailing list