[LinuxBIOS] Opteron caching of device memory

[Roman Kononov]

On 06/13/2007 09:39 AM, Myles Watson wrote:
> I’m using LinuxBIOS on my Tyan s2892.  I have a device that maps a lot 
> of the memory space, but I’m struggling trying to get the Opteron to 
> read and write to my device in larger blocks.  I have set the variable 
> MTRRs in the device driver to writeback (witnessed by /proc/mtrr), but I 
> still get 64-bit accesses instead of 64-byte (cache line).

Some time ago I officially asked an AMD's rep about this. Someone from AMD 
unofficially said that MemIO can be only Write-Combined. And I gave up with 
caching.

Below is what I asked. Maybe you can spot what I missed. Please let me know 
if you succeed.

Roman
--------------------------------------------------------------------
I have a difficulty related to cacheability of Memory-Mapped I/O memory in 
Opteron. Can you please assist me?

I have an Opteron-based system, where the Opteron and an FPGA are connected 
using a HyperTransport link. As a target of HT traffic, the FPGA responses 
to certain range of physical addresses. I am trying to configure the CPU and 
the North Bridge to treat the FPGA as Writeback memory. I cannot do this.

I've been able to make the FPGA Uncacheable, Write-Combining, Write-Protect 
and Writethrough. When I try to make it Writeback it behaves as if it were 
Writethrough.

Is it possible to make the FPGA Writeback? If yes, what should I change?

I have observability of the HT traffic from the FPGA point of view, so I 
know what is written and what is requested by the CPU.

Here is how I program the Opteron:

The processor is running in the Long Mode with paging enabled.

A pair of NB PCI Function 1, Memory-Mapped I/O Base and Limit registers is 
programmed with the FPGA physical address range, proper link and node IDs, 
posted.

A pair of the variable MTRR Base and Mask (MSR 0x200-0x20f) registers is 
programmed with the FPGA physical address range, valid bit and the desired 
caching method. Another pair of MTRR registers describes the DRAM, all 
others are disabled.

The PAT register (MSR 0x277) is 0x0606060606060606ull.

The Page Table Entry for the virtual address mapped to the FPGA has PAT, PCD 
and PWT bits cleared.

Bit CD in the CR0 register is cleared.