[LinuxBIOS] Setup of Memory Controller

[Linux Bios]

I'm a student researching digital forensics and acquisition of memory using hardware. I had a few ideas I'm hoping is ok that I run by everyone here.

I have Tyan s2891 motherboard with one dual core Opteron and an fpga in the second cpu socket. I would like to research two things that I would like to do in a customization to my own linuxbios setup.

1. Disable the Opteron's integrated memory controller and use my own fpga memory controller in the second cpu socket on the motherboard. In this case I would like CPU read/writes to memory to follow a path out of Northbridge to my memory controller in the second CPU socket. (I'm not worried about latency problems at this point.)

The FPGA Im talking about (actually DRC computers setup) is seen the to the OS as another PCI device  and not another CPU node. So what Im theorizing that needs to be done is disable the integrated memory controller in northbridge in the register called dramconfigreglo[24]=disinrcvrs (In AMD developer documentation) and change the DRAM address map to point to the HT link where my memory controller is.

-or-

2. Continue to use the Opterons integrated memory controller and send a copy of all memory writes (not reads) to my own fpga memory controller in the second cpu socket. I dont see anywhere in the developer documentation in which there is a register setting to realize this so Im not sure if this is possible in AMDs direct connect architecture.

Thanks to all for taking your time out to read this as well as your ideas and direction!

-nate

---------------------------------
Be a better pen pal. Text or chat with friends inside Yahoo! Mail. See how.

[ron minnich]

modify linuxbios early setup to route all memory requests from socket
0 to socket 1. Bypass (i.e. skip)  memory setup code.

I don't think it's too hard.

intriguing project. Where are you from?

ron

[Linux Bios]

Thanks Ron, I'm from the Air Force Institute of Technology in Ohio. I admit I am very simple minded. I am unable to see where I would change the routing in early setup. So far I have found myself in

src >> northbridge >> amd >> amdk8 >> raminit.c

This appears to be where northbridge is setup and where I would route memory requests to my memory conrtoller. However it looks to be coded by node_id. My FPGA would not be recognized necessarily as a node because its not an opteron. right?

Am I over thinking this? Where would I route all memory requests from socket
0 to socket 1?

Thanks for your help.

nate

ron minnich  wrote: modify linuxbios early setup to route all memory requests from socket
0 to socket 1. Bypass (i.e. skip)  memory setup code.

I don't think it's too hard.

intriguing project. Where are you from?

ron

[ron minnich]

On Nov 19, 2007 11:16 AM, Linux Bios  wrote:
> This appears to be where northbridge is setup and where I would route memory
> requests to my memory conrtoller. However it looks to be coded by node_id.
> My FPGA would not be recognized necessarily as a node because its not an
> opteron. right?

your fpga is a node. The routing doesn't care about what's there. You
jest tell it to route
all mem requests from node 0 to node 1 and done.
> Am I over thinking this? Where would I route all memory requests from socket
> 0 to socket 1?

early ht setup. The resource map code.

ron

[yhlu]

On Nov 19, 2007 11:17 AM, ron minnich  wrote:
> On Nov 19, 2007 11:16 AM, Linux Bios  wrote:
> > This appears to be where northbridge is setup and where I would route memory
> > requests to my memory conrtoller. However it looks to be coded by node_id.
> > My FPGA would not be recognized necessarily as a node because its not an
> > opteron. right?

the ram on your FPGA onboard should already be accessed via pref mem
64 above 4G.

So what is missed there?

YH

[Arne Georg Gleditsch]

Linux Bios  writes:
> Thanks Ron, I'm from the Air Force Institute of Technology in Ohio. I admit I
> am very simple minded. I am unable to see where I would change the routing in
> early setup. So far I have found myself in
>
> src >> northbridge >> amd >> amdk8 >> raminit.c
>
> This appears to be where northbridge is setup and where I would route memory
> requests to my memory conrtoller. However it looks to be coded by node_id. My
> FPGA would not be recognized necessarily as a node because its not an opteron.
> right?

That depends on whether the DRC is seen as a coherent device or a
non-coherent one.  In the first case you would map it as DRAM (toward
a node id), in the second as MMIO (toward a node:link).  I'm not
familiar with the DRC -- what kind of PCI Id is it assigned to?  I'm
not sure you can bypass the Opteron memory controller (in terms of
cache controller etc) completely, but not using the Opteron-connected
DRAM at all shouldn't be too hard.  Not sure how it would react to
having all of its memory mapped as MMIO, though...

[Nate]

> That depends on whether the DRC is seen as a coherent device or a non-coherent one. In the first case you would map it as DRAM (toward a node id), in the second as MMIO (toward a node:link).

As it is in the Tyan motherboard now, the FPGA is seen as a non-coherent HT cave device.
> I'm not familiar with the DRC -- what kind of PCI Id is it assigned to?

The "supported" intent of the DRC system is to use the FPGA as an application accelerator. Lspci output looks like this:

0000:40:01.0 Co-processor: DRC Computer Corp.: Unknown device 4200 (rev a1)
 0000:40:01.1 Co-processor: DRC Computer Corp.: Unknown device 4201
 0000:40:01.2 Co-processor: DRC Computer Corp.: Unknown device 4202
 0000:40:01.3 Co-processor: DRC Computer Corp.: Unknown device 4203

iomem:

fc400000-fc41ffff : 0000:40:01.0
 fcc0000000-fcdfffffff : 0000:40:01.0
> I'm not sure you can bypass the Opteron memory controller (in terms of cache controller etc) completely, but not using the Opteron-connected DRAM at all shouldn't be too hard. Not sure how it would react to having all of its memory mapped as MMIO, though...

Hmm... Your right. I dont want to use the Opteron connected DRAM and Im not looking to replace all of northbridge. I had thought about mapping all addresses to memory-mapped IO of the FPGA as the AMD docs say that "An address that maps to both DRAM and memory-mapped I/O will be routed to MMIO".  An avenue definitely worth testing. However I would rather the system see my memory controller as the gate to the sytems DRAM.

Im currently looking at the resourcemap.c for the tyan s2891 and looking at simply changing the hex value for the DRAM Limit and Base address for offset 44 and 40 to point to node 1 rather than node 0 and cross my fingers. Im also trying to figure out why some of the registers are commented out. Any ideas? It would leave me to believe these registers are setup by the kernel later on...

I should receive my biossavior tomorrow and commence testing over the break.

nate

Arne Georg Gleditsch  wrote: Linux Bios 
 writes:
> Thanks Ron, I'm from the Air Force Institute of Technology in Ohio. I admit I
> am very simple minded. I am unable to see where I would change the routing in
> early setup. So far I have found myself in
>
> src >> northbridge >> amd >> amdk8 >> raminit.c
>
> This appears to be where northbridge is setup and where I would route memory
> requests to my memory conrtoller. However it looks to be coded by node_id. My
> FPGA would not be recognized necessarily as a node because its not an opteron.
> right?

That depends on whether the DRC is seen as a coherent device or a
non-coherent one.  In the first case you would map it as DRAM (toward
a node id), in the second as MMIO (toward a node:link).  I'm not
familiar with the DRC -- what kind of PCI Id is it assigned to?  I'm
not sure you can bypass the Opteron memory controller (in terms of
cache controller etc) completely, but not using the Opteron-connected
DRAM at all shouldn't be too hard.  Not sure how it would react to
having all of its memory mapped as MMIO, though...

[Myles Watson]

> Im currently looking at the resourcemap.c for the tyan s2891 and looking at
> simply changing the hex value for the DRAM Limit and Base address for offset
> 44 and 40 to point to node 1 rather than node 0 and cross my fingers.

I've been wondering what would happen if I did that too.  I was
worried that it would try to send coherent packets if it thought the
region mapped to DRAM on another node.
> I should receive my biossavior tomorrow and commence testing over the
> break.

I'll be interested to see how it turns out for you.

Myles

[ron minnich]

On Nov 20, 2007 9:16 AM, Nate  wrote:
> Im currently looking at the resourcemap.c for the tyan s2891 and looking at
> simply changing the hex value for the DRAM Limit and Base address for offset
> 44 and 40 to point to node 1 rather than node 0 and cross my fingers. Im
> also trying to figure out why some of the registers are commented out. Any
> ideas? It would leave me to believe these registers are setup by the kernel
> later on...

they are probably commented due to being from a different port, or a
quick fix that stayed in as it was found to be needed, or as an
informative comment ... there are several reasons.

But yeah, just point 44 and 40 at node 1 and see how it goes.

ron

[Arne Georg Gleditsch]

Nate wrote:
> As it is in the Tyan motherboard now, the FPGA is seen as a non-coherent
> HT cave device.
> > I'm not familiar with the DRC -- what kind of PCI Id is it assigned to?
>
> The "supported" intent of the DRC system is to use the FPGA as an
> application accelerator. Lspci output looks like this:
>
> 0000:40:01.0 Co-processor: DRC Computer Corp.: Unknown device 4200 (rev a1)
> 0000:40:01.1 Co-processor: DRC Computer Corp.: Unknown device 4201
> 0000:40:01.2 Co-processor: DRC Computer Corp.: Unknown device 4202
> 0000:40:01.3 Co-processor: DRC Computer Corp.: Unknown device 4203

Yeah, definitely non-coherent.
> Hmm... Your right. I dont want to use the Opteron connected DRAM and Im
> not looking to replace all of northbridge. I had thought about mapping
> all addresses to memory-mapped IO of the FPGA as the AMD docs say that
> "An address that maps to both DRAM and memory-mapped I/O will be routed
> to MMIO".  An avenue definitely worth testing. However I would rather
> the system see my memory controller as the gate to the sytems DRAM.
>
> Im currently looking at the resourcemap.c for the tyan s2891 and looking
> at simply changing the hex value for the DRAM Limit and Base address for
> offset 44 and 40 to point to node 1 rather than node 0 and cross my
> fingers.

By all means try it, but I'd be more than a little surprised if it
works.  As long as the DRC is a non-coherent device, there's no "node 1"
on your system -- that's a coherent term.  I believe your best bet is to
populate one of the MMIO mapping register pairs with the relevant range
and point that towards the link where the DRC resides.  Depending on how
the DRC is put together, you might also need to configure its BAR to
make it respond to the same address range.  Additionally, you'll
probably need to adjust or short-circuit the PCI enumeration pass later
on, to make sure no overlapping MMIO mappings are defined and that the
DRC BAR is not reconfigured.  (Actually, you might need to set up more
than one MMIO map toward the DRC link, since you probably want to leave
a gap for your video memory and friends.)

Ok, perhaps it doesn't sound quite hassle free, but I still think it
should be fairly straight forward.  Good luck anyway, make sure to tell
us how it works out. :)