re: Prefmem of bus 3

[YhLu]

Eric,

I print out the allocating spew and found the align calculation seems not
correct.

PCI: 01:02.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align:
20 gran: 20
PCI: 03:03.0 10 *  [0x00000000 - 0x0fffffff] prefmem
PCI: 03:03.0 18 *  [0x10000000 - 0x13ffffff] prefmem
<null> compute_allocate_prefmem: base: 14000000 size: 14000000 align: 28
gran: 20 done

When computing limit in pci_set_device of pci_device.c, only using align.
So will produce limit=0xffffffff, instead of 0xf3ffffff.

So maybe need to using gran together with align and base to calculate the
limit.

Regards

YH

Allocating resources...
Root Device compute_allocate_io: base: 00000400 size: 00000000 align: 0
gran: 0
PCI: 00:18.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12
gran: 12
PCI: 01:01.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12
gran: 12
PCI: 02:0a.0 10 *  [0x00000000 - 0x000000ff] io
PCI: 02:0a.1 10 *  [0x00000400 - 0x000004ff] io
<null> compute_allocate_io: base: 00000500 size: 00001000 align: 12 gran: 12
done
PCI: 01:01.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align:
20 gran: 20
<null> compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20
gran: 20 done
PCI: 01:01.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20
gran: 20
PCI: 02:09.0 10 *  [0x00000000 - 0x0000ffff] mem
PCI: 02:09.0 18 *  [0x00010000 - 0x0001ffff] mem
PCI: 02:09.1 10 *  [0x00020000 - 0x0002ffff] mem
PCI: 02:09.1 18 *  [0x00030000 - 0x0003ffff] mem
PCI: 02:0a.0 14 *  [0x00040000 - 0x0004ffff] mem
PCI: 02:0a.0 1c *  [0x00050000 - 0x0005ffff] mem
PCI: 02:0a.1 14 *  [0x00060000 - 0x0006ffff] mem
PCI: 02:0a.1 1c *  [0x00070000 - 0x0007ffff] mem
<null> compute_allocate_mem: base: 00080000 size: 00100000 align: 20 gran:
20 done
PCI: 01:02.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12
gran: 12
<null> compute_allocate_io: base: 00000000 size: 00000000 align: 12 gran: 12
done
PCI: 01:02.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align:
20 gran: 20
PCI: 03:03.0 10 *  [0x00000000 - 0x0fffffff] prefmem
PCI: 03:03.0 18 *  [0x10000000 - 0x13ffffff] prefmem
<null> compute_allocate_prefmem: base: 14000000 size: 14000000 align: 28
gran: 20 done
PCI: 01:02.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20
gran: 20
<null> compute_allocate_mem: base: 00000000 size: 00000000 align: 20 gran:
20 done
PCI: 01:03.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12
gran: 12
PCI: 04:01.0 missing read_resources
PCI: 04:06.0 14 *  [0x00000000 - 0x000000ff] io
PCI: 04:05.0 18 *  [0x00000400 - 0x0000047f] io
PCI: 04:05.0 10 *  [0x00000480 - 0x000004bf] io
PCI: 04:05.0 14 *  [0x000004c0 - 0x000004cf] io
<null> compute_allocate_io: base: 000004d0 size: 00001000 align: 12 gran: 12
done
PCI: 01:03.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align:
20 gran: 20
PCI: 04:01.0 missing read_resources
<null> compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20
gran: 20 done
PCI: 01:03.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20
gran: 20
PCI: 04:01.0 missing read_resources
PCI: 04:06.0 10 *  [0x00000000 - 0x00ffffff] mem
PCI: 04:05.0 20 *  [0x01000000 - 0x0101ffff] mem
PCI: 04:00.0 10 *  [0x01020000 - 0x01020fff] mem
PCI: 04:00.1 10 *  [0x01021000 - 0x01021fff] mem
PCI: 04:05.0 1c *  [0x01022000 - 0x01022fff] mem
PCI: 04:06.0 18 *  [0x01023000 - 0x01023fff] mem
PCI: 04:00.2 10 *  [0x01024000 - 0x010240ff] mem
PCI: 04:00.2 14 *  [0x01025000 - 0x0102501f] mem
<null> compute_allocate_mem: base: 01025020 size: 01100000 align: 24 gran:
20 done
PCI: 01:04.5 missing read_resources
PCI: 01:04.6 missing read_resources
PCI: 01:01.0 1c *  [0x00000000 - 0x00000fff] io
PCI: 01:03.0 1c *  [0x00001000 - 0x00001fff] io
PCI: 01:02.0 1c *  [0x00002000 - 0x00001fff] io
PCI: 01:04.2 10 *  [0x00002000 - 0x0000201f] io
PCI: 01:04.1 20 *  [0x00002020 - 0x0000202f] io
<null> compute_allocate_io: base: 00002030 size: 00003000 align: 12 gran: 12
done
PCI: 00:18.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20
gran: 20
PCI: 01:04.5 missing read_resources
PCI: 01:04.6 missing read_resources
PCI: 01:02.0 24 *  [0x00000000 - 0x13ffffff] prefmem
PCI: 01:03.0 20 *  [0x14000000 - 0x150fffff] mem
PCI: 01:01.0 20 *  [0x15100000 - 0x151fffff] mem
PCI: 01:01.0 24 *  [0x15200000 - 0x151fffff] prefmem
PCI: 01:02.0 20 *  [0x15200000 - 0x151fffff] mem
PCI: 01:03.0 24 *  [0x15200000 - 0x151fffff] prefmem
PCI: 01:01.1 10 *  [0x15200000 - 0x15200fff] mem
PCI: 01:02.1 10 *  [0x15201000 - 0x15201fff] mem
<null> compute_allocate_mem: base: 15202000 size: 15300000 align: 28 gran:
20 done
PCI: 00:18.0 c0 *  [0x00001000 - 0x00003fff] io
<null> compute_allocate_io: base: 00004000 size: 00003c00 align: 12 gran: 0
done
Root Device compute_allocate_mem: base: 00000000 size: 00000000 align: 0
gran: 0
PCI: 00:18.0 b8 *  [0x00000000 - 0x152fffff] mem
<null> compute_allocate_mem: base: 15300000 size: 15300000 align: 28 gran: 0
done
Root Device compute_allocate_io: base: 00001000 size: 00003c00 align: 12
gran: 0
PCI: 00:18.0 c0 *  [0x00001000 - 0x00003fff] io
<null> compute_allocate_io: base: 00004000 size: 00003000 align: 12 gran: 0
done
Root Device compute_allocate_mem: base: e0000000 size: 15300000 align: 28
gran: 0
PCI: 00:18.0 b8 *  [0xe0000000 - 0xf52fffff] mem
<null> compute_allocate_mem: base: f5300000 size: 15300000 align: 28 gran: 0
done

-----邮件原件-----
发件人: YhLu 
发送时间: 2004年3月1日 13:10
收件人: '[e-mail]'; ron minnich
抄送: LinuxBIOS
主题: 答复: Prefmem of bus 3

Some time you said that arima board connect all PCI-X slots of amd 8131 irq
to amd 8111 ioapic intead of 8131.

So even disable ioapic of 8131, device on PCI-X still works properly.

After enable ioapic of 8131, devices that are not using prefmem works well.

YH.

-----邮件原件-----
发件人: [e-mail] [mailto:[e-mail]] 
发送时间: 2004年3月1日 10:48
收件人: ron minnich
抄送: YhLu; [e-mail]; LinuxBIOS
主题: Re: Prefmem of bus 3

ron minnich  writes:
> On Fri, 27 Feb 2004, YhLu wrote:
> > Have you ever tested myrinet card on K8 MB under LinuxBIOS?
>
> we have a 1408-node K8 machine with myrinet working under linuxbios, and a
> 256-node K8 machine working with linuxbios. Arima HDAMA mainboards.
>
> I wonder what's going on here, I have not had time to look in detail.

So the problem appears to be that the prefmem region on the upper bus
is to large.

I wonder if why this work on LANL clusters and not on others is the
result of code skew.

Lightning nodes with myrinet only seem to have one prefmem region.

Yet another reminder that I really need to finish syncing up the code bases.

Eric