Eric,
I print out the allocating spew and found the align calculation seems not correct.
PCI: 01:02.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20 gran: 20 PCI: 03:03.0 10 * [0x00000000 - 0x0fffffff] prefmem PCI: 03:03.0 18 * [0x10000000 - 0x13ffffff] prefmem <null> compute_allocate_prefmem: base: 14000000 size: 14000000 align: 28 gran: 20 done
When computing limit in pci_set_device of pci_device.c, only using align. So will produce limit=0xffffffff, instead of 0xf3ffffff.
So maybe need to using gran together with align and base to calculate the limit.
Regards
YH
Allocating resources... Root Device compute_allocate_io: base: 00000400 size: 00000000 align: 0 gran: 0 PCI: 00:18.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12 gran: 12 PCI: 01:01.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12 gran: 12 PCI: 02:0a.0 10 * [0x00000000 - 0x000000ff] io PCI: 02:0a.1 10 * [0x00000400 - 0x000004ff] io <null> compute_allocate_io: base: 00000500 size: 00001000 align: 12 gran: 12 done PCI: 01:01.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20 gran: 20 <null> compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20 gran: 20 done PCI: 01:01.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20 gran: 20 PCI: 02:09.0 10 * [0x00000000 - 0x0000ffff] mem PCI: 02:09.0 18 * [0x00010000 - 0x0001ffff] mem PCI: 02:09.1 10 * [0x00020000 - 0x0002ffff] mem PCI: 02:09.1 18 * [0x00030000 - 0x0003ffff] mem PCI: 02:0a.0 14 * [0x00040000 - 0x0004ffff] mem PCI: 02:0a.0 1c * [0x00050000 - 0x0005ffff] mem PCI: 02:0a.1 14 * [0x00060000 - 0x0006ffff] mem PCI: 02:0a.1 1c * [0x00070000 - 0x0007ffff] mem <null> compute_allocate_mem: base: 00080000 size: 00100000 align: 20 gran: 20 done PCI: 01:02.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12 gran: 12 <null> compute_allocate_io: base: 00000000 size: 00000000 align: 12 gran: 12 done PCI: 01:02.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20 gran: 20 PCI: 03:03.0 10 * [0x00000000 - 0x0fffffff] prefmem PCI: 03:03.0 18 * [0x10000000 - 0x13ffffff] prefmem <null> compute_allocate_prefmem: base: 14000000 size: 14000000 align: 28 gran: 20 done PCI: 01:02.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20 gran: 20 <null> compute_allocate_mem: base: 00000000 size: 00000000 align: 20 gran: 20 done PCI: 01:03.0 compute_allocate_io: base: 00000000 size: 00000000 align: 12 gran: 12 PCI: 04:01.0 missing read_resources PCI: 04:06.0 14 * [0x00000000 - 0x000000ff] io PCI: 04:05.0 18 * [0x00000400 - 0x0000047f] io PCI: 04:05.0 10 * [0x00000480 - 0x000004bf] io PCI: 04:05.0 14 * [0x000004c0 - 0x000004cf] io <null> compute_allocate_io: base: 000004d0 size: 00001000 align: 12 gran: 12 done PCI: 01:03.0 compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20 gran: 20 PCI: 04:01.0 missing read_resources <null> compute_allocate_prefmem: base: 00000000 size: 00000000 align: 20 gran: 20 done PCI: 01:03.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20 gran: 20 PCI: 04:01.0 missing read_resources PCI: 04:06.0 10 * [0x00000000 - 0x00ffffff] mem PCI: 04:05.0 20 * [0x01000000 - 0x0101ffff] mem PCI: 04:00.0 10 * [0x01020000 - 0x01020fff] mem PCI: 04:00.1 10 * [0x01021000 - 0x01021fff] mem PCI: 04:05.0 1c * [0x01022000 - 0x01022fff] mem PCI: 04:06.0 18 * [0x01023000 - 0x01023fff] mem PCI: 04:00.2 10 * [0x01024000 - 0x010240ff] mem PCI: 04:00.2 14 * [0x01025000 - 0x0102501f] mem <null> compute_allocate_mem: base: 01025020 size: 01100000 align: 24 gran: 20 done PCI: 01:04.5 missing read_resources PCI: 01:04.6 missing read_resources PCI: 01:01.0 1c * [0x00000000 - 0x00000fff] io PCI: 01:03.0 1c * [0x00001000 - 0x00001fff] io PCI: 01:02.0 1c * [0x00002000 - 0x00001fff] io PCI: 01:04.2 10 * [0x00002000 - 0x0000201f] io PCI: 01:04.1 20 * [0x00002020 - 0x0000202f] io <null> compute_allocate_io: base: 00002030 size: 00003000 align: 12 gran: 12 done PCI: 00:18.0 compute_allocate_mem: base: 00000000 size: 00000000 align: 20 gran: 20 PCI: 01:04.5 missing read_resources PCI: 01:04.6 missing read_resources PCI: 01:02.0 24 * [0x00000000 - 0x13ffffff] prefmem PCI: 01:03.0 20 * [0x14000000 - 0x150fffff] mem PCI: 01:01.0 20 * [0x15100000 - 0x151fffff] mem PCI: 01:01.0 24 * [0x15200000 - 0x151fffff] prefmem PCI: 01:02.0 20 * [0x15200000 - 0x151fffff] mem PCI: 01:03.0 24 * [0x15200000 - 0x151fffff] prefmem PCI: 01:01.1 10 * [0x15200000 - 0x15200fff] mem PCI: 01:02.1 10 * [0x15201000 - 0x15201fff] mem <null> compute_allocate_mem: base: 15202000 size: 15300000 align: 28 gran: 20 done PCI: 00:18.0 c0 * [0x00001000 - 0x00003fff] io <null> compute_allocate_io: base: 00004000 size: 00003c00 align: 12 gran: 0 done Root Device compute_allocate_mem: base: 00000000 size: 00000000 align: 0 gran: 0 PCI: 00:18.0 b8 * [0x00000000 - 0x152fffff] mem <null> compute_allocate_mem: base: 15300000 size: 15300000 align: 28 gran: 0 done Root Device compute_allocate_io: base: 00001000 size: 00003c00 align: 12 gran: 0 PCI: 00:18.0 c0 * [0x00001000 - 0x00003fff] io <null> compute_allocate_io: base: 00004000 size: 00003000 align: 12 gran: 0 done Root Device compute_allocate_mem: base: e0000000 size: 15300000 align: 28 gran: 0 PCI: 00:18.0 b8 * [0xe0000000 - 0xf52fffff] mem <null> compute_allocate_mem: base: f5300000 size: 15300000 align: 28 gran: 0 done
-----邮件原件----- 发件人: YhLu 发送时间: 2004年3月1日 13:10 收件人: 'ebiederman@lnxi.com'; ron minnich 抄送: LinuxBIOS 主题: 答复: Prefmem of bus 3
Some time you said that arima board connect all PCI-X slots of amd 8131 irq to amd 8111 ioapic intead of 8131.
So even disable ioapic of 8131, device on PCI-X still works properly.
After enable ioapic of 8131, devices that are not using prefmem works well.
YH.
-----邮件原件----- 发件人: ebiederman@lnxi.com [mailto:ebiederman@lnxi.com] 发送时间: 2004年3月1日 10:48 收件人: ron minnich 抄送: YhLu; ebiederman@lnxi.com; LinuxBIOS 主题: Re: Prefmem of bus 3
ron minnich rminnich@lanl.gov writes:
On Fri, 27 Feb 2004, YhLu wrote:
Have you ever tested myrinet card on K8 MB under LinuxBIOS?
we have a 1408-node K8 machine with myrinet working under linuxbios, and a
256-node K8 machine working with linuxbios. Arima HDAMA mainboards.
I wonder what's going on here, I have not had time to look in detail.
So the problem appears to be that the prefmem region on the upper bus is to large.
I wonder if why this work on LANL clusters and not on others is the result of code skew.
Lightning nodes with myrinet only seem to have one prefmem region.
Yet another reminder that I really need to finish syncing up the code bases.
Eric