pci resource allocation overlaps config_mmconf_base_address

[Scott Duplichan]

Hello,

I am testing with mahogany_fam10h and mmconf_base_address at e0000000.
I find that if I add a pci video card, coreboot assigns e0000000 to one
of the uma graphics bars. This range is already in use for pcie mmio.
Is there a mechanism intended to prevent this? The best I can come up
is to reduce the ffffffffff resource limit in northbridge.c to
mmconf_base_address-1, but that prevents the pci allocation from using
anything above this reserved range. It seems like it allocates from a
single contiguous range.

Thanks,
Scott

pci resource allocation overlaps config_mmconf_base_address

[Peter Stuge]

Scott Duplichan wrote:
> I am testing with mahogany_fam10h and mmconf_base_address at e0000000.
> I find that if I add a pci video card, coreboot assigns e0000000 to one
> of the uma graphics bars. This range is already in use for pcie mmio.
> Is there a mechanism intended to prevent this?

I think the best would be to create a fixed resource for UMA. I'm
afraid I can't say exactly how to do it though.

//Peter

[Myles Watson]

On Sat, Oct 16, 2010 at 3:37 PM, Scott Duplichan  wrote:
> Hello,
>
> I am testing with mahogany_fam10h and mmconf_base_address at e0000000.
> I find that if I add a pci video card, coreboot assigns e0000000 to one
> of the uma graphics bars. This range is already in use for pcie mmio.
> Is there a mechanism intended to prevent this? The best I can come up
> is to reduce the ffffffffff resource limit in northbridge.c to
> mmconf_base_address-1, but that prevents the pci allocation from using
> anything above this reserved range. It seems like it allocates from a
> single contiguous range.

Yes, it does.

You have a couple of options:
- Create a fixed resource for MMCONF.  You could place it at the very
top of the range so that it doesn't take up too much of your address
space
- Allow the resource allocator to allocate the MMCONF range
You'd have to be a little careful when it gets set to change the
address your PCI functions are using, but it should be doable.

Thanks,
Myles

pci resource allocation overlaps config_mmconf_base_address

[Rudolf Marek]

Hi all,

Scott, thanks on working on that! The resource overlap is bit pitty - it belongs 
to style - "Oh I haven't thought of that". You can han have a look how I did 
MMCONFIG for VIA k8t890, using dynamic allocation (mmconfig_set_resources in 
k8t890_traf_ctrl.c) or APIC which is fixed and this is what Myles suggest.

Thanks,
Rudolf

pci resource allocation overlaps config_mmconf_base_address

[Stefan Reinauer]

On 10/17/10 12:40 AM, Rudolf Marek wrote:
> Hi all,
>
> Scott, thanks on working on that! The resource overlap is bit pitty -
> it belongs to style - "Oh I haven't thought of that".

I think this was heavily discussed a while back, and lead to some
naughty work arounds in the i945 code. because having such big a "hole"
lead to no resources being placed after the MMCONF area. Ther should be
a better way to do it. Have to admit I have not particularly checked out
the k8t890 code yet. My experiments with allocating a resource for it in
the i945 northbridge code didn't keep the resource allocator from
putting other stuff on top of it, mainly because of the hierarchical way
the resource allocator works. Any insights are highly welcome.

Stefan

pci resource allocation overlaps config_mmconf_base_address

[Rudolf Marek]

> I think this was heavily discussed a while back, and lead to some
> naughty work arounds in the i945 code. because having such big a "hole"
> lead to no resources being placed after the MMCONF area.

I think with a dynamic resource there are some place above the address.  Which 
gets normally used.

Besides the mentioned file, the acpi_tables does  following:

For the acpi it takes it from resource system:

unsigned long acpi_fill_mcfg(unsigned long current)
{
         device_t dev;
         struct resource *res;

dev = dev_find_device(PCI_VENDOR_ID_VIA, PCI_DEVICE_ID_VIA_K8M890CE_5, 0);
         if (!dev)
                 return current;

res = find_resource(dev, K8T890_MMCONFIG_MBAR);
         if (res) {
                 current += acpi_create_mcfg_mmconfig((acpi_mcfg_mmconfig_t *)
                                 current, res->base, 0x0, 0x0, 0xff);
         }
         return current;
}

Thanks,
Rudolf

[Scott Duplichan]

]> Hi all,
]>
]> Scott, thanks on working on that! The resource overlap is bit pitty -
]> it belongs to style - "Oh I haven't thought of that".
]I think this was heavily discussed a while back, and lead to some
]naughty work arounds in the i945 code. because having such big a "hole"
]lead to no resources being placed after the MMCONF area. Ther should be
]a better way to do it. Have to admit I have not particularly checked out
]the k8t890 code yet. My experiments with allocating a resource for it in
]the i945 northbridge code didn't keep the resource allocator from
]putting other stuff on top of it, mainly because of the hierarchical way
]the resource allocator works. Any insights are highly welcome.
]
]Stefan

Thanks for the suggestions everyone. I like the k8t890 method, however
there is a concern. I think the dynamically obtained mmconf base address
would not be known or programmed until fairly late. But using mmconf
very early to access config space was the key change that solved the
unreliable AP startup on my project. The unreliable AP startup problem
was caused by simultaneous core access to cf8/cfc. I think there is a
real value in knowing the mmconf base very early, and that requires a
fixed resource assignment.

I found a reasonable scheme for handling fixed resources is already in
place. This is why the local apic space at fec00000 is never allocated
to a PCI device. Yes, the code allocates from a single range only. But
it knows to avoid fixed resources. As long as some care is used when
choosing a fixed resource location, the existing scheme seems fine.

Why does the current code for handling fixed resources allow the mmconf
space to get allocated to a PCI device? Function avoid_fixed_resources
calls function constrain_resources, which recursively searches the
device tree for fixed io and memory resources. The ioapic fixed memory
address is found and avoided during the recursive search because this
southbridge device is below the level where the search starts. On the
other hand, the mmconf fixed resource is added from the northbridge code,
and falls under 'APIC_CLUSTER: 0'. This device is not part of the search
for two reasons. One is that it is not at or below 'pci_domain 0' in the
device tree. Another reason is that its type is APIC_CLUSTER and not
PCI_DOMAIN.

Right now, each 'pci domain n' is treated as if it has a completely 
private resource address space. The real problem is that the resource
assignments of 'lapic cluster n' are completely ignored when assigning
resources to a pci domain. One solution might be to add an argument to
function avoid_fixed_resources that allows passing in apic cluster
in addition to the pci domain that is already passed in. Then
avoid_fixed_resources could consider the apic cluster resources when
adjusting its allocation limits to avoid fixed resources. I could
prototype something like this if it sounds reasonable.

Thanks,
Scott

pci resource allocation overlaps config_mmconf_base_address

[Rudolf Marek]

Hi Myles,

Please can you help here? Also I tried to have a look how the non-posted support 
could be added but it looks like it must be done somewhat inside the resource 
allocator. Please can you help with that too? Basically we would need to put 
some MMIO resources which have non-posted attribute set in resource (this must 
be done) to be allocated somewhat together so only one MMIO resource would be 
needed (and we need to modify the northbridge.c of K8 to set the non-posted bit)

Unfortunately I don't know how to do that. Please can you help or provide some 
hints?

Thanks,
Rudolf

pci resource allocation overlaps config_mmconf_base_address

[Arne Georg Gleditsch]

Rudolf Marek  writes:
> Please can you help here? Also I tried to have a look how the non-posted support 
> could be added but it looks like it must be done somewhat inside the resource 
> allocator. Please can you help with that too? Basically we would need to put 
> some MMIO resources which have non-posted attribute set in resource (this must 
> be done) to be allocated somewhat together so only one MMIO resource would be 
> needed (and we need to modify the northbridge.c of K8 to set the non-posted bit)

On a similar note: I'm looking at a southbridge that has its own notion
of what constitutes TOP_OF_DRAM.  Apparently, the IOMMU aperture needs
to be below this boundary, or DMA transactions towards the aperture are
terminated with master abort.  From the CPU's side, though, the IOMMU
aperture should be above TOP_OF_MEM, in order to avoid wasting the DRAM
behind it, no?  So, as well as being able to allocate posted and
non-posted memory resources in separate hunks: in this instance it would
be good to be able to allocate the GART aperture in a third hunk that
was placed below the other two, so that the SB TOM register could be
programmed to include it when needed.

Is this feasible, or should I approach this from a different angle?

[Myles Watson]

On Mon, Oct 18, 2010 at 6:48 AM, Arne Georg Gleditsch
 wrote:
> Rudolf Marek  writes:
> > Please can you help here? Also I tried to have a look how the non-posted support
> > could be added but it looks like it must be done somewhat inside the resource
> > allocator. Please can you help with that too? Basically we would need to put
> > some MMIO resources which have non-posted attribute set in resource (this must
> > be done) to be allocated somewhat together so only one MMIO resource would be
> > needed (and we need to modify the northbridge.c of K8 to set the non-posted bit)
>
> On a similar note: I'm looking at a southbridge that has its own notion
> of what constitutes TOP_OF_DRAM.  Apparently, the IOMMU aperture needs
> to be below this boundary, or DMA transactions towards the aperture are
> terminated with master abort.  From the CPU's side, though, the IOMMU
> aperture should be above TOP_OF_MEM, in order to avoid wasting the DRAM
> behind it, no?  So, as well as being able to allocate posted and
> non-posted memory resources in separate hunks: in this instance it would
> be good to be able to allocate the GART aperture in a third hunk that
> was placed below the other two, so that the SB TOM register could be
> programmed to include it when needed.
>
> Is this feasible, or should I approach this from a different angle?

It sounds like overkill to include this as a special case for the
resource allocator.  At first glance, the code would seem to be very
simple for that register.  Is this a one-time, one-resource problem?
Are there other hunks that need to be allocated together with it?

Thanks,
Myles

[Arne Georg Gleditsch]

Myles Watson  writes:
> On Mon, Oct 18, 2010 at 6:48 AM, Arne Georg Gleditsch
>  wrote:
> > On a similar note: I'm looking at a southbridge that has its own notion
> > of what constitutes TOP_OF_DRAM.  Apparently, the IOMMU aperture needs
> > to be below this boundary, or DMA transactions towards the aperture are
> > terminated with master abort.  From the CPU's side, though, the IOMMU
> > aperture should be above TOP_OF_MEM, in order to avoid wasting the DRAM
> > behind it, no?  So, as well as being able to allocate posted and
> > non-posted memory resources in separate hunks: in this instance it would
> > be good to be able to allocate the GART aperture in a third hunk that
> > was placed below the other two, so that the SB TOM register could be
> > programmed to include it when needed.
> >
> > Is this feasible, or should I approach this from a different angle?
>
> It sounds like overkill to include this as a special case for the
> resource allocator.  At first glance, the code would seem to be very
> simple for that register.  Is this a one-time, one-resource problem?
> Are there other hunks that need to be allocated together with it?

I'm not aware of any others, so arranging for a full "resource group"
might be overkill.  But as for very simple, I'm afraid I don't really
see it.  To be clear, I'm talking about the GART aperture that's
allocated in
src/northbridge/amd/amdfam10/misc_control.c:mcf3_read_resources.  What
would be the easy way to make sure this resource is allocated at the
lowest possible address in the IO hole (for this board)?

[Myles Watson]

On Mon, Oct 18, 2010 at 1:11 PM, Arne Georg Gleditsch
 wrote:
> Myles Watson  writes:
> > On Mon, Oct 18, 2010 at 6:48 AM, Arne Georg Gleditsch
> >  wrote:
> > > On a similar note: I'm looking at a southbridge that has its own notion
> > > of what constitutes TOP_OF_DRAM.  Apparently, the IOMMU aperture needs
> > > to be below this boundary, or DMA transactions towards the aperture are
> > > terminated with master abort.  From the CPU's side, though, the IOMMU
> > > aperture should be above TOP_OF_MEM, in order to avoid wasting the DRAM
> > > behind it, no?  So, as well as being able to allocate posted and
> > > non-posted memory resources in separate hunks: in this instance it would
> > > be good to be able to allocate the GART aperture in a third hunk that
> > > was placed below the other two, so that the SB TOM register could be
> > > programmed to include it when needed.
> > >
> > > Is this feasible, or should I approach this from a different angle?
> >
> > It sounds like overkill to include this as a special case for the
> > resource allocator.  At first glance, the code would seem to be very
> > simple for that register.  Is this a one-time, one-resource problem?
> > Are there other hunks that need to be allocated together with it?
>
> I'm not aware of any others, so arranging for a full "resource group"
> might be overkill.  But as for very simple, I'm afraid I don't really
> see it.  To be clear, I'm talking about the GART aperture that's
> allocated in
> src/northbridge/amd/amdfam10/misc_control.c:mcf3_read_resources.

Thanks for the clarification.  I misunderstood and spoke too quickly.
> What
> would be the easy way to make sure this resource is allocated at the
> lowest possible address in the IO hole (for this board)?

I guess I'm not sure.  It seems like you'll create a hole any way you do it...

By create a hole, I mean that since the resource allocator starts with
the largest areas and allocates until it reaches the smallest, trying
to put a resource that isn't the largest at the bottom will cause
larger ones to be shifted (leaving a gap) so that they can still be
aligned correctly.

Maybe the easiest thing to do is to make the GART aperture look like
the largest resource to the resource allocator, so that it gets
allocated first.  As long as you correctly store the resource, it
shouldn't break anything.

I've attached a patch to do it.  The size I picked was too large, but
I thought the patch would probably communicate the idea better than my
email.

Thanks,
Myles

[Myles Watson]

On Mon, Oct 18, 2010 at 12:12 AM, Rudolf Marek  wrote:
> Hi Myles,
>
> Please can you help here? Also I tried to have a look how the non-posted
> support could be added but it looks like it must be done somewhat inside the
> resource allocator. Please can you help with that too? Basically we would
> need to put some MMIO resources which have non-posted attribute set in
> resource (this must be done) to be allocated somewhat together so only one
> MMIO resource would be needed (and we need to modify the northbridge.c of K8
> to set the non-posted bit)
>
> Unfortunately I don't know how to do that. Please can you help or provide
> some hints?

As a first hack at it, you could try to (ab)use the
IORESOURCE_PREFETCH bit.  I know it means exactly the opposite of
non-posted, but it would force the resources to be allocated
together...

You could add a new bit IORESOURCE_NON_POSTED in src/include/device/resource.h

Then you would need to add a little more logic to the split in
src/device/device.c, so instead of just splitting resources based on
the PREFETCH bit, it added the NON_POSTED bit too.

Thanks,
Myles

[Myles Watson]

On Sun, Oct 17, 2010 at 11:17 PM, Scott Duplichan  wrote:
> ]> Hi all,
> ]>
> ]> Scott, thanks on working on that! The resource overlap is bit pitty -
> ]> it belongs to style - "Oh I haven't thought of that".
> ]I think this was heavily discussed a while back, and lead to some
> ]naughty work arounds in the i945 code. because having such big a "hole"
> ]lead to no resources being placed after the MMCONF area. Ther should be
> ]a better way to do it. Have to admit I have not particularly checked out
> ]the k8t890 code yet. My experiments with allocating a resource for it in
> ]the i945 northbridge code didn't keep the resource allocator from
> ]putting other stuff on top of it, mainly because of the hierarchical way
> ]the resource allocator works. Any insights are highly welcome.
> ]
> ]Stefan
>
> Thanks for the suggestions everyone. I like the k8t890 method, however
> there is a concern. I think the dynamically obtained mmconf base address
> would not be known or programmed until fairly late.

This is true.  You can use a hard-coded address until that point,
though.  There are many resources that must do this.
> But using mmconf
> very early to access config space was the key change that solved the
> unreliable AP startup on my project. The unreliable AP startup problem
> was caused by simultaneous core access to cf8/cfc. I think there is a
> real value in knowing the mmconf base very early, and that requires a
> fixed resource assignment.

It only has to be fixed until the allocation is done.  It would
require a little care when setting the new address, but I think it's
the right way to do it.
> I found a reasonable scheme for handling fixed resources is already in
> place. This is why the local apic space at fec00000 is never allocated
> to a PCI device. Yes, the code allocates from a single range only. But
> it knows to avoid fixed resources. As long as some care is used when
> choosing a fixed resource location, the existing scheme seems fine.

It causes a problem if the alignment doesn't match up well and leaves
a large hole.  How big is the MMCONF space? 256 MB?  What's the
highest you can put it without running into the other fixed resources?
> Why does the current code for handling fixed resources allow the mmconf
> space to get allocated to a PCI device?

Because there is no fixed resource defined for it?
> Function avoid_fixed_resources
> calls function constrain_resources, which recursively searches the
> device tree for fixed io and memory resources. The ioapic fixed memory
> address is found and avoided during the recursive search because this
> southbridge device is below the level where the search starts. On the
> other hand, the mmconf fixed resource is added from the northbridge code,
> and falls under 'APIC_CLUSTER: 0'. This device is not part of the search
> for two reasons. One is that it is not at or below 'pci_domain 0' in the
> device tree. Another reason is that its type is APIC_CLUSTER and not
> PCI_DOMAIN.

We can change the code to scan the entire tree if that makes more sense.
> Right now, each 'pci domain n' is treated as if it has a completely
> private resource address space.

I thought that was the definition of a PCI domain.
> The real problem is that the resource
> assignments of 'lapic cluster n' are completely ignored when assigning
> resources to a pci domain. One solution might be to add an argument to
> function avoid_fixed_resources that allows passing in apic cluster
> in addition to the pci domain that is already passed in. Then
> avoid_fixed_resources could consider the apic cluster resources when
> adjusting its allocation limits to avoid fixed resources. I could
> prototype something like this if it sounds reasonable.

Does the resource belong in the lapic cluster?

Thanks,
Myles