Resource Allocation discussion

[Myles Watson]

The resource patches I submitted work fairly well, but still need a little
help.

I haven't touched subtractively decoded resources, and that's where the
breakage is right now.

Questions:
- Who should have a resource that moves all other IO resources above 0x400
or 0x1000?
- Is one needed if subtractive resources are implemented correctly?
- Who is responsible for setting the top of the address range?
   - From an earlier thread with Carl-Daniel it sounded like there are fixed
locations for APICs that need to be avoided.  Where should these be avoided?

Thanks,
Myles

[Marc Jones]

Hi Myles,
I will try to help..... :)

On Mon, Dec 1, 2008 at 2:37 PM, Myles Watson  wrote:
> The resource patches I submitted work fairly well, but still need a little
> help.
>
> I haven't touched subtractively decoded resources, and that's where the
> breakage is right now.
>
> Questions:
> Who should have a resource that moves all other IO resources above 0x400
>
> or 0x1000?

What do you mean? By definition the subtractive port takes all
unclaimed cycles. There should be a LPC or ISA bridge in the
southbridge that forwards cycles that the legacy southbridge devices
don't claim.
> Is one needed if subtractive resources are implemented correctly?

I don't think so. You might need to track the the subtractive bit in
the bridges so you don't try to add those resources to the bridge BAR.
> Who is responsible for setting the top of the address range?

This should be set per mainboard but the chipset will probably have
them most influence on the value.
#define DEVICE_MEM_HIGH 0xFEBFFFFFUL is probably ok for most systems
but an easy way to override it would be good.
> From an earlier thread with Carl-Daniel it sounded like there are fixed
>
> locations for APICs that need to be avoided.  Where should these be avoided?

Yes, APICs are generally at FEC00000(IO) and FEE00000(local).

Marc

[Myles Watson]

> -----Original Message-----
> From: Marc Jones [mailto:[e-mail]]
> Sent: Monday, December 01, 2008 10:42 PM
> To: Myles Watson
> Cc: Coreboot
> Subject: Re: [coreboot] Resource Allocation discussion
>
> Hi Myles,
> I will try to help..... :)
>
> On Mon, Dec 1, 2008 at 2:37 PM, Myles Watson  wrote:
> > The resource patches I submitted work fairly well, but still need a
>
> little
> > help.
> >
> > I haven't touched subtractively decoded resources, and that's where the
> > breakage is right now.
> >
> > Questions:
> > Who should have a resource that moves all other IO resources above
>
> 0x400
> > or 0x1000?
>
> What do you mean? By definition the subtractive port takes all
> unclaimed cycles. There should be a LPC or ISA bridge in the
> southbridge that forwards cycles that the legacy southbridge devices
> don't claim.

Sorry.  I wasn't clear enough again.  What I meant was something more like:

Since the resource allocation algorithm can't avoid resources that it
doesn't know about, who decides if the subtractive area to avoid is
0x0-0x400, 0x0-0x1000, or some other range?  If I declare some device to
hold these addresses so that no other resource ends up there, should it be
subtractive?  I'm inclined to put it in the domain or in the SuperIO.
> > Is one needed if subtractive resources are implemented correctly?
>
> I don't think so. You might need to track the the subtractive bit in
> the bridges so you don't try to add those resources to the bridge BAR.

Right now subtractive resources are just skipped altogether.
> > Who is responsible for setting the top of the address range?
>
> This should be set per mainboard but the chipset will probably have
> them most influence on the value.

Are you aware of mainboards with the same chipset that have different
DEVICE_MEM_HIGH?
> #define DEVICE_MEM_HIGH 0xFEBFFFFFUL is probably ok for most systems
> but an easy way to override it would be good.

I'd like this to be in the domain as well.  If we set the domain's resource
range to 0x0-0xFEBFFFFF, then nothing will get allocated outside of that.
> > From an earlier thread with Carl-Daniel it sounded like there are
>
> fixed
> > locations for APICs that need to be avoided.  Where should these be
>
> avoided?
>
> Yes, APICs are generally at FEC00000(IO) and FEE00000(local).

So these should be created in the domain?

Thanks,
Myles

[Rudolf Marek]

> > Yes, APICs are generally at FEC00000(IO) and FEE00000(local).
>
> So these should be created in the domain?

Yes and no? Check this:

http://www.coreboot.org/Memory_map

Also MMCONFIG region should be somewhere. For VIAs, the static resources are in 
.h files in V2
http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-v2/src/sout...
http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-v2/src/sout...

Problem is that some resources (APICs can be only on arbitrary range, typically 
0xfec0 0000 - 0xfecf ffff

So DEVICE_MEM_HIGH should be lower then those .h files?? And IO_LOW would be 
nice too...

Rudolf

[Rudolf Marek]

Sorry I forgot to ask if the DEVICE_MEM_HIGH is in fact the HIGH limit for MMIO 
or not. Of course CPU must be instructed to decode the addresses as MMIO too. 
But in some range we need static allocation.

For MMIO decode:

DEVICE_MEM_LOW =< DEVICE_MEM_LOW_STATIC < DEVICE_MEM_HIGH_STATIC =< DEVICE_MEM_HIGH

And similar for IO decode.

Like here K8 resources:

MMIO map: #7 0x0040000000 - 0x00ff70ffff Access: R/W     Dstnode:0 DstLink 0
   IO map: #0  0x000000 - 0x000fff Access: /    Dstnode:0 DstLink 0
   IO map: #1  0x001000 - 0x0fffff Access: R/W   VGA Dstnode:0 DstLink 0
   IO map: #2  0x000000 - 0x000fff Access: /    Dstnode:0 DstLink 0
   IO map: #3  0x000000 - 0x000fff Access: /    Dstnode:0 DstLink 0

Rudolf

[Myles Watson]

> -----Original Message-----
> From: Rudolf Marek [mailto:[e-mail]]
> Sent: Tuesday, December 02, 2008 7:11 AM
> To: Myles Watson
> Cc: 'Marc Jones'; 'Coreboot'
> Subject: Re: [coreboot] Resource Allocation discussion
> > > Yes, APICs are generally at FEC00000(IO) and FEE00000(local).
> >
> > So these should be created in the domain?
>
> Yes and no? Check this:
>
> http://www.coreboot.org/Memory_map

Thanks for the pointers!
> Also MMCONFIG region should be somewhere. For VIAs, the static resources
> are in
> .h files in V2
> http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-
> v2/src/southbridge/via/k8t890/k8t890.h
> http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-
> v2/src/southbridge/via/vt8237r/vt8237r.h
>
> Problem is that some resources (APICs can be only on arbitrary range,
> typically
> 0xfec0 0000 - 0xfecf ffff
>
> So DEVICE_MEM_HIGH should be lower then those .h files?? And IO_LOW would
> be
> nice too...

I'm hoping that these defines can go away.  I'd like each board or each
domain to define their areas explicitly.  So, for example, the k8t890 has:

------- From k8t890_traf_ctrl.c ------

pci_dev_read_resources(dev);

res = new_resource(dev, 0x40);
    /* NB APIC fixed to this address. */
    res->base = K8T890_APIC_BASE;
    res->size = 256;
    res->limit = res->base + res->size - 1;
    res->align = 8;
    res->gran = 8;
    res->flags = IORESOURCE_MEM | IORESOURCE_FIXED |
    	     IORESOURCE_STORED | IORESOURCE_ASSIGNED;

/* Add an MMCONFIG resource. */
    res = new_resource(dev, K8T890_MMCONFIG_MBAR);
    res->size = 256 * 1024 * 1024;
    res->align = log2(res->size);
    res->gran = log2(res->size);
    res->limit = 0xffffffff;	/* 4G */
    res->flags = IORESOURCE_MEM;

------- end ----------

So it seems like we could add bridge resources that contain all the space
that can be used for allocations to the domain below this.
> -----Original Message-----
>
> Sorry I forgot to ask if the DEVICE_MEM_HIGH is in fact the HIGH limit for
> MMIO
> or not.

I don't know for sure.
> Of course CPU must be instructed to decode the addresses as MMIO
> too.

Yes.
> But in some range we need static allocation.
>
> For MMIO decode:
>
> DEVICE_MEM_LOW =< DEVICE_MEM_LOW_STATIC < DEVICE_MEM_HIGH_STATIC =<
> DEVICE_MEM_HIGH
>
> And similar for IO decode.
>
> Like here K8 resources:
>
> MMIO map: #7 0x0040000000 - 0x00ff70ffff Access: R/W     Dstnode:0 DstLink
> 0
>    IO map: #0  0x000000 - 0x000fff Access: /    Dstnode:0 DstLink 0
>    IO map: #1  0x001000 - 0x0fffff Access: R/W   VGA Dstnode:0 DstLink 0
>    IO map: #2  0x000000 - 0x000fff Access: /    Dstnode:0 DstLink 0
>    IO map: #3  0x000000 - 0x000fff Access: /    Dstnode:0 DstLink 0

If we explicitly declare the resources we use, I'm hoping that the
device-specific code can do the right thing.

Thanks,
Myles

[Rudolf Marek]

> ------- From k8t890_traf_ctrl.c ------
>
> pci_dev_read_resources(dev);
>
> res = new_resource(dev, 0x40);
>    /* NB APIC fixed to this address. */
>    res->base = K8T890_APIC_BASE;
>    res->size = 256;
>    res->limit = res->base + res->size - 1;
>    res->align = 8;
>    res->gran = 8;
>    res->flags = IORESOURCE_MEM | IORESOURCE_FIXED |
>    	     IORESOURCE_STORED | IORESOURCE_ASSIGNED;
>
> /* Add an MMCONFIG resource. */
>    res = new_resource(dev, K8T890_MMCONFIG_MBAR);
>    res->size = 256 * 1024 * 1024;
>    res->align = log2(res->size);
>    res->gran = log2(res->size);
>    res->limit = 0xffffffff;	/* 4G */
>    res->flags = IORESOURCE_MEM;
>
> ------- end ----------

Yes now I remember that I wrote the code ;) so please disregard mine previous 
mail about _LOW and _STATIC_HIGH.

Problem is that some resources needs to go to ACPI. And it is needed to

1) store them in some ACPI table during table construction, read resource should 
work.

2) worse case - store in AML code. This can be done by some SSDT hacks which is 
ugly or write AML code which re-reads the resources from hardware.

Ad 1)
http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-v2/src/main...
http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-v2/src/main...

Ad 2)

http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-v2/src/nort...
http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-v2/src/main...

In fact #2 as in ARUMA and proper DSDT code which handles _CSR (CRS?) resources 
is neccessary for Windows to boot properly. Check the Aruma DSDT:

http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-v2/src/main...

(In particular Method (_CRS, 0, NotSerialized) for PCI0).

Rudolf

[Myles Watson]

> -----Original Message-----
> From: Rudolf Marek [mailto:[e-mail]]
> Sent: Tuesday, December 02, 2008 7:56 AM
> To: Myles Watson
> Cc: 'Marc Jones'; 'Coreboot'
> Subject: Re: [coreboot] Resource Allocation discussion
> > ------- From k8t890_traf_ctrl.c ------
> >
> > pci_dev_read_resources(dev);
> >
> > res = new_resource(dev, 0x40);
> >    /* NB APIC fixed to this address. */
> >    res->base = K8T890_APIC_BASE;
> >    res->size = 256;
> >    res->limit = res->base + res->size - 1;
> >    res->align = 8;
> >    res->gran = 8;
> >    res->flags = IORESOURCE_MEM | IORESOURCE_FIXED |
> >    	     IORESOURCE_STORED | IORESOURCE_ASSIGNED;
> >
> > /* Add an MMCONFIG resource. */
> >    res = new_resource(dev, K8T890_MMCONFIG_MBAR);
> >    res->size = 256 * 1024 * 1024;
> >    res->align = log2(res->size);
> >    res->gran = log2(res->size);
> >    res->limit = 0xffffffff;	/* 4G */
> >    res->flags = IORESOURCE_MEM;
> >
> > ------- end ----------
>
> Yes now I remember that I wrote the code ;) so please disregard mine
> previous
> mail about _LOW and _STATIC_HIGH.

:)

I'm wondering why the size of the APIC region here is 256.  That's smaller
than any reference from the resource map link you sent.
> Problem is that some resources needs to go to ACPI. And it is needed to
> store them in some ACPI table during table construction, read resource
>
> should
> work.
> worse case - store in AML code. This can be done by some SSDT hacks
>
> which is
> ugly or write AML code which re-reads the resources from hardware.

Which device should logically own the ACPI resources?  We can just allocate
them there.  I vote for the domain or the mainboard (root device) to own
them.
> Ad 1)
> http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-
> v2/src/mainboard/asus/a8v-e_se/fadt.c
> http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-
> v2/src/mainboard/asus/a8v-e_se/acpi_tables.c
>
> Ad 2)
>
> http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-
> v2/src/northbridge/amd/amdk8/ssdt.dsl
> http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-
> v2/src/mainboard/agami/aruma/acpi_tables.c
>
> In fact #2 as in ARUMA and proper DSDT code which handles _CSR (CRS?)
> resources
> is neccessary for Windows to boot properly. Check the Aruma DSDT:
>
> http://tracker.coreboot.org/trac/coreboot/browser/trunk/coreboot-
> v2/src/mainboard/agami/aruma/dx/dsdt_lb.dsl
>
> (In particular Method (_CRS, 0, NotSerialized) for PCI0).

I'm in too deep here.  I'm hoping that we can figure out resource allocation
a little better before we move on to ACPI tables.

Thanks,
Myles

[Rudolf Marek]

Myles Watson wrote:
> > -----Original Message-----
> > From: Rudolf Marek [mailto:[e-mail]]
>
> I'm wondering why the size of the APIC region here is 256.  That's smaller
> than any reference from the resource map link you sent.

The K8M890 APIC decodes only 256 bytes. Some APICs can decode more.
> Which device should logically own the ACPI resources?  We can just allocate
> them there.  I vote for the domain or the mainboard (root device) to own
> them.

Here is something how the other resources from host bus and PCI bus are handled:

http://www.acpi.info/acpi_faq.htm

How do I report a CPU-to-PCI bridge (root PCI Bus) in the ACPI NameSpace?

And also some more about the ACPI representation in:
http://www.google.cz/search?hl=cs&q=%22PCI+Firmware+Specification%22+fil...
http://www.pcisig.com/specifications/conventional/pci_firmware/PCI_FW_Slot_P...
> I'm in too deep here.  I'm hoping that we can figure out resource allocation
> a little better before we move on to ACPI tables.

Yes ;)

Rudolf

[Myles Watson]

Here's the read_resources code from qemu after my latest patch.  Is this
what people are thinking?  Notice that I explicitly allocate VGA space and
APIC.  At the end of the code I'm pasting in a snippet of the log that shows
the resources after they are assigned values.

Thanks,
Myles

static void i440bx_read_resources(struct device *dev)
{
    struct resource *res;
    u32 tolmk;        /* Top of low mem, Kbytes. */
    int idx;

/* I/O resource. */
    res = new_resource(dev, 0);
    res->base = 0x1000UL;
    res->limit = 0xFFFFUL;
    res->flags = IORESOURCE_IO | IORESOURCE_BRIDGE;

/* read large mem memory descriptor
       for <16 MB read the more detailed small mem descriptor
       all values in kbytes */
    tolmk = ((inb_cmos(0x35)<<8) |inb_cmos(0x34)) * 64;
    if (tolmk <= 16 * 1024) {
        tolmk = (inb_cmos(0x31)<<8) |inb_cmos(0x30);
    }

printk(BIOS_WARNING, "Using CMOS settings of %d kB RAM.\n", tolmk);
    idx = 10;

/* 0 .. 640 kB */
    ram_resource(dev, idx++, 0, 640);

/* Hole for VGA (0xA0000-0xAFFFF) graphics and text mode
     * graphics (0xB8000-0xBFFFF) */
    res = new_resource(dev, IOINDEX_SUBTRACTIVE(0, 0));
    res->base = 0xA0000UL;
    res->size = 0x20000UL;
    res->limit = 0xBFFFUL;
    res->flags = IORESOURCE_MEM | IORESOURCE_SUBTRACTIVE |
             IORESOURCE_ASSIGNED | IORESOURCE_FIXED;

/* 768 kB .. Systop (in KB) */
    ram_resource(dev, idx++, 768, tolmk - 768);
}

static void i440bx_domain_read_resources(struct device *dev)
{
    struct resource *res;

/* Initialize the domain's I/O space constraints. */
    res = new_resource(dev, 0);
    res->base = 0x1000UL;
    res->limit = 0xffffUL;
    res->flags = IORESOURCE_IO | IORESOURCE_BRIDGE;

/* Initialize the system-wide memory resources constraints. */
    res = new_resource(dev, 1);
    res->limit = 0xffffffffULL;
    res->flags = IORESOURCE_MEM | IORESOURCE_BRIDGE;

/* Reserve space for the APIC. */
    res = new_resource(dev, 2);
    res->base = 0xfec00000UL;
    res->size = 0x100000UL;
    res->limit = 0xffffffffULL;
    res->flags = IORESOURCE_MEM | IORESOURCE_FIXED | IORESOURCE_STORED;

/* Reserve space for the APIC. */
    res = new_resource(dev, 2);
    res->base = 0xfee00000UL;
    res->size = 0x10000UL;
    res->limit = 0xffffffffULL;
    res->flags = IORESOURCE_MEM | IORESOURCE_FIXED | IORESOURCE_STORED;
}

Show resources in subtree (root)...After assigning values.
 Root Device links 1 child on link 0 cpus
 Root Device resource base 1000 size 410 align 8 gran 0 limit ffff flags
80100 index 0
 Root Device resource base fc000000 size 2001000 align 25 gran 0 limit
ffffffff flags 40080200 index 1
 Root Device resource base 0 size 0 align 0 gran 0 limit ffffffffffffffff
flags 81200 index 2
  CPU: 00 links 0 child on link 0 NULL
  PCI_DOMAIN: 0000 links 1 child on link 0 domain_0_pci_0_0
  PCI_DOMAIN: 0000 resource base 1000 size 410 align 8 gran 0 limit ffff
flags 40080100 index 0
  PCI_DOMAIN: 0000 resource base fc000000 size 2001000 align 25 gran 0 limit
ffffffff flags 40080200 index 1
  PCI_DOMAIN: 0000 resource base fee00000 size 10000 align 0 gran 0 limit
ffffffff flags a0000200 index 2
   PCI: 00:00.0 links 0 child on link 0 NULL
   PCI: 00:00.0 resource base 1000 size 0 align 0 gran 0 limit ffff flags
80100 index 0
   PCI: 00:00.0 resource base 0 size a0000 align 0 gran 0 limit 0 flags
e0004200 index a
   PCI: 00:00.0 resource base a0000 size 20000 align 0 gran 0 limit bfff
flags c0040200 index 10000000
   PCI: 00:00.0 resource base c0000 size 6f40000 align 0 gran 0 limit 0
flags e0004200 index b
   PCI: 00:01.0 links 0 child on link 0 NULL
   PCI: 00:01.1 links 0 child on link 0 NULL
   PCI: 00:01.1 resource base 1400 size 10 align 4 gran 4 limit ffff flags
40000100 index 20
   PCI: 00:01.3 links 0 child on link 0 NULL
   PCI: 00:02.0 links 0 child on link 0 NULL
   PCI: 00:02.0 resource base fc000000 size 2000000 align 25 gran 25 limit
ffffffff flags 40001200 index 10
   PCI: 00:02.0 resource base fe000000 size 1000 align 12 gran 12 limit
ffffffff flags 40000200 index 14
   PCI: 00:03.0 links 0 child on link 0 NULL
   PCI: 00:03.0 resource base 1000 size 100 align 8 gran 8 limit ffff flags
40000100 index 10
Phase 4: Setting resources...

[Marc Jones]

On Tue, Dec 2, 2008 at 11:06 AM, Myles Watson  wrote:
> Here's the read_resources code from qemu after my latest patch.  Is this
> what people are thinking?  Notice that I explicitly allocate VGA space and
> APIC.  At the end of the code I'm pasting in a snippet of the log that shows
> the resources after they are assigned values.

I think I like this. It makes sense to me. The legacy VGA space nadd
the legacy IO space should be reserved in the southbridge as you have
done. The IOAPIC address should be held by the southbridge since that
is where it is a located. The LAPIC should probably be in the
CPU/northbridge device.

It would be handy to show the fixed resources in the output.
> Show resources in subtree (root)...After assigning values.
>  Root Device links 1 child on link 0 cpus
>  Root Device resource base 1000 size 410 align 8 gran 0 limit ffff flags
> 80100 index 0

Coreboot allocates IO from the bottom (0x1000) up? I think it should
allocate top down as it does with memory. I think that the rule is
that the BIOS allocates top down and the OS allocates bottom up.

Marc

[Myles Watson]

On Tue, Dec 2, 2008 at 11:59 AM, Marc Jones  wrote:
> On Tue, Dec 2, 2008 at 11:06 AM, Myles Watson  wrote:
> > Here's the read_resources code from qemu after my latest patch.  Is this
> > what people are thinking?  Notice that I explicitly allocate VGA space
>
> and
> > APIC.  At the end of the code I'm pasting in a snippet of the log that
>
> shows
> > the resources after they are assigned values.
>
> I think I like this. It makes sense to me. The legacy VGA space nadd
> the legacy IO space should be reserved in the southbridge as you have
> done.

I actually think I just bumped up the base in the northbridge this time
around.
> The IOAPIC address should be held by the southbridge since that
> is where it is a located.

OK.

The LAPIC should probably be in the
> CPU/northbridge device.

So I got this one right?  No I put it in the domain.
> It would be handy to show the fixed resources in the output.

Because of a typo the two APIC reservations clobbered each other, so only
one is there.  The rest of the fixed allocations should be there.
> > Show resources in subtree (root)...After assigning values.
> >  Root Device links 1 child on link 0 cpus
> >  Root Device resource base 1000 size 410 align 8 gran 0 limit ffff flags
> > 80100 index 0
>
> Coreboot allocates IO from the bottom (0x1000) up?

Yes.
> I think it should
> allocate top down as it does with memory. I think that the rule is
> that the BIOS allocates top down and the OS allocates bottom up.

I don't think that would match v2 or the factory BIOS for serengeti or
qemu.  Is there a packing reason?

Thanks,
Myles

[Myles Watson]

On Tue, Dec 2, 2008 at 12:09 PM, Myles Watson  wrote:
> On Tue, Dec 2, 2008 at 11:59 AM, Marc Jones  wrote:
> > On Tue, Dec 2, 2008 at 11:06 AM, Myles Watson  wrote:
> > > Here's the read_resources code from qemu after my latest patch.  Is this
> > > what people are thinking?  Notice that I explicitly allocate VGA space
> >
> > and
> > > APIC.  At the end of the code I'm pasting in a snippet of the log that
> >
> > shows
> > > the resources after they are assigned values.
> >
> > I think I like this. It makes sense to me. The legacy VGA space nadd
> > the legacy IO space should be reserved in the southbridge as you have
> > done.
>
> I actually think I just bumped up the base in the northbridge this time
> around.

All right.  If I do this then I can't ignore fixed resources in the
allocation algorithm anymore.  It still works right now because 0-0xfff is
the largest resource, so it gets allocated first.  I think it would break if
you tried to allocate an IO resource larger than that.

Do we want to take fixed resources into account in the allocator?

Thanks,
Myles

[ron minnich]

On Tue, Dec 2, 2008 at 11:33 AM, Myles Watson  wrote:
> Do we want to take fixed resources into account in the allocator?

I would think so.

ron

[Myles Watson]

> -----Original Message-----
> From: ron minnich [mailto:[e-mail]]
> Sent: Tuesday, December 02, 2008 2:51 PM
> To: Myles Watson
> Cc: Marc Jones; Rudolf Marek; Coreboot
> Subject: Re: [coreboot] Resource Allocation discussion
>
> On Tue, Dec 2, 2008 at 11:33 AM, Myles Watson  wrote:
> > Do we want to take fixed resources into account in the allocator?
>
> I would think so.

I would too, but v2 didn't and did pretty well.  I'm interested to see an
idea for the resource allocation algorithm that allocates from largest to
smallest resource, except when it's skipping fixed resources.

I think it's doable, but nothing like what we have.

Thanks,
Myles

[Marc Jones]

On Tue, Dec 2, 2008 at 3:23 PM, Myles Watson  wrote:
> > -----Original Message-----
> > From: ron minnich [mailto:[e-mail]]
> > Sent: Tuesday, December 02, 2008 2:51 PM
> > To: Myles Watson
> > Cc: Marc Jones; Rudolf Marek; Coreboot
> > Subject: Re: [coreboot] Resource Allocation discussion
> >
> > On Tue, Dec 2, 2008 at 11:33 AM, Myles Watson  wrote:
> > > Do we want to take fixed resources into account in the allocator?
> >
> > I would think so.
>
> I would too, but v2 didn't and did pretty well.  I'm interested to see an
> idea for the resource allocation algorithm that allocates from largest to
> smallest resource, except when it's skipping fixed resources.
>
> I think it's doable, but nothing like what we have.

That surprises me. I thought it does, which is how/why you can get rid
of DEVICE_MEM_HIGH. Otherwise you would clobber the ROM and the APIC.

And that brings up where to reserve the ROM space. In the southbridge
subtractive device code I guess?

Marc

[Myles Watson]

> -----Original Message-----
> From: Marc Jones [mailto:[e-mail]]
> Sent: Tuesday, December 02, 2008 7:52 PM
> To: Myles Watson
> Cc: ron minnich; Rudolf Marek; Coreboot
> Subject: Re: [coreboot] Resource Allocation discussion
>
> On Tue, Dec 2, 2008 at 3:23 PM, Myles Watson  wrote:
> > > -----Original Message-----
> > > From: ron minnich [mailto:[e-mail]]
> > > Sent: Tuesday, December 02, 2008 2:51 PM
> > > To: Myles Watson
> > > Cc: Marc Jones; Rudolf Marek; Coreboot
> > > Subject: Re: [coreboot] Resource Allocation discussion
> > >
> > > On Tue, Dec 2, 2008 at 11:33 AM, Myles Watson
>
> wrote:
> > > > Do we want to take fixed resources into account in the allocator?
> > >
> > > I would think so.
> >
> > I would too, but v2 didn't and did pretty well.  I'm interested to see
>
> an
> > idea for the resource allocation algorithm that allocates from largest
>
> to
> > smallest resource, except when it's skipping fixed resources.
> >
> > I think it's doable, but nothing like what we have.
>
> That surprises me. I thought it does, which is how/why you can get rid
> of DEVICE_MEM_HIGH. Otherwise you would clobber the ROM and the APIC.

You're right.  I think it's broken.  That's why right now I'm setting the
base to 0xfc000000 manually.  Thanks for pointing it out.

I'd assumed that resource_max took fixed resources into account, but it
doesn't.  All resource_max does is align the resource based on its limit.

So maybe we need to traverse the tree looking for fixed spaces, and
constrain the domain's resource limit and base.  That wouldn't be too hard.
What I was saying would be hard is interleaving fixed resources with movable
resources.  In other words, if we put the VGA ROM between the two APIC
allocations to save space.  I think that's too ugly.

I'll add a traversal to the tree that doesn't ignore fixed resources.
> And that brings up where to reserve the ROM space. In the southbridge
> subtractive device code I guess?

Sure.  How much do we reserve?  Is it southbridge dependent?

Thanks,
Myles

[Myles Watson]

On Wed, Dec 3, 2008 at 7:46 AM, Myles Watson  wrote:
> > -----Original Message-----
> > From: Marc Jones [mailto:[e-mail]]
> > Sent: Tuesday, December 02, 2008 7:52 PM
> > To: Myles Watson
> > Cc: ron minnich; Rudolf Marek; Coreboot
> > Subject: Re: [coreboot] Resource Allocation discussion
> >
> > On Tue, Dec 2, 2008 at 3:23 PM, Myles Watson  wrote:
> > > > -----Original Message-----
> > > > From: ron minnich [mailto:[e-mail]]
> > > > Sent: Tuesday, December 02, 2008 2:51 PM
> > > > To: Myles Watson
> > > > Cc: Marc Jones; Rudolf Marek; Coreboot
> > > > Subject: Re: [coreboot] Resource Allocation discussion
> > > >
> > > > On Tue, Dec 2, 2008 at 11:33 AM, Myles Watson
> >
> > wrote:
> > > > > Do we want to take fixed resources into account in the allocator?
> > > >
> > > > I would think so.
> > >
> > > I would too, but v2 didn't and did pretty well.  I'm interested to see
> >
> > an
> > > idea for the resource allocation algorithm that allocates from largest
> >
> > to
> > > smallest resource, except when it's skipping fixed resources.
> > >
> > > I think it's doable, but nothing like what we have.
> >
> > That surprises me. I thought it does, which is how/why you can get rid
> > of DEVICE_MEM_HIGH. Otherwise you would clobber the ROM and the APIC.
>
> You're right.  I think it's broken.  That's why right now I'm setting the
> base to 0xfc000000 manually.  Thanks for pointing it out.
>
> I'd assumed that resource_max took fixed resources into account, but it
> doesn't.  All resource_max does is align the resource based on its limit.
>
> So maybe we need to traverse the tree looking for fixed spaces, and
> constrain the domain's resource limit and base.  That wouldn't be too hard.
> What I was saying would be hard is interleaving fixed resources with
> movable
> resources.  In other words, if we put the VGA ROM between the two APIC
> allocations to save space.  I think that's too ugly.
>
> I'll add a traversal to the tree that doesn't ignore fixed resources.

A question before the code:
1. Can anyone enumerate all the fixed resources that should be included in
QEMU?
Currently I have:
- DRAM (first 640K and 0xc0000-TOP-768K) Is this necessary?
- I/O first 0x400 in southbridge
- VGA 0xa0000-0xc0000
- APIC at 0xfec00000 and 0xfee00000
- ROM at 0xfff80000

Side note: Only the lowest and highest fixed addresses matter in my code.
The others get ignored.  I think it's too much pain to try to interleave
them to use the space between 0xfee00000 and 0xfff80000, for example.

All right.  I've added code that traverses each domain looking for fixed
resources.  Here's the pseudo code:

For each domain:
    call avoid_fixed_resources(domain_dev)

avoid_fixed_resources(dev)
     initialize limits to maximum size
     initialize the limits to dev's resources limits
     call constrain_resources(dev, limits)
     update dev's resources

constrain_resources(dev, limits)
     for each child
         call constrain_resources(child, limits)
         for each fixed resource
            if limits contain resource
               update limits to exclude resource (largest space possible)

And here's the code.  It's not a diff because it's all new.
struct constraints {
    struct resource pref, io, mem;
};

static void constrain_resources(struct device *dev, struct constraints*
limits)
{
    struct device *child;
    struct resource *res;
    struct resource *lim;
    int i;

#ifdef CONFIG_PCI_64BIT_PREF_MEM
    #define MEM_MASK (IORESOURCE_PREFETCH | IORESOURCE_MEM)
#else
    #define MEM_MASK (IORESOURCE_MEM)
#endif
#define IO_MASK (IORESOURCE_IO)
#define PREF_TYPE (IORESOURCE_PREFETCH | IORESOURCE_MEM)
#define MEM_TYPE (IORESOURCE_MEM)
#define IO_TYPE (IORESOURCE_IO)

/* Descend into every child and look for fixed resources. */
    for (child=dev->link[0].children; child; child = child->sibling) {
        constrain_resources(child, limits);
        for (i = 0; i<child->resources; i++) {
            res = &child->resource[i];
            if (!(res->flags & IORESOURCE_FIXED))
                continue;
            /* Choose larger space: above or below fixed resources. */

/* PREFETCH */
            if ((res->flags & MEM_MASK) == PREF_TYPE) {
                lim = &limits->pref;
                /* Is it already outside the limits? */
                if ((res->base + res->size -1) < lim->base ||
                    res->base > lim->limit)
                    continue;
                /* Is the space above larger? */
                if ((lim->limit - (res->base + res->size -1)) >
                    (res->base - lim->base))
                    lim->base = res->base + res->size;
                else
                    lim->limit = res->base -1;
            }

/* MEM */
            if ((res->flags & MEM_MASK) == MEM_TYPE) {
                lim = &limits->mem;
                /* Is it already outside the limits? */
                if ((res->base + res->size -1) < lim->base ||
                    res->base > lim->limit)
                    continue;
                /* Is the space above larger? */
                if ((lim->limit - (res->base + res->size -1)) >
                    (res->base - lim->base))
                    lim->base = res->base + res->size;
                else
                    lim->limit = res->base -1;
            }

/* I/O */
            if ((res->flags & IO_MASK) == IO_TYPE) {
                lim = &limits->io;
                /* Is it already outside the limits? */
                if ((res->base + res->size -1) < lim->base ||
                    res->base > lim->limit)
                    continue;
                /* Is the space above larger? */
                if ((lim->limit - (res->base + res->size -1)) >
                    (res->base - lim->base))
                    lim->base = res->base + res->size;
                else
                    lim->limit = res->base -1;
            }
        }
    }
}

static void avoid_fixed_resources(struct device *dev)
{
    struct constraints limits;
    struct resource *res;
    int i;

/* Initialize constraints to maximum size. */

limits.pref.base = 0;
    limits.pref.limit = 0xfffffffffffffffULL;
    limits.io.base = 0;
    limits.io.limit = 0xfffffffffffffffULL;
    limits.mem.base = 0;
    limits.mem.limit = 0xfffffffffffffffULL;

/* Initialize the limits from the device's resources. */
    for (i = 0; i<dev->resources; i++) {
        res = &dev->resource[i];

/* PREFETCH */
        if ((res->flags & MEM_MASK) == PREF_TYPE &&
            res->limit < limits.pref.limit )
            limits.pref.limit = res->limit;
        /* MEM */
        if ((res->flags & MEM_MASK) == MEM_TYPE &&
            res->limit < limits.mem.limit )
            limits.mem.limit = res->limit;
        /* I/O */
        if ((res->flags & IO_MASK) == IO_TYPE &&
            res->limit < limits.io.limit )
            limits.io.limit = res->limit;
    }

constrain_resources(dev, &limits);

for (i = 0; i<dev->resources; i++) {
        res = &dev->resource[i];
        /* Update the device's resources with new limits. */

/* PREFETCH */
        if ((res->flags & MEM_MASK) == PREF_TYPE) {
            /* Is the resource outside the limits? */
            if ( limits.pref.base > res->base )
                res->base = limits.pref.base;
            if ( res->limit > limits.pref.limit )
                res->limit = limits.pref.limit;
        }

/* MEM */
        if ((res->flags & MEM_MASK) == MEM_TYPE) {
            /* Is the resource outside the limits? */
            if ( limits.mem.base > res->base )
                res->base = limits.mem.base;
            if ( res->limit > limits.mem.limit )
                res->limit = limits.mem.limit;
        }

/* I/O */
        if ((res->flags & IO_MASK) == IO_TYPE) {
            /* Is the resource outside the limits? */
            if ( limits.io.base > res->base )
                res->base = limits.io.base;
            if ( res->limit > limits.io.limit )
                res->limit = limits.io.limit;
        }
    }
}

Thanks,
Myles

[Marc Jones]

On Wed, Dec 3, 2008 at 11:43 AM, Myles Watson  wrote:
> A question before the code:
> Can anyone enumerate all the fixed resources that should be included in
>
> QEMU?
> Currently I have:
> DRAM (first 640K and 0xc0000-TOP-768K) Is this necessary?
> I/O first 0x400 in southbridge
> VGA 0xa0000-0xc0000
> APIC at 0xfec00000 and 0xfee00000
> ROM at 0xfff80000

This looks normal. You could put the first 0x1000 in the SB IMO.
> Side note: Only the lowest and highest fixed addresses matter in my code.
> The others get ignored.  I think it's too much pain to try to interleave
> them to use the space between 0xfee00000 and 0xfff80000, for example.

This is probably ok for a normal system. There is a problem with this
if a device is already setup and should be treated as fixed but if
that is the case it should be seup in the dts....
> /* Initialize constraints to maximum size. */
>
> limits.pref.base = 0;
> limits.pref.limit = 0xfffffffffffffffULL;
> limits.io.base = 0;
> limits.io.limit = 0xfffffffffffffffULL;
> limits.mem.base = 0;
> limits.mem.limit = 0xfffffffffffffffULL;

The limit for memory and prefetch should be the same since they are in
the same address space. For mem.base I assume it gets set from TOM
(for K8) somehow?

Marc

[Myles Watson]

On Fri, Dec 5, 2008 at 3:08 PM, Marc Jones  wrote:
> On Wed, Dec 3, 2008 at 11:43 AM, Myles Watson  wrote:
> > A question before the code:
> > Can anyone enumerate all the fixed resources that should be included
>
> in
> > QEMU?
> > Currently I have:
> > DRAM (first 640K and 0xc0000-TOP-768K) Is this necessary?
> > I/O first 0x400 in southbridge
> > VGA 0xa0000-0xc0000
> > APIC at 0xfec00000 and 0xfee00000
> > ROM at 0xfff80000
>
> This looks normal. You could put the first 0x1000 in the SB IMO.

I changed it, thanks.
> > Side note: Only the lowest and highest fixed addresses matter in my code.
> > The others get ignored.  I think it's too much pain to try to interleave
> > them to use the space between 0xfee00000 and 0xfff80000, for example.
>
> This is probably ok for a normal system. There is a problem with this
> if a device is already setup and should be treated as fixed but if
> that is the case it should be seup in the dts....
> > /* Initialize constraints to maximum size. */
> >
> > limits.pref.base = 0;
> > limits.pref.limit = 0xfffffffffffffffULL;
> > limits.io.base = 0;
> > limits.io.limit = 0xfffffffffffffffULL;
> > limits.mem.base = 0;
> > limits.mem.limit = 0xfffffffffffffffULL;
>
> The limit for memory and prefetch should be the same since they are in
> the same address space. For mem.base I assume it gets set from TOM
> (for K8) somehow?

The problem comes when they are not in the same address space.  I need to be
able to allocate prefetchable memory above 4G.  The way the code works now,
you only get separate address spaces if the domain asks for it.  Otherwise
they end up in the same space.

Thanks,
Myles

[Marc Jones]

On Tue, Dec 2, 2008 at 12:09 PM, Myles Watson  wrote:
> > Coreboot allocates IO from the bottom (0x1000) up?
>
> Yes.
> > I think it should
> > allocate top down as it does with memory. I think that the rule is
> > that the BIOS allocates top down and the OS allocates bottom up.
>
> I don't think that would match v2 or the factory BIOS for serengeti or
> qemu.  Is there a packing reason?

No, It was a rule to stop device allocation from colliding between the
BIOS and OS The firmware is only required to setup boot devices and
the OS was expected to do the rest. I guess you can leave it since it
is working.

Marc

[Marc Jones]

On Tue, Dec 2, 2008 at 6:57 AM, Myles Watson  wrote:
> > -----Original Message-----
> > From: Marc Jones [mailto:[e-mail]]
> > Sent: Monday, December 01, 2008 10:42 PM
> > To: Myles Watson
> > Cc: Coreboot
> > Subject: Re: [coreboot] Resource Allocation discussion
> >
> > Hi Myles,
> > I will try to help..... :)
> >
> > On Mon, Dec 1, 2008 at 2:37 PM, Myles Watson  wrote:
> > > The resource patches I submitted work fairly well, but still need a
> >
> > little
> > > help.
> > >
> > > I haven't touched subtractively decoded resources, and that's where the
> > > breakage is right now.
> > >
> > > Questions:
> > > Who should have a resource that moves all other IO resources above
> >
> > 0x400
> > > or 0x1000?
> >
> > What do you mean? By definition the subtractive port takes all
> > unclaimed cycles. There should be a LPC or ISA bridge in the
> > southbridge that forwards cycles that the legacy southbridge devices
> > don't claim.
>
> Sorry.  I wasn't clear enough again.  What I meant was something more like:
>
> Since the resource allocation algorithm can't avoid resources that it
> doesn't know about, who decides if the subtractive area to avoid is
> 0x0-0x400, 0x0-0x1000, or some other range?  If I declare some device to
> hold these addresses so that no other resource ends up there, should it be
> subtractive?  I'm inclined to put it in the domain or in the SuperIO.

I think it belongs in the southbridge lpc/isa device which is in the
domain. It should be 0x0-0x1000.

Marc

[Myles Watson]

On Tue, Dec 2, 2008 at 11:44 AM, Marc Jones  wrote:
> On Tue, Dec 2, 2008 at 6:57 AM, Myles Watson  wrote:
> > > -----Original Message-----
> > > From: Marc Jones [mailto:[e-mail]]
> > > Sent: Monday, December 01, 2008 10:42 PM
> > > To: Myles Watson
> > > Cc: Coreboot
> > > Subject: Re: [coreboot] Resource Allocation discussion
> > >
> > > Hi Myles,
> > > I will try to help..... :)
> > >
> > > On Mon, Dec 1, 2008 at 2:37 PM, Myles Watson  wrote:
> > > > The resource patches I submitted work fairly well, but still need a
> > >
> > > little
> > > > help.
> > > >
> > > > I haven't touched subtractively decoded resources, and that's where
>
> the
> > > > breakage is right now.
> > > >
> > > > Questions:
> > > > Who should have a resource that moves all other IO resources above
> > >
> > > 0x400
> > > > or 0x1000?
> > >
> > > What do you mean? By definition the subtractive port takes all
> > > unclaimed cycles. There should be a LPC or ISA bridge in the
> > > southbridge that forwards cycles that the legacy southbridge devices
> > > don't claim.
> >
> > Sorry.  I wasn't clear enough again.  What I meant was something more
>
> like:
> > Since the resource allocation algorithm can't avoid resources that it
> > doesn't know about, who decides if the subtractive area to avoid is
> > 0x0-0x400, 0x0-0x1000, or some other range?  If I declare some device to
> > hold these addresses so that no other resource ends up there, should it
>
> be
> > subtractive?  I'm inclined to put it in the domain or in the SuperIO.
>
> I think it belongs in the southbridge lpc/isa device which is in the
> domain. It should be 0x0-0x1000.

So like this? For qemu:

static void i82371eb_isa_read_resources(struct device *dev)
{
    struct resource *res;
    res = new_resource(dev, IOINDEX_SUBTRACTIVE(0,0));
    res->base = 0x0UL;
    res->size = 0x1000UL;
    res->limit = 0xFFFUL;
    res->flags = IORESOURCE_MEM | IORESOURCE_SUBTRACTIVE |
             IORESOURCE_ASSIGNED | IORESOURCE_FIXED;
}

static void i82371eb_isa_set_resources(struct device *dev)
{
    struct resource *res;
    res = probe_resource(dev, IOINDEX_SUBTRACTIVE(0,0));
    res->flags |= IORESOURCE_STORED;
    pci_set_resources(dev);
}

Thanks for the help.

Myles

[Stefan Reinauer]

Myles Watson wrote:
> From an earlier thread with Carl-Daniel it sounded like there are
>
> fixed locations for APICs that need to be avoided.  Where should these
> be avoided?

The right way to do this is create devices for them that hold fixed
address resources.

[Carl-Daniel Hailfinger]

On 02.12.2008 12:36, Stefan Reinauer wrote:
> Myles Watson wrote:
> > From an earlier thread with Carl-Daniel it sounded like there are
> >
> > fixed locations for APICs that need to be avoided.  Where should these
> > be avoided?
>
> The right way to do this is create devices for them that hold fixed
> address resources.

Sounds good to me. We just have to make sure that fixed address
resources are allowed even if they are between DEVICE_MEM_HIGH and
0xFFFFFFFF.

Regards,
Carl-Daniel

[Myles Watson]

> -----Original Message-----
> From: Carl-Daniel Hailfinger [mailto:[e-mail]]
> Sent: Tuesday, December 02, 2008 4:45 AM
> To: Stefan Reinauer
> Cc: Myles Watson; Coreboot
> Subject: Re: [coreboot] Resource Allocation discussion
>
> On 02.12.2008 12:36, Stefan Reinauer wrote:
> > Myles Watson wrote:
> > > From an earlier thread with Carl-Daniel it sounded like there are
> > >
> > > fixed locations for APICs that need to be avoided.  Where should these
> > > be avoided?
> >
> > The right way to do this is create devices for them that hold fixed
> > address resources.
>
> Sounds good to me. We just have to make sure that fixed address
> resources are allowed even if they are between DEVICE_MEM_HIGH and
> 0xFFFFFFFF.

Fixed resources right now just get skipped.  They're not even considered by
the allocator.  I think it might be nice to do some sort of sanity check,
but ...

Thanks,
Myles

[Carl-Daniel Hailfinger]

On 02.12.2008 14:59, Myles Watson wrote:
> > -----Original Message-----
> > From: Carl-Daniel Hailfinger [mailto:[e-mail]]
> > Sent: Tuesday, December 02, 2008 4:45 AM
> > To: Stefan Reinauer
> > Cc: Myles Watson; Coreboot
> > Subject: Re: [coreboot] Resource Allocation discussion
> >
> > On 02.12.2008 12:36, Stefan Reinauer wrote:
> > > Myles Watson wrote:
> > > > From an earlier thread with Carl-Daniel it sounded like there are
> > > >
> > > > fixed locations for APICs that need to be avoided.  Where should these
> > > > be avoided?
> > >
> > > The right way to do this is create devices for them that hold fixed
> > > address resources.
> >
> > Sounds good to me. We just have to make sure that fixed address
> > resources are allowed even if they are between DEVICE_MEM_HIGH and
> > 0xFFFFFFFF.
>
> Fixed resources right now just get skipped.  They're not even considered by
> the allocator.  I think it might be nice to do some sort of sanity check,
> but ...

A nice warning message for overlapping resources would be helpful.

Regards,
Carl-Daniel

[Myles Watson]

> -----Original Message-----
> From: Stefan Reinauer [mailto:[e-mail]]
> Sent: Tuesday, December 02, 2008 4:36 AM
> To: Myles Watson
> Cc: Coreboot
> Subject: Re: [coreboot] Resource Allocation discussion
>
> Myles Watson wrote:
> > From an earlier thread with Carl-Daniel it sounded like there are
> >
> > fixed locations for APICs that need to be avoided.  Where should these
> > be avoided?
>
> The right way to do this is create devices for them that hold fixed
> address resources.

In the domain, or wherever they're implemented?

Thanks,
Myles