Change in coreboot[master]: device: Enable resource allocation above 4G boundary with allocator v4
Furquan Shaikh has uploaded this change for review. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
device: Enable resource allocation above 4G boundary with allocator v4
This change adds back CB:39487 which was reverted as part of CB:41412. Now that the resource allocator is split into old(v3) and new(v4), this change adds support for allocating resources above 4G boundary with the new allocator v4.
Original commit message: This change adds support for allocating resources above the 4G boundary by making use of memranges for resource windows enabled in the previous CL.
It adds a new resource flag IORESOURCE_ABOVE_4G which is used in the following ways: a) Downstream device resources can set this flag to indicate that they would like to have their resource allocation above the 4G boundary. These semantics will have to be enabled in the drivers managing the devices. It can also be extended to be enabled via devicetree. This flag is automatically propagated by the resource allocator from downstream devices to the upstream bridges in pass 1. It is done to ensure that the resource allocator has a global view of downstream requirements during pass 2 at domain level.
b) Bridges have a single resource window for each of mem and prefmem resource types. Thus, if any downstream resource of the bridge requests allocation above 4G boundary, all the other downstream resources of the same type under the bridge will be allocated above 4G boundary.
c) During pass 2, resource allocator at domain level splits IORESOURCE_MEM into two different memory ranges -- one for the window below 4G and other above 4G. Resource allocation happens separately for each of these windows.
d) At the bridge level, there is no extra logic required since the resource will live entirely above or below the 4G boundary. Hence, all downstream devices of any bridge will fall within the window allocated to the bridge resource. To handle this case separately from that of domain, initializing of memranges for a bridge is done differently than the domain.
Limitation: Resources of a given type at the bridge or downstream devices cannot live both above and below 4G boundary. Thus, if a bridge has some downstream resources requesting allocation for a given type above 4G boundary and other resources of the same type requesting allocation below 4G boundary, then all these resources of the same type get allocated above 4G boundary.
Change-Id: I92a5cf7cd1457f2f713e1ffd8ea31796ce3d0cce Signed-off-by: Furquan Shaikh furquan@google.com --- M src/device/resource_allocator_v4.c M src/include/device/resource.h 2 files changed, 113 insertions(+), 9 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/66/41466/1
diff --git a/src/device/resource_allocator_v4.c b/src/device/resource_allocator_v4.c index 64820a9..0e6061d 100644 --- a/src/device/resource_allocator_v4.c +++ b/src/device/resource_allocator_v4.c @@ -104,6 +104,19 @@ bridge_res->limit = child_res->limit;
/* + * Propagate the downstream resource request to allocate above 4G boundary to + * upstream bridge resource. This ensures that during pass 2, the resource + * allocator at domain level has a global view of all the downstream device + * requirements and thus address space is allocated as per updated flags in the + * bridge resource. + * + * Since the bridge resource is a single window, all the downstream resources of + * this bridge resource will be allocated space above 4G boundary. + */ + if (child_res->flags & IORESOURCE_ABOVE_4G) + bridge_res->flags |= IORESOURCE_ABOVE_4G; + + /* * Alignment value of 0 means that the child resource has no alignment * requirements and so the base value remains unchanged here. */ @@ -207,25 +220,99 @@ die("Unexpected resource type: flags(%d)!\n", res->flags); }
-static void initialize_memranges(struct memranges *ranges, const struct resource *res, - unsigned long memrange_type) +/* + * If the resource base is set to the limit, then it means that the resource is invalid and + * hence cannot be used for allocation. + */ +static bool is_resource_invalid(const struct resource *res) +{ + return res->base == res->limit; +} + +/* + * This function initializes memranges for domain device. If the resource crosses 4G boundary, + * then this function splits it into two ranges -- one for the window below 4G and the other for + * the window above 4G. The latter range has IORESOURCE_ABOVE_4G flag set to satisfy resource + * requests from downstream devices for allocations above 4G. + */ +static void initialize_domain_memranges(struct memranges *ranges, const struct resource *res, + unsigned long memrange_type) { resource_t res_base; resource_t res_limit; unsigned char align = get_alignment_by_resource_type(res); + const resource_t limit_4g = 0xffffffff;
memranges_init_empty_with_alignment(ranges, NULL, 0, align);
- if (res == NULL) + if ((res == NULL) || is_resource_invalid(res)) return;
res_base = res->base; res_limit = res->limit;
- if (res_base == res_limit) + /* + * Split the resource into two separate ranges if it crosses the 4G boundary. Memrange + * type is set differently to ensure that memrange does not merge these two ranges. For + * the range above 4G boundary, given memrange type is ORed with IORESOURCE_ABOVE_4G. + */ + if (res_base <= limit_4g) { + + resource_t range_limit; + + /* Clip the resource limit at 4G boundary if necessary. */ + range_limit = MIN(res_limit, limit_4g); + memranges_insert(ranges, res_base, range_limit - res_base + 1, memrange_type); + + /* + * If the resource lies completely below the 4G boundary, nothing more needs to + * be done. + */ + if (res_limit <= limit_4g) + return; + + /* + * If the resource window crosses the 4G boundary, then update res_base to add + * another entry for the range above the boundary. + */ + res_base = limit_4g + 1; + } + + if (res_base > res_limit) return;
- memranges_insert(ranges, res_base, res_limit - res_base + 1, memrange_type); + /* + * If resource lies completely above the 4G boundary or if the resource was clipped to + * add two separate ranges, the range above 4G boundary has the resource flag + * IORESOURCE_ABOVE_4G set. This allows domain to handle any downstream requests for + * resource allocation above 4G differently. + */ + memranges_insert(ranges, res_base, res_limit - res_base + 1, + memrange_type | IORESOURCE_ABOVE_4G); +} + +/* + * This function initializes memranges for bridge device. Unlike domain, bridge does not need to + * care about resource window crossing 4G boundary. This is handled by the resource allocator at + * domain level to ensure that all downstream bridges are allocated space either above or below + * 4G boundary as per the state of IORESOURCE_ABOVE_4G for the respective bridge resource. + * + * So, this function creates a single range of the entire resource window available for the + * bridge resource. Thus all downstream resources of the bridge for the given resource type get + * allocated space from the same window. If there is any downstream resource of the bridge which + * requests allocation above 4G, then all other downstream resources of the same type under the + * bridge get allocated above 4G. + */ +static void initialize_bridge_memranges(struct memranges *ranges, const struct resource *res, + unsigned long memrange_type) +{ + + memranges_init_empty(ranges, NULL, 0); + + if ((res == NULL) || is_resource_invalid(res)) + return; + + memranges_insert(ranges, res->base, res->limit - res->base + 1, memrange_type); }
static void print_resource_ranges(const struct memranges *ranges) @@ -368,10 +455,12 @@ dev_path(dev), resource2str(res), res->base, res->size, res->align, res->gran, res->limit);
- initialize_memranges(ranges, res, type); - - if (dev->path.type == DEVICE_PATH_DOMAIN) + if (dev->path.type == DEVICE_PATH_DOMAIN) { + initialize_domain_memranges(ranges, res, type); constrain_domain_resources(dev, ranges, type); + } else { + initialize_bridge_memranges(ranges, res, type); + }
print_resource_ranges(ranges); } @@ -477,12 +566,25 @@ * Domain does not distinguish between mem and prefmem resources. Thus, the resource * allocation at domain level considers mem and prefmem together when finding the best * fit based on the biggest resource requirement. + * + * However, resource requests for allocation above 4G boundary need to be handled + * separately if the domain resource window crosses this boundary. There is a single + * window for resource of type IORESOURCE_MEM. When creating memranges, this resource + * is split into two separate ranges -- one for the window below 4G boundary and other + * for the window above 4G boundary (with IORESOURCE_ABOVE_4G flag set). Thus, when + * allocating child resources, requests for below and above the 4G boundary are handled + * separately by setting the type_mask and type_match to allocate_child_resources() + * accordingly. */ res = find_domain_resource(domain, IORESOURCE_MEM); if (res) { setup_resource_ranges(domain, res, IORESOURCE_MEM, &ranges); - allocate_child_resources(domain->link_list, &ranges, IORESOURCE_TYPE_MASK, + allocate_child_resources(domain->link_list, &ranges, + IORESOURCE_TYPE_MASK | IORESOURCE_ABOVE_4G, IORESOURCE_MEM); + allocate_child_resources(domain->link_list, &ranges, + IORESOURCE_TYPE_MASK | IORESOURCE_ABOVE_4G, + IORESOURCE_MEM | IORESOURCE_ABOVE_4G); cleanup_resource_ranges(domain, &ranges, res); }
diff --git a/src/include/device/resource.h b/src/include/device/resource.h index c97b01d..42c7e6a 100644 --- a/src/include/device/resource.h +++ b/src/include/device/resource.h @@ -24,6 +24,8 @@ #define IORESOURCE_SUBTRACTIVE 0x00040000 /* The IO resource has a bus below it. */ #define IORESOURCE_BRIDGE 0x00080000 +/* This is a request to allocate resource about 4G boundary. */ +#define IORESOURCE_ABOVE_4G 0x00100000 /* The resource needs to be reserved in the coreboot table */ #define IORESOURCE_RESERVE 0x10000000 /* The IO resource assignment has been stored in the device */
Angel Pons has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 1: Code-Review+1
Hello build bot (Jenkins), Angel Pons,
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/41466
to look at the new patch set (#2).
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
device: Enable resource allocation above 4G boundary with allocator v4
This change adds back CB:39487 which was reverted as part of CB:41412. Now that the resource allocator is split into old(v3) and new(v4), this change adds support for allocating resources above 4G boundary with the new allocator v4.
Original commit message: This change adds support for allocating resources above the 4G boundary by making use of memranges for resource windows enabled in the previous CL.
It adds a new resource flag IORESOURCE_ABOVE_4G which is used in the following ways: a) Downstream device resources can set this flag to indicate that they would like to have their resource allocation above the 4G boundary. These semantics will have to be enabled in the drivers managing the devices. It can also be extended to be enabled via devicetree. This flag is automatically propagated by the resource allocator from downstream devices to the upstream bridges in pass 1. It is done to ensure that the resource allocator has a global view of downstream requirements during pass 2 at domain level.
b) Bridges have a single resource window for each of mem and prefmem resource types. Thus, if any downstream resource of the bridge requests allocation above 4G boundary, all the other downstream resources of the same type under the bridge will be allocated above 4G boundary.
c) During pass 2, resource allocator at domain level splits IORESOURCE_MEM into two different memory ranges -- one for the window below 4G and other above 4G. Resource allocation happens separately for each of these windows.
d) At the bridge level, there is no extra logic required since the resource will live entirely above or below the 4G boundary. Hence, all downstream devices of any bridge will fall within the window allocated to the bridge resource. To handle this case separately from that of domain, initializing of memranges for a bridge is done differently than the domain.
Limitation: Resources of a given type at the bridge or downstream devices cannot live both above and below 4G boundary. Thus, if a bridge has some downstream resources requesting allocation for a given type above 4G boundary and other resources of the same type requesting allocation below 4G boundary, then all these resources of the same type get allocated above 4G boundary.
Change-Id: I92a5cf7cd1457f2f713e1ffd8ea31796ce3d0cce Signed-off-by: Furquan Shaikh furquan@google.com --- M src/device/resource_allocator_v4.c M src/include/device/resource.h 2 files changed, 151 insertions(+), 39 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/66/41466/2
Nico Huber has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 4:
Is there a particular reason not to move all prefmem resources above 4G?
The idea came along with the thought of resource assignment from top to bottom (stealing from the end of the last memrange): If we'd always do that, we wouldn't even need a quirk (IORESOURCE_ ABOVE_4G). We could simply default to the upper-most possible placement and then opt out by constraining the domain resource or individual resource's `.limit`.
Furquan Shaikh has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 4:
Patch Set 4:
Is there a particular reason not to move all prefmem resources above 4G?
I don't think it is correct to assume that all components using these device resources support resources being above 4G boundary. Hence, the mainboard/device driver is given a choice to decide when/if it wants to allocate a resource allocation above 4G.
The idea came along with the thought of resource assignment from top to bottom (stealing from the end of the last memrange): If we'd always do that, we wouldn't even need a quirk (IORESOURCE_ ABOVE_4G). We could simply default to the upper-most possible placement and then opt out by constraining the domain resource or individual resource's `.limit`.
I don't think this is a domain decision to restrict the allocation. From a domain standpoint, it allows allocation from anywhere between 0 -> max addressable memory. It is upto the device driver and/or mainboard to determine for itself whether it wants the allocation to happen above 4G or not. A mainboard can have a mix of devices which require prefmem to be allocated above(e.g. USB4) and below (e.g. graphics) 4G. Hence, this flag is essential.
Kyösti Mälkki has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 4:
Patch Set 4:
Is there a particular reason not to move all prefmem resources above 4G?
The idea came along with the thought of resource assignment from top to bottom (stealing from the end of the last memrange): If we'd always do that, we wouldn't even need a quirk (IORESOURCE_ ABOVE_4G). We could simply default to the upper-most possible placement and then opt out by constraining the domain resource or individual resource's `.limit`.
In the PCI specification, is (PCI_BASE_ADDRESS_) MEM_PREFETCH allowed to be set without MEM_LIMIT_64 being set?
My expectation is 32-bit payloads will not be able to address any DRAM or MMIO located above 4 GiB but I have not experimented or investigated this any further. I believe linear framebuffers for graphics and some DMA queues/buffers for iPXE would fail. I have not checked if AHCI/XHCI etc advertise some BARs with PREFMEM too.
This also reminded me, samsung/lumpy has PCIe NIC that incorrectly advertises MMIO register bank BAR with PREFMEM. It needs a quirk to override such that the resource hits MTRR UC window, write-combining would fail. Remotely related CB:5169.
Nico Huber has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 4:
Patch Set 4:
Patch Set 4:
Is there a particular reason not to move all prefmem resources above 4G?
I don't think it is correct to assume that all components using these device resources support resources being above 4G boundary. Hence, the mainboard/device driver is given a choice to decide when/if it wants to allocate a resource allocation above 4G.
Sorry, that came out wrong. I meant to _default_ to put them above 4G. We already have an opt-out mechanism for it, the `.limit`. There is one reason, why I wouldn't default to always rely on the limit: chipset-integrated devices sometimes allow UC MMIO resources above 4G. We should avoid that.
The idea came along with the thought of resource assignment from top to bottom (stealing from the end of the last memrange): If we'd always do that, we wouldn't even need a quirk (IORESOURCE_ ABOVE_4G). We could simply default to the upper-most possible placement and then opt out by constraining the domain resource or individual resource's `.limit`.
I don't think this is a domain decision to restrict the allocation. From a domain standpoint, it allows allocation from anywhere between 0 -> max addressable memory.
For the very simple domains we handle currently, yes.
It is upto the device driver and/or mainboard to determine for itself whether it wants the allocation to happen above 4G or not. A mainboard can have a mix of devices which require prefmem to be allocated above(e.g. USB4) and below (e.g. graphics) 4G. Hence, this flag is essential.
An opt-out or opt-in is essential, yet another flag is not.
Nico Huber has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 4:
Is there a particular reason not to move all prefmem resources above 4G?
The idea came along with the thought of resource assignment from top to bottom (stealing from the end of the last memrange): If we'd always do that, we wouldn't even need a quirk (IORESOURCE_ ABOVE_4G). We could simply default to the upper-most possible placement and then opt out by constraining the domain resource or individual resource's `.limit`.
In the PCI specification, is (PCI_BASE_ADDRESS_) MEM_PREFETCH allowed to be set without MEM_LIMIT_64 being set?
I think it's allowed for legacy PCI devices. I don't think it matters, the allocator should always adhere to the `.limit`.
My expectation is 32-bit payloads will not be able to address any DRAM or MMIO located above 4 GiB but I have not experimented or investigated this any further. I believe linear framebuffers for graphics and some DMA queues/buffers for iPXE would fail. I have not checked if AHCI/XHCI etc advertise some BARs with PREFMEM too.
I think we should handle devices by class. With an opt-out for each, e.g.
"Place all video-device resources below 4GiB" "Place all network-device resources below 4GiB"
The primary video adapter should probably not have an option and should always stay below 4G. Hmm, unless we do LinuxBoot... sigh, so many options.
This also reminded me, samsung/lumpy has PCIe NIC that incorrectly advertises MMIO register bank BAR with PREFMEM. It needs a quirk to override such that the resource hits MTRR UC window, write-combining would fail. Remotely related CB:5169.
Nico Huber has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 4:
So, what was I'm going to say. No matter if we handle above 4G as an opt-in or an opt-out, I would prefer us to find a solution that doesn't need a quirk (yet another flag). Allocating top-down and using the `.limit` as opt-out could provide that, IMO.
Furquan Shaikh has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 4:
Patch Set 4:
So, what was I'm going to say. No matter if we handle above 4G as an opt-in or an opt-out, I would prefer us to find a solution that doesn't need a quirk (yet another flag). Allocating top-down and using the `.limit` as opt-out could provide that, IMO.
In my opinion, it's better to have an explicit flag than reusing the limit field for this. Limit field for discoverable resources of a device gets set based on the driver for that device. Example: PCI driver using its own mechanism sets the limit for the resources. Now, this driver can be common and used by all device resources that are discoverable by PCI specification.
In this case, we will have a mix of resource that will have limits set to either below or above 4G. If we were to reuse the limit field for deciding whether to allocate above or below 4G, then every device will need its own driver that can do the post-processing after pci read resource is done to set the limit correctly. Yes, we can enable helpers to always limit below 4G unless a device driver requires it. But, I don't like the idea of changing the limits from what they really are as advertised by the devices going into the allocator(i.e. before doing the allocation).
Instead having a separate flag makes it pretty clear what the device really provided as its requirement and what the device driver wants and the allocator can take action based on that.
Hello build bot (Jenkins), Angel Pons,
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/41466
to look at the new patch set (#5).
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
device: Enable resource allocation above 4G boundary with allocator v4
This change adds back CB:39487 which was reverted as part of CB:41412. Now that the resource allocator is split into old(v3) and new(v4), this change adds support for allocating resources above 4G boundary with the new allocator v4.
Original commit message: This change adds support for allocating resources above the 4G boundary by making use of memranges for resource windows enabled in the previous CL.
It adds a new resource flag IORESOURCE_ABOVE_4G which is used in the following ways: a) Downstream device resources can set this flag to indicate that they would like to have their resource allocation above the 4G boundary. These semantics will have to be enabled in the drivers managing the devices. It can also be extended to be enabled via devicetree. This flag is automatically propagated by the resource allocator from downstream devices to the upstream bridges in pass 1. It is done to ensure that the resource allocator has a global view of downstream requirements during pass 2 at domain level.
b) Bridges have a single resource window for each of mem and prefmem resource types. Thus, if any downstream resource of the bridge requests allocation above 4G boundary, all the other downstream resources of the same type under the bridge will be allocated above 4G boundary.
c) During pass 2, resource allocator at domain level splits IORESOURCE_MEM into two different memory ranges -- one for the window below 4G and other above 4G. Resource allocation happens separately for each of these windows.
d) At the bridge level, there is no extra logic required since the resource will live entirely above or below the 4G boundary. Hence, all downstream devices of any bridge will fall within the window allocated to the bridge resource. To handle this case separately from that of domain, initializing of memranges for a bridge is done differently than the domain.
Limitation: Resources of a given type at the bridge or downstream devices cannot live both above and below 4G boundary. Thus, if a bridge has some downstream resources requesting allocation for a given type above 4G boundary and other resources of the same type requesting allocation below 4G boundary, then all these resources of the same type get allocated above 4G boundary.
Change-Id: I92a5cf7cd1457f2f713e1ffd8ea31796ce3d0cce Signed-off-by: Furquan Shaikh furquan@google.com --- M src/device/resource_allocator_v4.c M src/include/device/resource.h 2 files changed, 115 insertions(+), 8 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/66/41466/5
Nico Huber has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 7:
(1 comment)
https://review.coreboot.org/c/coreboot/+/41466/7/src/device/resource_allocat... File src/device/resource_allocator_v4.c:
https://review.coreboot.org/c/coreboot/+/41466/7/src/device/resource_allocat... PS7, Line 310: memranges_init_empty(ranges, NULL, 0); The alignment is gone again. Maybe an error during rebase?
Hello build bot (Jenkins), Angel Pons,
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/41466
to look at the new patch set (#8).
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
device: Enable resource allocation above 4G boundary with allocator v4
This change adds back CB:39487 which was reverted as part of CB:41412. Now that the resource allocator is split into old(v3) and new(v4), this change adds support for allocating resources above 4G boundary with the new allocator v4.
Original commit message: This change adds support for allocating resources above the 4G boundary by making use of memranges for resource windows enabled in the previous CL.
It adds a new resource flag IORESOURCE_ABOVE_4G which is used in the following ways: a) Downstream device resources can set this flag to indicate that they would like to have their resource allocation above the 4G boundary. These semantics will have to be enabled in the drivers managing the devices. It can also be extended to be enabled via devicetree. This flag is automatically propagated by the resource allocator from downstream devices to the upstream bridges in pass 1. It is done to ensure that the resource allocator has a global view of downstream requirements during pass 2 at domain level.
b) Bridges have a single resource window for each of mem and prefmem resource types. Thus, if any downstream resource of the bridge requests allocation above 4G boundary, all the other downstream resources of the same type under the bridge will be allocated above 4G boundary.
c) During pass 2, resource allocator at domain level splits IORESOURCE_MEM into two different memory ranges -- one for the window below 4G and other above 4G. Resource allocation happens separately for each of these windows.
d) At the bridge level, there is no extra logic required since the resource will live entirely above or below the 4G boundary. Hence, all downstream devices of any bridge will fall within the window allocated to the bridge resource. To handle this case separately from that of domain, initializing of memranges for a bridge is done differently than the domain.
Limitation: Resources of a given type at the bridge or downstream devices cannot live both above and below 4G boundary. Thus, if a bridge has some downstream resources requesting allocation for a given type above 4G boundary and other resources of the same type requesting allocation below 4G boundary, then all these resources of the same type get allocated above 4G boundary.
Change-Id: I92a5cf7cd1457f2f713e1ffd8ea31796ce3d0cce Signed-off-by: Furquan Shaikh furquan@google.com --- M src/device/resource_allocator_v4.c M src/include/device/resource.h 2 files changed, 116 insertions(+), 8 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/66/41466/8
Furquan Shaikh has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 8:
(1 comment)
https://review.coreboot.org/c/coreboot/+/41466/7/src/device/resource_allocat... File src/device/resource_allocator_v4.c:
https://review.coreboot.org/c/coreboot/+/41466/7/src/device/resource_allocat... PS7, Line 310: memranges_init_empty(ranges, NULL, 0);
The alignment is gone again. […]
Done
Aaron Durbin has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 8: Code-Review+2
(1 comment)
https://review.coreboot.org/c/coreboot/+/41466/8/src/device/resource_allocat... File src/device/resource_allocator_v4.c:
https://review.coreboot.org/c/coreboot/+/41466/8/src/device/resource_allocat... PS8, Line 252: res_limit = res->limit; To avoid confusion to the reader you could check for IORESOURCE_IO here, insert the range, and bail out.
Furquan Shaikh has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 8:
(1 comment)
https://review.coreboot.org/c/coreboot/+/41466/8/src/device/resource_allocat... File src/device/resource_allocator_v4.c:
https://review.coreboot.org/c/coreboot/+/41466/8/src/device/resource_allocat... PS8, Line 252: res_limit = res->limit;
To avoid confusion to the reader you could check for IORESOURCE_IO here, insert the range, and bail […]
Done
Hello build bot (Jenkins), Angel Pons, Aaron Durbin,
I'd like you to reexamine a change. Please visit
https://review.coreboot.org/c/coreboot/+/41466
to look at the new patch set (#9).
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
device: Enable resource allocation above 4G boundary with allocator v4
This change adds back CB:39487 which was reverted as part of CB:41412. Now that the resource allocator is split into old(v3) and new(v4), this change adds support for allocating resources above 4G boundary with the new allocator v4.
Original commit message: This change adds support for allocating resources above the 4G boundary by making use of memranges for resource windows enabled in the previous CL.
It adds a new resource flag IORESOURCE_ABOVE_4G which is used in the following ways: a) Downstream device resources can set this flag to indicate that they would like to have their resource allocation above the 4G boundary. These semantics will have to be enabled in the drivers managing the devices. It can also be extended to be enabled via devicetree. This flag is automatically propagated by the resource allocator from downstream devices to the upstream bridges in pass 1. It is done to ensure that the resource allocator has a global view of downstream requirements during pass 2 at domain level.
b) Bridges have a single resource window for each of mem and prefmem resource types. Thus, if any downstream resource of the bridge requests allocation above 4G boundary, all the other downstream resources of the same type under the bridge will be allocated above 4G boundary.
c) During pass 2, resource allocator at domain level splits IORESOURCE_MEM into two different memory ranges -- one for the window below 4G and other above 4G. Resource allocation happens separately for each of these windows.
d) At the bridge level, there is no extra logic required since the resource will live entirely above or below the 4G boundary. Hence, all downstream devices of any bridge will fall within the window allocated to the bridge resource. To handle this case separately from that of domain, initializing of memranges for a bridge is done differently than the domain.
Limitation: Resources of a given type at the bridge or downstream devices cannot live both above and below 4G boundary. Thus, if a bridge has some downstream resources requesting allocation for a given type above 4G boundary and other resources of the same type requesting allocation below 4G boundary, then all these resources of the same type get allocated above 4G boundary.
Change-Id: I92a5cf7cd1457f2f713e1ffd8ea31796ce3d0cce Signed-off-by: Furquan Shaikh furquan@google.com --- M src/device/resource_allocator_v4.c M src/include/device/resource.h 2 files changed, 139 insertions(+), 12 deletions(-)
git pull ssh://review.coreboot.org:29418/coreboot refs/changes/66/41466/9
Aaron Durbin has posted comments on this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
Patch Set 9: Code-Review+2
Patrick Georgi has submitted this change. ( https://review.coreboot.org/c/coreboot/+/41466 )
Change subject: device: Enable resource allocation above 4G boundary with allocator v4 ......................................................................
device: Enable resource allocation above 4G boundary with allocator v4
This change adds back CB:39487 which was reverted as part of CB:41412. Now that the resource allocator is split into old(v3) and new(v4), this change adds support for allocating resources above 4G boundary with the new allocator v4.
Original commit message: This change adds support for allocating resources above the 4G boundary by making use of memranges for resource windows enabled in the previous CL.
It adds a new resource flag IORESOURCE_ABOVE_4G which is used in the following ways: a) Downstream device resources can set this flag to indicate that they would like to have their resource allocation above the 4G boundary. These semantics will have to be enabled in the drivers managing the devices. It can also be extended to be enabled via devicetree. This flag is automatically propagated by the resource allocator from downstream devices to the upstream bridges in pass 1. It is done to ensure that the resource allocator has a global view of downstream requirements during pass 2 at domain level.
b) Bridges have a single resource window for each of mem and prefmem resource types. Thus, if any downstream resource of the bridge requests allocation above 4G boundary, all the other downstream resources of the same type under the bridge will be allocated above 4G boundary.
c) During pass 2, resource allocator at domain level splits IORESOURCE_MEM into two different memory ranges -- one for the window below 4G and other above 4G. Resource allocation happens separately for each of these windows.
d) At the bridge level, there is no extra logic required since the resource will live entirely above or below the 4G boundary. Hence, all downstream devices of any bridge will fall within the window allocated to the bridge resource. To handle this case separately from that of domain, initializing of memranges for a bridge is done differently than the domain.
Limitation: Resources of a given type at the bridge or downstream devices cannot live both above and below 4G boundary. Thus, if a bridge has some downstream resources requesting allocation for a given type above 4G boundary and other resources of the same type requesting allocation below 4G boundary, then all these resources of the same type get allocated above 4G boundary.
Change-Id: I92a5cf7cd1457f2f713e1ffd8ea31796ce3d0cce Signed-off-by: Furquan Shaikh furquan@google.com Reviewed-on: https://review.coreboot.org/c/coreboot/+/41466 Tested-by: build bot (Jenkins) no-reply@coreboot.org Reviewed-by: Aaron Durbin adurbin@chromium.org --- M src/device/resource_allocator_v4.c M src/include/device/resource.h 2 files changed, 139 insertions(+), 12 deletions(-)
Approvals: build bot (Jenkins): Verified Aaron Durbin: Looks good to me, approved
diff --git a/src/device/resource_allocator_v4.c b/src/device/resource_allocator_v4.c index b65fc8b..ae3c764 100644 --- a/src/device/resource_allocator_v4.c +++ b/src/device/resource_allocator_v4.c @@ -104,6 +104,19 @@ bridge_res->limit = child_res->limit;
/* + * Propagate the downstream resource request to allocate above 4G boundary to + * upstream bridge resource. This ensures that during pass 2, the resource + * allocator at domain level has a global view of all the downstream device + * requirements and thus address space is allocated as per updated flags in the + * bridge resource. + * + * Since the bridge resource is a single window, all the downstream resources of + * this bridge resource will be allocated space above 4G boundary. + */ + if (child_res->flags & IORESOURCE_ABOVE_4G) + bridge_res->flags |= IORESOURCE_ABOVE_4G; + + /* * Alignment value of 0 means that the child resource has no alignment * requirements and so the base value remains unchanged here. */ @@ -207,22 +220,119 @@ die("Unexpected resource type: flags(%d)!\n", res->flags); }
-static void initialize_memranges(struct memranges *ranges, const struct resource *res, - unsigned long memrange_type) +/* + * If the resource is NULL or if the resource is not assigned, then it cannot be used for + * allocation for downstream devices. + */ +static bool is_resource_invalid(const struct resource *res) +{ + return (res == NULL) || !(res->flags & IORESOURCE_ASSIGNED); +} + +static void initialize_domain_io_resource_memranges(struct memranges *ranges, + const struct resource *res, + unsigned long memrange_type) +{ + memranges_insert(ranges, res->base, res->limit - res->base + 1, memrange_type); +} + +static void initialize_domain_mem_resource_memranges(struct memranges *ranges, + const struct resource *res, + unsigned long memrange_type) { resource_t res_base; resource_t res_limit; - unsigned char align = get_alignment_by_resource_type(res);
- memranges_init_empty_with_alignment(ranges, NULL, 0, align); - - if ((res == NULL) || !(res->flags & IORESOURCE_ASSIGNED)) - return; + const resource_t limit_4g = 0xffffffff;
res_base = res->base; res_limit = res->limit;
- memranges_insert(ranges, res_base, res_limit - res_base + 1, memrange_type); + /* + * Split the resource into two separate ranges if it crosses the 4G boundary. Memrange + * type is set differently to ensure that memrange does not merge these two ranges. For + * the range above 4G boundary, given memrange type is ORed with IORESOURCE_ABOVE_4G. + */ + if (res_base <= limit_4g) { + + resource_t range_limit; + + /* Clip the resource limit at 4G boundary if necessary. */ + range_limit = MIN(res_limit, limit_4g); + memranges_insert(ranges, res_base, range_limit - res_base + 1, memrange_type); + + /* + * If the resource lies completely below the 4G boundary, nothing more needs to + * be done. + */ + if (res_limit <= limit_4g) + return; + + /* + * If the resource window crosses the 4G boundary, then update res_base to add + * another entry for the range above the boundary. + */ + res_base = limit_4g + 1; + } + + if (res_base > res_limit) + return; + + /* + * If resource lies completely above the 4G boundary or if the resource was clipped to + * add two separate ranges, the range above 4G boundary has the resource flag + * IORESOURCE_ABOVE_4G set. This allows domain to handle any downstream requests for + * resource allocation above 4G differently. + */ + memranges_insert(ranges, res_base, res_limit - res_base + 1, + memrange_type | IORESOURCE_ABOVE_4G); +} + +/* + * This function initializes memranges for domain device. If the resource crosses 4G boundary, + * then this function splits it into two ranges -- one for the window below 4G and the other for + * the window above 4G. The latter range has IORESOURCE_ABOVE_4G flag set to satisfy resource + * requests from downstream devices for allocations above 4G. + */ +static void initialize_domain_memranges(struct memranges *ranges, const struct resource *res, + unsigned long memrange_type) +{ + unsigned char align = get_alignment_by_resource_type(res); + + memranges_init_empty_with_alignment(ranges, NULL, 0, align); + + if (is_resource_invalid(res)) + return; + + if (res->flags & IORESOURCE_IO) + initialize_domain_io_resource_memranges(ranges, res, memrange_type); + else + initialize_domain_mem_resource_memranges(ranges, res, memrange_type); +} + +/* + * This function initializes memranges for bridge device. Unlike domain, bridge does not need to + * care about resource window crossing 4G boundary. This is handled by the resource allocator at + * domain level to ensure that all downstream bridges are allocated space either above or below + * 4G boundary as per the state of IORESOURCE_ABOVE_4G for the respective bridge resource. + * + * So, this function creates a single range of the entire resource window available for the + * bridge resource. Thus all downstream resources of the bridge for the given resource type get + * allocated space from the same window. If there is any downstream resource of the bridge which + * requests allocation above 4G, then all other downstream resources of the same type under the + * bridge get allocated above 4G. + */ +static void initialize_bridge_memranges(struct memranges *ranges, const struct resource *res, + unsigned long memrange_type) +{ + unsigned char align = get_alignment_by_resource_type(res); + + memranges_init_empty_with_alignment(ranges, NULL, 0, align); + + if (is_resource_invalid(res)) + return; + + memranges_insert(ranges, res->base, res->limit - res->base + 1, memrange_type); }
static void print_resource_ranges(const struct memranges *ranges) @@ -360,10 +470,12 @@ dev_path(dev), resource2str(res), res->base, res->size, res->align, res->gran, res->limit);
- initialize_memranges(ranges, res, type); - - if (dev->path.type == DEVICE_PATH_DOMAIN) + if (dev->path.type == DEVICE_PATH_DOMAIN) { + initialize_domain_memranges(ranges, res, type); constrain_domain_resources(dev, ranges, type); + } else { + initialize_bridge_memranges(ranges, res, type); + }
print_resource_ranges(ranges); } @@ -469,12 +581,25 @@ * Domain does not distinguish between mem and prefmem resources. Thus, the resource * allocation at domain level considers mem and prefmem together when finding the best * fit based on the biggest resource requirement. + * + * However, resource requests for allocation above 4G boundary need to be handled + * separately if the domain resource window crosses this boundary. There is a single + * window for resource of type IORESOURCE_MEM. When creating memranges, this resource + * is split into two separate ranges -- one for the window below 4G boundary and other + * for the window above 4G boundary (with IORESOURCE_ABOVE_4G flag set). Thus, when + * allocating child resources, requests for below and above the 4G boundary are handled + * separately by setting the type_mask and type_match to allocate_child_resources() + * accordingly. */ res = find_domain_resource(domain, IORESOURCE_MEM); if (res) { setup_resource_ranges(domain, res, IORESOURCE_MEM, &ranges); - allocate_child_resources(domain->link_list, &ranges, IORESOURCE_TYPE_MASK, + allocate_child_resources(domain->link_list, &ranges, + IORESOURCE_TYPE_MASK | IORESOURCE_ABOVE_4G, IORESOURCE_MEM); + allocate_child_resources(domain->link_list, &ranges, + IORESOURCE_TYPE_MASK | IORESOURCE_ABOVE_4G, + IORESOURCE_MEM | IORESOURCE_ABOVE_4G); cleanup_resource_ranges(domain, &ranges, res); }
diff --git a/src/include/device/resource.h b/src/include/device/resource.h index c97b01d..42c7e6a 100644 --- a/src/include/device/resource.h +++ b/src/include/device/resource.h @@ -24,6 +24,8 @@ #define IORESOURCE_SUBTRACTIVE 0x00040000 /* The IO resource has a bus below it. */ #define IORESOURCE_BRIDGE 0x00080000 +/* This is a request to allocate resource about 4G boundary. */ +#define IORESOURCE_ABOVE_4G 0x00100000 /* The resource needs to be reserved in the coreboot table */ #define IORESOURCE_RESERVE 0x10000000 /* The IO resource assignment has been stored in the device */
-
Aaron Durbin (Code Review) -
Angel Pons (Code Review)
-
Furquan Shaikh (Code Review)
-
Kyösti Mälkki (Code Review)
-
Nico Huber (Code Review)
-
Patrick Georgi (Code Review)