Hi
e.g. if we got from HOB info that physical stack x has preallocated PCI buses 0x20..0x2f, io form 0x2000..0x2fff, mem 0xd0000000..0xdfffffff, mem 0x10000000000...0x1ffffffffff and there are 2 root buses 0x20 and 0x28 instead of adding one domain with "physical" stack we added two domains with "virtual" stacks: stack x1 with virtually preallocated PCI buses 0x20..0x27, ip form 0x2000..0x27ff, mem 0xd0000000..0xd7ffffff, mem 0x10000000000...0x17fffffffff stack x2 with virtually preallocated PCI buses 0x28..0x2f, ip form 0x2800..0x2fff, mem 0xd7000000..0xdfffffff, mem 0x18000000000...0x1ffffffffff
Each one with only one root bus without this link_list->next "complexity"
This only works if the downstream resources fit in this split virtual allocation, which you can't know before reading all downstream resources. Especially for mem32 resources the resource allocation is already tight so I think this can get ugly.
At some point of time I was thinking about something called "subdomains"
concept to cover this multiple root buses in one domain case so to make something like: domain 0 //domain domain 1 //subdomain first root bus from stack x and its downstream devices end domain 2 //subdomain second root bus from stack x and its downstream devices end end domain ... ... end ...
The way I understood it, domains are a set of resource windows to be constrained and then distributed over children and in this case children over multiple PCI root busses. I have some doubts that subdomains map the situation correctly/efficiently, because it has essentially the same problem as knowing how to split the resources between domains correctly.
OTOH, does it even make sense to map this in the devicetree? The way FSP reports stacks is generated at runtime and differs depending on the hardware configuration. So having a static structure mapping that may not be interesting?
Arthur
On Tue, Mar 22, 2022 at 9:58 AM Mariusz Szafrański via coreboot < coreboot@coreboot.org> wrote:
Hi Artur,
Multiple PCI root bus per domain gives us more control about resource allocation for downstream devices from poll preallocated by FSP to stack but adds this link_list->next looping complexity (maybe not much deal to handle that as you stated). Additional work will be needed for statically defining them in devicetree.
For us it was easiest to split them "virtually"
e.g. if we got from HOB info that physical stack x has preallocated PCI buses 0x20..0x2f, io form 0x2000..0x2fff, mem 0xd0000000..0xdfffffff, mem 0x10000000000...0x1ffffffffff and there are 2 root buses 0x20 and 0x28 instead of adding one domain with "physical" stack we added two domains with "virtual" stacks: stack x1 with virtually preallocated PCI buses 0x20..0x27, ip form 0x2000..0x27ff, mem 0xd0000000..0xd7ffffff, mem 0x10000000000...0x17fffffffff stack x2 with virtually preallocated PCI buses 0x28..0x2f, ip form 0x2800..0x2fff, mem 0xd7000000..0xdfffffff, mem 0x18000000000...0x1ffffffffff
Each one with only one root bus without this link_list->next "complexity"
It was just a "shortcut" for now.
At some point of time I was thinking about something called "subdomains" concept to cover this multiple root buses in one domain case so to make something like: domain 0 //domain domain 1 //subdomain first root bus from stack x and its downstream devices end domain 2 //subdomain second root bus from stack x and its downstream devices end end domain ... ... end ...
But finally didn`t tried to implement it.
Additional dirty "overflow" trick that I was used for some time was to use something like: domain 0 //first root bus pci 0:0.1....end .... //second root bus pci 0x20:0....end //overflow at 0x20 pci bus number boundary .... end
And dynamic update 0x20 -> real bus number at runtime
All above is what we have tried but all that is not a final solution. Maybe it will give someone else hint to find a better/easier way to handle this hw in coreboot.
Mariusz W dniu 22.03.2022 o 08:29, Arthur Heymans pisze:
Hi Mariusz
I was inspired by the multi domain approach doc and got quite far already. I decided to allocate and attach domains at runtime for the moment instead of statically via the devicetree. In the future I think having devicetree structures makes a lot of sense, e.g. to provide stack specific configuration or derive the IIO bifurcation configuration from it.
I see that FSP sometimes does this virtual splitting but not always and just reports multiple PCI roots on one stack (via a HOB). I don't think splitting up a reported stack in multiple domains in coreboot is a good idea. This means you need to be aware of the downstream resources when just constraining the domain resources to the ones reported to be allocated to the stack. This agains means bypassing or redoing a lot of the coreboot resources allocation so I doubt this approach makes sense. I'm currently thinking that the multiple PCI root bus per domain approach is the easiest. It already works with some minor allocator changes.
Kind regards
Arthur
On Tue, Mar 22, 2022 at 8:15 AM Mariusz Szafrański via coreboot < coreboot@coreboot.org> wrote:
Hi Arthur,
In our multidomain based PoC in this situation (multiple root busses on one stack) we "virtually" splitted this stack and its resource window to two or more virtual stacks and later handled as separate stacks.
Mariusz
W dniu 17.03.2022 o 19:03, Arthur Heymans pisze:
Hi
I've recently tried to improve the soc/intel/xeon_sp codebase. I want to make it use more native coreboot structures and codeflows instead of parsing the FSP HOB again and again to do things. Ideally the HOB is parsed only once in ramstage, parsed into adequate native coreboot structures (struct device, struct bus, chip_info, ...) and used later on.
The lowest hanging fruit in that effort is resource allocation. Currently the coreboot allocator is sort of hijacked by the soc code and done over again. The reason for this is that xeon_sp platforms operate a bit differently than most AMD and Intel Client hardware: there are multiple root busses. This means that there are PCI busses that are in use, but are not downstream from PCI bus 0. In hardware terminology those are the IIO and other type of Stacks.
Each Stack has its own range of usable PCI Bus numbers and decoded IO and MEM spaces below and above 4G. I tried to map these hardware concepts to the existing coreboot 'domain' structure. Each domain has resource windows that are used to allocate children devices on, which would be the PCI devices on the stacks. The allocator needs some tweaks to allow for multiple resources of a type (MEM or IO), but nothing major. See https://review.coreboot.org/c/coreboot/+/62353/ and https://review.coreboot.org/c/coreboot/+/62865 (allocator rewrite/improvement based on Nico's excellent unmerged v4.5 work) This seems to work really well and arguably even better than how it is now with more elegant handling of above and below 4G resources.
Now my question is the following: On some Stacks there are multiple root busses, but the resources need to be allocated on the same window. My initial idea was to add those root busses as separate struct bus in the domain->link_list. However currently the allocator assumes only one bus on domains (and bridges). In the code you'll see a lot of things like
for (child = domain->link_list->children; child; child = child->sibling) ....
This is fine if there is only one bus on the domain. Looping over link_list->next, struct bus'ses is certainly an option here, but I was told that having only one bus here was a design decision on the allocator v4 rewrite. I'm not sure how common that assumption is in the tree, so things could be broken in awkward ways.
Do you have any suggestions to move forward?
Kind regards
Arthur Heymans
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org