Hi,
since the Geode/CS553x platform emulates PCI headers for chipset/processor devices, we have the ability to switch off that emulation and hide those virtual PCI headers. Right now we do this in a fairly uncoordinated fashion: I counted at least three different open-coded code sequences with the same effect.
My current plan is: - encapsulate all those code sequences into calls of one or two functions - switch all calls from hide_vpci(some_value_calculated_by_hand) to hide_vpci_dev(struct device *) - replace the central array of hand-calculated values with a per-PCI-device "hidden" property.
This should result in an overall improvement of code readability and a reduction of code size. As a nice side benefit the dts is a lot easier to manipulate.
Questions? Comments?
Regards, Carl-Daniel
Carl-Daniel Hailfinger wrote:
Hi,
since the Geode/CS553x platform emulates PCI headers for chipset/processor devices, we have the ability to switch off that emulation and hide those virtual PCI headers. Right now we do this in a fairly uncoordinated fashion: I counted at least three different open-coded code sequences with the same effect.
My current plan is:
- encapsulate all those code sequences into calls of one or two functions
- switch all calls from hide_vpci(some_value_calculated_by_hand) to
hide_vpci_dev(struct device *)
- replace the central array of hand-calculated values with a
per-PCI-device "hidden" property.
This should result in an overall improvement of code readability and a reduction of code size. As a nice side benefit the dts is a lot easier to manipulate.
I like this idea a lot.
Maybe we could use the already existing "enabled" property for that? Usually if a device is not "enabled", coreboot would not assign resources to it. Instead, if the device driver knows how to disable the device, it could completely disable it. I think the i82801xx driver in v2 attempts a similar thing.
Stefan
On 08.05.2008 16:04, Stefan Reinauer wrote:
Carl-Daniel Hailfinger wrote:
Hi,
since the Geode/CS553x platform emulates PCI headers for chipset/processor devices, we have the ability to switch off that emulation and hide those virtual PCI headers. Right now we do this in a fairly uncoordinated fashion: I counted at least three different open-coded code sequences with the same effect.
My current plan is:
- encapsulate all those code sequences into calls of one or two functions
- switch all calls from hide_vpci(some_value_calculated_by_hand) to
hide_vpci_dev(struct device *)
- replace the central array of hand-calculated values with a
per-PCI-device "hidden" property.
This should result in an overall improvement of code readability and a reduction of code size. As a nice side benefit the dts is a lot easier to manipulate.
I like this idea a lot.
Maybe we could use the already existing "enabled" property for that? Usually if a device is not "enabled", coreboot would not assign resources to it. Instead, if the device driver knows how to disable the device, it could completely disable it. I think the i82801xx driver in v2 attempts a similar thing.
Interesting proposal. If I understand you correctly, enabled=0 would mean - disabled on all chipsets - hidden on those chipsets where hiding is supported.
There are two problems I have with that approach, though: - I don't understand PCI well enough to know how a device is disabled. Does disabling the device mean disabling the resources as well? Does it mean the device won't respond to config cycles outside a special "enable cycle" anymore? - With automatic hiding, we lose the ability to leave a device untouched. Do we need an enum(disabled, hidden, enabled, untouched) for a full representation of possible states?
Regards, Carl-Daniel
you need an enum or you need a new variable. Hiding and enabled are different.
not enable does not imply not visible. So it's 2 bits on way or another :-)
I like the idea. I like the way you guys are pushing for really using the device tree. thanks
ron
ron minnich wrote:
you need an enum or you need a new variable. Hiding and enabled are different.
Why would you not hide a non-enabled device?
not enable does not imply not visible. So it's 2 bits on way or another :-)
In Linux it is, isn't it? I thought to remember it won't show devices with no BARs set up..
On Thu, May 8, 2008 at 7:22 AM, Stefan Reinauer stepan@coresystems.de wrote:
Why would you not hide a non-enabled device?
in some cases we actually have to allocate the BARs and so on and leave it disabled -- we don't know if the OS wants it, but if the OS wants it, it has to work. This is one of the less intuitive bits of the device bringup process.
In Linux it is, isn't it? I thought to remember it won't show devices with no BARs set up..
yep, we actually have to set up the BARs even if it is not enabled in the tree. We can't leave the OS in a situtation where it can't use the device.
thanks
ron
ron minnich wrote:
On Thu, May 8, 2008 at 7:22 AM, Stefan Reinauer stepan@coresystems.de wrote:
Why would you not hide a non-enabled device?
in some cases we actually have to allocate the BARs and so on and leave it disabled -- we don't know if the OS wants it, but if the OS wants it, it has to work. This is one of the less intuitive bits of the device bringup process.
Which device is that?
In Linux it is, isn't it? I thought to remember it won't show devices with no BARs set up..
yep, we actually have to set up the BARs even if it is not enabled in the tree. We can't leave the OS in a situtation where it can't use the device.
What does pci xx.x off end do then?
On Thu, May 8, 2008 at 7:51 AM, Stefan Reinauer stepan@coresystems.de wrote:
Which device is that?
Almost all of them. Consider a bridge.
What does pci xx.x off end do then?
in v2, it means that in the init function, the CMD register is un-enabled (I think; I have not looked at this code for a bit and it always confuses me).
Look in pci_probe_dev for example: device is unconditionally enabled, so we can scan it.
IIRC one case that drove this was multiple VGA cards -- you want to set up the BARs etc. for all but you really don't know what the OS will do -- maybe nothing -- but you want them there if it will do something.
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
ron
ron minnich wrote:
On Thu, May 8, 2008 at 7:51 AM, Stefan Reinauer stepan@coresystems.de wrote:
in some cases we actually have to allocate the BARs and so on and leave it disabled -- we don't know if the OS wants it, but if the OS wants it, it has to work. This is one of the less intuitive bits of the device bringup process.
Which device is that?
Almost all of them. Consider a bridge.
I still think a bridge that gets disabled in Config.lb or the dts should not be set up. Otherwise it might be bridging, which would really not be what we want if we disable the device.
I can't find any bridge in v2 from a quick scan that would stay un-enabled.
If we don't know whether the OS wants the device, we enable it, not leave it or disable it.
We disable or hide the device, if that functionality is not there on the _board_, ie. because it is not wired.
If we start thinking about what the OS wants in this context, we have to call this ACPI and make distinctions between OSes in the DTS. ;-)
What does pci xx.x off end do then?
in v2, it means that in the init function, the CMD register is un-enabled (I think; I have not looked at this code for a bit and it always confuses me).
Look in pci_probe_dev for example: device is unconditionally enabled, so we can scan it.
IIRC one case that drove this was multiple VGA cards -- you want to set up the BARs etc. for all but you really don't know what the OS will do -- maybe nothing -- but you want them there if it will do something.
Unless the bios disables one of the devices, in which case the OS should find nothing.
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
So disabled devices do show up in lspci?
On Thu, May 8, 2008 at 8:14 AM, Stefan Reinauer stepan@coresystems.de wrote:
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
So disabled devices do show up in lspci?
That's what my memory tells me, now I have to find an example :-)
ron
On 08.05.2008 17:37, ron minnich wrote:
On Thu, May 8, 2008 at 8:14 AM, Stefan Reinauer stepan@coresystems.de wrote:
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
So disabled devices do show up in lspci?
That's what my memory tells me, now I have to find an example :-)
IIRC this happens at least on the M57SLI.
Regards, Carl-Daniel
Carl-Daniel Hailfinger wrote:
On 08.05.2008 17:37, ron minnich wrote:
On Thu, May 8, 2008 at 8:14 AM, Stefan Reinauer stepan@coresystems.de wrote:
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
So disabled devices do show up in lspci?
That's what my memory tells me, now I have to find an example :-)
IIRC this happens at least on the M57SLI.
Can you verify?
On Thu, May 08, 2008 at 08:37:40AM -0700, ron minnich wrote:
On Thu, May 8, 2008 at 8:14 AM, Stefan Reinauer stepan@coresystems.de wrote:
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
So disabled devices do show up in lspci?
That's what my memory tells me, now I have to find an example :-)
I guess this depends a bit (not sure on _what_ it depends). I've seen devices show up even if disabled via Config.lb IIRC, but I've also seen a box where I can say 'off' in Config.lb (e.g. for an AC'97 modem) and it will not show up in lspci...
Uwe.
On Thu, May 8, 2008 at 8:58 AM, Uwe Hermann uwe@hermann-uwe.de wrote:
I guess this depends a bit (not sure on _what_ it depends). I've seen devices show up even if disabled via Config.lb IIRC, but I've also seen a box where I can say 'off' in Config.lb (e.g. for an AC'97 modem) and it will not show up in lspci...
PC hardware is defined as much by its exceptions as its rules :-)
ron
On 08.05.2008 17:14, Stefan Reinauer wrote:
ron minnich wrote:
On Thu, May 8, 2008 at 7:51 AM, Stefan Reinauer stepan@coresystems.de wrote:
in some cases we actually have to allocate the BARs and so on and leave it disabled -- we don't know if the OS wants it, but if the OS wants it, it has to work. This is one of the less intuitive bits of the device bringup process.
Which device is that?
Almost all of them. Consider a bridge.
I still think a bridge that gets disabled in Config.lb or the dts should not be set up. Otherwise it might be bridging, which would really not be what we want if we disable the device.
Are devices behind a disabled bridge completely invisible or do their PCI headers show up?
I can't find any bridge in v2 from a quick scan that would stay un-enabled.
If we don't know whether the OS wants the device, we enable it, not leave it or disable it.
We disable or hide the device, if that functionality is not there on the _board_, ie. because it is not wired.
Commerical BIOS sometimes have the ability to disable/hide otherwise functional onboard devices, which is useful if the machine is severely resource constrained in the IRQ area and/or the user doesn't want the OS to drive the device.
If we start thinking about what the OS wants in this context, we have to call this ACPI and make distinctions between OSes in the DTS. ;-)
Mind bleach! Now!
What does pci xx.x off end do then?
in v2, it means that in the init function, the CMD register is un-enabled (I think; I have not looked at this code for a bit and it always confuses me).
Look in pci_probe_dev for example: device is unconditionally enabled, so we can scan it.
IIRC one case that drove this was multiple VGA cards -- you want to set up the BARs etc. for all but you really don't know what the OS will do -- maybe nothing -- but you want them there if it will do something.
Unless the bios disables one of the devices, in which case the OS should find nothing.
What about devices which cause periodic interrupts once they are enabled? Enabling them has the undesirable effect of causing unhandled interrupts which irritate Linux a lot.
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
So disabled devices do show up in lspci?
I could try to dig up a lspci from a mail someone sent last year, but if Uwe has seen this on one of his machines, I'd be glad if he could track it down.
Regards, Carl-Daniel
Ron/Stefan?
I'm not sure if we can perform generic hiding of devices behind a bridge. Could you look at the mail below?
Thanks! Carl-Daniel
On 08.05.2008 18:06, Carl-Daniel Hailfinger wrote:
On 08.05.2008 17:14, Stefan Reinauer wrote:
ron minnich wrote:
On Thu, May 8, 2008 at 7:51 AM, Stefan Reinauer stepan@coresystems.de wrote:
Are devices behind a disabled bridge completely invisible or do their PCI headers show up?
I can't find any bridge in v2 from a quick scan that would stay un-enabled.
If we don't know whether the OS wants the device, we enable it, not leave it or disable it.
We disable or hide the device, if that functionality is not there on the _board_, ie. because it is not wired.
Commerical BIOS sometimes have the ability to disable/hide otherwise functional onboard devices, which is useful if the machine is severely resource constrained in the IRQ area and/or the user doesn't want the OS to drive the device.
If we start thinking about what the OS wants in this context, we have to call this ACPI and make distinctions between OSes in the DTS. ;-)
Mind bleach! Now!
Look in pci_probe_dev for example: device is unconditionally enabled, so we can scan it.
IIRC one case that drove this was multiple VGA cards -- you want to set up the BARs etc. for all but you really don't know what the OS will do -- maybe nothing -- but you want them there if it will do something.
Unless the bios disables one of the devices, in which case the OS should find nothing.
What about devices which cause periodic interrupts once they are enabled? Enabling them has the undesirable effect of causing unhandled interrupts which irritate Linux a lot.
There really is a difference between un-enabled (or disabled) and hidden; as you pointed out, hidden means no config space, or "will not show up in lspci".
So disabled devices do show up in lspci?
I could try to dig up a lspci from a mail someone sent last year, but if Uwe has seen this on one of his machines, I'd be glad if he could track it down.
Carl-Daniel Hailfinger wrote:
Interesting proposal. If I understand you correctly, enabled=0 would mean
- disabled on all chipsets
- hidden on those chipsets where hiding is supported.
There are two problems I have with that approach, though:
- I don't understand PCI well enough to know how a device is disabled.
On a high level:
Disabling the device means we disable its memory and io space.
Hiding the device means we disable its config space.
So with any of the three resource types disabled, the device is not really usable. They're just decoded differently.
Does disabling the device mean disabling the resources as well? Does it mean the device won't respond to config cycles outside a special "enable cycle" anymore?
absolutely. If you have a special disable function, you will also need a special enable function. But that won't happen, as you don't re-enable a device that you disabled on purpose.
- With automatic hiding, we lose the ability to leave a device
untouched. Do we need an enum(disabled, hidden, enabled, untouched) for a full representation of possible states?
hidden implies disabled, since when you disable the config space of the device, the BARs are gone, too.
untouched is not an option we provided at any time before, but basically it implies disabled, too.
On 08.05.2008 16:29, Stefan Reinauer wrote:
Carl-Daniel Hailfinger wrote:
Interesting proposal. If I understand you correctly, enabled=0 would mean
- disabled on all chipsets
- hidden on those chipsets where hiding is supported.
There are two problems I have with that approach, though:
- I don't understand PCI well enough to know how a device is disabled.
On a high level:
Thanks for the explanations!
Disabling the device means we disable its memory and io space.
But it can still do DMA and issue IRQs, right?
Also, does disabling memory and IO space set a special "disabled" flag or do we simply set each resource with a size of zero?
Hiding the device means we disable its config space.
So with any of the three resource types disabled, the device is not really usable. They're just decoded differently.
not usable != disabled.
Does disabling the device mean disabling the resources as well? Does it mean the device won't respond to config cycles outside a special "enable cycle" anymore?
absolutely. If you have a special disable function, you will also need a special enable function. But that won't happen, as you don't re-enable a device that you disabled on purpose.
Do we need special per-device functions for en-/disabling or is this doable with a generic PCI function?
- With automatic hiding, we lose the ability to leave a device
untouched. Do we need an enum(disabled, hidden, enabled, untouched) for a full representation of possible states?
hidden implies disabled, since when you disable the config space of the device, the BARs are gone, too.
Well, if you can hide a device, can it issue IRQs or perform DMA on its own initiative?
untouched is not an option we provided at any time before, but basically it implies disabled, too.
I still think untouched could be useful. For one, it would allow us to minimize setup during early porting and we could do the rest of the setup from a network-loaded payload. That's not neccessary if you have a ROM emulator, but a big plus for those developers without one.
Regards, Carl-Daniel
On Thu, May 8, 2008 at 9:15 AM, Carl-Daniel Hailfinger
Disabling the device means we disable its memory and io space.
But it can still do DMA and issue IRQs, right?
not dma unless bus master is set, and no BIOS should ever set that (but I bet EFI does as it has drivers ...)
Also, does disabling memory and IO space set a special "disabled" flag or do we simply set each resource with a size of zero?
no, what should happen is you allocate the BARs but don't enable it. Many OSes can re-enable a disabled device, but the can't handle BARs with a value of 0.
Hiding the device means we disable its config space.
So with any of the three resource types disabled, the device is not really usable. They're just decoded differently.
not usable != disabled.
if the device is disabled, the OS can re-enable it. I've actually re-enabled disabled bridges via setpci, for example. If the device is hidden, the OS can do nothing to get to it.
absolutely. If you have a special disable function, you will also need a special enable function. But that won't happen, as you don't re-enable a device that you disabled on purpose.
Do we need special per-device functions for en-/disabling or is this doable with a generic PCI function?
Not quite! You can re-enable any disabled device with setpci -- I've done it. But you can't re-enable a hidden device -- pci can't see it.
Well, if you can hide a device, can it issue IRQs or perform DMA on its own initiative?
IRQs -- unclear. That's a little tricky. DMA, no, BUSMASTERing won't be set for one thing.
I think if we add to many states we're going to be in PCI hell. Let's be careful.
untouched is not an option we provided at any time before, but basically it implies disabled, too.
I still think untouched could be useful. For one, it would allow us to minimize setup during early porting and we could do the rest of the setup from a network-loaded payload. That's not neccessary if you have a ROM emulator, but a big plus for those developers without one.
interesting idea but you have to make sure you know the implications of this for bridges, etc. etc.
And if you don't allocate BARs, and the system comes up, you may find there is no way to allocate the address space for the BARs -- there may not be room. This happened on more than one occasion with, e.g., Quadrics.
ron
Carl-Daniel Hailfinger wrote:
My current plan is:
- encapsulate all those code sequences into calls of one or two functions
- switch all calls from hide_vpci(some_value_calculated_by_hand) to
hide_vpci_dev(struct device *)
OK.
- replace the central array of hand-calculated values with a
per-PCI-device "hidden" property.
I think that this will add complications that you don't intend.
There are two reasons to hide a device. 1. Your platform doesn't use the device and so there is no reason for it to hang about. This option was addressed with the array of device location that you noted, but I don't think that anyone used it. (OLPC doesn't count)
2. There is shared hardware and only one device can be present at a time. An example of this is IDE or flash and 5536 USB configuration.
Adding a hidden property could handle the first case but then every device will have a dts to contain a property that isn't used.
In the second case, the logic to hide the device is based on other configurations and the hidden property would either be pointless or confuse the issue of if or when the device should be hidden.
Marc
On Thu, May 08, 2008 at 12:16:08PM -0600, Marc Jones wrote:
Carl-Daniel Hailfinger wrote:
My current plan is:
- encapsulate all those code sequences into calls of one or two functions
- switch all calls from hide_vpci(some_value_calculated_by_hand) to
hide_vpci_dev(struct device *)
OK.
- replace the central array of hand-calculated values with a
per-PCI-device "hidden" property.
I think that this will add complications that you don't intend.
There are two reasons to hide a device.
- Your platform doesn't use the device and so there is no reason for it
to hang about. This option was addressed with the array of device location that you noted, but I don't think that anyone used it. (OLPC doesn't count)
Actually, we use it in v3 on the alix.2c3 to disable the VGA device which has no physical connector present.
Thanks, Ward.
On Thu, May 08, 2008 at 12:16:08PM -0600, Marc Jones wrote:
There are two reasons to hide a device.
Good. I like starting the reasoning from this point of view.
- Your platform doesn't use the device and so there is no reason
for it to hang about.
I think this is common..
This option was addressed with the array of device location that you noted, but I don't think that anyone used it.
What is this refering to?
- There is shared hardware and only one device can be present at a
time. An example of this is IDE or flash
--8<-- 5536 book 5.18 Flash Controller This interface is shared with the IDE interface, using the same balls. NOR or NAND Flash may co-exist with IDE devices using PIO (Programmed I/O) mode. -->8--
This suggests that both devices can be visible and enabled with shared balls.
and 5536 USB configuration.
Port 4 HCI vs. UDC? Are they separate PCI functions?
Adding a hidden property could handle the first case but then every device will have a dts to contain a property that isn't used.
Not all devices will be hideable. Did we make a policy to allow implicit defaults (implemented in code I guess) in device dts files? If not then yes, every device dts needs the hidden=0 setting.
In the second case, the logic to hide the device is based on other configurations and the hidden property would either be pointless or confuse the issue of if or when the device should be hidden.
This is important. We must make dts cope with this type of mutually exclusive configuration. It is simply not good enough for the system to permit an incompatible/impossible configuration.
//Peter
recall that "device hiding" is very special to the geode, and the LX is the last Geode. Be careful about generalizing what is, after all, a special case.
ron
-
Carl-Daniel Hailfinger -
Marc Jones -
Peter Stuge -
ron minnich -
Stefan Reinauer -
Uwe Hermann -
Ward Vandewege