QEMU emulates two of the three PCI buses found on real hardware because some clients seem to need both and fail with only one present, but OpenBIOS only handles a single PCI bus and initialises and puts in the device tree only one of these: the second one which is where devices are connected and also marks it bus 0. However, clients getting info from the device tree may not know about this and thinking there is only one PCI bus they erroneously use the address of the first bus to access PCI config registers for devices on the second bus which silently fails as these requests will go to the other empty bus emulated and return invalid values. Devices mapped via MMIO still appear to work but they may not be correctly initialised and some cards are not detected because of this.
Until support for multiple PCI buses is implemented add an empty node in the device tree for the uninitialised bus to let clients know about it. This is still not entirely correct as bus-range property does not match real hardware but this fixes detecting PCI devices (such as USB) under MorphOS and may also fix enabling the bus master bit needed with some network cards and allow the workarund for this to be reverted.
Signed-off-by: BALATON Zoltan balaton@eik.bme.hu --- arch/ppc/qemu/tree.fs | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/arch/ppc/qemu/tree.fs b/arch/ppc/qemu/tree.fs index 5b6bbc6..e1708de 100644 --- a/arch/ppc/qemu/tree.fs +++ b/arch/ppc/qemu/tree.fs @@ -50,6 +50,18 @@ new-device h# ff800000 encode-int encode+ " ranges" property finish-device
+\ Add empty node for first pci bus +\ Remove this when pci driver is fixed to handle multiple buses +new-device + " pci" device-name + " pci" device-type + h# f0000000 encode-int 02000000 encode-int encode+ " reg" property + 3 encode-int " #address-cells" property + 2 encode-int " #size-cells" property + 1 encode-int " #interrupt-cells" property + 0 encode-int 0 encode-int encode+ " bus-range" property +finish-device + \ ------------------------------------------------------------- \ /packages \ -------------------------------------------------------------
On 20/11/16 18:04, BALATON Zoltan wrote:
QEMU emulates two of the three PCI buses found on real hardware because some clients seem to need both and fail with only one present, but OpenBIOS only handles a single PCI bus and initialises and puts in the device tree only one of these: the second one which is where devices are connected and also marks it bus 0. However, clients getting info from the device tree may not know about this and thinking there is only one PCI bus they erroneously use the address of the first bus to access PCI config registers for devices on the second bus which silently fails as these requests will go to the other empty bus emulated and return invalid values. Devices mapped via MMIO still appear to work but they may not be correctly initialised and some cards are not detected because of this.
Until support for multiple PCI buses is implemented add an empty node in the device tree for the uninitialised bus to let clients know about it. This is still not entirely correct as bus-range property does not match real hardware but this fixes detecting PCI devices (such as USB) under MorphOS and may also fix enabling the bus master bit needed with some network cards and allow the workarund for this to be reverted.
Signed-off-by: BALATON Zoltan balaton@eik.bme.hu
I've given this a spin around my PPC images and I don't see any regressions so I think this is going in the right direction - I think it may need a couple of minor tweaks though.
Firstly, I now see the following PCI BAR warnings on boot for my old openSUSE 11 boot ISO (openSUSE-11.1-NET-ppc.iso):
SuSE Linux zImage starting: loaded at 02000000-0325cd30 (4000000/0/fff025d0; sp: fffb9d40) uncompressing ELF header done. (00000100 bytes) Allocated 00900c04 bytes for kernel @ 03400000 Leave 00fd16ab bytes for initrd @ 0227ef52 uncompressing kernel done. (005a3590 bytes) entering kernel at 03410000(227ef52/fd16ab/fff025d0) OF stdout device is: /pci@f2000000/mac-io@c/escc@13000/ch-a@13020 command line: memory layout at init: alloc_bottom : 03a15000 alloc_top : 20000000 alloc_top_hi : 20000000 rmo_top : 20000000 ram_top : 20000000 Looking for displays found display : /pci@f2000000/QEMU,VGA@e, opening ... done copying OF device tree ... Building dt strings... Building dt structure... Device tree strings 0x03d16000 -> 0x03d165c0 Device tree struct 0x03d17000 -> 0x03d1a000 Calling quiesce ... returning from prom_init Hello World ! setup_arch: bootmem arch: exit
I/O resource not set for host bridge 0 Memory resource not set for host bridge 0 pci 0001:10:0e.0: BAR 0: can't allocate mem resource [0x0-0xffffff] pci 0001:10:0c.0: BAR 0: can't allocate mem resource [0x0-0x7ffff] pci 0001:10:0f.0: BAR 6: can't allocate mem resource [0x82040000-0x8207ffff] pci 0001:10:0e.0: BAR 6: can't allocate mem resource [0x82010000-0x8201ffff] pci 0001:10:0e.0: BAR 2: can't allocate mem resource [0x0-0xfff] pci 0001:10:0d.0: BAR 0: can't allocate mem resource [0x0-0xff] ohci_hcd 0001:10:0d.0: init 0001:10:0d.0 fail, -16 Moving into tmpfs... done. Integrating /parts/00_lib insmod /modules/squashfs.ko mount: /parts/00_lib: we need a loop device mount: using /dev/loop0 Integrating /parts/01_usr mount: /parts/01_usr: we need a loop device mount: using /dev/loop1
This is compared to the original below:
SuSE Linux zImage starting: loaded at 02000000-0325cd30 (4000000/0/fff025d0; sp: fffb9d40) uncompressing ELF header done. (00000100 bytes) Allocated 00900c04 bytes for kernel @ 03400000 Leave 00fd16ab bytes for initrd @ 0227ef52 uncompressing kernel done. (005a3590 bytes) entering kernel at 03410000(227ef52/fd16ab/fff025d0) OF stdout device is: /pci@f2000000/mac-io@c/escc@13000/ch-a@13020 command line: memory layout at init: alloc_bottom : 03a15000 alloc_top : 20000000 alloc_top_hi : 20000000 rmo_top : 20000000 ram_top : 20000000 Looking for displays found display : /pci@f2000000/QEMU,VGA@e, opening ... done copying OF device tree ... Building dt strings... Building dt structure... Device tree strings 0x03d16000 -> 0x03d165c0 Device tree struct 0x03d17000 -> 0x03d1a000 Calling quiesce ... returning from prom_init Hello World ! setup_arch: bootmem arch: exit
Moving into tmpfs... done. Integrating /parts/00_lib insmod /modules/squashfs.ko mount: /parts/00_lib: we need a loop device mount: using /dev/loop0 Integrating /parts/01_usr mount: /parts/01_usr: we need a loop device mount: using /dev/loop1
So perhaps you might need to fake up a few extra properties to get rid of the warnings on boot?
Secondly while the address of the extra PCI node looks fine for -M mac99 in the device tree, I'm not sure it is correct for -M g3beige:
Welcome to OpenBIOS v1.1 built on Dec 31 2016 16:18
0 > show-devs fff47c54 / fff47d70 /aliases fff47e14 /openprom (BootROM) fff4dec0 /openprom/client-services fff47fbc /options fff48034 /chosen fff480e4 /builtin fff48188 /builtin/console fff4dbcc /packages fff4edb0 /packages/cmdline fff4eef0 /packages/disk-label fff50920 /packages/terminal-emulator fff51884 /packages/deblocker fff51bd8 /packages/hfsplus-files fff51e78 /packages/hfs-files fff52114 /packages/ext2-files fff5234c /packages/iso9660-files fff52584 /packages/grubfs-files fff527bc /packages/mac-parts fff52a34 /packages/pc-parts fff52c68 /packages/xcoff-loader fff52d34 /packages/elf-loader fff52dfc /packages/bootinfo-loader fff50508 /cpus fff568f8 /cpus/PowerPC,750@0 (cpu) fff50608 /memory@0 (memory) fff506d0 /rom@ff800000 fff507a8 /pci@f0000000 (pci) fff52ec8 /pci@80000000 (pci) fff532f8 /pci@80000000/QEMU,VGA@1 (display) fff53d34 /pci@80000000/NE2000@2 (network) fff540f0 /pci@80000000/mac-io@3 (mac-io) fff5458c /pci@80000000/mac-io@3/via-cuda@16000 (via-cuda) fff547a8 /pci@80000000/mac-io@3/via-cuda@16000/adb (adb) fff54910 /pci@80000000/mac-io@3/via-cuda@16000/adb/keyboard@8 (keyboard) fff54aec /pci@80000000/mac-io@3/via-cuda@16000/adb/mouse@9 (mouse) fff54c78 /pci@80000000/mac-io@3/via-cuda@16000/rtc (rtc) fff54e1c /pci@80000000/mac-io@3/via-cuda@16000/power-mgt (power-mgt) fff55014 /pci@80000000/mac-io@3/nvram@60000 (nvram) fff55234 /pci@80000000/mac-io@3/escc@13000 (escc) fff55354 /pci@80000000/mac-io@3/escc@13000/ch-a@13020 (serial) fff55628 /pci@80000000/mac-io@3/escc@13000/ch-b@13000 (serial) fff558d0 /pci@80000000/mac-io@3/escc-legacy@12000 (escc-legacy) fff559f4 /pci@80000000/mac-io@3/escc-legacy@12000/ch-a@12002 (serial) fff55c50 /pci@80000000/mac-io@3/escc-legacy@12000/ch-b@12000 (serial) fff55e80 /pci@80000000/mac-io@3/ata-3@20000 (ata) fff56118 /pci@80000000/mac-io@3/ata-3@21000 (ata) fff563b0 /pci@80000000/mac-io@3/ata-3@21000/cdrom@0 (block)
Perhaps the extra /pci node needs to be wrapped in an [IFDEF] block or similar so that it only appears for -M mac99?
ATB,
Mark.
On Sat, 31 Dec 2016, Mark Cave-Ayland wrote:
I've given this a spin around my PPC images and I don't see any regressions so I think this is going in the right direction - I think it
Good to hear. Have you also tested if it makes a difference with irq-s on mac99? I've found that USB irq-s did not work correctly without this patch so it may help with problems with that (e.g. sound and mouse problems) and likely makes the patch in OpenBIOS to set the bus master for RTL8139 unnecessary as well but I haven't tested these.
may need a couple of minor tweaks though.
Firstly, I now see the following PCI BAR warnings on boot for my old openSUSE 11 boot ISO (openSUSE-11.1-NET-ppc.iso):
[...]
I/O resource not set for host bridge 0 Memory resource not set for host bridge 0 pci 0001:10:0e.0: BAR 0: can't allocate mem resource [0x0-0xffffff] pci 0001:10:0c.0: BAR 0: can't allocate mem resource [0x0-0x7ffff] pci 0001:10:0f.0: BAR 6: can't allocate mem resource [0x82040000-0x8207ffff] pci 0001:10:0e.0: BAR 6: can't allocate mem resource [0x82010000-0x8201ffff] pci 0001:10:0e.0: BAR 2: can't allocate mem resource [0x0-0xfff] pci 0001:10:0d.0: BAR 0: can't allocate mem resource [0x0-0xff] ohci_hcd 0001:10:0d.0: init 0001:10:0d.0 fail, -16
Maybe these were silently failing before but I'm not sure what it is. It's hard to see these messages on normal boot as it seems to be printing dark gray text on black. Is there some way to get output on serial and enable more detailed logs? I'm not familiar with openSUSE on PPC or yaboot.
I've also noticed this color problem with Finnix and wonder how this shows up on real Macs. I thought Macs have a light background by default but this happens after the boot loader has cleared the screen and set up its own colors so maybe it's not the default background but there's a problem somewhere later. (After all the OpenBIOS background is also light but it's cleared black before the text is printed.)
So perhaps you might need to fake up a few extra properties to get rid of the warnings on boot?
I'm not sure. Maybe only the first two warnings for bridge 0 could be eliminated by that but the other warnings refer to cards on the second bus and not the one my patch touches. Maybe the patch just uncovers problems with the other bus that was hidden so far because these requests were previously sent to the first bus and got empty replies earlier before it could get to the place where the warnings are issued. These warnings may show that the cards are actually tried to be initialised now but there are some problems. These may need to be fixed somewhere else on the other bus not in the empty PCI bus my patch adds as that does not have these cards at all.
Secondly while the address of the extra PCI node looks fine for -M mac99 in the device tree, I'm not sure it is correct for -M g3beige:
[...]
Perhaps the extra /pci node needs to be wrapped in an [IFDEF] block or similar so that it only appears for -M mac99?
It's likely wrong for g3beige but not sure an [IFDEF] would be enough to make it only appear on mac99. [IFDEF]s seem to test CONFIG_* options but mac99 is a runtime parameter so we would need a way to detect if we're running on mac99 or g3beige (the same openbios-elf.ppc compiled with the same CONFIG_* options can run on both so I can't use [IFDEF] for that). We would need to test the value of machine_id from arch/ppc/qemu/init.c for this but I'm not sure how to do that from Forth. Maybe moving the patch from Forth to C would be easier.
But instead of that we could also aim for a more complete solution and consider doing it similar to spapr and SLOF and generate an fdt in QEMU and pass it to OpenBIOS and patch/amend it as needed after converting to a usual device tree (we could use the code from SLOF for this). I've checked how it's done with spapr and there a call to the hypervisor is used to get the fdt. We don't have that but we already use FWCFG so we could use FW_CFG_SETUP_* to pass the fdt. I think this is doable but I'm still missing some details and haven't tried to implement it. What do you think about such a solution? Basiacally we would need to add creating the fdt and making it available via FWCFG to the mac machines in QEMU, then implement getting it and using it in OpenBIOS. This way we could also solve the problem of keeping OpenBIOS and QEMU in sync because a change in the fdt could be made entirely in QEMU without needing a corresponding change in OpenBIOS. Also adding new machines could be easier. This is a bigger task but maybe worth doing if it could make the current code cleaner.
Regards, BALATON Zoltan
On 01/01/17 22:22, BALATON Zoltan wrote:
On Sat, 31 Dec 2016, Mark Cave-Ayland wrote:
I've given this a spin around my PPC images and I don't see any regressions so I think this is going in the right direction - I think it
Good to hear. Have you also tested if it makes a difference with irq-s on mac99? I've found that USB irq-s did not work correctly without this patch so it may help with problems with that (e.g. sound and mouse problems) and likely makes the patch in OpenBIOS to set the bus master for RTL8139 unnecessary as well but I haven't tested these.
As USB hasn't previously been working that well on mac99, I must confess that it's not something I routinely test. If you think that adding a USB mouse/keyboard is enough then I can include that as part of my tests?
may need a couple of minor tweaks though.
Firstly, I now see the following PCI BAR warnings on boot for my old openSUSE 11 boot ISO (openSUSE-11.1-NET-ppc.iso):
[...]
I/O resource not set for host bridge 0 Memory resource not set for host bridge 0 pci 0001:10:0e.0: BAR 0: can't allocate mem resource [0x0-0xffffff] pci 0001:10:0c.0: BAR 0: can't allocate mem resource [0x0-0x7ffff] pci 0001:10:0f.0: BAR 6: can't allocate mem resource [0x82040000-0x8207ffff] pci 0001:10:0e.0: BAR 6: can't allocate mem resource [0x82010000-0x8201ffff] pci 0001:10:0e.0: BAR 2: can't allocate mem resource [0x0-0xfff] pci 0001:10:0d.0: BAR 0: can't allocate mem resource [0x0-0xff] ohci_hcd 0001:10:0d.0: init 0001:10:0d.0 fail, -16
Maybe these were silently failing before but I'm not sure what it is. It's hard to see these messages on normal boot as it seems to be printing dark gray text on black. Is there some way to get output on serial and enable more detailed logs? I'm not familiar with openSUSE on PPC or yaboot.
Sure, just fire up QEMU with -nographic and you can then grab the output from a standard text terminal (or at least that worked for me here).
I've also noticed this color problem with Finnix and wonder how this shows up on real Macs. I thought Macs have a light background by default but this happens after the boot loader has cleared the screen and set up its own colors so maybe it's not the default background but there's a problem somewhere later. (After all the OpenBIOS background is also light but it's cleared black before the text is printed.)
I did look at this a few years back when I did the big display rewrite. My last guess was that the writes to the VGA colour registers in vga.fs should be forced LE, however when I traced it through to QEMU the values seemed to come through correctly, and the defaults were being overridden from the bootloader.
If you are interested to take a further look at this, I can point you towards the relevant sections of code. I'm fairly sure that once the cause is found the fix should be reasonably trivial.
So perhaps you might need to fake up a few extra properties to get rid of the warnings on boot?
I'm not sure. Maybe only the first two warnings for bridge 0 could be eliminated by that but the other warnings refer to cards on the second bus and not the one my patch touches. Maybe the patch just uncovers problems with the other bus that was hidden so far because these requests were previously sent to the first bus and got empty replies earlier before it could get to the place where the warnings are issued. These warnings may show that the cards are actually tried to be initialised now but there are some problems. These may need to be fixed somewhere else on the other bus not in the empty PCI bus my patch adds as that does not have these cards at all.
Yes you could be right here. The one thing I did think was strange was that both the original and new /pci nodes have a bus range of 0 which could lead to some confusion?
Secondly while the address of the extra PCI node looks fine for -M mac99 in the device tree, I'm not sure it is correct for -M g3beige:
[...]
Perhaps the extra /pci node needs to be wrapped in an [IFDEF] block or similar so that it only appears for -M mac99?
It's likely wrong for g3beige but not sure an [IFDEF] would be enough to make it only appear on mac99. [IFDEF]s seem to test CONFIG_* options but mac99 is a runtime parameter so we would need a way to detect if we're running on mac99 or g3beige (the same openbios-elf.ppc compiled with the same CONFIG_* options can run on both so I can't use [IFDEF] for that). We would need to test the value of machine_id from arch/ppc/qemu/init.c for this but I'm not sure how to do that from Forth. Maybe moving the patch from Forth to C would be easier.
Probably the simplest option here is to hook is_newworld() into Forth using bind_func() as "is-newworld"?
But instead of that we could also aim for a more complete solution and consider doing it similar to spapr and SLOF and generate an fdt in QEMU and pass it to OpenBIOS and patch/amend it as needed after converting to a usual device tree (we could use the code from SLOF for this). I've checked how it's done with spapr and there a call to the hypervisor is used to get the fdt. We don't have that but we already use FWCFG so we could use FW_CFG_SETUP_* to pass the fdt. I think this is doable but I'm still missing some details and haven't tried to implement it. What do you think about such a solution? Basiacally we would need to add creating the fdt and making it available via FWCFG to the mac machines in QEMU, then implement getting it and using it in OpenBIOS. This way we could also solve the problem of keeping OpenBIOS and QEMU in sync because a change in the fdt could be made entirely in QEMU without needing a corresponding change in OpenBIOS. Also adding new machines could be easier. This is a bigger task but maybe worth doing if it could make the current code cleaner.
There was a recent thread about this with Thomas and my takeaway from this was that since the DT binds many functions (and not just properties) into the DT that it's not clear this would be much of a win. Feel free to pick up the thread at https://www.coreboot.org/pipermail/openbios/2016-November/009846.html if you have any further thoughts.
ATB,
Mark.
On Mon, 2 Jan 2017, Mark Cave-Ayland wrote:
As USB hasn't previously been working that well on mac99, I must confess that it's not something I routinely test. If you think that adding a USB mouse/keyboard is enough then I can include that as part of my tests?
I'm not sure USB is completely fixed yet (as the error about init-ing ohci_hcd in your SUSE test suggests) but the goal of my patch was to make USB mouse work and found that previously the USB hardware was not initialised correctly because it was trying to poke PCI config registers on the other empty bus address and failing as these did not get to the emulated card. Adding both bus to the device tree made the driver find the card and USB keyboard and mouse started working in MorphOS. Linux may be missing some more and MacOS seemed to work even without this patch.
Yes you could be right here. The one thing I did think was strange was that both the original and new /pci nodes have a bus range of 0 which could lead to some confusion?
Maybe but I don't know how these properties are really used. Also the first bus may not be completely emulated by QEMU so I've only added enough of the device tree node so that the OS knows there is a (possibly non-working) bus there and then goes on to the second bus with the devices. (Otherwise it might try to set up AGP at the first bus and I'm not sure that is emulated by QEMU yet.)
Probably the simplest option here is to hook is_newworld() into Forth using bind_func() as "is-newworld"?
That looks ugly just for using it at one place. It looks better to me to move the patch to C instead. Ideally we should have less Forth-C bindings not more I think.
There was a recent thread about this with Thomas and my takeaway from this was that since the DT binds many functions (and not just properties) into the DT that it's not clear this would be much of a win. Feel free to pick up the thread at https://www.coreboot.org/pipermail/openbios/2016-November/009846.html if you have any further thoughts.
I have saved that message for replying it later, I'll try to summarise my thoughts on this. Of course the idea is only to provide a skeleton fdt from QEMU describing the devices emulated, then OpenBIOS would convert it to its own format and add funtions, nodes or other details as needed. The win would be that this tree would always be in sync with what QEMU emulates without needing to change OpenBIOS whenever something in QEMU is changed. (Eventually we could also get rid of the several FW_CFG and hard coded values in OpenBIOS as these could be added by QEMU matching what's emulated so it also simplifies OpenBIOS.)
Regards, BALATON Zoltan
-
BALATON Zoltan -
Mark Cave-Ayland