Hello,
I'm investigating the idea of a new QEMU chrp machine derived from mac99. The background is that there are ofmem issues when running Haiku on the g3beige machine, whereas running AIX on mac99 fails with the fix in place that makes Haiku go further. I assume this is due to differences in memory layout so that I need to move OpenBIOS to another location in RAM and anything else that clobbers the memory locations AIX assumes to be able to claim for itself.
Not having (low-level) access to an IBM Power System / System p / pSeries, the best source I could find was a JS20 device tree [1]. (Some older RS/6000 ones for comparison are available at [2].)
Interestingly the nvram is at 0x00000100ff800000 and pci at 0x08000000f8000000, i.e. above 32-bit address space. With sizeof(void*) == 4, drivers/macio.c:nvram is unable to save and access such an address.
ofmem cannot deal with 64-bit virtual/physical addresses either, since it uses 32-bit ucell in translation_t and range_t. The MMU translations on mac99 don't match Apple's OpenFirmware on my G5 size- wise.
What's the plan for fixing this? Will we need an openbios-ppc64 after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/ init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Regards,
Andreas
[1] http://www.ibm.com/developerworks/forums/thread.jspa?threadID=73270 [2] http://www.tibit.org/ppc/
On Thu, Sep 30, 2010 at 5:02 PM, Andreas Färber andreas.faerber@web.de wrote:
Hello,
I'm investigating the idea of a new QEMU chrp machine derived from mac99. The background is that there are ofmem issues when running Haiku on the g3beige machine, whereas running AIX on mac99 fails with the fix in place that makes Haiku go further. I assume this is due to differences in memory layout so that I need to move OpenBIOS to another location in RAM and anything else that clobbers the memory locations AIX assumes to be able to claim for itself.
Not having (low-level) access to an IBM Power System / System p / pSeries, the best source I could find was a JS20 device tree [1]. (Some older RS/6000 ones for comparison are available at [2].)
Interestingly the nvram is at 0x00000100ff800000 and pci at 0x08000000f8000000, i.e. above 32-bit address space. With sizeof(void*) == 4, drivers/macio.c:nvram is unable to save and access such an address.
ofmem cannot deal with 64-bit virtual/physical addresses either, since it uses 32-bit ucell in translation_t and range_t. The MMU translations on mac99 don't match Apple's OpenFirmware on my G5 size-wise.
It looks like the cell size is 32 bits, but both #address-cells and #size-cells is 2.
What's the plan for fixing this? Will we need an openbios-ppc64 after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
If you don't have to implement JS20, are there any other machines where all devices are located inside the first 4G?
There are more trees here: http://penguinppc.org/historical/dev-trees-html/trees-index.html
On 30.09.2010, at 19:02, Andreas Färber wrote:
Hello,
I'm investigating the idea of a new QEMU chrp machine derived from mac99. The background is that there are ofmem issues when running Haiku on the g3beige machine, whereas running AIX on mac99 fails with the fix in place that makes Haiku go further. I assume this is due to differences in memory layout so that I need to move OpenBIOS to another location in RAM and anything else that clobbers the memory locations AIX assumes to be able to claim for itself.
Not having (low-level) access to an IBM Power System / System p / pSeries, the best source I could find was a JS20 device tree [1]. (Some older RS/6000 ones for comparison are available at [2].)
I would certainly vote for going with a YDL PowerStation here. Maybe I can try to get you access on one of these even.
Interestingly the nvram is at 0x00000100ff800000 and pci at 0x08000000f8000000, i.e. above 32-bit address space. With sizeof(void*) == 4, drivers/macio.c:nvram is unable to save and access such an address.
ofmem cannot deal with 64-bit virtual/physical addresses either, since it uses 32-bit ucell in translation_t and range_t. The MMU translations on mac99 don't match Apple's OpenFirmware on my G5 size-wise.
If I understand the U3 and U4 designs correctly, PCI is always bound to be within the first 32 bits. Copying Ben for verification. I don't really remember having seen 64 bit cells on any Power system either, but I could be wrong. Again, let's ask the expert here :).
What's the plan for fixing this? Will we need an openbios-ppc64 after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Those are pure fixups to get back to 32 bit mode from the interrupt handlers, as they go into 64 bit mode by default.
Alex
Regards,
Andreas
[1] http://www.ibm.com/developerworks/forums/thread.jspa?threadID=73270 [2] http://www.tibit.org/ppc/
On Fri, 2010-10-01 at 00:03 +0200, Alexander Graf wrote:
If I understand the U3 and U4 designs correctly, PCI is always bound to be within the first 32 bits. Copying Ben for verification. I don't really remember having seen 64 bit cells on any Power system either, but I could be wrong. Again, let's ask the expert here :).
Yes. However, on js2x, the firmware plays funny remapping tricks...
In general, however, OpenBIOS should be fixed. It's getting more common to have even 32-bit machines with physical devices above 32-bit (even ARM is now growing an equivalent of PAE for that, and almost all 4xx powerpc do that).
What's the plan for fixing this? Will we need an openbios-ppc64
after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Those are pure fixups to get back to 32 bit mode from the interrupt handlers, as they go into 64 bit mode by default.
OpenBIOS should be generally fixed regardless of being 32 or 64-bit for being capable of handling 64-bit physical addresses. As to accessing those from 32-bit mode, for that I suppose it either
- runs in virtual mode exclusively on such machines so it can create special mappings in the MMU (4xx)
- use a little asm stub that switch mode & back on real 64-bit machines to access locations in the 64-bit address space
I'm no fan of the later due to the "issues" with doing I/Os in real mode on many powerpc chips but that's how IBM own firmware works so ..
Cheers, Ben.
Alex
Regards,
Andreas
[1]
http://www.ibm.com/developerworks/forums/thread.jspa?threadID=73270
Am 30.09.2010 um 20:46 schrieb Blue Swirl:
On Thu, Sep 30, 2010 at 5:02 PM, Andreas Färber <andreas.faerber@web.de
wrote: Not having (low-level) access to an IBM Power System / System p / pSeries, the best source I could find was a JS20 device tree [1]. (Some older RS/6000 ones for comparison are available at [2].)
If you don't have to implement JS20, are there any other machines where all devices are located inside the first 4G?
It doesn't need to be a JS20; any POWER4, PPC970, POWER5, POWER6 or POWER7 CHRP system from IBM should do. Maybe Alex has access to one for KVM?
Finding such a chrp system doesn't solve the mac99 issue though. Haiku is happy that on OpenBIOS/ppc64 everything is like on 32-bit since it hadn't been ported to ppc64 yet, so I'd rather have qemu-system-ppc64 for testing the actual differences than hiding them. As you confirmed, it's not a question of cell size but of cell count. I guess I'll have to try using a new type like phys_addr_t to replace ucell in the ofmem backend.
There are more trees here: http://penguinppc.org/historical/dev-trees-html/trees-index.html
Yeah, those were linked from tibit.org, but they're all Mac trees and equally old.
Andreas
On 01.10.2010, at 00:09, Benjamin Herrenschmidt wrote:
On Fri, 2010-10-01 at 00:03 +0200, Alexander Graf wrote:
If I understand the U3 and U4 designs correctly, PCI is always bound to be within the first 32 bits. Copying Ben for verification. I don't really remember having seen 64 bit cells on any Power system either, but I could be wrong. Again, let's ask the expert here :).
Yes. However, on js2x, the firmware plays funny remapping tricks...
In general, however, OpenBIOS should be fixed. It's getting more common to have even 32-bit machines with physical devices above 32-bit (even ARM is now growing an equivalent of PAE for that, and almost all 4xx powerpc do that).
So how does the firmware tell the OS that a device is on a 64 bit physical address if it only knows about 32 bit wide cells?
What's the plan for fixing this? Will we need an openbios-ppc64
after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Those are pure fixups to get back to 32 bit mode from the interrupt handlers, as they go into 64 bit mode by default.
OpenBIOS should be generally fixed regardless of being 32 or 64-bit for being capable of handling 64-bit physical addresses. As to accessing those from 32-bit mode, for that I suppose it either
- runs in virtual mode exclusively on such machines so it can create
special mappings in the MMU (4xx)
- use a little asm stub that switch mode & back on real 64-bit machines
to access locations in the 64-bit address space
I'm no fan of the later due to the "issues" with doing I/Os in real mode on many powerpc chips but that's how IBM own firmware works so ..
Actually we should be able to do the former even for 64-bit Book3S systems. After all, we run in virtual mode always and own the HTAB. And that contains 64 bit addresses.
Alex
On Fri, 2010-10-01 at 00:25 +0200, Alexander Graf wrote:
On 01.10.2010, at 00:09, Benjamin Herrenschmidt wrote:
On Fri, 2010-10-01 at 00:03 +0200, Alexander Graf wrote:
If I understand the U3 and U4 designs correctly, PCI is always bound to be within the first 32 bits. Copying Ben for verification. I don't really remember having seen 64 bit cells on any Power system either, but I could be wrong. Again, let's ask the expert here :).
Yes. However, on js2x, the firmware plays funny remapping tricks...
In general, however, OpenBIOS should be fixed. It's getting more common to have even 32-bit machines with physical devices above 32-bit (even ARM is now growing an equivalent of PAE for that, and almost all 4xx powerpc do that).
So how does the firmware tell the OS that a device is on a 64 bit physical address if it only knows about 32 bit wide cells?
#address-cells & #size-cells indicate the number of cells for an address or a size :-)
What's the plan for fixing this? Will we need an openbios-ppc64
after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Those are pure fixups to get back to 32 bit mode from the interrupt handlers, as they go into 64 bit mode by default.
OpenBIOS should be generally fixed regardless of being 32 or 64-bit for being capable of handling 64-bit physical addresses. As to accessing those from 32-bit mode, for that I suppose it either
- runs in virtual mode exclusively on such machines so it can create
special mappings in the MMU (4xx)
- use a little asm stub that switch mode & back on real 64-bit machines
to access locations in the 64-bit address space
I'm no fan of the later due to the "issues" with doing I/Os in real mode on many powerpc chips but that's how IBM own firmware works so ..
Actually we should be able to do the former even for 64-bit Book3S systems. After all, we run in virtual mode always and own the HTAB. And that contains 64 bit addresses.
Apple OF runs in virtual mode. IBM in real. Dunno if AIX can cope with virt.
Cheers, Ben.
Alex
Am 01.10.2010 um 00:25 schrieb Alexander Graf:
On 01.10.2010, at 00:09, Benjamin Herrenschmidt wrote:
On Fri, 2010-10-01 at 00:03 +0200, Alexander Graf wrote:
If I understand the U3 and U4 designs correctly, PCI is always bound to be within the first 32 bits. Copying Ben for verification. I don't really remember having seen 64 bit cells on any Power system either, but I could be wrong. Again, let's ask the expert here :).
Yes. However, on js2x, the firmware plays funny remapping tricks...
In general, however, OpenBIOS should be fixed. It's getting more common to have even 32-bit machines with physical devices above 32-bit (even ARM is now growing an equivalent of PAE for that, and almost all 4xx powerpc do that).
So how does the firmware tell the OS that a device is on a 64 bit physical address if it only knows about 32 bit wide cells?
By concatenating two 32 bit wide cells in Big Endian order, it seems. :)
Andreas
On 2010-9-30 6:25 PM, Alexander Graf wrote:
So how does the firmware tell the OS that a device is on a 64 bit physical address if it only knows about 32 bit wide cells?
In general, the /memory node's "reg" and "available" properties tell you how big your physical memory addressing is. The root node's ("/") #address-cells and #size-cells tells you how to read those two properties.
If you have a 32-bit virtual addressing range, you'll obviously need some kind of MMU/IOMMU to address anything above ffff.ffff .
For IO, in particular PCI/PCIX/PCIE, you'll see that the root IO node has a ranges property indicating where the physical addressing of the IO chunks are (IO, mem32, mem64).
Oh, and one confusion I occasionally hear about - "cells" strictly speaking refers to the native size of values on the stack (and thus the size operated on with ! and @). The same term "cell" is used for items in a property, but 1275 defines that property cells are explicitly 32-bits. A 64-bit address in a property is by definition two property-cells long.
On Fri, 2010-10-01 at 00:29 +0200, Andreas Färber wrote:
So how does the firmware tell the OS that a device is on a 64 bit physical address if it only knows about 32 bit wide cells?
By concatenating two 32 bit wide cells in Big Endian order, it seems. :)
This is actually all well documented in 1275 :-)
#address-cells defines the number of cells for an address for children of the node containing that property. #size-cells does the same for region "sizes". Thus, "reg" properties size is a multiple of the parent's #address-cells + #size-cells.
Then ranges properties mix the parent and child sizes to defined translations accross domains, etc...
Cheers, Ben
On 01.10.2010, at 00:28, Benjamin Herrenschmidt wrote:
On Fri, 2010-10-01 at 00:25 +0200, Alexander Graf wrote:
On 01.10.2010, at 00:09, Benjamin Herrenschmidt wrote:
On Fri, 2010-10-01 at 00:03 +0200, Alexander Graf wrote:
If I understand the U3 and U4 designs correctly, PCI is always bound to be within the first 32 bits. Copying Ben for verification. I don't really remember having seen 64 bit cells on any Power system either, but I could be wrong. Again, let's ask the expert here :).
Yes. However, on js2x, the firmware plays funny remapping tricks...
In general, however, OpenBIOS should be fixed. It's getting more common to have even 32-bit machines with physical devices above 32-bit (even ARM is now growing an equivalent of PAE for that, and almost all 4xx powerpc do that).
So how does the firmware tell the OS that a device is on a 64 bit physical address if it only knows about 32 bit wide cells?
#address-cells & #size-cells indicate the number of cells for an address or a size :-)
Ah - that makes sense. So I guess cell size really does stay 32 bit on POWER. Very good :).
What's the plan for fixing this? Will we need an openbios-ppc64
after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Those are pure fixups to get back to 32 bit mode from the interrupt handlers, as they go into 64 bit mode by default.
OpenBIOS should be generally fixed regardless of being 32 or 64-bit for being capable of handling 64-bit physical addresses. As to accessing those from 32-bit mode, for that I suppose it either
- runs in virtual mode exclusively on such machines so it can create
special mappings in the MMU (4xx)
- use a little asm stub that switch mode & back on real 64-bit machines
to access locations in the 64-bit address space
I'm no fan of the later due to the "issues" with doing I/Os in real mode on many powerpc chips but that's how IBM own firmware works so ..
Actually we should be able to do the former even for 64-bit Book3S systems. After all, we run in virtual mode always and own the HTAB. And that contains 64 bit addresses.
Apple OF runs in virtual mode. IBM in real. Dunno if AIX can cope with virt.
Phew. What does SLOF run in? Btw - that'd be the alternative: emulate a U4 board and have SLOF run inside.
Alex
Am 01.10.2010 um 00:03 schrieb Alexander Graf:
On 30.09.2010, at 19:02, Andreas Färber wrote:
What's the plan for fixing this? Will we need an openbios-ppc64 after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/ qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Those are pure fixups to get back to 32 bit mode from the interrupt handlers, as they go into 64 bit mode by default.
I'd really hate to see code duplication between an arch/ppc and arch/ ppc64, since we'll want ppc and ppc64 versions of New World Macs and CHRP systems.
The easiest would be if I would find out how to configure a multi-lib ppc/ppc64 gcc-4.4.x, so that we can compile a ppc64 version using `make EXTRACFLAGS=-m64`. Otherwise we'd need to provide a separate cross-ppc64 config file and hack switch-arch to drop the "64" at some point so that it builds in obj-ppc64 but uses code from ppc...
Andreas
On 01.10.2010, at 00:37, Andreas Färber wrote:
Am 01.10.2010 um 00:03 schrieb Alexander Graf:
On 30.09.2010, at 19:02, Andreas Färber wrote:
What's the plan for fixing this? Will we need an openbios-ppc64 after all, or is there a way to fix these issues in openbios-ppc using uint64_t and some assembler magic? The 970 init (arch/ppc/qemu/init.c:ppc64_patch_handlers) apparently does something clever for the interrupt handlers and seems able to use 64-bit ld/std instructions in start.S. That still wouldn't help with ucell based structs and C functions of course...
Those are pure fixups to get back to 32 bit mode from the interrupt handlers, as they go into 64 bit mode by default.
I'd really hate to see code duplication between an arch/ppc and arch/ppc64, since we'll want ppc and ppc64 versions of New World Macs and CHRP systems.
I'm not fully convinced that this would be a bad thing, but ok.
The easiest would be if I would find out how to configure a multi-lib ppc/ppc64 gcc-4.4.x, so that we can compile a ppc64 version using `make EXTRACFLAGS=-m64`. Otherwise we'd need to provide a separate cross-ppc64 config file and hack switch-arch to drop the "64" at some point so that it builds in obj-ppc64 but uses code from ppc...
You don't need to compile 64 bit code to address 64 bit physical addresses. Just map them to 32 bit addresses and you're good from the existing code base. The only thing that needs to change is the physical address variable type of the alloc function. That has to become uint64_t.
Alex
Am 01.10.2010 um 00:35 schrieb Alexander Graf:
On 01.10.2010, at 00:28, Benjamin Herrenschmidt wrote:
Apple OF runs in virtual mode. IBM in real. Dunno if AIX can cope with virt.
So far it could. But we're mostly using identity translations anyway.
OpenBIOS/ppc turns on the MMU without respecting real-mode? option it seems. I was going to prepare a patch the weekend, since RTAS mandates real mode according to the CHRP spec.
Andreas
Am 01.10.2010 um 00:42 schrieb Alexander Graf:
On 01.10.2010, at 00:37, Andreas Färber wrote:
The easiest would be if I would find out how to configure a multi- lib ppc/ppc64 gcc-4.4.x, so that we can compile a ppc64 version using `make EXTRACFLAGS=-m64`. Otherwise we'd need to provide a separate cross-ppc64 config file and hack switch-arch to drop the "64" at some point so that it builds in obj-ppc64 but uses code from ppc...
You don't need to compile 64 bit code to address 64 bit physical addresses. Just map them to 32 bit addresses and you're good from the existing code base.
Not in real mode iiuc. Unless of course we keep switching on and off the MMU, which sounds kind of scary to me.
The idea is that OpenBIOS contains RTAS code (start.S), AIX allocates memory for it, calls rtas-instantiate (methods.c), we copy it there and later AIX might even unmap OpenBIOS while continuing to use RTAS. It might as well change the MMU mappings at that point, so we do need code that works in real mode.
Andreas
The only thing that needs to change is the physical address variable type of the alloc function. That has to become uint64_t.
Alex
-- OpenBIOS http://openbios.org/ Mailinglist: http://lists.openbios.org/mailman/listinfo Free your System - May the Forth be with you
-
Alexander Graf -
Andreas Färber -
Benjamin Herrenschmidt -
Blue Swirl -
Tarl Neustaedter