I'm trying to solve the last bigger outstanding problem for MorphOS to run with OpenBIOS. This is that it modifies exception vectors without first turning the MMU off (probably expecting it to be off as happens on real hardware). OpenBIOS on the other hand turns the MMU on and needs this for client interface calls to work so calling the client boot code with MMU off results in a hang. There is disabled code in the quiesce method to clear the MMU bits but this is not sufficient/correct because MorphOS continues to call client interface callbacks after calling quiesce. Thus I was trying to save MSR, turn MMU on then restore MSR on client callbacks with the patch below but it does not work and leads to writes to wrong addresses and a crash on restore. (Probably the stack location is wrong after the patch?) Can anyone tell why it's not working and what could be done instead?
Regards, BALATON Zoltan
diff --git a/openbios-devel/arch/ppc/qemu/start.S b/openbios-devel/arch/ppc/qemu index ae2fd53..384a8a6 100644 --- a/openbios-devel/arch/ppc/qemu/start.S +++ b/openbios-devel/arch/ppc/qemu/start.S @@ -515,13 +515,13 @@ _GLOBAL(call_elf): LOAD_REG_IMMEDIATE(r5, of_client_callback) // r5 = callback li r6,0 // r6 = address of client program argume li r7,0 // r7 = length of client program argumen - li r0,MSR_FP | MSR_ME | MSR_DR | MSR_IR + li r0,MSR_FP | MSR_ME MTMSRD(r0) blrl
#ifdef CONFIG_PPC64 /* Restore SF bit */ - LOAD_REG_IMMEDIATE(r0, MSR_SF | MSR_FP | MSR_ME | MSR_DR | MSR_IR) + LOAD_REG_IMMEDIATE(r0, MSR_SF | MSR_FP | MSR_ME) MTMSRD(r0) #endif LOAD_REG_IMMEDIATE(r8, saved_stack) // restore stack pointer @@ -535,10 +535,10 @@ _GLOBAL(call_elf):
#ifdef __powerpc64__ #define STKOFF STACKFRAME_MINSIZE -#define SAVE_SPACE 320 +#define SAVE_SPACE 328 #else #define STKOFF 8 -#define SAVE_SPACE 144 +#define SAVE_SPACE 148 #endif GLOBL(of_client_callback):
@@ -607,6 +607,10 @@ GLOBL(of_client_callback): PPC_STL r29, (STKOFF + 30 * ULONG_SIZE)(r1) PPC_STL r30, (STKOFF + 31 * ULONG_SIZE)(r1) PPC_STL r31, (STKOFF + 32 * ULONG_SIZE)(r1) + mfmsr r2 + PPC_STL r2, (STKOFF + 33 * ULONG_SIZE)(r1) + ori r2, r2, (MSR_DR | MSR_IR) + mtmsr r2
#ifdef CONFIG_PPC64 LOAD_REG_IMMEDIATE(r2, of_client_interface) @@ -643,6 +647,8 @@ GLOBL(of_client_callback): PPC_LL r29, (STKOFF + 30 * ULONG_SIZE)(r1) PPC_LL r30, (STKOFF + 31 * ULONG_SIZE)(r1) PPC_LL r31, (STKOFF + 32 * ULONG_SIZE)(r1) + PPC_LL r2, (STKOFF + 33 * ULONG_SIZE)(r1) + mtmsr r2
/* restore ctr, cr and xer */
This results in:
Unassigned mem write 00000000100fff6c = 0x0 Unassigned mem write 00000000100fff74 = 0xfde7e90 Unassigned mem write 00000000100fff78 = 0x0 Unassigned mem write 00000000100fff7c = 0x4311d0 Unassigned mem write 00000000100fff80 = 0x0 Unassigned mem write 00000000100fff84 = 0x22000042 Unassigned mem write 00000000100fff88 = 0x20000000 Unassigned mem write 00000000100fff8c = 0x0 Unassigned mem write 00000000100fff90 = 0x8 Unassigned mem write 00000000100fff94 = 0x2 Unassigned mem write 00000000100fff98 = 0x0 Unassigned mem write 00000000100fff9c = 0x680000 Unassigned mem write 00000000100fffa0 = 0xfde7f98 Unassigned mem write 00000000100fffa4 = 0xfdf7e70 Unassigned mem write 00000000100fffa8 = 0x22000042 Unassigned mem write 00000000100fffac = 0x0 Unassigned mem write 00000000100fffb0 = 0xfff32687 Unassigned mem write 00000000100fffb4 = 0xfde7f60 Unassigned mem write 00000000100fffb8 = 0xfde7f20 Unassigned mem write 00000000100fffbc = 0x0 Unassigned mem write 00000000100fffc0 = 0xfffb0000 Unassigned mem write 00000000100fffc4 = 0x688000 Unassigned mem write 00000000100fffc8 = 0x3000 Unassigned mem write 00000000100fffcc = 0xfffb601c Unassigned mem write 00000000100fffd0 = 0x0 Unassigned mem write 00000000100fffd4 = 0xfffb601c Unassigned mem write 00000000100fffd8 = 0xfffb0000 Unassigned mem write 00000000100fffdc = 0x0 Unassigned mem write 00000000100fffe0 = 0xfff02620 Unassigned mem write 00000000100fffe4 = 0xfde7f60 Unassigned mem write 00000000100fffe8 = 0xfde7f20 Unassigned mem write 00000000100fffec = 0x688000 Unassigned mem write 00000000100ffff0 = 0xfffb0000 Unassigned mem write 00000000100ffff4 = 0x67bd80 Unassigned mem write 00000000100ffff8 = 0x3000 Unassigned mem read 0000000060000000 qemu: fatal: Trying to execute code outside RAM or ROM at 0x60000000
NIP 60000000 LR 60000000 CTR 00000000 XER 00000000 MSR 00000000 HID0 00000000 HF 00000000 idx 1 TB 00000000 836520711 DECR 3458446955 GPR00 0000000000000000 0000000060000000 0000000000000000 0000000000000000 GPR04 000000004bfffffc 0000000000000000 0000000000000000 0000000000000000 GPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000 CR 00000000 [ - - - - - - - - ] RES ffffffff FPR00 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR04 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000 FPSCR 00000000 SRR0 fff0db1c SRR1 00003030 PVR 000c0209 VRSAVE 00000000 SPRG0 0fe00000 SPRG1 ffffff6c SPRG2 22000042 SPRG3 00000000 SPRG4 00000000 SPRG5 00000000 SPRG6 00000000 SPRG7 00000000 SDR1 0fe00000 DAR ffffff3c DSISR 42000000
end of dissassembly before the error:
0xfff02750: lwz r31,136(r1) 0xfff02754: lwz r2,140(r1) 0xfff02758: mtmsr r2
IN: 0xfff0275c: lwz r2,20(r1) 0xfff02760: mtctr r2 0xfff02764: lwz r2,24(r1) 0xfff02768: mtcr r2 0xfff0276c: lwz r2,28(r1) 0xfff02770: mtxer r2 0xfff02774: lwz r2,12(r1) 0xfff02778: lwz r0,16(r1) 0xfff0277c: lwz r1,8(r1) 0xfff02780: lwz r4,4(r1) 0xfff02784: mtlr r4 0xfff02788: lwz r4,8(r1) 0xfff0278c: lwz r1,0(r1) 0xfff02790: blr
qemu: fatal: Trying to execute code outside RAM or ROM at 0x60000000
On 06/06/14 01:55, BALATON Zoltan wrote:
I'm trying to solve the last bigger outstanding problem for MorphOS to run with OpenBIOS. This is that it modifies exception vectors without first turning the MMU off (probably expecting it to be off as happens on real hardware). OpenBIOS on the other hand turns the MMU on and needs this for client interface calls to work so calling the client boot code with MMU off results in a hang. There is disabled code in the quiesce method to clear the MMU bits but this is not sufficient/correct because MorphOS continues to call client interface callbacks after calling quiesce. Thus I was trying to save MSR, turn MMU on then restore MSR on client callbacks with the patch below but it does not work and leads to writes to wrong addresses and a crash on restore. (Probably the stack location is wrong after the patch?) Can anyone tell why it's not working and what could be done instead?
I really don't think that this is the correct way to handle this at all. Firstly I believe that real-mode? is set to false by default on real hardware, e.g. from articles like http://www.dialectronics.com/Words/OF_Part_III.shtml.
Secondly, for context I quote from one of your earlier emails:
<quote> MorphOS does not set a callback but seems to call OF functions before starting to take over memory management and it probably only calls exit after that which may work according to the above document. During MMU take over what happens though is this: First MorphOS copies some code to 0x0000-0x1fff without first disabling the memory management interrupts. (Maybe it expects them to be off until enabled.) The code it copies is not correct yet as it has jumps pointing to somewhere near the end of ROM area. Then it fixes up these jumps with addresses pointing to its current location at its load address (it may change this back later again but I have not got there yet). Then it enables the MMU bits in the MSR. Surely enough on QEMU it hits a DSI exception just after it copied its vectors but before it fixed them up which leads to a jump to somewhere in ROM and eventually hits an invalid opcode again. This does not happen if I disable the MMU bits in MSR when the vectors are first copied and let MorphOS enable them later after it fixed the vectors up. I think this either again just works by chance on real hardware or the OF implementations there disable these bits when calling the boot loader and do not need working MMU for client functions. It could be verified by checking the value of MSR on real hardware but I have no access to any. In either case if MorphOS had not assumed that no MMU exception happens while the vectors are wrong but explicitely disabled MMU bits before touching the vectors it would work. The exception happening here during MMU take over is not caused by openbios but by accessing memory by the boot code but openbios needs working MMU for client functions earlier so it cannot disable these bits itself before calling the bootloader unless it could enable during callbacks. So I see no easy way to fix this within openbios. </quote>
It seems to me that if real-mode? is set to false on real hardware, then your hypothesis that MorphOS expects MMU interrupts to be disabled can't be true; as soon as your hit an MMU fault you have to service it. Now I can think of two possible things here:
1) Incorrect/not enough state saving on CIF entry/exit
Does MorphOS call CIF between updating the trap table and fixing it up? This was similar to a recent bug I had with SPARC64, in that I had to modify the CIF entry/exit to save more processor state to prevent traps. This is because *BSDs assume they could call CIF at a similar point before final takeover without realising that a window fill/spill trap could occur between these two points.
2) Trap table is copied from OpenFirmware directly
Maybe the MorphOS developers have found the address of the default trap table in a real OF ROM and copy those values into the real vectors as a starting point? Then they can just fix up the values into their executable as required while during the transition any "unfixed" traps would be handled by the PROM as before? Can you provide more details as to exactly which addresses are copied from, and whether this is done on a "per vector" basis or whether the entire 0x0000-0x1fff block is copied as a single chunk and from where? I think this may be the more likely scenario.
ATB,
Mark.
On Fri, 6 Jun 2014, Mark Cave-Ayland wrote:
On 06/06/14 01:55, BALATON Zoltan wrote: I really don't think that this is the correct way to handle this at all. Firstly I believe that real-mode? is set to false by default on real hardware, e.g. from articles like http://www.dialectronics.com/Words/OF_Part_III.shtml.
From the device trees I've seen this seems to be the case but that does
not mean that the client code is also run with enabled MMU. OF could enable it only during callbacks as my patch tried. Without real hardware I can't find out though.
Secondly, for context I quote from one of your earlier emails:
<quote> MorphOS does not set a callback but seems to call OF functions before starting to take over memory management and it probably only calls exit after that which may work according to the above document. During MMU take over what happens though is this: First MorphOS copies some code to 0x0000-0x1fff without first disabling the memory management interrupts. (Maybe it expects them to be off until enabled.) The code it copies is not correct yet as it has jumps pointing to somewhere near the end of ROM area. Then it fixes up these jumps with addresses pointing to its current location at its load address (it may change this back later again but I have not got there yet). Then it enables the MMU bits in the MSR. Surely enough on QEMU it hits a DSI exception just after it copied its vectors but before it fixed them up which leads to a jump to somewhere in ROM and eventually hits an invalid opcode again. This does not happen if I disable the MMU bits in MSR when the vectors are first copied and let MorphOS enable them later after it fixed the vectors up. I think this either again just works by chance on real hardware or the OF implementations there disable these bits when calling the boot loader and do not need working MMU for client functions. It could be verified by checking the value of MSR on real hardware but I have no access to any. In either case if MorphOS had not assumed that no MMU exception happens while the vectors are wrong but explicitely disabled MMU bits before touching the vectors it would work. The exception happening here during MMU take over is not caused by openbios but by accessing memory by the boot code but openbios needs working MMU for client functions earlier so it cannot disable these bits itself before calling the bootloader unless it could enable during callbacks. So I see no easy way to fix this within openbios. </quote>
It seems to me that if real-mode? is set to false on real hardware, then your hypothesis that MorphOS expects MMU interrupts to be disabled can't be true;
Except if OF does what I said above. Either that or on real hardware no MMU interrupt is generated while overwriting the vectors until they are correct again for some other reason. (Like because page zero is alredy mapped unlike with OpenBIOS.)
as soon as your hit an MMU fault you have to service it. Now I can think of two possible things here:
- Incorrect/not enough state saving on CIF entry/exit
Does MorphOS call CIF between updating the trap table and fixing it up? This
No it doesn't. All CIF calls are before it touches the vectors and they work all right as long as MMU is enabled.
was similar to a recent bug I had with SPARC64, in that I had to modify the CIF entry/exit to save more processor state to prevent traps. This is because *BSDs assume they could call CIF at a similar point before final takeover without realising that a window fill/spill trap could occur between these two points.
So this suggests those traps are probably not happening on real hardware. Maybe because the MMU is off while client code is running?
- Trap table is copied from OpenFirmware directly
Maybe the MorphOS developers have found the address of the default trap table in a real OF ROM and copy those values into the real vectors as a starting point? Then they can just fix up the values into their executable as required while during the transition any "unfixed" traps would be handled by the PROM as before?
I don't think this is what happens (it would also break when the OF ROM is updated so I can't imagine they'd do that). Instead the addresses they copy first is where the final vectors will be after they set up their memory map. But before that they use the vectors from the loaded boot executable and this is what the fixup does. (I've said above the first vectors point to somewhere near the end of the ROM area but this is later replaced with their routines after the MMU is taken over. It just happens to be OpenBIOS ROM at this point.)
Can you provide more details as to exactly which addresses are copied from, and whether this is done on a "per vector" basis or whether the entire 0x0000-0x1fff block is copied as a single chunk and from where? I think this may be the more likely scenario.
The entire region is copied with default addresses pointing to the final location of exception handlers but at the time of copying these are not correct yet and point into the OF ROM. These are then immediately replaced by addresses pointing to the current location of the handlers in the boot code but on QEMU a DSI is happening during this which jumps off to the wrong address. If I disable the MMU at the time when the first vectors are copied it works. No CIF calls are made after this point but they are made after quiesce and before setting up the vectors so diabling MMU at call_elf or quiesce does not work.
There is some disassembly of the part replacing the vectors at the end of this message and how it fails when getting a DSI during it.
Regards, BALATON Zoltan
IN: 0x0040091c: rlwinm r0,r26,0,0,19 0x00400920: lis r3,104 0x00400924: lis r4,66 0x00400928: stw r0,40(r2) 0x0040092c: addi r3,r3,-18952 0x00400930: addi r4,r4,4068 0x00400934: li r5,8192 0x00400938: bl 0x41f004
IN: 0x0041f004: stwu r1,-32(r1) 0x0041f008: mflr r0 0x0041f00c: stw r29,20(r1) 0x0041f010: stw r30,24(r1) 0x0041f014: stw r31,28(r1) 0x0041f018: stw r0,36(r1) 0x0041f01c: mr r29,r3 0x0041f020: mr r30,r4 0x0041f024: mr r31,r5 0x0041f028: li r3,0 0x0041f02c: crclr 4*cr1+eq 0x0041f030: bl 0x441bb0
IN: 0x00441bb0: rlwinm. r0,r5,27,5,31 0x00441bb4: mr r11,r3 0x00441bb8: mtctr r0 0x00441bbc: beq- 0x441c14
IN: 0x00441bc0: rlwinm r0,r5,0,0,26 0x00441bc4: subf r5,r0,r5 0x00441bc8: lwz r0,0(r4) 0x00441bcc: stw r0,0(r3) 0x00441bd0: lwz r9,4(r4) 0x00441bd4: stw r9,4(r3) 0x00441bd8: lwz r0,8(r4) 0x00441bdc: stw r0,8(r3) 0x00441be0: lwz r9,12(r4) 0x00441be4: stw r9,12(r3) 0x00441be8: lwz r0,16(r4) 0x00441bec: stw r0,16(r3) 0x00441bf0: lwz r9,20(r4) 0x00441bf4: stw r9,20(r3) 0x00441bf8: lwz r0,24(r4) 0x00441bfc: stw r0,24(r3) 0x00441c00: lwz r9,28(r4) 0x00441c04: stw r9,28(r3) 0x00441c08: addi r4,r4,32 0x00441c0c: addi r3,r3,32 0x00441c10: bdnz+ 0x441bc8
IN: 0x00441bc8: lwz r0,0(r4) 0x00441bcc: stw r0,0(r3) 0x00441bd0: lwz r9,4(r4) 0x00441bd4: stw r9,4(r3) 0x00441bd8: lwz r0,8(r4) 0x00441bdc: stw r0,8(r3) 0x00441be0: lwz r9,12(r4) 0x00441be4: stw r9,12(r3) 0x00441be8: lwz r0,16(r4) 0x00441bec: stw r0,16(r3) 0x00441bf0: lwz r9,20(r4) 0x00441bf4: stw r9,20(r3) 0x00441bf8: lwz r0,24(r4) 0x00441bfc: stw r0,24(r3) 0x00441c00: lwz r9,28(r4) 0x00441c04: stw r9,28(r3) 0x00441c08: addi r4,r4,32 0x00441c0c: addi r3,r3,32 0x00441c10: bdnz+ 0x441bc8
IN: 0x00441bc8: lwz r0,0(r4) 0x00441bcc: stw r0,0(r3) 0x00441bd0: lwz r9,4(r4) 0x00441bd4: stw r9,4(r3) 0x00441bd8: lwz r0,8(r4) 0x00441bdc: stw r0,8(r3) 0x00441be0: lwz r9,12(r4) 0x00441be4: stw r9,12(r3) 0x00441be8: lwz r0,16(r4) 0x00441bec: stw r0,16(r3) 0x00441bf0: lwz r9,20(r4) 0x00441bf4: stw r9,20(r3) 0x00441bf8: lwz r0,24(r4) 0x00441bfc: stw r0,24(r3) 0x00441c00: lwz r9,28(r4) 0x00441c04: stw r9,28(r3) 0x00441c08: addi r4,r4,32 0x00441c0c: addi r3,r3,32 0x00441c10: bdnz+ 0x441bc8
IN: 0x00441bcc: stw r0,0(r3)
IN: 0x00441bd0: lwz r9,4(r4)
IN: 0x00441bd4: stw r9,4(r3) 0x00441bd8: lwz r0,8(r4) 0x00441bdc: stw r0,8(r3) 0x00441be0: lwz r9,12(r4) 0x00441be4: stw r9,12(r3) 0x00441be8: lwz r0,16(r4) 0x00441bec: stw r0,16(r3) 0x00441bf0: lwz r9,20(r4) 0x00441bf4: stw r9,20(r3) 0x00441bf8: lwz r0,24(r4) 0x00441bfc: stw r0,24(r3) 0x00441c00: lwz r9,28(r4) 0x00441c04: stw r9,28(r3) 0x00441c08: addi r4,r4,32 0x00441c0c: addi r3,r3,32 0x00441c10: bdnz+ 0x441bc8
IN: 0x00441bc8: lwz r0,0(r4) 0x00441bcc: stw r0,0(r3) 0x00441bd0: lwz r9,4(r4) 0x00441bd4: stw r9,4(r3) 0x00441bd8: lwz r0,8(r4) 0x00441bdc: stw r0,8(r3) 0x00441be0: lwz r9,12(r4) 0x00441be4: stw r9,12(r3) 0x00441be8: lwz r0,16(r4) 0x00441bec: stw r0,16(r3) 0x00441bf0: lwz r9,20(r4) 0x00441bf4: stw r9,20(r3) 0x00441bf8: lwz r0,24(r4) 0x00441bfc: stw r0,24(r3) 0x00441c00: lwz r9,28(r4) 0x00441c04: stw r9,28(r3) 0x00441c08: addi r4,r4,32 0x00441c0c: addi r3,r3,32 0x00441c10: bdnz+ 0x441bc8
IN: 0x00441bc8: lwz r0,0(r4) 0x00441bcc: stw r0,0(r3) 0x00441bd0: lwz r9,4(r4) 0x00441bd4: stw r9,4(r3) 0x00441bd8: lwz r0,8(r4) 0x00441bdc: stw r0,8(r3) 0x00441be0: lwz r9,12(r4) 0x00441be4: stw r9,12(r3) 0x00441be8: lwz r0,16(r4) 0x00441bec: stw r0,16(r3) 0x00441bf0: lwz r9,20(r4) 0x00441bf4: stw r9,20(r3) 0x00441bf8: lwz r0,24(r4) 0x00441bfc: stw r0,24(r3) 0x00441c00: lwz r9,28(r4) 0x00441c04: stw r9,28(r3) 0x00441c08: addi r4,r4,32 0x00441c0c: addi r3,r3,32 0x00441c10: bdnz+ 0x441bc8
Raise exception at 00441bcc => 00000002 (00) IN: 0x00000300: b 0xffffc3a0
invalid/unsupported opcode: 00 - 00 - 00 (00000000) ffffc3a0 0 IN: 0xffffc3a0: .long 0x0
Raise exception at ffffc3a4 => 00000006 (21) IN: 0x00000700: mtsprg 2,r2 0x00000704: li r2,7 0x00000708: b 0xffffe0f0
invalid/unsupported opcode: 00 - 00 - 00 (00000000) ffffe0f0 0 IN: 0xffffe0f0: .long 0x0
Raise exception at ffffe0f4 => 00000006 (21) Raise exception at ffffe0f4 => 00000006 (21)
Vectors as set up by OpenBIOS and copied to page zero (this is what it looked like before MorphOS replaced them):
Disassembly of section .text.vectors:
fff00000 <.text.vectors>: fff00000: 60 00 00 00 nop fff00004: 60 00 00 00 nop fff00008: 4b ff ff fc b 0xfff00004 ... fff00100: 48 00 24 20 b 0xfff02520 fff00104: 3c 20 80 00 lis r1,-32768 fff00108: 7c 21 0a 15 add. r1,r1,r1 fff0010c: 41 82 00 10 beq 0xfff0011c fff00110: 7c 20 00 a6 mfmsr r1 fff00114: 78 21 00 40 clrldi r1,r1,1 fff00118: 7c 20 01 64 mtmsrd r1 fff0011c: 7c 68 02 a6 mflr r3 fff00120: 3c 80 ff f1 lis r4,-15 fff00124: 38 84 8e 98 addi r4,r4,-29032 fff00128: 7c 89 03 a6 mtctr r4 fff0012c: 4e 80 04 20 bctr ... fff00200: 4b ff ff 05 bl 0xfff00104 ... fff00300: 48 00 20 8c b 0xfff0238c ... fff00380: 4b ff fd 85 bl 0xfff00104 ... fff00400: 48 00 20 28 b 0xfff02428 ... fff00480: 4b ff fc 85 bl 0xfff00104 ... fff00500: 4b ff fc 05 bl 0xfff00104 ... fff00600: 4b ff fb 05 bl 0xfff00104 ... fff00700: 4b ff fa 05 bl 0xfff00104
I really don't think that this is the correct way to handle this at all. Firstly I believe that real-mode? is set to false by default on real hardware, e.g. from articles like http://www.dialectronics.com/Words/OF_Part_III.shtml.
From the device trees I've seen this seems to be the case but that does
not mean that the client code is also run with enabled MMU. OF could enable it only during callbacks as my patch tried. Without real hardware I can't find out though.
When you call the client interface you have to restore the MMU to the state OF expects (and then when it returns back to what the OS wants). This is as defined in the PowerPC binding.
And yes, Apple OF runs by default with the MMU on (and it doesn't work properly with it off, fwiw).
It seems to me that if real-mode? is set to false on real hardware, then your hypothesis that MorphOS expects MMU interrupts to be disabled can't be true;
It is impossible to disable MMU exceptions. Well, you can disable the MMU of course ;-)
Except if OF does what I said above. Either that or on real hardware no MMU interrupt is generated while overwriting the vectors until they are correct again for some other reason. (Like because page zero is alredy mapped unlike with OpenBIOS.)
All exceptions are taken in real mode, so that's not it.
I suspect MorphOS simply does the wrong thing, and there is no (sane) working around it; just like it expects a particular name for the root node (which can be worked around easily, as long as MorphOS is the only client with this particular weirdosity).
- Trap table is copied from OpenFirmware directly
Maybe the MorphOS developers have found the address of the default trap table in a real OF ROM and copy those values into the real vectors as a starting point? Then they can just fix up the values into their executable as required while during the transition any "unfixed" traps would be handled by the PROM as before?
Apple OF constructs the exception code at runtime. It is at a fixed address (namely, 0), and it doesn't just copy a table.
I don't think this is what happens (it would also break when the OF ROM is updated so I can't imagine they'd do that). Instead the addresses they copy first is where the final vectors will be after they set up their memory map. But before that they use the vectors from the loaded boot executable and this is what the fixup does. (I've said above the first vectors point to somewhere near the end of the ROM area but this is later replaced with their routines after the MMU is taken over. It just happens to be OpenBIOS ROM at this point.)
If it takes over memory management without using the OF callbacks for that, it can from that point on not call the client interface anymore.
IN: 0x0040091c: rlwinm r0,r26,0,0,19 0x00400920: lis r3,104 0x00400924: lis r4,66 0x00400928: stw r0,40(r2) 0x0040092c: addi r3,r3,-18952 0x00400930: addi r4,r4,4068 0x00400934: li r5,8192 0x00400938: bl 0x41f004
[Is this GDB output? Please use the disas "/r" modifier, it makes things more readable. Thanks!]
It is unclear to me where exactly it faults; it seems to be on a write to 00421084? While the write to four bytes earlier worked? How curious.
Try to get the full context where the DSI happened (all register contents, etc.)
Segher
On Fri, 6 Jun 2014, Segher Boessenkool wrote:
I really don't think that this is the correct way to handle this at all. Firstly I believe that real-mode? is set to false by default on real hardware, e.g. from articles like http://www.dialectronics.com/Words/OF_Part_III.shtml.
From the device trees I've seen this seems to be the case but that does
not mean that the client code is also run with enabled MMU. OF could enable it only during callbacks as my patch tried. Without real hardware I can't find out though.
When you call the client interface you have to restore the MMU to the state OF expects (and then when it returns back to what the OS wants). This is as defined in the PowerPC binding.
This is what I was trying to do with my patch (at least enabling the MMU and disabling it on return, not restoring the vectors too). Any idea why that does not work?
And yes, Apple OF runs by default with the MMU on (and it doesn't work properly with it off, fwiw).
The question is not if OF itself runs with the MMU on but if it runs OS boot code also with MMU bits enabled in the MSR or not. Since MorphOS boots on real hardware there the MMU bits are either off or no exception is happening at the point where it does on QEMU.
It seems to me that if real-mode? is set to false on real hardware, then your hypothesis that MorphOS expects MMU interrupts to be disabled can't be true;
It is impossible to disable MMU exceptions. Well, you can disable the MMU of course ;-)
What do you call clearing the MSR_DR and MSR_IR bits then? If I clear these bits at the point where MorphOS starts to write to the vectors (just when it overwrites the DSI vector) it boots because it will replace and fixup the vectors and then enable the bits itself (and not call OF functions afterwards). If the bits remain enabled it jumps to an invalid vector and crashes.
Except if OF does what I said above. Either that or on real hardware no MMU interrupt is generated while overwriting the vectors until they are correct again for some other reason. (Like because page zero is alredy mapped unlike with OpenBIOS.)
All exceptions are taken in real mode, so that's not it.
I don't understand this.
I suspect MorphOS simply does the wrong thing, and there is no (sane) working around it; just like it expects a particular name for the root node (which can be worked around easily, as long as MorphOS is the only client with this particular weirdosity).
That may be but they won't fix it and it works on real hardware so there must be a way around it.
Apple OF constructs the exception code at runtime. It is at a fixed address (namely, 0), and it doesn't just copy a table.
Do you have more details? I could not get how it works from just this.
If it takes over memory management without using the OF callbacks for that, it can from that point on not call the client interface anymore.
And it doesn't. It calls all calbacks before overwriting the vectors and taking over memory management.
IN: 0x0040091c: rlwinm r0,r26,0,0,19 0x00400920: lis r3,104 0x00400924: lis r4,66 0x00400928: stw r0,40(r2) 0x0040092c: addi r3,r3,-18952 0x00400930: addi r4,r4,4068 0x00400934: li r5,8192 0x00400938: bl 0x41f004
[Is this GDB output? Please use the disas "/r" modifier, it makes things more readable. Thanks!]
It's qemu in_asm debug output. I know of no options there.
It is unclear to me where exactly it faults; it seems to be on a write to 00421084? While the write to four bytes earlier worked? How curious.
It says: Raise exception at 00441bcc => 00000002 (00)
Where 0x00441bcc: stw r0,0(r3)
part of a memcpy function that writes to the vectors.
Try to get the full context where the DSI happened (all register contents, etc.)
Breakpoint 2, 0x00000300 in ?? () (gdb) bt #0 0x00000300 in ?? () #1 0x0041f034 in ?? () #2 0x0040093c in ?? () #3 0xfff02604 in ?? () (gdb) info registers r0 0x7c52f2a6 2085810854 r1 0xfde7eb0 266239664 r2 0x684c60 6835296 r3 0x1000 4096 r4 0x421fe4 4333540 r5 0x0 0 r6 0x68507c 6836348 r7 0x6850b8 6836408 r8 0xfde7f60 266239840 r9 0x0 0 r10 0x400000 4194304 r11 0x0 0 r12 0x680000 6815744 r13 0x0 0 r14 0xfff32687 4294125191 r15 0xfde7f60 266239840 r16 0xfde7f20 266239776 r17 0xfde7f64 266239844 r18 0xfde7f24 266239780 r19 0x988000 9994240 r20 0x3030 12336 r21 0x684348 6832968 r22 0x688284 6849156 r23 0x0 0 r24 0x30000 196608 r25 0x688a34 6851124 r26 0x10000000 268435456 r27 0x0 0 r28 0x684c80 6835328 r29 0x67b5f8 6796792 r30 0x420fe4 4329444 r31 0x2000 8192 pc 0x300 0x300 msr 0x1000 4096 cr 0x44002044 1140858948 lr 0x41f034 0x41f034 ctr 0x80 128 xer 0x0 0
Seems like the exception is happening when writing to 0x1000.
Regards, BALATON Zoltan
On Fri, Jun 06, 2014 at 10:13:57PM +0200, BALATON Zoltan wrote:
On Fri, 6 Jun 2014, Segher Boessenkool wrote:
I really don't think that this is the correct way to handle this at all. Firstly I believe that real-mode? is set to false by default on real hardware, e.g. from articles like http://www.dialectronics.com/Words/OF_Part_III.shtml.
From the device trees I've seen this seems to be the case but that does
not mean that the client code is also run with enabled MMU. OF could enable it only during callbacks as my patch tried. Without real hardware I can't find out though.
When you call the client interface you have to restore the MMU to the state OF expects (and then when it returns back to what the OS wants). This is as defined in the PowerPC binding.
This is what I was trying to do with my patch (at least enabling the MMU and disabling it on return, not restoring the vectors too). Any idea why that does not work?
The OS should do it, not the client interface itself.
And yes, Apple OF runs by default with the MMU on (and it doesn't work properly with it off, fwiw).
The question is not if OF itself runs with the MMU on but if it runs OS boot code also with MMU bits enabled in the MSR or not. Since MorphOS boots on real hardware there the MMU bits are either off or no exception is happening at the point where it does on QEMU.
If real-mode? is false the client program is started with MMU on.
It seems to me that if real-mode? is set to false on real hardware, then your hypothesis that MorphOS expects MMU interrupts to be disabled can't be true;
It is impossible to disable MMU exceptions. Well, you can disable the MMU of course ;-)
What do you call clearing the MSR_DR and MSR_IR bits then?
"Disabling the MMU". "Translation off".
Except if OF does what I said above. Either that or on real hardware no MMU interrupt is generated while overwriting the vectors until they are correct again for some other reason. (Like because page zero is alredy mapped unlike with OpenBIOS.)
All exceptions are taken in real mode, so that's not it.
I don't understand this.
Whenever an exception is taken MSR[DR] and MSR[IR] are cleared. So it doesn't matter one iota what state the MMU is in when a DSI happens, it will be handled in the same way always.
But reading again, perhaps you meant it influences whether an exception hapens at all. Well, certainly :-)
I suspect MorphOS simply does the wrong thing, and there is no (sane) working around it; just like it expects a particular name for the root node (which can be worked around easily, as long as MorphOS is the only client with this particular weirdosity).
That may be but they won't fix it and it works on real hardware so there must be a way around it.
If they in fact do things wrong, then they should fix it. There is not terribly much that can be done on the OF side here.
Apple OF constructs the exception code at runtime. It is at a fixed address (namely, 0), and it doesn't just copy a table.
Do you have more details? I could not get how it works from just this.
When PowerPC takes an exception, MSR is written (DR=0 IR=0 EE=0 etc.) and the program counter is set to e.g. 0x0300 for a DSI. Apple OF creates the code at 0x0300 at init time; it is not copied from a block in ROM or anything.
It is unclear to me where exactly it faults; it seems to be on a write to 00421084? While the write to four bytes earlier worked? How curious.
It says: Raise exception at 00441bcc => 00000002 (00)
Where 0x00441bcc: stw r0,0(r3)
part of a memcpy function that writes to the vectors.
Try to get the full context where the DSI happened (all register contents, etc.)
[snip]
r3 0x1000 4096
Seems like the exception is happening when writing to 0x1000.
Yeah. So it is writing to that page without having it mapped. It should either have it mapped or access it with the MMU off. AFAIK there is notthing that requires the OF implementation to have it mapped.
Segher
On Sat, 7 Jun 2014, Segher Boessenkool wrote:
When you call the client interface you have to restore the MMU to the state OF expects (and then when it returns back to what the OS wants). This is as defined in the PowerPC binding.
This is what I was trying to do with my patch (at least enabling the MMU and disabling it on return, not restoring the vectors too). Any idea why that does not work?
The OS should do it, not the client interface itself.
MorphOS does not change MMU bits before taking over memory management so there's nothing to restore and CIF calls work. But the default setting does not work on QEMU though during MMU take over.
I suspect MorphOS simply does the wrong thing, and there is no (sane) working around it; just like it expects a particular name for the root node (which can be worked around easily, as long as MorphOS is the only client with this particular weirdosity).
That may be but they won't fix it and it works on real hardware so there must be a way around it.
If they in fact do things wrong, then they should fix it. There is not terribly much that can be done on the OF side here.
They won't fix it. They did not seem very supportive of the idea of MorphOS running on QEMU for unknown reasons. All they ever said is that it is not supported. So it must be fixed on the OF side. (And it can be if it works on real hardware.)
When PowerPC takes an exception, MSR is written (DR=0 IR=0 EE=0 etc.) and the program counter is set to e.g. 0x0300 for a DSI. Apple OF creates the code at 0x0300 at init time; it is not copied from a block in ROM or anything.
OK, this is an implementation detail that is not important here. It does not matter how the vectors end up at their addresses.
r3 0x1000 4096
Seems like the exception is happening when writing to 0x1000.
Yeah. So it is writing to that page without having it mapped. It should either have it mapped or access it with the MMU off. AFAIK there is notthing that requires the OF implementation to have it mapped.
MorphOS expects the OF implementation to behave as Apple's does and apparently it is either mapped there or the MMU is off when the boot loader that accesses this page is running. (This is whay I'm saying for some time.) MorphOS won't map the page itself but does not expect DSI exceptions writing there.
Why should it be already mapped on OpenBIOS too? Well, the device tree says:
/cpus/PowerPC,G4 translations 00001000 00003000 00001000 00000000 0fc58000 001b8000 0fc58000 00000000 fff00000 00100000 0ff00000 00000000
So MorphOS can expect it to be mapped. Can this be fixed somehow? Where?
(Mark also said earlier that the whole memory should be mapped one to one at startup. If so why MMU exceptions are happening at all?)
Regards, BALATON Zoltan
On 07/06/14 13:28, BALATON Zoltan wrote:
r3 0x1000 4096
Seems like the exception is happening when writing to 0x1000.
Yeah. So it is writing to that page without having it mapped. It should either have it mapped or access it with the MMU off. AFAIK there is notthing that requires the OF implementation to have it mapped.
MorphOS expects the OF implementation to behave as Apple's does and apparently it is either mapped there or the MMU is off when the boot loader that accesses this page is running. (This is whay I'm saying for some time.) MorphOS won't map the page itself but does not expect DSI exceptions writing there.
Why should it be already mapped on OpenBIOS too? Well, the device tree says:
/cpus/PowerPC,G4 translations 00001000 00003000 00001000 00000000 0fc58000 001b8000 0fc58000 00000000 fff00000 00100000 0ff00000 00000000
So MorphOS can expect it to be mapped. Can this be fixed somehow? Where?
(Mark also said earlier that the whole memory should be mapped one to one at startup. If so why MMU exceptions are happening at all?)
Silly question, but if the DSI is happening when writing to 0x1000 then surely MorphOS has already overwritten the block 0x0 - 0xfff and so the exception handler at 0x300 won't be pointing to OpenBIOS' handler anyway? If it's not pointing to OpenBIOS' handler then there's no way it can handle the translation.
ATB,
Mark.
On Sat, 7 Jun 2014, Mark Cave-Ayland wrote:
On 07/06/14 13:28, BALATON Zoltan wrote:
r3 0x1000 4096
Seems like the exception is happening when writing to 0x1000.
Yeah. So it is writing to that page without having it mapped. It should either have it mapped or access it with the MMU off. AFAIK there is notthing that requires the OF implementation to have it mapped.
MorphOS expects the OF implementation to behave as Apple's does and apparently it is either mapped there or the MMU is off when the boot loader that accesses this page is running. (This is whay I'm saying for some time.) MorphOS won't map the page itself but does not expect DSI exceptions writing there.
Why should it be already mapped on OpenBIOS too? Well, the device tree says:
/cpus/PowerPC,G4 translations 00001000 00003000 00001000 00000000 0fc58000 001b8000 0fc58000 00000000 fff00000 00100000 0ff00000 00000000
So MorphOS can expect it to be mapped. Can this be fixed somehow? Where?
(Mark also said earlier that the whole memory should be mapped one to one at startup. If so why MMU exceptions are happening at all?)
Silly question, but if the DSI is happening when writing to 0x1000 then surely MorphOS has already overwritten the block 0x0 - 0xfff and so the exception handler at 0x300 won't be pointing to OpenBIOS' handler anyway? If it's not pointing to OpenBIOS' handler then there's no way it can handle the translation.
Right. That's why it fails and because it writes an incorrect vector in the first step that it then replaces with the current location of the handler but if the MMU is enabled then that is not reached due to this exception calling an invalid handler. So either the MMU bits should be off or the area should already be mapped so no exception happens.
I'm not sure how it should work but on real hardware the mapped area seems to be 0x0000-0x3000 while with OpenBIOS it's only 0x1000-0x3000. Could we just modify it to be 0x2000-0x3000 to avoid the exception? Or can we match what's on real hardware?
Regards, BALATON Zoltan
On Sat, 7 Jun 2014, BALATON Zoltan wrote:
I'm not sure how it should work but on real hardware the mapped area seems to be 0x0000-0x3000 while with OpenBIOS it's only 0x1000-0x3000. Could we just modify it to be 0x2000-0x3000 to avoid the exception? Or can we match what's on real hardware?
Alternatively maybe the translation could be locked in using dbat registers as maybe the DSI happens because the entry is not in the TLB any more by the time it is accessed. Please decide on one solution and let me know which one you prefer.
I'd really like to get this in before the freeze because it is probably the biggest obstacle now preventing more people to try running MorphOS on QEMU. (The other one is in QEMU but that's less prohibiting because unlike building OpenBIOS that does not need a cross-compiler environment.)
Regards, BALATON Zoltan
On 07/06/14 20:31, BALATON Zoltan wrote:
Silly question, but if the DSI is happening when writing to 0x1000 then surely MorphOS has already overwritten the block 0x0 - 0xfff and so the exception handler at 0x300 won't be pointing to OpenBIOS' handler anyway? If it's not pointing to OpenBIOS' handler then there's no way it can handle the translation.
Right. That's why it fails and because it writes an incorrect vector in the first step that it then replaces with the current location of the handler but if the MMU is enabled then that is not reached due to this exception calling an invalid handler. So either the MMU bits should be off or the area should already be mapped so no exception happens.
I'm not sure how it should work but on real hardware the mapped area seems to be 0x0000-0x3000 while with OpenBIOS it's only 0x1000-0x3000. Could we just modify it to be 0x2000-0x3000 to avoid the exception? Or can we match what's on real hardware?
The only way an exception won't happen here is if the mapping is already in the TLB, and my understanding is that being able to lock a TLB entry in place is *not* a standard feature of PPC CPUs so you're reliant on the inbuilt LRU algorithm.
It might be that MorphOS just got lucky that this works on real hardware without faulting (similar to the bug I had for SPARC64) but some further analysis would help. Does QEMU ppc have a DEBUG_TLB or similar option so you can see when the eviction is triggered? Sadly I think the options here are quite limited :/
ATB,
Mark.
On 08/06/14 13:15, BALATON Zoltan wrote:
On Sat, 7 Jun 2014, BALATON Zoltan wrote:
I'm not sure how it should work but on real hardware the mapped area seems to be 0x0000-0x3000 while with OpenBIOS it's only 0x1000-0x3000. Could we just modify it to be 0x2000-0x3000 to avoid the exception? Or can we match what's on real hardware?
Alternatively maybe the translation could be locked in using dbat registers as maybe the DSI happens because the entry is not in the TLB any more by the time it is accessed. Please decide on one solution and let me know which one you prefer.
Actually just thinking about this, I've just remembered that in OpenBIOS we use a round-robin hash table slot eviction scheme that forces a tlbie (i.e. invalidate the TLB entry) at the end so perhaps we are invalidating the entry early? See arch/ppc/qemu/ofmem.c in hash_page32().
Quick and dirty hack alert: you may be able to tweak the code so that any hash entry with a virtual address which references physical page 0 is never chosen for eviction (and hence never forcibly invalidated with tlbie) and see if that helps at all? Have a play and see what happens.
I'd really like to get this in before the freeze because it is probably the biggest obstacle now preventing more people to try running MorphOS on QEMU. (The other one is in QEMU but that's less prohibiting because unlike building OpenBIOS that does not need a cross-compiler environment.)
Understood. I appreciate the time you have spent working on this, we just need to make sure that we don't just scatter MorphOSisms around the code in a way that makes maintenance harder for those of us who won't run it everyday, and isn't detrimental to all the other supported architectures.
ATB,
Mark.
On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
It might be that MorphOS just got lucky that this works on real hardware without faulting (similar to the bug I had for SPARC64) but some further analysis would help. Does QEMU ppc have a DEBUG_TLB or similar option so you can see when the eviction is triggered? Sadly I think the options here are quite limited :/
There are some debug options in qemu/target-ppc/mmu_helper.c but they don't give much clue and the DUMP_PAGE_TABLES option does not work because it does not compile if enabled (something seems to have changed leaving debug code not updated in this file; since I don't know what was the change it's not trivial to fix).
On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
Actually just thinking about this, I've just remembered that in OpenBIOS we use a round-robin hash table slot eviction scheme that forces a tlbie (i.e. invalidate the TLB entry) at the end so perhaps we are invalidating the entry early? See arch/ppc/qemu/ofmem.c in hash_page32().
Or maybe it was never in the TLB because the copying of the vectors that write here was before enabling the MMU but I don't completely understand what will cause an exception and what doesn't. The vectors have been used but that may put it in the instruction cache not in the data one.
Quick and dirty hack alert: you may be able to tweak the code so that any hash entry with a virtual address which references physical page 0 is never chosen for eviction (and hence never forcibly invalidated with tlbie) and see if that helps at all? Have a play and see what happens.
I don't see how this could be done and this seems to be a very dirty hack that I'd like to avoid.
Understood. I appreciate the time you have spent working on this, we just need to make sure that we don't just scatter MorphOSisms around the code in a way that makes maintenance harder for those of us who won't run it everyday, and isn't detrimental to all the other supported architectures.
I agree with this completely and I hope we can come up with a solution that works and can be accepted. I did some more experiments but could not make it work. So far the only working solution seems to run this code with MMU off. I tried this patch:
--- a/openbios-devel/arch/ppc/qemu/start.S +++ b/openbios-devel/arch/ppc/qemu/start.S @@ -517,6 +517,10 @@ _GLOBAL(call_elf): li r7,0 // r7 = length of client program arguments li r0,MSR_FP | MSR_ME | MSR_DR | MSR_IR MTMSRD(r0) + LOAD_REG_IMMEDIATE(r8, __vectors) + li r9,0 + lwz r10,4096(r8) + stw r10,4096(r9) blrl
#ifdef CONFIG_PPC64
which writes to 0x1000 after enabling the MMU and before calling the client code to make sure the translation is in the table. With this it can get past the write that caused an exception before but hits another one later and crashes. Looks like the code where this happens sets up MMU related registers and TLB entries so the handlers are probably not yet working and it cannot handle the exception even if the right routine is called. This is where it crashes with the above patch:
IN: 0x00400acc: mfdbsr r0 0x00400ad0: mfl2cr r11 0x00400ad4: addis r9,r10,-1 0x00400ad8: addi r9,r9,32767 0x00400adc: cmplwi r9,1 0x00400ae0: bgt- 0x400af4
IN: 0x00400af4: mficcr r0 0x00400af8: mr r4,r16 0x00400afc: mr r5,r15 0x00400b00: addi r3,r1,16 0x00400b04: li r6,0 0x00400b08: bl 0x41cce8
IN: 0x0041cce8: stwu r1,-96(r1) 0x0041ccec: stmw r13,20(r1) 0x0041ccf0: lwz r0,0(r3) 0x0041ccf4: sync 0x0041ccf8: mtsr 0,r0 0x0041ccfc: isync
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2 0x00000404: li r2,4 0x00000408: b 0x41f0d4
IN: 0x0041f0d4: mtsprg 1,r1 0x0041f0d8: mfmsr r1 0x0041f0dc: ori r1,r1,12336 0x0041f0e0: sync 0x0041f0e4: mtmsr r1
Raise exception at 0041f0e8 => 00000003 (40000000) Raise exception at 0041f0e8 => 00000003 (40000000)
Actually after writing the sr0-sr15 and sdr1 registers it clears the MMU bits and then writes to the ibat and dbat registers and seems to set up the TLB and then enable the bits in the MSR. So maybe Apple relies on translations in these registers and that's why it works there. Unfortunately these are only implemented in 32bit processors so it would not work everywhere but still this seems to be the only other solution I can think of short of disabling the MMU somehow during this code. Any ideas?
Regards, BALATON Zoltan
On 09/06/14 01:27, BALATON Zoltan wrote:
On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
It might be that MorphOS just got lucky that this works on real hardware without faulting (similar to the bug I had for SPARC64) but some further analysis would help. Does QEMU ppc have a DEBUG_TLB or similar option so you can see when the eviction is triggered? Sadly I think the options here are quite limited :/
There are some debug options in qemu/target-ppc/mmu_helper.c but they don't give much clue and the DUMP_PAGE_TABLES option does not work because it does not compile if enabled (something seems to have changed leaving debug code not updated in this file; since I don't know what was the change it's not trivial to fix).
A quick look around shows that you probably want to look at DEBUG_MMU/DEBUG_BAT in target-ppc/mmu_has32.c. And I'm not even a PPC guy :)
On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
Actually just thinking about this, I've just remembered that in OpenBIOS we use a round-robin hash table slot eviction scheme that forces a tlbie (i.e. invalidate the TLB entry) at the end so perhaps we are invalidating the entry early? See arch/ppc/qemu/ofmem.c in hash_page32().
Or maybe it was never in the TLB because the copying of the vectors that write here was before enabling the MMU but I don't completely understand what will cause an exception and what doesn't. The vectors have been used but that may put it in the instruction cache not in the data one.
It's easy to get lost in the details, but MMUs are actually really simple - they do exactly what they say on the tin. As soon as the MMU is enabled, the basic process looks like this:
1) Does the instruction/data fetch address exist in the TLB (cache)? If so, use it.
2) If it doesn't exist in the cache, throw a DSI (data) or ISI (instruction) exception and switch to real mode
3) The exception handler then adds the lookup to the hash table
4) The CPU retries the faulting load/store which will now succeed, and stores the result in the TLB for later use
Therefore for a load/store not to fault on real hardware then that means the entry is already in the TLB, i.e. it must have already been accessed AND also not been previously evicted by hash collision (see hash_page32).
Hence you need to trace through previous accesses to the addresses that fault and determine why the entry is no longer in the TLB in QEMU/OpenBIOS when it should be there on real hardware.
Quick and dirty hack alert: you may be able to tweak the code so that any hash entry with a virtual address which references physical page 0 is never chosen for eviction (and hence never forcibly invalidated with tlbie) and see if that helps at all? Have a play and see what happens.
I don't see how this could be done and this seems to be a very dirty hack that I'd like to avoid.
I'm not saying this is for production use, I'm just giving a suggestion as to things you can do to understand what is happening. There is no substitute for hard work when debugging issues like this I'm afraid.
Understood. I appreciate the time you have spent working on this, we just need to make sure that we don't just scatter MorphOSisms around the code in a way that makes maintenance harder for those of us who won't run it everyday, and isn't detrimental to all the other supported architectures.
I agree with this completely and I hope we can come up with a solution that works and can be accepted. I did some more experiments but could not make it work. So far the only working solution seems to run this code with MMU off. I tried this patch:
--- a/openbios-devel/arch/ppc/qemu/start.S +++ b/openbios-devel/arch/ppc/qemu/start.S @@ -517,6 +517,10 @@ _GLOBAL(call_elf): li r7,0 // r7 = length of client program arguments li r0,MSR_FP | MSR_ME | MSR_DR | MSR_IR MTMSRD(r0)
LOAD_REG_IMMEDIATE(r8, __vectors)li r9,0lwz r10,4096(r8)stw r10,4096(r9) blrl#ifdef CONFIG_PPC64
which writes to 0x1000 after enabling the MMU and before calling the client code to make sure the translation is in the table.
Okay, so this should make sense with what I said above.
With this it can get past the write that caused an exception before but hits another one later and crashes. Looks like the code where this happens sets up MMU related registers and TLB entries so the handlers are probably not yet working and it cannot handle the exception even if the right routine is called. This is where it crashes with the above patch:
IN: 0x00400acc: mfdbsr r0 0x00400ad0: mfl2cr r11 0x00400ad4: addis r9,r10,-1 0x00400ad8: addi r9,r9,32767 0x00400adc: cmplwi r9,1 0x00400ae0: bgt- 0x400af4
IN: 0x00400af4: mficcr r0 0x00400af8: mr r4,r16 0x00400afc: mr r5,r15 0x00400b00: addi r3,r1,16 0x00400b04: li r6,0 0x00400b08: bl 0x41cce8
IN: 0x0041cce8: stwu r1,-96(r1) 0x0041ccec: stmw r13,20(r1) 0x0041ccf0: lwz r0,0(r3) 0x0041ccf4: sync 0x0041ccf8: mtsr 0,r0 0x0041ccfc: isync
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2 0x00000404: li r2,4 0x00000408: b 0x41f0d4
IN: 0x0041f0d4: mtsprg 1,r1 0x0041f0d8: mfmsr r1 0x0041f0dc: ori r1,r1,12336 0x0041f0e0: sync 0x0041f0e4: mtmsr r1
Raise exception at 0041f0e8 => 00000003 (40000000) Raise exception at 0041f0e8 => 00000003 (40000000)
Actually after writing the sr0-sr15 and sdr1 registers it clears the MMU bits and then writes to the ibat and dbat registers and seems to set up the TLB and then enable the bits in the MSR. So maybe Apple relies on translations in these registers and that's why it works there.
Not that I have the definitive answer, but I can't see why on earth Apple would map the trap table with an IBAT/DBAT. My understanding is that these are designed for optimally mapping large blocks of memory, e.g. kernels, so why would you implement a second form of translation in a BIOS environment when you already have limited resources? It just doesn't make any sense. Plus you still get a fault once you effectively "preload" the trap table into the TLB with your fake access above, so something else is still missing from the picture.
Unfortunately these are only implemented in 32bit processors so it would not work everywhere but still this seems to be the only other solution I can think of short of disabling the MMU somehow during this code. Any ideas?
Not at the moment. All I can re-emphasise is that debugging issues like these takes a great deal of time/effort, but also the rewards gained are similarly great.
ATB,
Mark.
On Mon, 9 Jun 2014, Mark Cave-Ayland wrote:
A quick look around shows that you probably want to look at DEBUG_MMU/DEBUG_BAT in target-ppc/mmu_has32.c. And I'm not even a PPC guy :)
Me neither... These give some more debug output but they are only useful for debugging BATs not debugging translation via the TLB as it only prints TLB address and hashes so I can only see if the TLB was consulted but not the address that was looked up.
Therefore for a load/store not to fault on real hardware then that means the entry is already in the TLB, i.e. it must have already been accessed AND also not been previously evicted by hash collision (see hash_page32).
Unfortunately PPC has many other ways to translate an address not only via the TLB so this is not that simple. I suspect the TLB is not used on Macs during boot but BAT translation is because MorphOS only disables the MMU during replacing the BAT registers.
Hence you need to trace through previous accesses to the addresses that fault and determine why the entry is no longer in the TLB in QEMU/OpenBIOS when it should be there on real hardware.
The vectors are only accessed by OpenBIOS during the inital copying of the vectors before setting up the MMU then by MorphOS when copying it's vectors that cause the exception which crashes. No other access inbetween so it's not evicted from TLB as it never gets into it.
Actually after writing the sr0-sr15 and sdr1 registers it clears the MMU bits and then writes to the ibat and dbat registers and seems to set up the TLB and then enable the bits in the MSR. So maybe Apple relies on translations in these registers and that's why it works there.
Not that I have the definitive answer, but I can't see why on earth Apple would map the trap table with an IBAT/DBAT. My understanding is that these are designed for optimally mapping large blocks of memory, e.g. kernels, so why would you implement a second form of translation in a BIOS environment when you already have limited resources? It just doesn't make any sense. Plus you still get a fault once you effectively "preload" the trap table into the TLB with your fake access above, so something else is still missing from the picture.
After more experiments I think I got a bit closer to the solution. The other exeption I got with my last patch while setting the sr0 register was because the value was read from a local variable on the stack which generated the exception. With this patch:
--- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -531,6 +531,32 @@ setup_mmu(unsigned long ramsize) asm volatile("mtsrin %0,%1" :: "r" (sr_base + i), "r" (j)); }
+ /* Add permanent translations for the first 128MB and the client stack area + * via BAT registers on 32bit CPUs where this is supported */ + if (!is_ppc64()) { + unsigned long stack_addr; + mBAT bat; + + memset(&bat, 0, sizeof(bat)); + bat.batu.bl = 0x3ff; /* 128 MB */ + bat.batu.vs = 1; /* supervisor access */ + bat.batl.pp = 2; /* read/write */ + mtspr(S_DBAT0L, bat.batl); + mtspr(S_IBAT0L, bat.batl); + mtspr(S_DBAT0U, bat.batu); + mtspr(S_IBAT0U, bat.batu); + + stack_addr = get_heap_top(); + /* only add if not overlapping the previous one */ + if (stack_addr > 0x8000000) { + bat.batu.bepi = stack_addr >> 17; + bat.batu.bl = 1; /* 256 kB */ + bat.batl.brpn = stack_addr >> 17; + mtspr(S_DBAT1L, bat.batl); + mtspr(S_DBAT1U, bat.batu); + } + asm volatile("isync"); + } #endif
ofmem = ofmem_arch_get_private();
it boots but only with 256MB of memory.
With 128MB it will overwrite the stack as I've found earlier here: http://www.openfirmware.info/pipermail/openbios/2014-March/008186.html
With 512MB it will freeze after clearing the TLB and replacing BAT registers with it's own values like this:
Set IBAT0l to 00000000 (0041ce54) Set IBAT0u to 00000000 (0041ce54) Flush BAT from 00000000 to 00020000 (00000000) Flush done Flush BAT from 00000000 to 00020000 (00000000) Flush done Set IBAT1l to f0000012 (0041ce54) Set IBAT1u to f0001ffe (0041ce54) Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Flush BAT from f0000000 to 00000000 (0ffe0000) Flush done Set IBAT2l to 00000000 (0041ce54) Set IBAT2u to 00000000 (0041ce54) Set IBAT3l to 00000012 (0041ce54) Set IBAT3u to 00000ffe (0041ce54) Flush BAT from 00000000 to 08000000 (07fe0000) Flush done Flush BAT from 00000000 to 08000000 (07fe0000) Flush done Set DBAT0l to f000002a (0041ce54) Set DBAT0u to f0001ffe (0041ce54) Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Flush BAT from f0000000 to 00000000 (0ffe0000) Flush done Set DBAT1l to 00000012 (0041ce54) Set DBAT1u to 00001ffe (0041ce54) Flush BAT from 1fdc0000 to 2fdc0000 (0ffe0000) Flush done Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Set DBAT2l to 00000000 (0041ce54) Set DBAT2u to 00000000 (0041ce54) Set DBAT3l to 8000002a (0041ce54) Set DBAT3u to 80001ffe (0041ce54)
and then trying to access a variable on the stack:
0x0041cf40: lmw r13,20(r1) 0x0041cf44: addi r1,r1,96 0x0041cf48: blr
htab_base 0000000000000000 htab_mask 000000000000ffff hash 000000000000fde7 0 htab=0000000000000000/000000000000ffff vsid=0 ptem=3f hash=000000000000fde7 1 htab=0000000000000000/000000000000ffff vsid=0 api=3f hash=ffffffffffff0218 Raise exception at 0041cf40 => 00000002 (00)
r0 0x3030 12336 r1 0x1fde7e70 534675056
which is not covered by any translation now. So I think the stack must be in the first 256MB to which a translation is added to the BAT and this is why it works with 256MB of memory. Thus to be complete the stack should be moved from the end of the memory where it is now (described in a comment in arch/ppc/qemu/start.S) to somwhere else. But where? There's another comment in arch/ppc/qemu/ofmem.c that seems to originate from here:
http://opensource.apple.com/source/BootX/BootX-81/bootx.tproj/include.subpro...
but seems it has changed at the origin since it was copied. Does anyone know a good location that is left free by the boot loaders we care about where the stack could be placed within the first 256MB? MoprhOS seems to start using memory from the end so maybe the stack should not be there. Any more ideas?
Regards, BALATON Zoltan
On Mon, 9 Jun 2014, BALATON Zoltan wrote:
Therefore for a load/store not to fault on real hardware then that means the entry is already in the TLB, i.e. it must have already been accessed AND also not been previously evicted by hash collision (see hash_page32).
Unfortunately PPC has many other ways to translate an address not only via the TLB so this is not that simple. I suspect the TLB is not used on Macs during boot but BAT translation is because MorphOS only disables the MMU during replacing the BAT registers.
In the tests I wrote about here:
http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg02828.html
I've also tried to find answers to these questions. The results so far are here:
http://goliat.eik.bme.hu/~balaton/oftest/results/
It seems BAT registers are not used only a TLB which is near the end of the memory like in OpenBIOS.
Hence you need to trace through previous accesses to the addresses that fault and determine why the entry is no longer in the TLB in QEMU/OpenBIOS when it should be there on real hardware.
The vectors are only accessed by OpenBIOS during the inital copying of the vectors before setting up the MMU then by MorphOS when copying it's vectors that cause the exception which crashes. No other access inbetween so it's not evicted from TLB as it never gets into it.
Actually after writing the sr0-sr15 and sdr1 registers it clears the MMU bits and then writes to the ibat and dbat registers and seems to set up the TLB and then enable the bits in the MSR. So maybe Apple relies on translations in these registers and that's why it works there.
Not that I have the definitive answer, but I can't see why on earth Apple would map the trap table with an IBAT/DBAT. My understanding is that these are designed for optimally mapping large blocks of memory, e.g. kernels, so why would you implement a second form of translation in a BIOS environment when you already have limited resources? It just doesn't make any sense. Plus you still get a fault once you effectively "preload" the trap table into the TLB with your fake access above, so something else is still missing from the picture.
The fake access would still be needed I think because otherwise a translation for the page starting at 0x1000 would not be in the TLB. The fault I got with the fake access was not when writing to 0x1000 like without it but accessing the stack that the fake access did not preload.
With 128MB it will overwrite the stack as I've found earlier here: http://www.openfirmware.info/pipermail/openbios/2014-March/008186.html
With 512MB it will freeze after clearing the TLB and replacing BAT registers with it's own values like this:
Set IBAT0l to 00000000 (0041ce54) Set IBAT0u to 00000000 (0041ce54) Flush BAT from 00000000 to 00020000 (00000000) Flush done Flush BAT from 00000000 to 00020000 (00000000) Flush done Set IBAT1l to f0000012 (0041ce54) Set IBAT1u to f0001ffe (0041ce54) Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Flush BAT from f0000000 to 00000000 (0ffe0000) Flush done Set IBAT2l to 00000000 (0041ce54) Set IBAT2u to 00000000 (0041ce54) Set IBAT3l to 00000012 (0041ce54) Set IBAT3u to 00000ffe (0041ce54) Flush BAT from 00000000 to 08000000 (07fe0000) Flush done Flush BAT from 00000000 to 08000000 (07fe0000) Flush done Set DBAT0l to f000002a (0041ce54) Set DBAT0u to f0001ffe (0041ce54) Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Flush BAT from f0000000 to 00000000 (0ffe0000) Flush done Set DBAT1l to 00000012 (0041ce54) Set DBAT1u to 00001ffe (0041ce54) Flush BAT from 1fdc0000 to 2fdc0000 (0ffe0000) Flush done Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Set DBAT2l to 00000000 (0041ce54) Set DBAT2u to 00000000 (0041ce54) Set DBAT3l to 8000002a (0041ce54) Set DBAT3u to 80001ffe (0041ce54)
and then trying to access a variable on the stack:
0x0041cf40: lmw r13,20(r1) 0x0041cf44: addi r1,r1,96 0x0041cf48: blr
htab_base 0000000000000000 htab_mask 000000000000ffff hash 000000000000fde7 0 htab=0000000000000000/000000000000ffff vsid=0 ptem=3f hash=000000000000fde7 1 htab=0000000000000000/000000000000ffff vsid=0 api=3f hash=ffffffffffff0218 Raise exception at 0041cf40 => 00000002 (00)
r0 0x3030 12336 r1 0x1fde7e70 534675056
which is not covered by any translation now. So I think the stack must be in the first 256MB to which a translation is added to the BAT and this is why it works with 256MB of memory. Thus to be complete the stack should be moved from the end of the memory where it is now (described in a comment in arch/ppc/qemu/start.S) to somwhere else. But where? There's another comment in arch/ppc/qemu/ofmem.c that seems to originate from here:
http://opensource.apple.com/source/BootX/BootX-81/bootx.tproj/include.subpro...
but seems it has changed at the origin since it was copied. Does anyone know a good location that is left free by the boot loaders we care about where the stack could be placed within the first 256MB? MoprhOS seems to start using memory from the end so maybe the stack should not be there. Any more ideas?
From the test results it seems that Apple puts the stack after the image
(and probably clears it or otherwise adds a mapping in the TLB for it so accessing it will not generate exceptions). How could this be implemented in OpenBIOS? I've tried to look at the code to find a good place but there seems to be different cases for different architectures and using preloaded kernel image seems to be an additional complication. Could the file-size stored in saved-program-state be used as a base to place the stack in call_elf? (this would need adding an additional parameter to it in the ppc/qemu case because accessing Forth from assembly is not something I'd try).
Or are my conclusions wrong or you have a better idea? (Also what about my other patches on the list? This place seemed abandoned for the last week.)
Regards, BALATON Zoltan
On 17/06/14 01:01, BALATON Zoltan wrote:
On Mon, 9 Jun 2014, BALATON Zoltan wrote:
Therefore for a load/store not to fault on real hardware then that means the entry is already in the TLB, i.e. it must have already been accessed AND also not been previously evicted by hash collision (see hash_page32).
Unfortunately PPC has many other ways to translate an address not only via the TLB so this is not that simple. I suspect the TLB is not used on Macs during boot but BAT translation is because MorphOS only disables the MMU during replacing the BAT registers.
In the tests I wrote about here:
http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg02828.html
I've also tried to find answers to these questions. The results so far are here:
http://goliat.eik.bme.hu/~balaton/oftest/results/
It seems BAT registers are not used only a TLB which is near the end of the memory like in OpenBIOS.
I had a feeling that would be the case, however I'm glad at least we have some evidence to back this up. BTW you mean hash table, not TLB here, right?
Hence you need to trace through previous accesses to the addresses that fault and determine why the entry is no longer in the TLB in QEMU/OpenBIOS when it should be there on real hardware.
The vectors are only accessed by OpenBIOS during the inital copying of the vectors before setting up the MMU then by MorphOS when copying it's vectors that cause the exception which crashes. No other access inbetween so it's not evicted from TLB as it never gets into it.
Actually after writing the sr0-sr15 and sdr1 registers it clears the MMU bits and then writes to the ibat and dbat registers and seems to set up the TLB and then enable the bits in the MSR. So maybe Apple relies on translations in these registers and that's why it works there.
Not that I have the definitive answer, but I can't see why on earth Apple would map the trap table with an IBAT/DBAT. My understanding is that these are designed for optimally mapping large blocks of memory, e.g. kernels, so why would you implement a second form of translation in a BIOS environment when you already have limited resources? It just doesn't make any sense. Plus you still get a fault once you effectively "preload" the trap table into the TLB with your fake access above, so something else is still missing from the picture.
The fake access would still be needed I think because otherwise a translation for the page starting at 0x1000 would not be in the TLB. The fault I got with the fake access was not when writing to 0x1000 like without it but accessing the stack that the fake access did not preload.
With 128MB it will overwrite the stack as I've found earlier here: http://www.openfirmware.info/pipermail/openbios/2014-March/008186.html
With 512MB it will freeze after clearing the TLB and replacing BAT registers with it's own values like this:
Set IBAT0l to 00000000 (0041ce54) Set IBAT0u to 00000000 (0041ce54) Flush BAT from 00000000 to 00020000 (00000000) Flush done Flush BAT from 00000000 to 00020000 (00000000) Flush done Set IBAT1l to f0000012 (0041ce54) Set IBAT1u to f0001ffe (0041ce54) Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Flush BAT from f0000000 to 00000000 (0ffe0000) Flush done Set IBAT2l to 00000000 (0041ce54) Set IBAT2u to 00000000 (0041ce54) Set IBAT3l to 00000012 (0041ce54) Set IBAT3u to 00000ffe (0041ce54) Flush BAT from 00000000 to 08000000 (07fe0000) Flush done Flush BAT from 00000000 to 08000000 (07fe0000) Flush done Set DBAT0l to f000002a (0041ce54) Set DBAT0u to f0001ffe (0041ce54) Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Flush BAT from f0000000 to 00000000 (0ffe0000) Flush done Set DBAT1l to 00000012 (0041ce54) Set DBAT1u to 00001ffe (0041ce54) Flush BAT from 1fdc0000 to 2fdc0000 (0ffe0000) Flush done Flush BAT from 00000000 to 10000000 (0ffe0000) Flush done Set DBAT2l to 00000000 (0041ce54) Set DBAT2u to 00000000 (0041ce54) Set DBAT3l to 8000002a (0041ce54) Set DBAT3u to 80001ffe (0041ce54)
and then trying to access a variable on the stack:
0x0041cf40: lmw r13,20(r1) 0x0041cf44: addi r1,r1,96 0x0041cf48: blr
htab_base 0000000000000000 htab_mask 000000000000ffff hash 000000000000fde7 0 htab=0000000000000000/000000000000ffff vsid=0 ptem=3f hash=000000000000fde7 1 htab=0000000000000000/000000000000ffff vsid=0 api=3f hash=ffffffffffff0218 Raise exception at 0041cf40 => 00000002 (00)
r0 0x3030 12336 r1 0x1fde7e70 534675056
which is not covered by any translation now. So I think the stack must be in the first 256MB to which a translation is added to the BAT and this is why it works with 256MB of memory. Thus to be complete the stack should be moved from the end of the memory where it is now (described in a comment in arch/ppc/qemu/start.S) to somwhere else. But where? There's another comment in arch/ppc/qemu/ofmem.c that seems to originate from here:
http://opensource.apple.com/source/BootX/BootX-81/bootx.tproj/include.subpro...
but seems it has changed at the origin since it was copied. Does anyone know a good location that is left free by the boot loaders we care about where the stack could be placed within the first 256MB? MoprhOS seems to start using memory from the end so maybe the stack should not be there. Any more ideas?
From the test results it seems that Apple puts the stack after the image
(and probably clears it or otherwise adds a mapping in the TLB for it so accessing it will not generate exceptions). How could this be implemented in OpenBIOS? I've tried to look at the code to find a good place but there seems to be different cases for different architectures and using preloaded kernel image seems to be an additional complication. Could the file-size stored in saved-program-state be used as a base to place the stack in call_elf? (this would need adding an additional parameter to it in the ppc/qemu case because accessing Forth from assembly is not something I'd try).
FWIW I don't know much about the PPC side of this, however for SPARC the client stack is set as part of the context initialisation (see arch/sparc64/context.c for an example of this). This allows the caller to initialise a context (including specifying the stack) all in C before dropping into asm to finally execute the loaded file.
Or are my conclusions wrong or you have a better idea? (Also what about my other patches on the list? This place seemed abandoned for the last week.)
Well, as I mentioned last week I've been away on holiday. Plus it's QEMU soft freeze time which means everybody is busy trying to get their patches applied in preparation for 2.1. This generally means people are very busy.
ATB,
Mark.
On Tue, 17 Jun 2014, Mark Cave-Ayland wrote:
It seems BAT registers are not used only a TLB which is near the end of the memory like in OpenBIOS.
I had a feeling that would be the case, however I'm glad at least we have some evidence to back this up. BTW you mean hash table, not TLB here, right?
Uhmm yes, hash table for TLB to be precise.
FWIW I don't know much about the PPC side of this, however for SPARC the client stack is set as part of the context initialisation (see arch/sparc64/context.c for an example of this). This allows the caller to initialise a context (including specifying the stack) all in C before dropping into asm to finally execute the loaded file.
This seems to use a static array defined in that file as the client stack. That would still be mapped in the ROM area and would fail when MorphOS takes over the MMU and only adds translation for the first 256 MB of memory at first. (Otherwise I could just clear the stack area before calling the client code to preload translations like for the 0x1000 case but this is not enough and seems the stack should be moved where Apple puts it after the loaded image to fit within 256 MBs.
Regards, BALATON Zoltan
On Tue, 17 Jun 2014, BALATON Zoltan wrote:
From the test results it seems that Apple puts the stack after the image (and probably clears it or otherwise adds a mapping in the TLB for it so accessing it will not generate exceptions). How could this be implemented in OpenBIOS? I've tried to look at the code to find a good place but there seems to be different cases for different architectures and using preloaded kernel image seems to be an additional complication. Could the file-size stored in saved-program-state be used as a base to place the stack in call_elf? (this would need adding an additional parameter to it in the ppc/qemu case because accessing Forth from assembly is not something I'd try).
Or are my conclusions wrong or you have a better idea? (Also what about my other patches on the list? This place seemed abandoned for the last week.)
With the patch below I don't get DSI-s but it crashes in an ISI while trying to replace the sr registers. How could this possibly work on real hardware without getting any exceptions during replacing the vectors? Is there a way to preload the TLB with code addresses too so we can avoid the crash? How? (For data accessing it before calling the boot code is enough to preload translations but how to avoid ISI-s?)
Regards, BALATON Zoltan
diff --git a/openbios-devel/arch/ppc/qemu/init.c b/openbios-devel/arch/ppc/qemu/init.c index 8d5ef8e..2872c7b 100644 --- a/openbios-devel/arch/ppc/qemu/init.c +++ b/openbios-devel/arch/ppc/qemu/init.c @@ -594,12 +594,20 @@ static void go(void); static void go(void) { - ucell addr; + ucell addr, end;
feval("saved-program-state >sps.entry @"); addr = POP(); + feval("saved-program-state >sps.end @"); + end = POP();
- call_elf(0, 0, addr); + if (end) { + ofmem_claim(end, 64 * 1024, 0); + memset((void *)end, 0, 64 * 1024); + end += 64 * 1024; + } + + call_elf(0, 0, addr, end); }
static void kvm_of_init(void) diff --git a/openbios-devel/arch/ppc/qemu/kernel.h b/openbios-devel/arch/ppc/qemu/kernel.h index fe9be83..464aa14 100644 --- a/openbios-devel/arch/ppc/qemu/kernel.h +++ b/openbios-devel/arch/ppc/qemu/kernel.h @@ -22,7 +22,8 @@ extern void exit( int status ) __attribute__ ((noreturn)); /* start.S */ extern void flush_icache_range( char *start, char *stop ); extern char of_rtas_start[], of_rtas_end[]; -extern void call_elf( unsigned long arg1, unsigned long arg2, unsigned long elf_entry ); +extern void call_elf( unsigned long arg1, unsigned long arg2, + unsigned long elf_entry, unsigned long stack_addr );
/* methods.c */ extern void node_methods_init( const char *cpuname ); diff --git a/openbios-devel/arch/ppc/qemu/main.c b/openbios-devel/arch/ppc/qemu/main.c index 44b1666..ac45c6c 100644 --- a/openbios-devel/arch/ppc/qemu/main.c +++ b/openbios-devel/arch/ppc/qemu/main.c @@ -61,7 +61,7 @@ static void check_preloaded_kernel(void) ph = find_dev("/chosen"); set_property(ph, "bootargs", strdup(kernel_cmdline), strlen(kernel_cmdline) + 1); } - call_elf(initrd_image, initrd_size, kernel_image); + call_elf(initrd_image, initrd_size, kernel_image, 0); } }
diff --git a/openbios-devel/arch/ppc/qemu/start.S b/openbios-devel/arch/ppc/qemu/start.S index ae2fd53..2edf7ba 100644 --- a/openbios-devel/arch/ppc/qemu/start.S +++ b/openbios-devel/arch/ppc/qemu/start.S @@ -501,7 +501,7 @@ real_entry: saved_stack: DATA_LONG(0) .previous - /* void call_elf( arg1, arg2, entry ) */ + /* void call_elf( arg1, arg2, entry, stack_addr ) */ _GLOBAL(call_elf): mflr r0 PPC_STLU r1, -STACKFRAME_MINSIZE(r1) @@ -512,6 +512,13 @@ _GLOBAL(call_elf): mfsdr1 r1 addi r1, r1, -32768 /* - 32 KiB exception stack */ addis r1, r1, -1 /* - 64 KiB stack */ + cmpwi r6,0 + beq 1f + mr r1,r6 // use stack_addr if not NULL +1: LOAD_REG_IMMEDIATE(r8, __vectors) // access 0x1000 to preload TLB + li r9,0 + lwz r10,4096(r8) + stw r10,4096(r9) LOAD_REG_IMMEDIATE(r5, of_client_callback) // r5 = callback li r6,0 // r6 = address of client program arguments (unused) li r7,0 // r7 = length of client program arguments (unused) diff --git a/openbios-devel/forth/debugging/client.fs b/openbios-devel/forth/debugging/client.fs index f374404..f06d2d3 100644 --- a/openbios-devel/forth/debugging/client.fs +++ b/openbios-devel/forth/debugging/client.fs @@ -21,6 +21,7 @@ struct ( saved-program-state ) /n field >sps.entry /n field >sps.file-size /n field >sps.file-type + /n field >sps.end constant saved-program-state.size create saved-program-state saved-program-state.size allot
diff --git a/openbios-devel/libopenbios/aout_load.c b/openbios-devel/libopenbios/aout_load.c index e9d2002..01edb88 100644 --- a/openbios-devel/libopenbios/aout_load.c +++ b/openbios-devel/libopenbios/aout_load.c @@ -160,6 +160,8 @@ aout_load(struct sys_info *info, ihandle_t dev) PUSH(size); feval("saved-program-state >sps.file-size !"); feval("aout saved-program-state >sps.file-type !"); + PUSH(entry + size); + feval("saved-program-state >sps.end !");
feval("-1 state-valid !");
diff --git a/openbios-devel/libopenbios/bootcode_load.c b/openbios-devel/libopenbios/bootcode_load.c index 0fabf55..5c09c87 100644 --- a/openbios-devel/libopenbios/bootcode_load.c +++ b/openbios-devel/libopenbios/bootcode_load.c @@ -89,6 +89,8 @@ bootcode_load(ihandle_t dev) PUSH(size); feval("saved-program-state >sps.file-size !"); feval("bootcode saved-program-state >sps.file-type !"); + PUSH(bootcode); + feval("saved-program-state >sps.end !");
feval("-1 state-valid !");
diff --git a/openbios-devel/libopenbios/elf_load.c b/openbios-devel/libopenbios/elf_load.c index 9c7850e..236b893 100644 --- a/openbios-devel/libopenbios/elf_load.c +++ b/openbios-devel/libopenbios/elf_load.c @@ -478,6 +478,7 @@ elf_init_program(void) Elf_phdr *phdr; size_t size, total_size = 0; char *addr; + char *end = NULL; uintptr_t tmp;
/* TODO: manage ELF notes section */ @@ -519,6 +520,8 @@ elf_init_program(void)
memcpy(addr, base + phdr[i].p_offset, size);
+ if (addr + phdr[i].p_memsz > end) + end = addr + phdr[i].p_memsz; total_size += size;
#ifdef CONFIG_PPC @@ -532,6 +535,8 @@ elf_init_program(void) PUSH(total_size); feval("saved-program-state >sps.file-size !"); feval("elf saved-program-state >sps.file-type !"); + PUSH((ucell)end); + feval("saved-program-state >sps.end !");
feval("-1 state-valid !"); } diff --git a/openbios-devel/libopenbios/fcode_load.c b/openbios-devel/libopenbios/fcode_load.c index f4eb7bf..3f3fbbf 100644 --- a/openbios-devel/libopenbios/fcode_load.c +++ b/openbios-devel/libopenbios/fcode_load.c @@ -81,6 +81,8 @@ fcode_load(ihandle_t dev) PUSH(size); feval("saved-program-state >sps.file-size !"); feval("fcode saved-program-state >sps.file-type !"); + PUSH(start + size); + feval("saved-program-state >sps.end !");
feval("-1 state-valid !");
diff --git a/openbios-devel/libopenbios/forth_load.c b/openbios-devel/libopenbios/forth_load.c index c3a1929..403f1e5 100644 --- a/openbios-devel/libopenbios/forth_load.c +++ b/openbios-devel/libopenbios/forth_load.c @@ -69,6 +69,8 @@ int forth_load(ihandle_t dev) PUSH((ucell)forthsize); feval("saved-program-state >sps.file-size !"); feval("forth saved-program-state >sps.file-type !"); + PUSH(0); + feval("saved-program-state >sps.end !");
feval("-1 state-valid !");
diff --git a/openbios-devel/libopenbios/xcoff_load.c b/openbios-devel/libopenbios/xcoff_load.c index 0dcb28c..3eb8574 100644 --- a/openbios-devel/libopenbios/xcoff_load.c +++ b/openbios-devel/libopenbios/xcoff_load.c @@ -60,6 +60,7 @@ xcoff_init_program(void) uint32_t offset; size_t total_size = 0; int i; + ucell end = 0;
feval("0 state-valid !");
@@ -113,6 +114,8 @@ xcoff_init_program(void)
memcpy((char*)(uintptr_t)shdr->s_vaddr, base + shdr->s_scnptr, shdr->s_size); + if (shdr->s_vaddr + shdr->s_size > end) + end = shdr->s_vaddr + shdr->s_size; total_size += shdr->s_size; #ifdef CONFIG_PPC flush_icache_range((char*)(uintptr_t)shdr->s_vaddr, @@ -122,11 +125,15 @@ xcoff_init_program(void)
memcpy((char*)(uintptr_t)shdr->s_vaddr, base + shdr->s_scnptr, shdr->s_size); + if (shdr->s_vaddr + shdr->s_size > end) + end = shdr->s_vaddr + shdr->s_size; total_size += shdr->s_size;
} else if (strcmp(shdr->s_name, ".bss") == 0) {
memset((void *)(uintptr_t)shdr->s_vaddr, 0, shdr->s_size); + if (shdr->s_vaddr + shdr->s_size > end) + end = shdr->s_vaddr + shdr->s_size; total_size += shdr->s_size; } else { DPRINTF(" Skip '%s' section\n", shdr->s_name); @@ -142,6 +149,8 @@ xcoff_init_program(void) PUSH(total_size); feval("saved-program-state >sps.file-size !"); feval("xcoff saved-program-state >sps.file-type !"); + PUSH(end); + feval("saved-program-state >sps.end !");
feval("-1 state-valid !"); }
On 19/06/14 22:19, BALATON Zoltan wrote:
With the patch below I don't get DSI-s but it crashes in an ISI while trying to replace the sr registers. How could this possibly work on real hardware without getting any exceptions during replacing the vectors? Is there a way to preload the TLB with code addresses too so we can avoid the crash? How? (For data accessing it before calling the boot code is enough to preload translations but how to avoid ISI-s?)
Unfortunately, as far as I know there are only a few PPC processors that allow you to lock entries in the TLB and none of them were used in Macs.
As for the code that generates the ISIs, is this in MorphOS as opposed to OpenBIOS? I guess something must have previously accessed an entry on the same page before the registers were updated, or maybe there is some kind of hardware readahead?
ATB,
Mark.
On Fri, 20 Jun 2014, Mark Cave-Ayland wrote:
On 19/06/14 22:19, BALATON Zoltan wrote:
With the patch below I don't get DSI-s but it crashes in an ISI while trying to replace the sr registers. How could this possibly work on real hardware without getting any exceptions during replacing the vectors? Is there a way to preload the TLB with code addresses too so we can avoid the crash? How? (For data accessing it before calling the boot code is enough to preload translations but how to avoid ISI-s?)
Unfortunately, as far as I know there are only a few PPC processors that allow you to lock entries in the TLB and none of them were used in Macs.
As for the code that generates the ISIs, is this in MorphOS as opposed to OpenBIOS? I guess something must have previously accessed an entry on the same page before the registers were updated, or maybe there is some kind of hardware readahead?
The code is in the MorphOS boot loader and what it does is trying to take over memory management. Unfortunately it seems there is a period when it already replaced the vectors but have not set up the TLB hash table yet so it cannot actually handle exceptions. I could prevent DSIs but running the code during this period generates ISI-s. If the code is run in the same order on real hardware then it's not likely that the page is accessed there and not on QEMU. A readahead could explain it but I don't know if that happens. I have no better idea now than manually generating faults for all pages where the client code is loaded before calling it. I'll try to implement that unless someone can suggest a better solution.
Regards, BALATON Zoltan
On Sat, 21 Jun 2014, BALATON Zoltan wrote:
On Fri, 20 Jun 2014, Mark Cave-Ayland wrote:
As for the code that generates the ISIs, is this in MorphOS as opposed to OpenBIOS? I guess something must have previously accessed an entry on the same page before the registers were updated, or maybe there is some kind of hardware readahead?
The code is in the MorphOS boot loader and what it does is trying to take over memory management. Unfortunately it seems there is a period when it already replaced the vectors but have not set up the TLB hash table yet so it cannot actually handle exceptions. I could prevent DSIs but running the code during this period generates ISI-s. If the code is run in the same order on real hardware then it's not likely that the page is accessed there and not on QEMU. A readahead could explain it but I don't know if that happens. I have no better idea now than manually generating faults for all pages where the client code is loaded before calling it. I'll try to implement that unless someone can suggest a better solution.
Experimenting with it some more I could not make it work that way probably because by the time the ISI happens the vectors are already replaced and the sr0 register is overwritten so even if I manage to put translations in our hash table for that code that won't be correct any more by the time the ISI that causes a crash is handled.
So the only things that might work are using IBATs or disabling the MMU bits when MorphOS overwrites the vectors. I think this latter option is a bit cleaner but how can I get an interrupt when a memory address is written to? (Apart from setting up a watch point for it.) In the OpenBIOS code there are comments stating that page 0 is not mapped to catch NULL pointer dereferences but this does not seem to work as MorphOS can write over page 0 and only get an exception when reaching the next page. (Also I've found documented in one version of the PPC programming environments document that 0x00-0xff is reserved for operating system use so they can legally write there.) I'm sure I'm missing something here again.
It could possibly work on real hardware because of caches and that's where the read ahead might happen during the critical part. This is what seems to happen during MorphOS's MMU take over as far as I understand:
1. memcpy to 0, len=0x2000 2. fixup jumps and write base address to 0x80 (this is what's zeroed at the earlier write at the beginning) 3. Set MSR_BE and tweak HID0 (this may cause code to be preloaded in cache?) 4. Set sr0-15 from stack variables 5. Values for IBAT and DBAT registers are loaded from the stack and SDR1 is set to 0 dropping the hash table 6. MSR_DR and MSR_IR are cleared 7. BAT registers are set up as shown in this message: http://www.openfirmware.info/pipermail/openbios/2014-June/008419.html which means that the stack should be within the first 256MB 8. TLB entries are invalidated then MMU bits are re-enabled (but hashed page tables are not there yet so it relies on BATs at this point) 9. After doing some other stuff eventually a function is called that sets up the hash table 10. Vectors are replaced again later during boot after the microkernel has started its servers
Currently we get a DSI at 0x1000 in step 1. and crash due to our handler already replaced, probably we don't get a DSI earlier due to a prevoius zeroing of 0x80 at the very beginning while our handler still works so we have a translation for that. If we managed to get through this we get a DSI in step 5. while accessing the stack variable and crash there or get an ISI running code that replaces sr0 and crash due to that when managed to avoid DSIs. I have patches to move the stack to after the executable as was shown in my oftest results and clearing it plus writing to 0x1000 before calling the client code can avoid DSI-s but I still get the ISI. Alternatively adding code to ofmem_arch_map_pages which is called from ofmem_map to add translations to the hash table like this:
void ofmem_arch_map_pages(phys_addr_t phys, ucell virt, ucell size, ucell mode) { unsigned int i;
if (virt < OF_CODE_START) { for (i = 0; i * PAGE_SIZE < size; i++) { hash_page(virt + i * PAGE_SIZE, phys + i * PAGE_SIZE, mode); } } }
also seems to help with DSIs but cannot avoid the ISI (maybe because sr0 is replaced that could possible be used to look up the hash table).
Can anyone see a way through this or has an idea what would be an acceptable hack to allow this code run? I'm out of ideas now.
Regards, BALATON Zoltan
On 25.06.14 01:21, BALATON Zoltan wrote:
On Sat, 21 Jun 2014, BALATON Zoltan wrote:
On Fri, 20 Jun 2014, Mark Cave-Ayland wrote:
As for the code that generates the ISIs, is this in MorphOS as opposed to OpenBIOS? I guess something must have previously accessed an entry on the same page before the registers were updated, or maybe there is some kind of hardware readahead?
The code is in the MorphOS boot loader and what it does is trying to take over memory management. Unfortunately it seems there is a period when it already replaced the vectors but have not set up the TLB hash table yet so it cannot actually handle exceptions. I could prevent DSIs but running the code during this period generates ISI-s. If the code is run in the same order on real hardware then it's not likely that the page is accessed there and not on QEMU. A readahead could explain it but I don't know if that happens. I have no better idea now than manually generating faults for all pages where the client code is loaded before calling it. I'll try to implement that unless someone can suggest a better solution.
Experimenting with it some more I could not make it work that way probably because by the time the ISI happens the vectors are already replaced and the sr0 register is overwritten so even if I manage to put translations in our hash table for that code that won't be correct any more by the time the ISI that causes a crash is handled.
So the only things that might work are using IBATs or disabling the MMU bits when MorphOS overwrites the vectors. I think this latter option is a bit cleaner but how can I get an interrupt when a memory address is written to? (Apart from setting up a watch point for it.) In the OpenBIOS code there are comments stating that page 0 is not mapped to catch NULL pointer dereferences but this does not seem to work as MorphOS can write over page 0 and only get an exception when reaching the next page. (Also I've found documented in one version of the PPC programming environments document that 0x00-0xff is reserved for operating system use so they can legally write there.) I'm sure I'm missing something here again.
It could possibly work on real hardware because of caches and that's where the read ahead might happen during the critical part. This is what seems to happen during MorphOS's MMU take over as far as I understand:
- memcpy to 0, len=0x2000
- fixup jumps and write base address to 0x80 (this is what's zeroed at the earlier write at the beginning)
- Set MSR_BE and tweak HID0 (this may cause code to be preloaded in cache?)
- Set sr0-15 from stack variables
- Values for IBAT and DBAT registers are loaded from the stack and SDR1 is set to 0 dropping the hash table
- MSR_DR and MSR_IR are cleared
- BAT registers are set up as shown in this message:
http://www.openfirmware.info/pipermail/openbios/2014-June/008419.html which means that the stack should be within the first 256MB 8. TLB entries are invalidated then MMU bits are re-enabled (but hashed page tables are not there yet so it relies on BATs at this point) 9. After doing some other stuff eventually a function is called that sets up the hash table 10. Vectors are replaced again later during boot after the microkernel has started its servers
Shouldn't the above sequence work fine if it's executed with MSR_IR=0 MSR_DR=0 from the beginning?
Alex
On 25.06.14 01:30, Alexander Graf wrote:
On 25.06.14 01:21, BALATON Zoltan wrote:
On Sat, 21 Jun 2014, BALATON Zoltan wrote:
On Fri, 20 Jun 2014, Mark Cave-Ayland wrote:
As for the code that generates the ISIs, is this in MorphOS as opposed to OpenBIOS? I guess something must have previously accessed an entry on the same page before the registers were updated, or maybe there is some kind of hardware readahead?
The code is in the MorphOS boot loader and what it does is trying to take over memory management. Unfortunately it seems there is a period when it already replaced the vectors but have not set up the TLB hash table yet so it cannot actually handle exceptions. I could prevent DSIs but running the code during this period generates ISI-s. If the code is run in the same order on real hardware then it's not likely that the page is accessed there and not on QEMU. A readahead could explain it but I don't know if that happens. I have no better idea now than manually generating faults for all pages where the client code is loaded before calling it. I'll try to implement that unless someone can suggest a better solution.
Experimenting with it some more I could not make it work that way probably because by the time the ISI happens the vectors are already replaced and the sr0 register is overwritten so even if I manage to put translations in our hash table for that code that won't be correct any more by the time the ISI that causes a crash is handled.
So the only things that might work are using IBATs or disabling the MMU bits when MorphOS overwrites the vectors. I think this latter option is a bit cleaner but how can I get an interrupt when a memory address is written to? (Apart from setting up a watch point for it.) In the OpenBIOS code there are comments stating that page 0 is not mapped to catch NULL pointer dereferences but this does not seem to work as MorphOS can write over page 0 and only get an exception when reaching the next page. (Also I've found documented in one version of the PPC programming environments document that 0x00-0xff is reserved for operating system use so they can legally write there.) I'm sure I'm missing something here again.
It could possibly work on real hardware because of caches and that's where the read ahead might happen during the critical part. This is what seems to happen during MorphOS's MMU take over as far as I understand:
- memcpy to 0, len=0x2000
- fixup jumps and write base address to 0x80 (this is what's zeroed at the earlier write at the beginning)
- Set MSR_BE and tweak HID0 (this may cause code to be preloaded in cache?)
- Set sr0-15 from stack variables
- Values for IBAT and DBAT registers are loaded from the stack and SDR1 is set to 0 dropping the hash table
- MSR_DR and MSR_IR are cleared
- BAT registers are set up as shown in this message:
http://www.openfirmware.info/pipermail/openbios/2014-June/008419.html which means that the stack should be within the first 256MB 8. TLB entries are invalidated then MMU bits are re-enabled (but hashed page tables are not there yet so it relies on BATs at this point) 9. After doing some other stuff eventually a function is called that sets up the hash table 10. Vectors are replaced again later during boot after the microkernel has started its servers
Shouldn't the above sequence work fine if it's executed with MSR_IR=0 MSR_DR=0 from the beginning?
Or if you want to keep using virtual paging, just add iBAT and dBAT entries for the first 256MB when you see MorphOS? Maybe it assumes that it gets booted with firmware bolting itself via BATs, not the HTAB.
Alex
On Wed, 25 Jun 2014, Alexander Graf wrote:
On 25.06.14 01:21, BALATON Zoltan wrote:
On Sat, 21 Jun 2014, BALATON Zoltan wrote:
On Fri, 20 Jun 2014, Mark Cave-Ayland wrote:
As for the code that generates the ISIs, is this in MorphOS as opposed to OpenBIOS? I guess something must have previously accessed an entry on the same page before the registers were updated, or maybe there is some kind of hardware readahead?
The code is in the MorphOS boot loader and what it does is trying to take over memory management. Unfortunately it seems there is a period when it already replaced the vectors but have not set up the TLB hash table yet so it cannot actually handle exceptions. I could prevent DSIs but running the code during this period generates ISI-s. If the code is run in the same order on real hardware then it's not likely that the page is accessed there and not on QEMU. A readahead could explain it but I don't know if that happens. I have no better idea now than manually generating faults for all pages where the client code is loaded before calling it. I'll try to implement that unless someone can suggest a better solution.
Experimenting with it some more I could not make it work that way probably because by the time the ISI happens the vectors are already replaced and the sr0 register is overwritten so even if I manage to put translations in our hash table for that code that won't be correct any more by the time the ISI that causes a crash is handled.
So the only things that might work are using IBATs or disabling the MMU bits when MorphOS overwrites the vectors. I think this latter option is a bit cleaner but how can I get an interrupt when a memory address is written to? (Apart from setting up a watch point for it.) In the OpenBIOS code there are comments stating that page 0 is not mapped to catch NULL pointer dereferences but this does not seem to work as MorphOS can write over page 0 and only get an exception when reaching the next page. (Also I've found documented in one version of the PPC programming environments document that 0x00-0xff is reserved for operating system use so they can legally write there.) I'm sure I'm missing something here again.
It could possibly work on real hardware because of caches and that's where the read ahead might happen during the critical part. This is what seems to happen during MorphOS's MMU take over as far as I understand:
- memcpy to 0, len=0x2000
- fixup jumps and write base address to 0x80 (this is what's zeroed at the earlier write at the beginning)
- Set MSR_BE and tweak HID0 (this may cause code to be preloaded in cache?)
- Set sr0-15 from stack variables
- Values for IBAT and DBAT registers are loaded from the stack and SDR1 is set to 0 dropping the hash table
- MSR_DR and MSR_IR are cleared
- BAT registers are set up as shown in this message:
http://www.openfirmware.info/pipermail/openbios/2014-June/008419.html which means that the stack should be within the first 256MB 8. TLB entries are invalidated then MMU bits are re-enabled (but hashed page tables are not there yet so it relies on BATs at this point) 9. After doing some other stuff eventually a function is called that sets up the hash table 10. Vectors are replaced again later during boot after the microkernel has started its servers
Shouldn't the above sequence work fine if it's executed with MSR_IR=0 MSR_DR=0 from the beginning?
Yes it should and does if I do that from a debugger. Problem is that if I disable MSR_IR and MSR_DR in call_elf (or quiesce) it freezes in the next client callback because OpenBIOS needs these enabled. So I'd need to disable these bits when the vectors are overwritten but I don't seem to get an interrupt there where I could check for this only after it's too late.
On Wed, 25 Jun 2014, Alexander Graf wrote:
Or if you want to keep using virtual paging, just add iBAT and dBAT entries for the first 256MB when you see MorphOS?
I've done this in the patch referred to in step 7. and it seems to work (plus it needs the stack to be moved). But how can I tell if we are running MorphOS?
Maybe it assumes that it gets booted with firmware bolting itself via BATs, not the HTAB.
Not according to the result of oftest which shows that on Apple BATs are not used and MSR is 0x3030 as on OpenBIOS.
Regards, BALATON Zoltan
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
On Sat, 21 Jun 2014, BALATON Zoltan wrote:
On Fri, 20 Jun 2014, Mark Cave-Ayland wrote:
As for the code that generates the ISIs, is this in MorphOS as opposed to OpenBIOS? I guess something must have previously accessed an entry on the same page before the registers were updated, or maybe there is some kind of hardware readahead?
The code is in the MorphOS boot loader and what it does is trying to take over memory management. Unfortunately it seems there is a period when it already replaced the vectors but have not set up the TLB hash table yet so it cannot actually handle exceptions. I could prevent DSIs but running the code during this period generates ISI-s. If the code is run in the same order on real hardware then it's not likely that the page is accessed there and not on QEMU. A readahead could explain it but I don't know if that happens. I have no better idea now than manually generating faults for all pages where the client code is loaded before calling it. I'll try to implement that unless someone can suggest a better solution.
Experimenting with it some more I could not make it work that way probably because by the time the ISI happens the vectors are already replaced and the sr0 register is overwritten so even if I manage to put translations in our hash table for that code that won't be correct any more by the time the ISI that causes a crash is handled.
So the only things that might work are using IBATs or disabling the MMU bits when MorphOS overwrites the vectors. I think this latter option is a bit cleaner but how can I get an interrupt when a memory address is written to? (Apart from setting up a watch point for it.) In the OpenBIOS code there are comments stating that page 0 is not mapped to catch NULL pointer dereferences but this does not seem to work as MorphOS can write over page 0 and only get an exception when reaching the next page. (Also I've found documented in one version of the PPC programming environments document that 0x00-0xff is reserved for operating system use so they can legally write there.) I'm sure I'm missing something here again.
It could possibly work on real hardware because of caches and that's where the read ahead might happen during the critical part. This is what seems to happen during MorphOS's MMU take over as far as I understand:
- memcpy to 0, len=0x2000
- fixup jumps and write base address to 0x80 (this is what's zeroed at
the earlier write at the beginning) 3. Set MSR_BE and tweak HID0 (this may cause code to be preloaded in cache?) 4. Set sr0-15 from stack variables 5. Values for IBAT and DBAT registers are loaded from the stack and SDR1 is set to 0 dropping the hash table 6. MSR_DR and MSR_IR are cleared 7. BAT registers are set up as shown in this message: http://www.openfirmware.info/pipermail/openbios/2014-June/008419.html which means that the stack should be within the first 256MB 8. TLB entries are invalidated then MMU bits are re-enabled (but hashed page tables are not there yet so it relies on BATs at this point) 9. After doing some other stuff eventually a function is called that sets up the hash table 10. Vectors are replaced again later during boot after the microkernel has started its servers
Currently we get a DSI at 0x1000 in step 1. and crash due to our handler already replaced, probably we don't get a DSI earlier due to a prevoius zeroing of 0x80 at the very beginning while our handler still works so we have a translation for that. If we managed to get through this we get a DSI in step 5. while accessing the stack variable and crash there or get an ISI running code that replaces sr0 and crash due to that when managed to avoid DSIs. I have patches to move the stack to after the executable as was shown in my oftest results and clearing it plus writing to 0x1000 before calling the client code can avoid DSI-s but I still get the ISI. Alternatively adding code to ofmem_arch_map_pages which is called from ofmem_map to add translations to the hash table like this:
void ofmem_arch_map_pages(phys_addr_t phys, ucell virt, ucell size, ucell mode) { unsigned int i;
if (virt < OF_CODE_START) { for (i = 0; i * PAGE_SIZE < size; i++) { hash_page(virt + i * PAGE_SIZE, phys + i * PAGE_SIZE, mode); } } }
also seems to help with DSIs but cannot avoid the ISI (maybe because sr0 is replaced that could possible be used to look up the hash table).
Can anyone see a way through this or has an idea what would be an acceptable hack to allow this code run? I'm out of ideas now.
I don't get how it works and why I'm still getting an ISI. Can someone shed some light on it please? Here's what I see:
OpenBIOS sets up MMU like this:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
helper_store_sr: reg=1 20000401 00000000 helper_store_sr: reg=2 20000402 00000000 helper_store_sr: reg=3 20000403 00000000 helper_store_sr: reg=4 20000404 00000000 helper_store_sr: reg=5 20000405 00000000 helper_store_sr: reg=6 20000406 00000000 helper_store_sr: reg=7 20000407 00000000 helper_store_sr: reg=8 20000408 00000000 helper_store_sr: reg=9 20000409 00000000 helper_store_sr: reg=10 2000040a 00000000 helper_store_sr: reg=11 2000040b 00000000 helper_store_sr: reg=12 2000040c 00000000 helper_store_sr: reg=13 2000040d 00000000 helper_store_sr: reg=14 2000040e 00000000 helper_store_sr: reg=15 2000040f 00000000
I've also added code to ofmem_arch_map_pages to add all mapped pages to the hashed page table now like this:
void ofmem_arch_map_pages(phys_addr_t phys, ucell virt, ucell size, ucell mode) { unsigned int i;
for (i = 0; i * PAGE_SIZE < size; i++) { hash_page(pa2va(phys + i * PAGE_SIZE), phys + i * PAGE_SIZE, mode); } }
It now should have a translation in the page table for the executable and stack as those are mapped when loading like this:
OFMEM: ofmem_claim_phys phys=81000000 size=01000000 align=00000000 OFMEM: ofmem_claim_virt virt=81000000 size=01000000 align=00000000 OFMEM: ofmem_map_page_range 81000000 -> 81000000 01000000 mode 0000006a filesz: 00284348 memsz: 0028731C p_offset: 00010000 p_vaddr 00400000 OFMEM: ofmem_claim 00400000 0028731c 00000000 OFMEM: ofmem_map_page_range 00400000 -> 00400000 00288000 mode 00000002 OFMEM: ofmem_claim 0068731c 00010000 00000000 OFMEM: ofmem_map_page_range 00687000 -> 00687000 00011000 mode 00000002
Actually the whole memory except page 0 is mapped in ofmem_init so everything should be translatable via the hash table. Despite of this I still get DSIs and ISIs although less frequently (why?) which are handled by OpenBIOS's handlers until MorphOS tries to take over the MMU like this:
IN: 0x004009ac: mfdbsr r0 ; I think this is really HID0 on a G4 CPU ; objdump says: 7c 10 fa a6 mfspr r0,1008 0x004009b0: oris r0,r0,32912 0x004009b4: li r9,-516 0x004009b8: ori r0,r0,132 0x004009bc: and r0,r0,r9 0x004009c0: isync
IN: 0x004009c4: sync 0x004009c8: isync
IN: 0x004009cc: mtdbsr r0 0x004009d0: isync
IN: 0x004009d4: sync 0x004009d8: isync
IN: 0x004009dc: b 0x400acc
IN: 0x00400acc: mfdbsr r0 0x00400ad0: mfl2cr r11 0x00400ad4: addis r9,r10,-1 0x00400ad8: addi r9,r9,32767 0x00400adc: cmplwi r9,1 0x00400ae0: bgt- 0x400af4
IN: 0x00400af4: mficcr r0 0x00400af8: mr r4,r16 0x00400afc: mr r5,r15 0x00400b00: addi r3,r1,16 0x00400b04: li r6,0 0x00400b08: bl 0x41cce8
IN: 0x0041cce8: stwu r1,-96(r1) 0x0041ccec: stmw r13,20(r1) 0x0041ccf0: lwz r0,0(r3) 0x0041ccf4: sync 0x0041ccf8: mtsr 0,r0 0x0041ccfc: isync
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
The instruction where the exception happens would load the next value for sr1 but it is an ISI so it's not because of data access:
41cd00: 80 03 00 04 lwz r0,4(r3) 41cd04: 7c 00 04 ac sync 41cd08: 7c 01 01 a4 mtsr 1,r0 41cd0c: 4c 00 01 2c isync 41cd10: 80 03 00 08 lwz r0,8(r3) 41cd14: 7c 00 04 ac sync 41cd18: 7c 02 01 a4 mtsr 2,r0
The rest just shows the path taken instead and the crash in the replaced exception handlers which reenable the MMU and get another exception infinitely:
IN: 0x00000400: mtsprg 2,r2 0x00000404: li r2,4 0x00000408: b 0x41f0d4
IN: 0x00000400: mtsprg 2,r2
IN: 0x00000404: li r2,4
IN: 0x00000408: b 0x41f0d4
IN: 0x0041f0d4: mtsprg 1,r1
IN: 0x0041f0d8: mfmsr r1
IN: 0x0041f0dc: ori r1,r1,12336
IN: 0x0041f0e0: sync
IN: 0x0041f0e4: mtmsr r1
Raise exception at 0041f0e8 => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2
Regards, BALATON Zoltan
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
[...]
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
I was mistaken about the values being the same as it is zeroing sr0. So can this explain why translation via the page table fails after this and why an ISI is generated? Why are the sr registers set up with the values above by OpenBIOS? Could they be 0 instead?
Regards, BALATON Zoltan
On 25.06.14 12:40, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
[...]
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
I was mistaken about the values being the same as it is zeroing sr0. So can this explain why translation via the page table fails after this and why an ISI is generated? Why are the sr registers set up with the values above by OpenBIOS? Could they be 0 instead?
SR registers are used to translate EAs to VAs. If you set them all to 0 they would end up getting the same VSID.
Alex
On Wed, 25 Jun 2014, Alexander Graf wrote:
On 25.06.14 12:40, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
[...]
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
I was mistaken about the values being the same as it is zeroing sr0. So can this explain why translation via the page table fails after this and why an ISI is generated? Why are the sr registers set up with the values above by OpenBIOS? Could they be 0 instead?
SR registers are used to translate EAs to VAs. If you set them all to 0 they would end up getting the same VSID.
OK but why is SEGR_BASE defined as 0x0400 in arch/ppc/qemu/ofmem.c? If I change this to 0 I get closer and it survives zeroing the sr registers but it cannot survive zeroing SDR1 and crashes after that happens:
IN: 0x0041cde0: lwz r0,60(r3)
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cde4: sync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cde8: mtsr 15,r0
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cdec: isync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 0000000000000697 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=0000000000000697 found PTE at offset 0000a5e0 PTE access granted !
helper_store_sr: reg=15 00000000 2000000f
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c found PTE at offset 00000728 PTE access granted !
IN: 0x0041cdf0: sync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cdf4: mtsdr1 r6
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cdf8: isync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
ppc_store_sdr1: 00000000
htab_base 0000000000000000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=0000000000000000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c 1 htab=0000000000000000/000000000000ffff vsid=0 api=1 hash=fffffffffffffbe3 Raise exception at 0041cdfc => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2
What comes after this is loading some values from the stack then disabling MMU bits in MSR (then putting the values loaded into BAT regs). If I could get these run without getting an ISI inbetween it would boot:
41ce0c: 82 84 00 14 lwz r20,20(r4) 41ce10: 82 a4 00 10 lwz r21,16(r4) 41ce14: 82 c4 00 1c lwz r22,28(r4) 41ce18: 82 e4 00 18 lwz r23,24(r4) 41ce1c: 83 05 00 04 lwz r24,4(r5) 41ce20: 83 25 00 00 lwz r25,0(r5) 41ce24: 83 45 00 0c lwz r26,12(r5) 41ce28: 83 65 00 08 lwz r27,8(r5) 41ce2c: 83 85 00 14 lwz r28,20(r5) 41ce30: 83 a5 00 10 lwz r29,16(r5) 41ce34: 83 c5 00 1c lwz r30,28(r5) 41ce38: 83 e5 00 18 lwz r31,24(r5) 41ce3c: 7c 00 04 ac sync 41ce40: 7c 00 00 a6 mfmsr r0 41ce44: 70 00 ff cf andi. r0,r0,65487 41ce48: 7c 00 04 ac sync 41ce4c: 7c 00 01 24 mtmsr r0 41ce50: 4c 00 01 2c isync
Why are we getting ISIs on QEMU and why can this work on real hardware without crashing?
Regards, BALATON Zoltan
On 26.06.14 01:36, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, Alexander Graf wrote:
On 25.06.14 12:40, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
[...]
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
I was mistaken about the values being the same as it is zeroing sr0. So can this explain why translation via the page table fails after this and why an ISI is generated? Why are the sr registers set up with the values above by OpenBIOS? Could they be 0 instead?
SR registers are used to translate EAs to VAs. If you set them all to 0 they would end up getting the same VSID.
OK but why is SEGR_BASE defined as 0x0400 in arch/ppc/qemu/ofmem.c?
I guess to make it easier to debug vs empty SR registers and to make sure it doesn't collide with a guest that sets SR registers to 0 temporarily?
What does real Apple firmware use for their VSIDs?
If I change this to 0 I get closer and it survives zeroing the sr registers but it cannot survive zeroing SDR1 and crashes after that happens:
IN: 0x0041cde0: lwz r0,60(r3)
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cde4: sync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cde8: mtsr 15,r0
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cdec: isync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 0000000000000697 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=0000000000000697 found PTE at offset 0000a5e0 PTE access granted !
helper_store_sr: reg=15 00000000 2000000f
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c found PTE at offset 00000728 PTE access granted !
IN: 0x0041cdf0: sync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cdf4: mtsdr1 r6
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
0x0041cdf8: isync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
ppc_store_sdr1: 00000000
htab_base 0000000000000000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=0000000000000000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c 1 htab=0000000000000000/000000000000ffff vsid=0 api=1 hash=fffffffffffffbe3 Raise exception at 0041cdfc => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2
What comes after this is loading some values from the stack then disabling MMU bits in MSR (then putting the values loaded into BAT regs). If I could get these run without getting an ISI inbetween it would boot:
41ce0c: 82 84 00 14 lwz r20,20(r4) 41ce10: 82 a4 00 10 lwz r21,16(r4) 41ce14: 82 c4 00 1c lwz r22,28(r4) 41ce18: 82 e4 00 18 lwz r23,24(r4) 41ce1c: 83 05 00 04 lwz r24,4(r5) 41ce20: 83 25 00 00 lwz r25,0(r5) 41ce24: 83 45 00 0c lwz r26,12(r5) 41ce28: 83 65 00 08 lwz r27,8(r5) 41ce2c: 83 85 00 14 lwz r28,20(r5) 41ce30: 83 a5 00 10 lwz r29,16(r5) 41ce34: 83 c5 00 1c lwz r30,28(r5) 41ce38: 83 e5 00 18 lwz r31,24(r5) 41ce3c: 7c 00 04 ac sync 41ce40: 7c 00 00 a6 mfmsr r0 41ce44: 70 00 ff cf andi. r0,r0,65487 41ce48: 7c 00 04 ac sync 41ce4c: 7c 00 01 24 mtmsr r0 41ce50: 4c 00 01 2c isync
Why are we getting ISIs on QEMU and why can this work on real hardware without crashing?
We are getting ISIs because the HTAB is now 0 bytes long, so it doesn't contain any entries. Maybe it works on real hardware by accident because the translations are still in the TLB?
I think with MorphOS you're best off detecting that you are running MorphOS somehow (checksum over loaded files, specific client interface calls only it does) and set MSR.DR=0 and MSR.IR=0.
Alex
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 01:36, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, Alexander Graf wrote:
On 25.06.14 12:40, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
[...]
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
I was mistaken about the values being the same as it is zeroing sr0. So can this explain why translation via the page table fails after this and why an ISI is generated? Why are the sr registers set up with the values above by OpenBIOS? Could they be 0 instead?
SR registers are used to translate EAs to VAs. If you set them all to 0 they would end up getting the same VSID.
OK but why is SEGR_BASE defined as 0x0400 in arch/ppc/qemu/ofmem.c?
I guess to make it easier to debug vs empty SR registers and to make sure it doesn't collide with a guest that sets SR registers to 0 temporarily?
What does real Apple firmware use for their VSIDs?
I'll modify oftest to print sr regs and try to rerun it to find out.
0x0041cdf8: isync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
ppc_store_sdr1: 00000000
htab_base 0000000000000000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=0000000000000000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c 1 htab=0000000000000000/000000000000ffff vsid=0 api=1 hash=fffffffffffffbe3 Raise exception at 0041cdfc => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2
What comes after this is loading some values from the stack then disabling MMU bits in MSR (then putting the values loaded into BAT regs). If I could get these run without getting an ISI inbetween it would boot:
41ce0c: 82 84 00 14 lwz r20,20(r4) 41ce10: 82 a4 00 10 lwz r21,16(r4) 41ce14: 82 c4 00 1c lwz r22,28(r4) 41ce18: 82 e4 00 18 lwz r23,24(r4) 41ce1c: 83 05 00 04 lwz r24,4(r5) 41ce20: 83 25 00 00 lwz r25,0(r5) 41ce24: 83 45 00 0c lwz r26,12(r5) 41ce28: 83 65 00 08 lwz r27,8(r5) 41ce2c: 83 85 00 14 lwz r28,20(r5) 41ce30: 83 a5 00 10 lwz r29,16(r5) 41ce34: 83 c5 00 1c lwz r30,28(r5) 41ce38: 83 e5 00 18 lwz r31,24(r5) 41ce3c: 7c 00 04 ac sync 41ce40: 7c 00 00 a6 mfmsr r0 41ce44: 70 00 ff cf andi. r0,r0,65487 41ce48: 7c 00 04 ac sync 41ce4c: 7c 00 01 24 mtmsr r0 41ce50: 4c 00 01 2c isync
Why are we getting ISIs on QEMU and why can this work on real hardware without crashing?
We are getting ISIs because the HTAB is now 0 bytes long, so it doesn't contain any entries. Maybe it works on real hardware by accident because the translations are still in the TLB?
If so then why is not that the case on QEMU and can it be changed? Is the TLB flushed differently than real hardware? But I suspect the cache to be a more likely explanation.
I think with MorphOS you're best off detecting that you are running MorphOS somehow (checksum over loaded files, specific client interface calls only it does) and set MSR.DR=0 and MSR.IR=0.
This is extremely fragile and dependent on an exact version of executable that I like to avoid. Is there a problem with always disabling MMU bits when our vectors are overwritten? I guess that a well behaved guest should disable these before overwriting the vectors and enable them afterwards and for a buggy guest this would prevent random crashes or may make it work as in case of MorphOS. The problem is I don't know how to detect if the vectors are overwritten. I though about checking it from dsi_exception handler, but here I don't get more than one hits not like the ISI-s I don't want to happen. Do any of you have an idea about how could this be done?
Regards, BALATON Zoltan
On 26.06.14 13:10, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 01:36, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, Alexander Graf wrote:
On 25.06.14 12:40, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
[...]
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
I was mistaken about the values being the same as it is zeroing sr0. So can this explain why translation via the page table fails after this and why an ISI is generated? Why are the sr registers set up with the values above by OpenBIOS? Could they be 0 instead?
SR registers are used to translate EAs to VAs. If you set them all to 0 they would end up getting the same VSID.
OK but why is SEGR_BASE defined as 0x0400 in arch/ppc/qemu/ofmem.c?
I guess to make it easier to debug vs empty SR registers and to make sure it doesn't collide with a guest that sets SR registers to 0 temporarily?
What does real Apple firmware use for their VSIDs?
I'll modify oftest to print sr regs and try to rerun it to find out.
0x0041cdf8: isync
htab_base 000000000fe00000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=000000000fe00000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c
ppc_store_sdr1: 00000000
htab_base 0000000000000000 htab_mask 000000000000ffff hash 000000000000041c 0 htab=0000000000000000/000000000000ffff vsid=0 ptem=1 hash=000000000000041c 1 htab=0000000000000000/000000000000ffff vsid=0 api=1 hash=fffffffffffffbe3 Raise exception at 0041cdfc => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2
What comes after this is loading some values from the stack then disabling MMU bits in MSR (then putting the values loaded into BAT regs). If I could get these run without getting an ISI inbetween it would boot:
41ce0c: 82 84 00 14 lwz r20,20(r4) 41ce10: 82 a4 00 10 lwz r21,16(r4) 41ce14: 82 c4 00 1c lwz r22,28(r4) 41ce18: 82 e4 00 18 lwz r23,24(r4) 41ce1c: 83 05 00 04 lwz r24,4(r5) 41ce20: 83 25 00 00 lwz r25,0(r5) 41ce24: 83 45 00 0c lwz r26,12(r5) 41ce28: 83 65 00 08 lwz r27,8(r5) 41ce2c: 83 85 00 14 lwz r28,20(r5) 41ce30: 83 a5 00 10 lwz r29,16(r5) 41ce34: 83 c5 00 1c lwz r30,28(r5) 41ce38: 83 e5 00 18 lwz r31,24(r5) 41ce3c: 7c 00 04 ac sync 41ce40: 7c 00 00 a6 mfmsr r0 41ce44: 70 00 ff cf andi. r0,r0,65487 41ce48: 7c 00 04 ac sync 41ce4c: 7c 00 01 24 mtmsr r0 41ce50: 4c 00 01 2c isync
Why are we getting ISIs on QEMU and why can this work on real hardware without crashing?
We are getting ISIs because the HTAB is now 0 bytes long, so it doesn't contain any entries. Maybe it works on real hardware by accident because the translations are still in the TLB?
If so then why is not that the case on QEMU and can it be changed? Is the TLB flushed differently than real hardware? But I suspect the cache to be a more likely explanation.
IIRC real hardware has a TLB and an ERAT, similar to L1 and L2 caches. QEMU only models a TLB - and even then takes the liberty of flushing entries whenever it wants to.
I think with MorphOS you're best off detecting that you are running MorphOS somehow (checksum over loaded files, specific client interface calls only it does) and set MSR.DR=0 and MSR.IR=0.
This is extremely fragile and dependent on an exact version of executable that I like to avoid. Is there a problem with always disabling MMU bits when our vectors are overwritten? I guess that a well behaved guest should disable these before overwriting the vectors and enable them afterwards and for a buggy guest this would prevent random crashes or may make it work as in case of MorphOS. The problem is I don't know how to detect if the vectors are overwritten. I though about checking it from dsi_exception handler, but here I don't get more than one hits not like the ISI-s I don't want to happen. Do any of you have an idea about how could this be done?
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Alex
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Regards, BALATON Zoltan
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch? I don't think any other OS really accesses these ranges, but only time will tell ;).
Alex
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch?
Definitely too early as this is the first thing it does.
I don't think any other OS really accesses these ranges, but only time will tell ;).
Don't know, according to previous replies they may. Then how about always clearing the translation for page zero the next time our handler is called so I always get a new exception when it's accessed? (I have to add it for the access to succeed but I may remove it at the next possible occasion.) In the lack of a better idea I'll try this.
Regards, BALATON Zoltan
On 26.06.14 13:36, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch?
Definitely too early as this is the first thing it does.
Too bad.
I don't think any other OS really accesses these ranges, but only time will tell ;).
Don't know, according to previous replies they may. Then how about always clearing the translation for page zero the next time our handler is called so I always get a new exception when it's accessed? (I have to add it for the access to succeed but I may remove it at the next possible occasion.) In the lack of a better idea I'll try this.
That sounds odd. Why don't you just implement a tiny instruction emulator? We only need to catch stores here, no?
Alex
On Thu, 26 Jun 2014, Alexander Graf wrote:
I don't think any other OS really accesses these ranges, but only time will tell ;).
Don't know, according to previous replies they may. Then how about always clearing the translation for page zero the next time our handler is called so I always get a new exception when it's accessed? (I have to add it for the access to succeed but I may remove it at the next possible occasion.) In the lack of a better idea I'll try this.
That sounds odd. Why don't you just implement a tiny instruction emulator? We only need to catch stores here, no?
I assume yes but I'm not sure and I don't know how to do that. Do you know about an example somewhere?
Regards, BALATON Zoltan
On 26.06.14 14:03, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
I don't think any other OS really accesses these ranges, but only time will tell ;).
Don't know, according to previous replies they may. Then how about always clearing the translation for page zero the next time our handler is called so I always get a new exception when it's accessed? (I have to add it for the access to succeed but I may remove it at the next possible occasion.) In the lack of a better idea I'll try this.
That sounds odd. Why don't you just implement a tiny instruction emulator? We only need to catch stores here, no?
I assume yes but I'm not sure and I don't know how to do that. Do you know about an example somewhere?
http://www.mail-archive.com/kvm@vger.kernel.org/msg104190.html
vaddr_accessed is really just DAR at the time of the interrupt. I think you can reuse pretty much all of that code and remove everything related to load instructions.
Alex
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch? I don't think any other OS really accesses these ranges, but only time will tell ;).
As a start I've tried this patch:
--- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) { - unsigned long dar, dsisr; + unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : ); - + asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode); - hash_page(dar, phys, mode); + + if (dsisr & BIT(1)) { + /* handle page fault */ + hash_page(dar, phys, mode); + } + + if (dsisr & BIT(4) && dar == 0) { + /* handle protection violation */ + hash_page(dar, phys, mode); + srr1 &= ~(MSR_IR | MSR_DR); + asm volatile("mtsrr1 %0" :: "r" (srr1)); + } }
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */ + /* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0); + hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the OpenBIOS ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top()
which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunatly it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects.
I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
Regards, BALATON Zoltan
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 01:36, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, Alexander Graf wrote:
On 25.06.14 12:40, BALATON Zoltan wrote:
On Wed, 25 Jun 2014, BALATON Zoltan wrote:
ppc_store_sdr1: 0fe00000 helper_store_sr: reg=0 20000400 00000000
[...]
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000)
^^^ This exception should not happen. It is trying to handle it but the handlers are not working yet and gets in an infinite loop. It boots if MMU is disabled while this part runs but MorphOS does not disable it yet and according to my oftest results they are enabled on Apple too. How does it work on real hardware and why does it fail on QEMU? (Note the the value of sr0 is identical to the one set by OpenBIOS and SDR1 is unchanged so translations via the page table should still work, shouldn't it?)
I was mistaken about the values being the same as it is zeroing sr0. So can this explain why translation via the page table fails after this and why an ISI is generated? Why are the sr registers set up with the values above by OpenBIOS? Could they be 0 instead?
SR registers are used to translate EAs to VAs. If you set them all to 0 they would end up getting the same VSID.
OK but why is SEGR_BASE defined as 0x0400 in arch/ppc/qemu/ofmem.c?
I guess to make it easier to debug vs empty SR registers and to make sure it doesn't collide with a guest that sets SR registers to 0 temporarily?
What does real Apple firmware use for their VSIDs?
I updated oftest to print sr registers too. From a test on iMac,1 it seems the values are just sr0=0, sr1=1, ... sr15=15.
Regards, BALATON Zoltan
Hello,
Great to see progress with the Mac99 emulation. I got less time for it lately but can someone please answer this question?
On Fri, 27 Jun 2014, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch? I don't think any other OS really accesses these ranges, but only time will tell ;).
As a start I've tried this patch:
--- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the OpenBIOS ROM
in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top()
which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunately it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects.
I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
Also what about my OpenBIOS patches that are not yet merged? Will someone get to that eventually?
Regards, BALATON Zoltan
On 27.06.14 23:34, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch? I don't think any other OS really accesses these ranges, but only time will tell ;).
As a start I've tried this patch:
--- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the
OpenBIOS ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top()
which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunatly it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects.
I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
They have to be somewhere, because we have to swap them all back in after the interrupt.
Alex
On 14.07.14 01:03, BALATON Zoltan wrote:
Hello,
Great to see progress with the Mac99 emulation. I got less time for it lately but can someone please answer this question?
On Fri, 27 Jun 2014, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch? I don't think any other OS really accesses these ranges, but only time will tell ;).
As a start I've tried this patch:
--- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the
OpenBIOS ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top()
which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunately it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects.
I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
Also what about my OpenBIOS patches that are not yet merged? Will someone get to that eventually?
Mark, would you like to pick them up? My patch apply workflow for OpenBIOS is ... suboptimal ;).
Alex
On 28/07/14 14:38, Alexander Graf wrote:
Also what about my OpenBIOS patches that are not yet merged? Will someone get to that eventually?
Mark, would you like to pick them up? My patch apply workflow for OpenBIOS is ... suboptimal ;).
Ha ;) I've no objection to patch, I just would like someone who knows about licensing to give the okay before bringing the libpayload code into OpenBIOS. I'll try Stefan again...
ATB,
Mark.
On Mon, 28 Jul 2014, Alexander Graf wrote:
On 27.06.14 23:34, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch? I don't think any other OS really accesses these ranges, but only time will tell ;).
As a start I've tried this patch:
--- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the OpenBIOS
ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top()
which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunatly it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects.
I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
They have to be somewhere, because we have to swap them all back in after the interrupt.
They are probably on a stack somewhere but the handlers replace the stack if I remember correctly so it's not trivial to access that from C code. But my question was primarily about why the exception is retriggering? Do I need to do something else to signal to the CPU that the protection violation is handled or how can I ignore an exception so the execution continues?
Regards, BALATON Zoltan
Am 30.07.2014 um 16:42 schrieb BALATON Zoltan balaton@eik.bme.hu:
On Mon, 28 Jul 2014, Alexander Graf wrote:
On 27.06.14 23:34, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote:
On 26.06.14 13:20, BALATON Zoltan wrote:
On Thu, 26 Jun 2014, Alexander Graf wrote: You could enable write protection on the first page before you enter the payload. Then you could unprotect it and disable interrupts as soon as someone wrote to it. I guess that'd be a pretty solid hack.
Good idea, I'll look into this. But 0x00-0xff is reserved for operating system use and MorphOS does write to 0x80 before touching the vectors (that's why I can't just check from the DSI handler). Can I selectively enable writes on a write protected page? (I'll need to read about it some more.)
Not easily. You could emulate the 0x80 write maybe. Is it too early if you just disable DR/IR on the 0x80 touch? I don't think any other OS really accesses these ranges, but only time will tell ;).
As a start I've tried this patch: --- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the OpenBIOS ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top()
which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunatly it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects. I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
They have to be somewhere, because we have to swap them all back in after the interrupt.
They are probably on a stack somewhere but the handlers replace the stack if I remember correctly so it's not trivial to access that from C code.
Then pass a pointer to them to the C handler code?
But my question was primarily about why the exception is retriggering? Do I need to do something else to signal to the CPU that the protection violation is handled or how can I ignore an exception so the execution continues?
If you don't fix up the reason why the interrupt occured in the first place it will simply happen again.
Alex
On Wed, 30 Jul 2014, Alexander Graf wrote:
Am 30.07.2014 um 16:42 schrieb BALATON Zoltan balaton@eik.bme.hu:
As a start I've tried this patch: --- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the OpenBIOS ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top()
which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunatly it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects. I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
They have to be somewhere, because we have to swap them all back in after the interrupt.
They are probably on a stack somewhere but the handlers replace the stack if I remember correctly so it's not trivial to access that from C code.
Then pass a pointer to them to the C handler code?
Good idea, but I'd like to make it work first before starting emulating instructions.
But my question was primarily about why the exception is retriggering? Do I need to do something else to signal to the CPU that the protection violation is handled or how can I ignore an exception so the execution continues?
If you don't fix up the reason why the interrupt occured in the first place it will simply happen again.
I don't get it. The exception happens because of a write to 0x80 which is in a protected page. (The value written is the same that is already in that memory cell by the way so no action is needed.) What could I do to "fix up the reason" so that the exception does not happen again after returning from the handler? I thought that simply ignoring the write and returning from the handler should make it continue execution but it does not seem to work that way. I'm clearly missing something but I don't know what. Can you spot a problem in the code above?
Regards, BALATON Zoltan
On 30.07.14 17:47, BALATON Zoltan wrote:
On Wed, 30 Jul 2014, Alexander Graf wrote:
Am 30.07.2014 um 16:42 schrieb BALATON Zoltan balaton@eik.bme.hu:
As a start I've tried this patch: --- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the
OpenBIOS ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top() which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunatly it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects. I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
They have to be somewhere, because we have to swap them all back in after the interrupt.
They are probably on a stack somewhere but the handlers replace the stack if I remember correctly so it's not trivial to access that from C code.
Then pass a pointer to them to the C handler code?
Good idea, but I'd like to make it work first before starting emulating instructions.
But my question was primarily about why the exception is retriggering? Do I need to do something else to signal to the CPU that the protection violation is handled or how can I ignore an exception so the execution continues?
If you don't fix up the reason why the interrupt occured in the first place it will simply happen again.
I don't get it. The exception happens because of a write to 0x80 which is in a protected page. (The value written is the same that is already in that memory cell by the way so no action is needed.) What could I do to "fix up the reason" so that the exception does not happen again after returning from the handler? I thought that simply ignoring the write and returning from the handler should make it continue execution but it does not seem to work that way. I'm clearly missing something but I don't know what. Can you spot a problem in the code above?
If you just return you return to the same instruction that caused the interrupt, so it will trigger again. You need to return to SRR0+4 if you want to "skip" the instruction.
The code above doesn't skip - it turns off paging. First off I'm not sure whether a & b && c actually does what you like it to do. Other that than, I guess your best bet would be gdbstub and a breakpoint to see whether you hit that point ;)
Alex
On Wed, 30 Jul 2014, Alexander Graf wrote:
On 30.07.14 17:47, BALATON Zoltan wrote:
On Wed, 30 Jul 2014, Alexander Graf wrote:
Am 30.07.2014 um 16:42 schrieb BALATON Zoltan balaton@eik.bme.hu:
As a start I've tried this patch: --- a/openbios-devel/arch/ppc/qemu/ofmem.c +++ b/openbios-devel/arch/ppc/qemu/ofmem.c @@ -460,15 +460,26 @@ static void hash_page(unsigned long ea, phys_addr_t phys, void dsi_exception(void) {
- unsigned long dar, dsisr;
unsigned long dar, dsisr, srr1; ucell mode; phys_addr_t phys;
asm volatile("mfdar %0" : "=r" (dar) : ); asm volatile("mfdsisr %0" : "=r" (dsisr) : );
- asm volatile("mfsrr1 %0" : "=r" (srr1) : ); phys = ea_to_phys(dar, &mode);
- hash_page(dar, phys, mode);
- if (dsisr & BIT(1)) {
/* handle page fault */hash_page(dar, phys, mode);- }
- if (dsisr & BIT(4) && dar == 0) {
/* handle protection violation */hash_page(dar, phys, mode);srr1 &= ~(MSR_IR | MSR_DR);asm volatile("mtsrr1 %0" :: "r" (srr1));- }
}
void @@ -554,9 +565,10 @@ ofmem_init(void) ofmem_claim_virt(PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0); ofmem_map(PAGE_SIZE, PAGE_SIZE, get_ram_bottom() - PAGE_SIZE, 0);
- /* Mark the first page as non-free */
/* Mark the first page as non-free and write protect it */ ofmem_claim_phys(0, PAGE_SIZE, 0); ofmem_claim_virt(0, PAGE_SIZE, 0);
hash_page(0, 0, 3);
/* Map everything at the top of physical RAM 1:1, minus the
OpenBIOS ROM in ofmem_claim_phys(get_ram_top(), get_hash_base() + HASH_SIZE - get_ram_top() which does not break Finnix but does not work with MorphOS because it catches the write to 0x80 as it should but instead of ignoring it this protection violation exception is always retrigerring infinitely and it does not go further. What am I missing to ignore protection violations for writes to page 0 without emulating the writes at the moment. (The initial write to 0x80 is setting it to 0 which is the value it already has.) Unfortunatly it is hard to debug because if I call printk from the exception handler it seems to break beyond repair possibly due to side effects. I've also looked at the code you've referred to but that uses kvm functions not included in the patch so it may be more complicated than that. If I get it correctly I can get the instruction from the address in srr0, the target memory cell from dar but still need to find out the source which is probably a register that does not contain the value by the time I get it so it may not be trivial to emulate the write.
They have to be somewhere, because we have to swap them all back in after the interrupt.
They are probably on a stack somewhere but the handlers replace the stack if I remember correctly so it's not trivial to access that from C code.
Then pass a pointer to them to the C handler code?
Good idea, but I'd like to make it work first before starting emulating instructions.
But my question was primarily about why the exception is retriggering? Do I need to do something else to signal to the CPU that the protection violation is handled or how can I ignore an exception so the execution continues?
If you don't fix up the reason why the interrupt occured in the first place it will simply happen again.
I don't get it. The exception happens because of a write to 0x80 which is in a protected page. (The value written is the same that is already in that memory cell by the way so no action is needed.) What could I do to "fix up the reason" so that the exception does not happen again after returning from the handler? I thought that simply ignoring the write and returning from the handler should make it continue execution but it does not seem to work that way. I'm clearly missing something but I don't know what. Can you spot a problem in the code above?
If you just return you return to the same instruction that caused the interrupt, so it will trigger again. You need to return to SRR0+4 if you want to "skip" the instruction.
Ahh, OK. Probably this it what I was missing.
The code above doesn't skip - it turns off paging. First off I'm not sure
What I intended is to turn off page protection on first write to address 0 and should ignore the write to 0x80 before that.
whether a & b && c actually does what you like it to do. Other that than, I
The & should be higher precedence than && but adding paranthesis to make it clearer would not hurt.
guess your best bet would be gdbstub and a breakpoint to see whether you hit that point ;)
The handler is called but I could experiment with what happens after returning. Thanks for the ideas, I'll try to make it work if I find some time for it.
Regards, BALATON Zoltan
On 28/07/14 14:38, Alexander Graf wrote:
Also what about my OpenBIOS patches that are not yet merged? Will someone get to that eventually?
Mark, would you like to pick them up? My patch apply workflow for OpenBIOS is ... suboptimal ;).
Well I've finally had confirmation from Stefan that the libpayload license is compatible with OpenBIOS so I'm okay for the USB driver to go in if you are? Does it require any simultaneous changes to QEMU in order for things not to break?
ATB,
Mark.
On Mon, 4 Aug 2014, Mark Cave-Ayland wrote:
On 28/07/14 14:38, Alexander Graf wrote:
Also what about my OpenBIOS patches that are not yet merged? Will someone get to that eventually?
Mark, would you like to pick them up? My patch apply workflow for OpenBIOS is ... suboptimal ;).
Well I've finally had confirmation from Stefan that the libpayload license is compatible with OpenBIOS so I'm okay for the USB driver to go in if you are? Does it require any simultaneous changes to QEMU in order for things not to break?
I don't know about any needed changes to QEMU. All my QEMU patches are in the latest release already and I have only one further patch to use USB keyboard in mac99 after the driver is taken in OpenBIOS and reaches QEMU (until then it should still work with ADB on 32bit version and 64bit already uses USB).
There are some more pending patches for OpenBIOS though. After handling the USB driver can you look at those too please? These are:
http://www.openfirmware.info/pipermail/openbios/2014-June/008448.html (This depends on PCI_COMMAND_BUS_MASTER added by the USB driver.)
http://www.openfirmware.info/pipermail/openbios/2014-June/008428.html
http://www.openfirmware.info/pipermail/openbios/2014-June/008427.html or http://www.openfirmware.info/pipermail/openbios/2014-June/008369.html whichever you prefer.
Regards, BALATON Zoltan
-
Alexander Graf -
BALATON Zoltan -
Mark Cave-Ayland -
Segher Boessenkool