On 09/06/14 01:27, BALATON Zoltan wrote:
On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
It might be that MorphOS just got lucky that this works on real hardware without faulting (similar to the bug I had for SPARC64) but some further analysis would help. Does QEMU ppc have a DEBUG_TLB or similar option so you can see when the eviction is triggered? Sadly I think the options here are quite limited :/
There are some debug options in qemu/target-ppc/mmu_helper.c but they don't give much clue and the DUMP_PAGE_TABLES option does not work because it does not compile if enabled (something seems to have changed leaving debug code not updated in this file; since I don't know what was the change it's not trivial to fix).
A quick look around shows that you probably want to look at DEBUG_MMU/DEBUG_BAT in target-ppc/mmu_has32.c. And I'm not even a PPC guy :)
On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
Actually just thinking about this, I've just remembered that in OpenBIOS we use a round-robin hash table slot eviction scheme that forces a tlbie (i.e. invalidate the TLB entry) at the end so perhaps we are invalidating the entry early? See arch/ppc/qemu/ofmem.c in hash_page32().
Or maybe it was never in the TLB because the copying of the vectors that write here was before enabling the MMU but I don't completely understand what will cause an exception and what doesn't. The vectors have been used but that may put it in the instruction cache not in the data one.
It's easy to get lost in the details, but MMUs are actually really simple - they do exactly what they say on the tin. As soon as the MMU is enabled, the basic process looks like this:
1) Does the instruction/data fetch address exist in the TLB (cache)? If so, use it.
2) If it doesn't exist in the cache, throw a DSI (data) or ISI (instruction) exception and switch to real mode
3) The exception handler then adds the lookup to the hash table
4) The CPU retries the faulting load/store which will now succeed, and stores the result in the TLB for later use
Therefore for a load/store not to fault on real hardware then that means the entry is already in the TLB, i.e. it must have already been accessed AND also not been previously evicted by hash collision (see hash_page32).
Hence you need to trace through previous accesses to the addresses that fault and determine why the entry is no longer in the TLB in QEMU/OpenBIOS when it should be there on real hardware.
Quick and dirty hack alert: you may be able to tweak the code so that any hash entry with a virtual address which references physical page 0 is never chosen for eviction (and hence never forcibly invalidated with tlbie) and see if that helps at all? Have a play and see what happens.
I don't see how this could be done and this seems to be a very dirty hack that I'd like to avoid.
I'm not saying this is for production use, I'm just giving a suggestion as to things you can do to understand what is happening. There is no substitute for hard work when debugging issues like this I'm afraid.
Understood. I appreciate the time you have spent working on this, we just need to make sure that we don't just scatter MorphOSisms around the code in a way that makes maintenance harder for those of us who won't run it everyday, and isn't detrimental to all the other supported architectures.
I agree with this completely and I hope we can come up with a solution that works and can be accepted. I did some more experiments but could not make it work. So far the only working solution seems to run this code with MMU off. I tried this patch:
--- a/openbios-devel/arch/ppc/qemu/start.S +++ b/openbios-devel/arch/ppc/qemu/start.S @@ -517,6 +517,10 @@ _GLOBAL(call_elf): li r7,0 // r7 = length of client program arguments li r0,MSR_FP | MSR_ME | MSR_DR | MSR_IR MTMSRD(r0)
LOAD_REG_IMMEDIATE(r8, __vectors)li r9,0lwz r10,4096(r8)stw r10,4096(r9) blrl#ifdef CONFIG_PPC64
which writes to 0x1000 after enabling the MMU and before calling the client code to make sure the translation is in the table.
Okay, so this should make sense with what I said above.
With this it can get past the write that caused an exception before but hits another one later and crashes. Looks like the code where this happens sets up MMU related registers and TLB entries so the handlers are probably not yet working and it cannot handle the exception even if the right routine is called. This is where it crashes with the above patch:
IN: 0x00400acc: mfdbsr r0 0x00400ad0: mfl2cr r11 0x00400ad4: addis r9,r10,-1 0x00400ad8: addi r9,r9,32767 0x00400adc: cmplwi r9,1 0x00400ae0: bgt- 0x400af4
IN: 0x00400af4: mficcr r0 0x00400af8: mr r4,r16 0x00400afc: mr r5,r15 0x00400b00: addi r3,r1,16 0x00400b04: li r6,0 0x00400b08: bl 0x41cce8
IN: 0x0041cce8: stwu r1,-96(r1) 0x0041ccec: stmw r13,20(r1) 0x0041ccf0: lwz r0,0(r3) 0x0041ccf4: sync 0x0041ccf8: mtsr 0,r0 0x0041ccfc: isync
helper_store_sr: reg=0 00000000 20000400 Raise exception at 0041cd00 => 00000003 (40000000) IN: 0x00000400: mtsprg 2,r2 0x00000404: li r2,4 0x00000408: b 0x41f0d4
IN: 0x0041f0d4: mtsprg 1,r1 0x0041f0d8: mfmsr r1 0x0041f0dc: ori r1,r1,12336 0x0041f0e0: sync 0x0041f0e4: mtmsr r1
Raise exception at 0041f0e8 => 00000003 (40000000) Raise exception at 0041f0e8 => 00000003 (40000000)
Actually after writing the sr0-sr15 and sdr1 registers it clears the MMU bits and then writes to the ibat and dbat registers and seems to set up the TLB and then enable the bits in the MSR. So maybe Apple relies on translations in these registers and that's why it works there.
Not that I have the definitive answer, but I can't see why on earth Apple would map the trap table with an IBAT/DBAT. My understanding is that these are designed for optimally mapping large blocks of memory, e.g. kernels, so why would you implement a second form of translation in a BIOS environment when you already have limited resources? It just doesn't make any sense. Plus you still get a fault once you effectively "preload" the trap table into the TLB with your fake access above, so something else is still missing from the picture.
Unfortunately these are only implemented in 32bit processors so it would not work everywhere but still this seems to be the only other solution I can think of short of disabling the MMU somehow during this code. Any ideas?
Not at the moment. All I can re-emphasise is that debugging issues like these takes a great deal of time/effort, but also the rewards gained are similarly great.
ATB,
Mark.