[OpenBIOS] Running client with MMU off

[Mark Cave-Ayland]

On 09/06/14 01:27, BALATON Zoltan wrote:

> On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
>> It might be that MorphOS just got lucky that this works on real
>> hardware without faulting (similar to the bug I had for SPARC64) but
>> some further analysis would help. Does QEMU ppc have a DEBUG_TLB or
>> similar option so you can see when the eviction is triggered? Sadly I
>> think the options here are quite limited :/
>
> There are some debug options in qemu/target-ppc/mmu_helper.c but they
> don't give much clue and the DUMP_PAGE_TABLES option does not work
> because it does not compile if enabled (something seems to have changed
> leaving debug code not updated in this file; since I don't know what was
> the change it's not trivial to fix).

A quick look around shows that you probably want to look at 
DEBUG_MMU/DEBUG_BAT in target-ppc/mmu_has32.c. And I'm not even a PPC guy :)

> On Sun, 8 Jun 2014, Mark Cave-Ayland wrote:
>> Actually just thinking about this, I've just remembered that in
>> OpenBIOS we use
>> a round-robin hash table slot eviction scheme that forces a tlbie (i.e.
>> invalidate the TLB entry) at the end so perhaps we are invalidating
>> the entry
>> early? See arch/ppc/qemu/ofmem.c in hash_page32().
>
> Or maybe it was never in the TLB because the copying of the vectors that
> write here was before enabling the MMU but I don't completely understand
> what will cause an exception and what doesn't. The vectors have been
> used but that may put it in the instruction cache not in the data one.

It's easy to get lost in the details, but MMUs are actually really 
simple - they do exactly what they say on the tin. As soon as the MMU is 
enabled, the basic process looks like this:

1) Does the instruction/data fetch address exist in the TLB (cache)? If 
so, use it.

2) If it doesn't exist in the cache, throw a DSI (data) or ISI 
(instruction) exception and switch to real mode

3) The exception handler then adds the lookup to the hash table

4) The CPU retries the faulting load/store which will now succeed, and 
stores the result in the TLB for later use

Therefore for a load/store not to fault on real hardware then that means 
the entry is already in the TLB, i.e. it must have already been accessed 
AND also not been previously evicted by hash collision (see hash_page32).

Hence you need to trace through previous accesses to the addresses that 
fault and determine why the entry is no longer in the TLB in 
QEMU/OpenBIOS when it should be there on real hardware.

>> Quick and dirty hack alert: you may be able to tweak the code so that
>> any hash
>> entry with a virtual address which references physical page 0 is never
>> chosen
>> for eviction (and hence never forcibly invalidated with tlbie) and see
>> if that
>> helps at all? Have a play and see what happens.
>
> I don't see how this could be done and this seems to be a very dirty
> hack that I'd like to avoid.

I'm not saying this is for production use, I'm just giving a suggestion 
as to things you can do to understand what is happening. There is no 
substitute for hard work when debugging issues like this I'm afraid.

>> Understood. I appreciate the time you have spent working on this, we
>> just need
>> to make sure that we don't just scatter MorphOSisms around the code in
>> a way
>> that makes maintenance harder for those of us who won't run it
>> everyday, and
>> isn't detrimental to all the other supported architectures.
>
> I agree with this completely and I hope we can come up with a solution
> that works and can be accepted. I did some more experiments but could
> not make it work. So far the only working solution seems to run this
> code with MMU off. I tried this patch:
>
> --- a/openbios-devel/arch/ppc/qemu/start.S
> +++ b/openbios-devel/arch/ppc/qemu/start.S
> @@ -517,6 +517,10 @@ _GLOBAL(call_elf):
>          li      r7,0                    // r7 = length of client
> program arguments
>          li      r0,MSR_FP | MSR_ME | MSR_DR | MSR_IR
>          MTMSRD(r0)
> +       LOAD_REG_IMMEDIATE(r8, __vectors)
> +       li      r9,0
> +       lwz     r10,4096(r8)
> +       stw     r10,4096(r9)
>          blrl
>
>   #ifdef CONFIG_PPC64
>
> which writes to 0x1000 after enabling the MMU and before calling the
> client code to make sure the translation is in the table.

Okay, so this should make sense with what I said above.

> With this it
> can get past the write that caused an exception before but hits another
> one later and crashes. Looks like the code where this happens sets up
> MMU related registers and TLB entries so the handlers are probably not
> yet working and it cannot handle the exception even if the right routine
> is called. This is where it crashes with the above patch:
>
> IN:
> 0x00400acc:  mfdbsr  r0
> 0x00400ad0:  mfl2cr  r11
> 0x00400ad4:  addis   r9,r10,-1
> 0x00400ad8:  addi    r9,r9,32767
> 0x00400adc:  cmplwi  r9,1
> 0x00400ae0:  bgt-    0x400af4
>
> IN:
> 0x00400af4:  mficcr  r0
> 0x00400af8:  mr      r4,r16
> 0x00400afc:  mr      r5,r15
> 0x00400b00:  addi    r3,r1,16
> 0x00400b04:  li      r6,0
> 0x00400b08:  bl      0x41cce8
>
> IN:
> 0x0041cce8:  stwu    r1,-96(r1)
> 0x0041ccec:  stmw    r13,20(r1)
> 0x0041ccf0:  lwz     r0,0(r3)
> 0x0041ccf4:  sync
> 0x0041ccf8:  mtsr    0,r0
> 0x0041ccfc:  isync
>
> helper_store_sr: reg=0 00000000 20000400
> Raise exception at 0041cd00 => 00000003 (40000000)
> IN:
> 0x00000400:  mtsprg  2,r2
> 0x00000404:  li      r2,4
> 0x00000408:  b       0x41f0d4
>
> IN:
> 0x0041f0d4:  mtsprg  1,r1
> 0x0041f0d8:  mfmsr   r1
> 0x0041f0dc:  ori     r1,r1,12336
> 0x0041f0e0:  sync
> 0x0041f0e4:  mtmsr   r1
>
> Raise exception at 0041f0e8 => 00000003 (40000000)
> Raise exception at 0041f0e8 => 00000003 (40000000)
>
> Actually after writing the sr0-sr15 and sdr1 registers it clears the MMU
> bits and then writes to the ibat and dbat registers and seems to set up
> the TLB and then enable the bits in the MSR. So maybe Apple relies on
> translations in these registers and that's why it works there.

Not that I have the definitive answer, but I can't see why on earth 
Apple would map the trap table with an IBAT/DBAT. My understanding is 
that these are designed for optimally mapping large blocks of memory, 
e.g. kernels, so why would you implement a second form of translation in 
a BIOS environment when you already have limited resources? It just 
doesn't make any sense. Plus you still get a fault once you effectively 
"preload" the trap table into the TLB with your fake access above, so 
something else is still missing from the picture.

> Unfortunately these are only implemented in 32bit processors so it would
> not work everywhere but still this seems to be the only other solution I
> can think of short of disabling the MMU somehow during this code. Any
> ideas?

Not at the moment. All I can re-emphasise is that debugging issues like 
these takes a great deal of time/effort, but also the rewards gained are 
similarly great.


ATB,

Mark.