Hi,
I was playing a bit with vfio-based PCI device assignment of VGA in qemu and I seem to be hitting a wall just trying to jump into the VGA BIOS. I'm booting qemu with -vga none and assigning a radeon hd5450 via vfio-pci with some extra code to handle passing legacy accesses through to the host. Legacy access hardly seems to matter though as the experiment quickly dies when the vcpu starts executing zero'd memory. gdb shows me something like this:
0x000f257c <__callrom+72>: 66 c7 44 24 16 ff ff movw $0xffff,0x16(%esp) 0x000f2583 <__callrom+79>: 66 c7 44 24 1a ff ff movw $0xffff,0x1a(%esp) 0x000f258a <__callrom+86>: 66 c7 44 24 08 00 f0 movw $0xf000,0x8(%esp) 0x000f2591 <__callrom+93>: b8 80 d1 0f 00 mov $0xfd180,%eax 0x000f2596 <__callrom+98>: 66 89 44 24 0a mov %ax,0xa(%esp) 0x000f259b <__callrom+103>: c1 e5 10 shl $0x10,%ebp 0x000f259e <__callrom+106>: 0f b7 d7 movzwl %di,%edx 0x000f25a1 <__callrom+109>: 09 ea or %ebp,%edx 0x000f25a3 <__callrom+111>: 89 54 24 26 mov %edx,0x26(%esp) 0x000f25a7 <__callrom+115>: 89 e0 mov %esp,%eax 0x000f25a9 <__callrom+117>: 3d 00 70 00 00 cmp $0x7000,%eax 0x000f25ae <__callrom+122>: 76 0a jbe 0xf25ba <__callrom+134> 0x000f25ba <__callrom+134>: 89 f0 mov %esi,%eax 0x000f25bc <__callrom+136>: bb 58 68 00 00 mov $0x6858,%ebx 0x000f25c1 <__callrom+141>: e8 31 98 00 00 call 0xfbdf7 0x000fbdf7: ba 01 be 00 00 mov $0xbe01,%edx 0x000fbdfc: e9 0e ff ff ff jmp 0xfbd0f 0x000fbd0f: 89 c1 mov %eax,%ecx 0x000fbd11: b8 30 00 00 00 mov $0x30,%eax 0x000fbd16: 8e d8 mov %eax,%ds 0x000fbd18: 8e c0 mov %eax,%es 0x000fbd1a: 8e d0 mov %eax,%ss 0x000fbd1c: 8e e0 mov %eax,%fs 0x000fbd1e: 8e e8 mov %eax,%gs 0x000fbd20: 66 ea 26 bd 28 00 ljmpw $0x28,$0xbd26 0x0000bd26: 00 00 add %al,(%eax)
(qemu) xp /16x 0x0000bd26 000000000000bd26: 0x00000000 0x00000000 0x00000000 0x00000000 000000000000bd36: 0x00000000 0x00000000 0x00000000 0x00000000 000000000000bd46: 0x00000000 0x00000000 0x00000000 0x00000000 000000000000bd56: 0x00000000 0x00000000 0x00000000 0x00000000
(qemu) xp /16x 0x000c0000 00000000000c0000: 0xe975aa55 0x00000221 0x00000000 0x00000000 00000000000c0010: 0x00000000 0x00000000 0x000001d4 0x42490000 00000000000c0020: 0x0000254d 0x00000000 0x00000000 0x04000000 00000000000c0030: 0x31363720 0x35353932 0x00003032 0x00000000
Trying to follow the code into __callrom(), I'm really confused how the option rom init vector is actually used since callrom() passes the option rom header offset to the init vector rather than anything actually resembling the value of the init vector. I really don't know x86 though, so maybe I'm missing something.
The option rom is loaded as a PCI expansion rom from the device, it appears to be non-PnP. The seabios log shows this:
Scan for VGA option rom Attempting to init PCI bdf 00:02.0 (vd 1002:68f9) Attempting to map option rom on dev 00:02.0 Option rom sizing returned febe0000 fffe0000 Inspecting possible rom at 0xfebe0000 (vd=1002:68f9 bdf=00:02.0) Copying option rom (size 59904) from 0xfebe0000 to 0x000c0000 handle_08 Checking rom 0x000c0000 (sig aa55 size 117) Running option rom at c000:0003 <here we go into the weeds, I'll continue to see periodic handle_08 if I let it run>
Appreciate any hints to make this work. Thanks,
Alex
On Thu, Jan 03, 2013 at 02:31:43PM -0700, Alex Williamson wrote:
Hi,
I was playing a bit with vfio-based PCI device assignment of VGA in qemu and I seem to be hitting a wall just trying to jump into the VGA BIOS. I'm booting qemu with -vga none and assigning a radeon hd5450 via vfio-pci with some extra code to handle passing legacy accesses through to the host. Legacy access hardly seems to matter though as the experiment quickly dies when the vcpu starts executing zero'd memory. gdb shows me something like this:
[...]
0x000fbd20: 66 ea 26 bd 28 00 ljmpw $0x28,$0xbd26 0x0000bd26: 00 00 add %al,(%eax)
Everything looks okay except for here. I'd guess it's likely just a reporting issue. The code being run is actually at 0xfbd26 - as part of jumping into real-mode, the code has a segment offset (0xf0000) that must be added in.
Trying to follow the code into __callrom(), I'm really confused how the option rom init vector is actually used since callrom() passes the option rom header offset to the init vector rather than anything actually resembling the value of the init vector. I really don't know x86 though, so maybe I'm missing something.
The option rom should actually have code (eg, a jmp instruction) at offset 3 of the option rom. So, the goal really is to jump to the 3rd byte of the option rom to execute it.
-Kevin
On Thu, 2013-01-03 at 19:13 -0500, Kevin O'Connor wrote:
On Thu, Jan 03, 2013 at 02:31:43PM -0700, Alex Williamson wrote:
Hi,
I was playing a bit with vfio-based PCI device assignment of VGA in qemu and I seem to be hitting a wall just trying to jump into the VGA BIOS. I'm booting qemu with -vga none and assigning a radeon hd5450 via vfio-pci with some extra code to handle passing legacy accesses through to the host. Legacy access hardly seems to matter though as the experiment quickly dies when the vcpu starts executing zero'd memory. gdb shows me something like this:
[...]
0x000fbd20: 66 ea 26 bd 28 00 ljmpw $0x28,$0xbd26 0x0000bd26: 00 00 add %al,(%eax)
Everything looks okay except for here. I'd guess it's likely just a reporting issue. The code being run is actually at 0xfbd26 - as part of jumping into real-mode, the code has a segment offset (0xf0000) that must be added in.
Trying to follow the code into __callrom(), I'm really confused how the option rom init vector is actually used since callrom() passes the option rom header offset to the init vector rather than anything actually resembling the value of the init vector. I really don't know x86 though, so maybe I'm missing something.
The option rom should actually have code (eg, a jmp instruction) at offset 3 of the option rom. So, the goal really is to jump to the 3rd byte of the option rom to execute it.
Yes!
(gdb) x/i 0xc0003 0xc0003: jmp 0xc0229
(gdb) x/10i 0xc0229 0xc0229: push %ax 0xc022b: push %cx 0xc022d: push %dx 0xc022f: push %bx 0xc0231: push %bp 0xc0233: push %si 0xc0235: push %di 0xc0237: push %cs 0xc0238: pop %ds 0xc0239: mov %eax,0xe8c01c2
Thanks for filling in that piece of the puzzle for me. If the above is just a reporting problem, how do I fix it so I can actually step through the rom? Thanks,
Alex
On Thu, Jan 03, 2013 at 06:41:35PM -0700, Alex Williamson wrote:
Thanks for filling in that piece of the puzzle for me. If the above is just a reporting problem, how do I fix it so I can actually step through the rom? Thanks,
This excerpt from the README might help:
------
Note that gdb seems to get breakpoints confused when the cpu is in 16-bit real mode. This makes stepping through the program difficult (though 'step instruction' still works). Also, one may need to set 16bit break points at both the cpu address and memory address (eg, break *0x1234 ; break *0xf1234).
------
Though I wrote the above a few years back so it may no longer be accurate.
Otherwise, all I remember is that it was a pain.
-Kevin
On Thu, Jan 03, 2013 at 06:41:35PM -0700, Alex Williamson wrote:
Thanks for filling in that piece of the puzzle for me. If the above is just a reporting problem, how do I fix it so I can actually step through the rom? Thanks,
One other note - I think Darmawan may have been doing something similar with PCI rom debugging - you may wish to reach out to him.
-Kevin
On Sat, Nov 12, 2011 at 03:41:15PM +0700, Darmawan Salihun wrote:
I made a blogpost detailing the steps to debug PCI Option ROM with Coreboot+SeaBIOS and a GDB-server-compatible debugger:
http://bioshacking.blogspot.com/search/label/PCI%20Option%20ROM
Hopefully would help those in need because it takes quite a while to get it right. Thanks to Kevin O'Connor for the helps :-)
Regards,
Darmawan
On Thu, 2013-01-03 at 20:57 -0500, Kevin O'Connor wrote:
On Thu, Jan 03, 2013 at 06:41:35PM -0700, Alex Williamson wrote:
Thanks for filling in that piece of the puzzle for me. If the above is just a reporting problem, how do I fix it so I can actually step through the rom? Thanks,
One other note - I think Darmawan may have been doing something similar with PCI rom debugging - you may wish to reach out to him.
Thanks for the connection. For now the link Daniel provided has given me enough to make progress. It looks like this is a case where the VGA bios manages to get the physical address of the device through a legacy VGA register (0x3c3) and shoots itself by using that rather than the emulated address found through config space. I can kludge the offset and get the VGA option rom to finish, but something is still wrong since it doesn't trigger the monitor to sync. Still digging. Thanks for the help,
Alex
On Sat, Nov 12, 2011 at 03:41:15PM +0700, Darmawan Salihun wrote:
I made a blogpost detailing the steps to debug PCI Option ROM with Coreboot+SeaBIOS and a GDB-server-compatible debugger:
http://bioshacking.blogspot.com/search/label/PCI%20Option%20ROM
Hopefully would help those in need because it takes quite a while to get it right. Thanks to Kevin O'Connor for the helps :-)
Regards,
Darmawan
On Thu, 2013-01-03 at 20:49 -0700, Alex Williamson wrote:
On Thu, 2013-01-03 at 20:57 -0500, Kevin O'Connor wrote:
On Thu, Jan 03, 2013 at 06:41:35PM -0700, Alex Williamson wrote:
Thanks for filling in that piece of the puzzle for me. If the above is just a reporting problem, how do I fix it so I can actually step through the rom? Thanks,
One other note - I think Darmawan may have been doing something similar with PCI rom debugging - you may wish to reach out to him.
Thanks for the connection. For now the link Daniel provided has given me enough to make progress. It looks like this is a case where the VGA bios manages to get the physical address of the device through a legacy VGA register (0x3c3) and shoots itself by using that rather than the emulated address found through config space. I can kludge the offset and get the VGA option rom to finish, but something is still wrong since it doesn't trigger the monitor to sync. Still digging. Thanks for the help,
I enabled unassigned memory debugging in qemu and get some peculiar output during the vga bios execution. Given this state:
---------------------------[ STACK ]--- 6E02 4942 B5D4 B5E5 6DAE 6DB2 0000 0000 B56A B572 6DBA 0000 6E0A 6DA6 8001 0000 ---------------------------[ DS:SI ]--- C0000000: 55 AA 75 E9 21 02 00 00 00 00 00 00 00 00 00 00 U.u.!........... C0000010: 00 00 00 00 00 00 00 00 D4 01 00 00 00 00 49 42 ..............IB C0000020: 4D 25 00 00 00 00 00 00 00 00 00 00 00 00 00 04 M%.............. C0000030: 20 37 36 31 32 39 35 35 32 30 00 00 00 00 00 00 .761295520...... ---------------------------[ ES:DI ]--- 00006E0A: 1A 6E 00 00 00 20 28 03 E8 FD 00 00 E8 FD 00 00 .n....(......... 00006E1A: 00 00 A4 0D 10 00 00 00 10 00 00 00 6E 02 28 17 ............n.(. 00006E2A: 00 00 90 D1 00 00 00 00 00 00 00 00 00 00 FF FF ................ 00006E3A: 00 00 FF FF 00 00 00 00 00 00 10 00 00 00 00 F0 ................ ----------------------------[ CPU ]---- AX: 6DBA BX: B5E5 CX: 0000 DX: 0001 SI: 0000 DI: 6E0A SP: 6DA2 BP: 6E02 CS: C000 DS: C000 ES: 0000 SS: 0000
IP: 460E EIP:0000460E CS:IP: C000:460E (0xC460E) SS:SP: 0000:6DA2 (0x06DA2) SS:BP: 0000:6E02 (0x06E02) OF <0> DF <0> IF <1> TF <0> SF <1> ZF <0> AF <1> PF <1> CF <1> ID <0> VIP <0> VIF <0> AC <0> VM <0> RF <0> NT <0> IOPL <0> ---------------------------[ CODE ]---- 0xc460e: mov bp,sp 0xc4610: push bx 0xc4611: push cx 0xc4612: push dx 0xc4613: push di 0xc4614: push ax 0xc4615: mov bx,ax 0xc4617: mov cx,ss 0xc4619: mov es,cx 0xc461b: mov si,WORD PTR es:[bx+0x2]
How does that mov generate this:
Unassigned mem read 00000000b5e5b5d4
Real-mode tcg bug? Here's the next state:
---------------------------[ STACK ]--- 6E02 4942 B5D4 B5E5 6DAE 6DB2 0000 0000 B56A B572 6DBA 0000 6E0A 6DA6 8001 0000 ---------------------------[ DS:SI ]--- C0000000: 55 AA 75 E9 21 02 00 00 00 00 00 00 00 00 00 00 U.u.!........... C0000010: 00 00 00 00 00 00 00 00 D4 01 00 00 00 00 49 42 ..............IB C0000020: 4D 25 00 00 00 00 00 00 00 00 00 00 00 00 00 04 M%.............. C0000030: 20 37 36 31 32 39 35 35 32 30 00 00 00 00 00 00 .761295520...... ---------------------------[ ES:DI ]--- 00006E0A: 1A 6E 00 00 00 20 28 03 E8 FD 00 00 E8 FD 00 00 .n....(......... 00006E1A: 00 00 A4 0D 10 00 00 00 10 00 00 00 6E 02 28 17 ............n.(. 00006E2A: 00 00 90 D1 00 00 00 00 00 00 00 00 00 00 FF FF ................ 00006E3A: 00 00 FF FF 00 00 00 00 00 00 10 00 00 00 00 F0 ................ ----------------------------[ CPU ]---- AX: 6DBA BX: B5E5 CX: 0000 DX: 0001 SI: 0000 DI: 6E0A SP: 6DA2 BP: 6DA2 CS: C000 DS: C000 ES: 0000 SS: 0000
IP: 4610 EIP:00004610 CS:IP: C000:4610 (0xC4610) SS:SP: 0000:6DA2 (0x06DA2) SS:BP: 0000:6DA2 (0x06DA2) OF <0> DF <0> IF <1> TF <0> SF <1> ZF <0> AF <1> PF <1> CF <1> ID <0> VIP <0> VIF <0> AC <0> VM <0> RF <0> NT <0> IOPL <0> ---------------------------[ CODE ]---- 0xc4610: push bx 0xc4611: push cx 0xc4612: push dx 0xc4613: push di 0xc4614: push ax 0xc4615: mov bx,ax 0xc4617: mov cx,ss 0xc4619: mov es,cx 0xc461b: mov si,WORD PTR es:[bx+0x2] 0xc461f: mov si,WORD PTR es:[si+0x2]
Here's another odd one, this state:
---------------------------[ STACK ]--- 6E02 4942 B5D4 B5E7 6DAE 6DB2 0000 0000 B56A B572 6DBA 0000 6E0A 6DA6 8001 0000 ---------------------------[ DS:SI ]--- C0000004: 21 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 !............... C0000014: 00 00 00 00 D4 01 00 00 00 00 49 42 4D 25 00 00 ..........IBM%.. C0000024: 00 00 00 00 00 00 00 00 00 00 00 04 20 37 36 31 .............761 C0000034: 32 39 35 35 32 30 00 00 00 00 00 00 19 02 00 00 295520.......... ---------------------------[ ES:DI ]--- 00006E0A: 1A 6E 00 00 00 20 28 03 E8 FD 00 00 E8 FD 00 00 .n....(......... 00006E1A: 00 00 A4 0D 10 00 00 00 10 00 00 00 6E 02 28 17 ............n.(. 00006E2A: 00 00 90 D1 00 00 00 00 00 00 00 00 00 00 FF FF ................ 00006E3A: 00 00 FF FF 00 00 00 00 00 00 10 00 00 00 00 F0 ................ ----------------------------[ CPU ]---- AX: 0001 BX: B5E5 CX: 0000 DX: 0001 SI: 0004 DI: 6E0A SP: 6DA2 BP: 6DA2 CS: C000 DS: C000 ES: 0000 SS: 0000
IP: 4785 EIP:00004785 CS:IP: C000:4785 (0xC4785) SS:SP: 0000:6DA2 (0x06DA2) SS:BP: 0000:6DA2 (0x06DA2) OF <0> DF <0> IF <1> TF <0> SF <0> ZF <0> AF <0> PF <0> CF <0> ID <0> VIP <0> VIF <0> AC <0> VM <0> RF <0> NT <0> IOPL <0> ---------------------------[ CODE ]---- 0xc4785: pop bp 0xc4786: ret 0xc4787: push bx 0xc4788: push si 0xc4789: mov bx,ax 0xc478b: mov si,WORD PTR es:[bx+0x2] 0xc478f: mov ax,si 0xc4791: mov si,WORD PTR [si+0x48] 0xc4794: add si,ax 0xc4796: mov bx,ax
Generates:
Unassigned mem read 00000000f000c000
Next state:
---------------------------[ STACK ]--- 4942 B5D4 B5E7 6DAE 6DB2 0000 0000 B56A B572 6DBA 0000 6E0A 6DA6 8001 0000 8001 ---------------------------[ DS:SI ]--- C0000004: 21 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 !............... C0000014: 00 00 00 00 D4 01 00 00 00 00 49 42 4D 25 00 00 ..........IBM%.. C0000024: 00 00 00 00 00 00 00 00 00 00 00 04 20 37 36 31 .............761 C0000034: 32 39 35 35 32 30 00 00 00 00 00 00 19 02 00 00 295520.......... ---------------------------[ ES:DI ]--- 00006E0A: 1A 6E 00 00 00 20 28 03 E8 FD 00 00 E8 FD 00 00 .n....(......... 00006E1A: 00 00 A4 0D 10 00 00 00 10 00 00 00 6E 02 28 17 ............n.(. 00006E2A: 00 00 90 D1 00 00 00 00 00 00 00 00 00 00 FF FF ................ 00006E3A: 00 00 FF FF 00 00 00 00 00 00 10 00 00 00 00 F0 ................ ----------------------------[ CPU ]---- AX: 0001 BX: B5E5 CX: 0000 DX: 0001 SI: 0004 DI: 6E0A SP: 6DA4 BP: 6E02 CS: C000 DS: C000 ES: 0000 SS: 0000
IP: 4786 EIP:00004786 CS:IP: C000:4786 (0xC4786) SS:SP: 0000:6DA4 (0x06DA4) SS:BP: 0000:6E02 (0x06E02) OF <0> DF <0> IF <1> TF <0> SF <0> ZF <0> AF <0> PF <0> CF <0> ID <0> VIP <0> VIF <0> AC <0> VM <0> RF <0> NT <0> IOPL <0> ---------------------------[ CODE ]---- 0xc4786: ret 0xc4787: push bx 0xc4788: push si 0xc4789: mov bx,ax 0xc478b: mov si,WORD PTR es:[bx+0x2] 0xc478f: mov ax,si 0xc4791: mov si,WORD PTR [si+0x48] 0xc4794: add si,ax 0xc4796: mov bx,ax 0xc4798: mov si,WORD PTR [si+0x1e]
It looks like maybe these innocuous since the next state looks correct. Again, thanks for any insight you might have into this,
Alex
On Thu, Jan 3, 2013 at 2:31 PM, Alex Williamson alex.williamson@redhat.com wrote:
Hi,
I was playing a bit with vfio-based PCI device assignment of VGA in qemu and I seem to be hitting a wall just trying to jump into the VGA BIOS. I'm booting qemu with -vga none and assigning a radeon hd5450 via vfio-pci with some extra code to handle passing legacy accesses through to the host. Legacy access hardly seems to matter though as the experiment quickly dies when the vcpu starts executing zero'd memory. gdb shows me something like this:
0x000f257c <__callrom+72>: 66 c7 44 24 16 ff ff movw $0xffff,0x16(%esp) 0x000f2583 <__callrom+79>: 66 c7 44 24 1a ff ff movw $0xffff,0x1a(%esp) 0x000f258a <__callrom+86>: 66 c7 44 24 08 00 f0 movw $0xf000,0x8(%esp) 0x000f2591 <__callrom+93>: b8 80 d1 0f 00 mov $0xfd180,%eax 0x000f2596 <__callrom+98>: 66 89 44 24 0a mov %ax,0xa(%esp) 0x000f259b <__callrom+103>: c1 e5 10 shl $0x10,%ebp 0x000f259e <__callrom+106>: 0f b7 d7 movzwl %di,%edx 0x000f25a1 <__callrom+109>: 09 ea or %ebp,%edx 0x000f25a3 <__callrom+111>: 89 54 24 26 mov %edx,0x26(%esp) 0x000f25a7 <__callrom+115>: 89 e0 mov %esp,%eax 0x000f25a9 <__callrom+117>: 3d 00 70 00 00 cmp $0x7000,%eax 0x000f25ae <__callrom+122>: 76 0a jbe 0xf25ba <__callrom+134> 0x000f25ba <__callrom+134>: 89 f0 mov %esi,%eax 0x000f25bc <__callrom+136>: bb 58 68 00 00 mov $0x6858,%ebx 0x000f25c1 <__callrom+141>: e8 31 98 00 00 call 0xfbdf7 0x000fbdf7: ba 01 be 00 00 mov $0xbe01,%edx 0x000fbdfc: e9 0e ff ff ff jmp 0xfbd0f 0x000fbd0f: 89 c1 mov %eax,%ecx 0x000fbd11: b8 30 00 00 00 mov $0x30,%eax 0x000fbd16: 8e d8 mov %eax,%ds 0x000fbd18: 8e c0 mov %eax,%es 0x000fbd1a: 8e d0 mov %eax,%ss 0x000fbd1c: 8e e0 mov %eax,%fs 0x000fbd1e: 8e e8 mov %eax,%gs 0x000fbd20: 66 ea 26 bd 28 00 ljmpw $0x28,$0xbd26 0x0000bd26: 00 00 add %al,(%eax)
I think GDB is probably getting lost at the far jump (ljmpw). To my knowledge, GDB does not understand the x86 segmented real-mode memory model.
The trace above is __callrom() -> farcall16big() -> call16big() -> __call16big() -> transition16big in romlayout.S, which contains the ljmpw in question.
Trying to follow the code into __callrom(), I'm really confused how the option rom init vector is actually used since callrom() passes the option rom header offset to the init vector rather than anything actually resembling the value of the init vector. I really don't know x86 though, so maybe I'm missing something.
The far address of the ROM initialization vector (seg:offset) is being loaded into br.code, which will get used later on in the farcall helper functions noted above. The offset is set to OPTION_ROM_INITVECTOR by callrom(), which is 3; this is relative to the segment of the ROM header (hence the FLATPTR_TO_SEG in __callrom). The ROM header should have a jump at that location to the real initialization code.
I don't have any real tips on how to make this work - hopefully someone else will know the magic incantation to clue in GDB to the new code segment.
This blog post also looks relevant: http://crustynation.net/wiki/doku.php?id=blog:gdb_real_mode
-- Daniel
-
Alex Williamson -
Daniel Verkamp -
Kevin O'Connor