[SeaBIOS] [PATCH] vgabios: Avoid extra stack if it appears a modern OS is in use

[Kevin O'Connor]

If a VBE mode set call is made, stop using the vgabios extra stack.
This works under the premise that only a modern OS would invoke the
VBE mode changing facilities and a modern OS would always call the
vgabios with sufficient stack space.

This is an ugly hack to work around a problem Windows Vista (and
possibly later Windows releases) has with the VGA BIOS using a stack
in the e-segment.

Reported-by: Richard Laager 
Signed-off-by: Kevin O'Connor 
---

I ran some further tests with the Windows emulator crashing problem.
I eventually came up with the targeted solution above.  Basically, I
tried to find a signature that indicates Windows is running and
disable the extra stack in that case.  It's a hack for sure, but it
seems to work.

For the curious, some other tests that I ran that didn't yield a
solution:

* The "concurrent dos" image I have actually uses vm86 mode and it
  still expects the vgabios to only use a tiny amount of stack space.
  So, just disabling the extra stack in vm86 mode breaks this
  "concurrent dos" image.

* Windows faults on the first write access to the e-segment, so
  testing if the extra stack is read/writable doesn't work (as the
  test itself causes a fault which halts the emulation).

* Windows actually copies the first 1 Meg before starting the
  emulation, so putting some kind of signature in the extra stack
  doesn't help detect if it's read-only.

* When these troublesome vgabios calls are made, Windows is using a
  copy of the first 1 Meg instead of accessing it in place.  If a log
  is written to the writable part of the first 1 Meg, then when the
  log is inspected later it doesn't show any of the troublesome calls.

---
 vgasrc/vbe.c     |  1 +
 vgasrc/vgabios.h |  1 +
 vgasrc/vgainit.c | 15 +++++++++++++++
 3 files changed, 17 insertions(+)

diff --git a/vgasrc/vbe.c b/vgasrc/vbe.c
index af3d0cc..834d7b3 100644
--- a/vgasrc/vbe.c
+++ b/vgasrc/vbe.c
@@ -205,6 +205,7 @@ static void
 vbe_104f02(struct bregs *regs)
 {
     dprintf(1, "VBE mode set: %x\n", regs->bx);
+    check_extra_stack();

int mode = regs->bx & ~MF_VBEFLAGS;
     int flags = regs->bx & MF_VBEFLAGS;
diff --git a/vgasrc/vgabios.h b/vgasrc/vgabios.h
index fd796f2..985bbbb 100644
--- a/vgasrc/vgabios.h
+++ b/vgasrc/vgabios.h
@@ -100,6 +100,7 @@ extern u8 vgafont16alt[];
 // vgainit.c
 extern struct video_save_pointer_s video_save_pointer_table;
 extern struct video_param_s video_param_table[29];
+void check_extra_stack(void);

// vgabios.c
 extern int VgaBDF;
diff --git a/vgasrc/vgainit.c b/vgasrc/vgainit.c
index 8d12261..4616883 100644
--- a/vgasrc/vgainit.c
+++ b/vgasrc/vgainit.c
@@ -86,6 +86,21 @@ allocate_extra_stack(void)
     }
 }

+void
+check_extra_stack(void)
+{
+    if (!CONFIG_VGA_ALLOCATE_EXTRA_STACK)
+        return;
+    extern void entry_10(void);
+    if (GET_IVT(0x10).offset == (u32)entry_10)
+        return;
+    // Disable the extra stack if a VBE mode set call is made.  This
+    // works around a bug in Windows Vista when the stack is in the
+    // e-segment.
+    dprintf(1, "Disabling SeaVGABIOS extra stack.\n");
+    SET_IVT(0x10, SEGOFF(get_global_seg(), (u32)entry_10));
+}
+

/****************************************************************
  * Timer hook