Hi,
After I posted the seavgabios stuff above, it occurred to me that another way to tackle this would be to keep the code in seabios and harden the sercon assembler entry point. Specifically, the assembler code could check that the last mode set wasn't from a vesa mode set call and it could check that calll (vs callw) works.
Additionally we need to hook the sercon code only into a few subcalls, and with the exception of 00h (aka set-mode) none of them are useful for software which wants program vesa modes. So, yes, I think that could work.
I'll go have a look, starting with old linux xorg+x86emu. Do you remember which windows versions have problems too?
cheers, Gerd