Eric,
I admit that my current situation is out of the mainstream. But it's uncovered an issue that could affect others doing more mainstream things.
You write:
The sane thing for an application to expect in 32bit protected mode is flat 32bit segments with a base of 0. That is enough to get things done. If you need something more I would like to hear why.
The code I'm having trouble with is defensive in nature. It is attempting to put the processor into a known state, in case some of the segment registers or flags were not initialized in the switch to protected mode. The code fails because LinuxBIOS is leaving the processor in an inconsistent state (ring 0 protected mode with an invalid GDT backing it). I think this demonstrates both the need for defensive programming by payload writers, and the potential pitfalls of LinuxBIOS doing things that are unexpected.
It is always good to ask whether the cost of NOT making a change outweighs the cost of making it. As Ron points out, I'm the first to trip over this (lucky me). If LB continues as it is, will I be the last? No. That's the "any problem you've ever run into is somewhere on the Internet" syndrome.
I can certainly resolve my problem by writing a wrapper that sets up a GDT before branching to the "problem" code. But that only helps me. I'd prefer to have a solution that helps everyone.
The cost of changing LB to maintain a consistent GDT is that one developer has to make the changes to accomplish this. Others probably review the change. That's less work now than it will be in future, because the issues are fresh.
The cost of NOT changing LB is that N future payload developers discover the problem and have to implement workarounds. The larger part of that cost is probably tracking down what's causing the problem in the first place.
I would argue that while N might be increasing very slowly, eventually the cost to the LB community of NOT fixing the GDT will outweigh the cost of fixing it.
An alternative (but to me, a less desirable one) would be to find some way to make it obvious to payload developers what kind of environment they can count on (i.e. document the GDT issue, making it a 'feature' rather than a 'bug'). You could argue that this e-mail thread does that, but I don't think it will help anyone who hasn't already run into the issue and spent time finding its cause.
Steve