I made a program that tells what control characters are in the Mac OS ROM file. It reported that every single control character is in that file. Then I had it only look at the bootscript part. The carriage return and line feed characters are the only control characters found there.
http://www.ascii-code.com/ This page will tell you what the control characters are in case you don't know.
I added some code to the bootinfo_init_program() function in bootinfo_load.c file that changes all control characters to '\n'. This made Mac OS 9 boot. But when I changed the control characters to a single space, Mac OS 9 failed to boot. This seems strange to me. The space character is the ultimate whitespace character, so why would it fail when '\n' succeeds? My only guess right now is a bug with one of the words in Forth.
Here is the code I used. It is found around line 190 in the file:
while (current < size) {
c = base[current++];
// remove all control characters if (c >= 0 && c <= 31) { c = '\n'; }
if (c == '<') { script = 0; tag = 1; taglen = 0;
On 2016-Apr-16 19:17 , Programmingkid wrote:
I added some code to the bootinfo_init_program() function in bootinfo_load.c file that changes all control characters to '\n'. This made Mac OS 9 boot. But when I changed the control characters to a single space, Mac OS 9 failed to boot. This seems strange to me. The space character is the ultimate whitespace character, so why would it fail when '\n' succeeds? My only guess right now is a bug with one of the words in Forth.
Well, changing end-of-line to space will probably fail for the following standard words (doing a search through IEEE 1275 for EOL):
\ dev " parse load help setenv devalias boot set-default printenv
The last seven are supposed to be interactive commands, so no surprise they need an EOL. "dev" is supposed to be as well, but it's too-widely used in code (they should be using '" /" find-device' instead of 'dev /', but poor programming isn't limited to C language). It's rare to see a quoted string ended by end-of-line, but I have run into it.
I suspect that as Mark was suggesting, the CR vs LF is a red herring. The issue is likely to be what happens to the return stack between commands - the forth interpreter might trash the return stack between commands - it could well be that in the process of parsing strings from the boot code, it's resetting the return stack, and that's where we're getting tripped up. Maybe some parsing code goes through a different path for CR vs LF, for example. Apple's Forth code is known to take advantage of Apple's interpreter's quirks, to the degree that plug-in card manufacturers had to produce different code for Sun, IBM and Apple systems.
It could also be that the parsing code is getting tangled up with this code's use of "here", where it is allocating from the heap in the middle of parsing. Perhaps the Forth parser is using the heap itself and things are tripping over each other.
On the other hand, I can't think of much bad that will happen by replacing all CR (0x0D) with LF (0x0A), other than some formatting problems if you have code which is trying to be fancy with a spinning wheel (sequence of / - \ |). The main problem with that approach is that we know something is wrong and we may be just hiding the problem.
One possible solution; take the boot-script code, and rather than interpret it directly, try compiling it. E.g., before starting the boot code, add:
: boot-script-container <add boot code here> ; boot-script-container
That way we could separate the parsing of the forth source from the execution of the forth. You'd have to watch out for the cases where there are colon definitions already in the boot code, because the openbios interpreter probably doesn't allow nested colon definitions. So you might end up with something like
: boot-script-container1 <code from "here" to "release-load-area"> ; boot-script-container1
: do-translate ... through : release-virt ...
: boot-script-container2 <code from "1000 constant" to "go"> ; boot-script-container2
On Apr 16, 2016, at 8:09 PM, Tarl Neustaedter wrote:
On 2016-Apr-16 19:17 , Programmingkid wrote:
I added some code to the bootinfo_init_program() function in bootinfo_load.c file that changes all control characters to '\n'. This made Mac OS 9 boot. But when I changed the control characters to a single space, Mac OS 9 failed to boot. This seems strange to me. The space character is the ultimate whitespace character, so why would it fail when '\n' succeeds? My only guess right now is a bug with one of the words in Forth.
Well, changing end-of-line to space will probably fail for the following standard words (doing a search through IEEE 1275 for EOL):
\ dev " parse load help setenv devalias boot set-default printenv
The last seven are supposed to be interactive commands, so no surprise they need an EOL. "dev" is supposed to be as well, but it's too-widely used in code (they should be using '" /" find-device' instead of 'dev /', but poor programming isn't limited to C language). It's rare to see a quoted string ended by end-of-line, but I have run into it.
I suspect that as Mark was suggesting, the CR vs LF is a red herring.
You think so? This looks like the solution Mark and I have been wanting. An explanation as to why EOL a.k.a. '\n' substitution works. I can finally tell him it is because the standard requires it.
Thank you very much for solving this problem.
The issue is likely to be what happens to the return stack between commands - the forth interpreter might trash the return stack between commands - it could well be that in the process of parsing strings from the boot code, it's resetting the return stack, and that's where we're getting tripped up. Maybe some parsing code goes through a different path for CR vs LF, for example. Apple's Forth code is known to take advantage of Apple's interpreter's quirks, to the degree that plug-in card manufacturers had to produce different code for Sun, IBM and Apple systems.
I do have Apple's Forth implementation available to test. All I need is a test program that does something to the return stack.
After looking at the bootscript, I think return stack trashing is not a problem. There are three calls to >r and then three calls the r>. They balance each other out. To show that this is ok, just run this program:
: main 1 2 3 \ place these numbers on the stack
r r r
." doing something" cr r> r> r> ." You should see 3 2 1: " . . . cr ;
You will see at the end 3 2 1.
It could also be that the parsing code is getting tangled up with this code's use of "here", where it is allocating from the heap in the middle of parsing. Perhaps the Forth parser is using the heap itself and things are tripping over each other.
On the other hand, I can't think of much bad that will happen by replacing all CR (0x0D) with LF (0x0A), other than some formatting problems if you have code which is trying to be fancy with a spinning wheel (sequence of / - \ |). The main problem with that approach is that we know something is wrong and we may be just hiding the problem.
When all operating systems boot properly, we can say there is no problem.
One possible solution; take the boot-script code, and rather than interpret it directly, try compiling it. E.g., before starting the boot code, add:
: boot-script-container
<add boot code here> ; boot-script-container
That way we could separate the parsing of the forth source from the execution of the forth. You'd have to watch out for the cases where there are colon definitions already in the boot code, because the openbios interpreter probably doesn't allow nested colon definitions. So you might end up with something like
: boot-script-container1 <code from "here" to "release-load-area"> ; boot-script-container1
: do-translate ... through : release-virt ...
: boot-script-container2 <code from "1000 constant" to "go"> ; boot-script-container2
I really like this idea.
On 2016-Apr-16 21:07 , Programmingkid wrote:
After looking at the bootscript, I think return stack trashing is not a problem. There are three calls to >r and then three calls the r>. They balance each other out. To show that this is ok, just run this program:
: main 1 2 3 \ place these numbers on the stack
r r r
." doing something" cr r> r> r> ." You should see 3 2 1: " . . . cr ;
You will see at the end 3 2 1.
The above is guaranteed to work, it's all inside a colon definition (which means it's compiled as a whole before executing). The issue comes up when commands using the return stack are used outside colon definitions. E.g., at the ok prompt or in a boot-script string:
ok 3 >r ok <do some fancy parsing> ok r> .
If the parser uses the return stack itself, it might reset the stack before returning to the ok prompt, causing the third line to do something unexpected. In the case of Sun's OBP, when parsing things at the ok prompt, if something issued a "throw", the return stack would be reset to empty before returning to the ok prompt. I don't know what cases the Openbios parser might do something similar.
[boot-script-container stuff to try to separate compiling from executing forth]
I really like this idea.
It's worth trying to see it gives us further insight on what's going on. If it's a problem with "here" being used by both the code and the parser and overwriting each other, the pre-compile will prevent the problem from occurring by separating the two usages. Good luck!
On Apr 16, 2016, at 9:34 PM, Tarl Neustaedter wrote:
On 2016-Apr-16 21:07 , Programmingkid wrote:
After looking at the bootscript, I think return stack trashing is not a problem. There are three calls to >r and then three calls the r>. They balance each other out. To show that this is ok, just run this program:
: main 1 2 3 \ place these numbers on the stack
r r r
." doing something" cr r> r> r> ." You should see 3 2 1: " . . . cr ;
You will see at the end 3 2 1.
The above is guaranteed to work, it's all inside a colon definition (which means it's compiled as a whole before executing). The issue comes up when commands using the return stack are used outside colon definitions. E.g., at the ok prompt or in a boot-script string:
ok 3 >r ok <do some fancy parsing> ok r> .
I see what you mean.
If the parser uses the return stack itself, it might reset the stack before returning to the ok prompt, causing the third line to do something unexpected. In the case of Sun's OBP, when parsing things at the ok prompt, if something issued a "throw", the return stack would be reset to empty before returning to the ok prompt. I don't know what cases the Openbios parser might do something similar.
It might be the case that the parser doesn't use the return stack. If that is the case, we don't have to worry about this issue.
If using the return stack is suppose to cause problems when in boot-script reading mode, why does Mac OS 9 successfully boots?
[boot-script-container stuff to try to separate compiling from executing forth]
I really like this idea.
It's worth trying to see it gives us further insight on what's going on. If it's a problem with "here" being used by both the code and the parser and overwriting each other, the pre-compile will prevent the problem from occurring by separating the two usages. Good luck!
Seeing that the EOL character is required by the specs, I just don't see a need to do this anymore. It looks like your research has solved the problem. Also doesn't code executing in a colon definition behave differently than it would in interpretation mode?
On Sat, Apr 16, 2016 at 08:09:47PM -0400, Tarl Neustaedter wrote:
On 2016-Apr-16 19:17 , Programmingkid wrote:
I added some code to the bootinfo_init_program() function in bootinfo_load.c file that changes all control characters to '\n'. This made Mac OS 9 boot. But when I changed the control characters to a single space, Mac OS 9 failed to boot. This seems strange to me. The space character is the ultimate whitespace character, so why would it fail when '\n' succeeds? My only guess right now is a bug with one of the words in Forth.
Well, changing end-of-line to space will probably fail for the following standard words (doing a search through IEEE 1275 for EOL):
\ dev " parse load help setenv devalias boot set-default printenv
The last seven are supposed to be interactive commands, so no surprise they need an EOL. "dev" is supposed to be as well, but it's too-widely used in code (they should be using '" /" find-device' instead of 'dev /', but poor programming isn't limited to C language). It's rare to see a quoted string ended by end-of-line, but I have run into it.
Yes. Anything that calls PARSE (or WORD or PARSE-WORD or NAME or whatever similar words you have) depends on this. That is a much longer list than what you found, for example, also ( is in that list.
EVALUATE makes its whole argument string the input buffer, as one string. It does not handle multiple lines. If you want to use it to handle the boot script, you should break things into lines yourself.
I suspect that as Mark was suggesting, the CR vs LF is a red herring. The issue is likely to be what happens to the return stack between commands - the forth interpreter might trash the return stack between commands - it could well be that in the process of parsing strings from the boot code, it's resetting the return stack, and that's where we're getting tripped up. Maybe some parsing code goes through a different path for CR vs LF, for example. Apple's Forth code is known to take advantage of Apple's interpreter's quirks, to the degree that plug-in card manufacturers had to produce different code for Sun, IBM and Apple systems.
Between words is the first part of the problem; it's not very hard to make you parser loop play tricks so you can mess with the return stack as well.
But the OS9 boot code also messes with the return stack between _lines_. And allowing that is not feasible.
The usual solution is to implement a separate R stack for use in interpretation mode (just an array in Forth, not some magic system-provided something). The only limitation is that you cannot put something in the R stack in interpret mode and then access it in compile mode (or the other way around). This isn't a problem in practice.
On the other hand, I can't think of much bad that will happen by replacing all CR (0x0D) with LF (0x0A),
Yeah. Many systems use the "parsing 20 means parsing anything <= 20", i.e., parsing a space allows all cosntrol chars as well. I think that is an OF requirement even? Then (on code where the line delimiter is 0d 0a and there are no other 0d or 0a in the file) you just get an extra blank line after every line, which usually is harmless.
One possible solution; take the boot-script code, and rather than interpret it directly, try compiling it.
That does not work. The OS9 bootscript contains many definitions as well. Not just colon definitions.
In summary, the two things you need:
-- Parse the bootscript line by line. -- Have a separate interpretation mode R stack.
Segher
On 2016-Apr-16 21:14 , Segher Boessenkool wrote:
One possible solution; take the boot-script code, and rather than interpret it directly, try compiling it.
That does not work. The OS9 bootscript contains many definitions as well. Not just colon definitions.
I thought the other definitions should work inside a colon definition. Unfortunately, I no longer have access to an openboot to test against (I retired late last year).
That's too bad.
In summary, the two things you need:
-- Parse the bootscript line by line. -- Have a separate interpretation mode R stack.
The separate R stack is probably the harder issue.
On Sat, Apr 16, 2016 at 09:40:04PM -0400, Tarl Neustaedter wrote:
On 2016-Apr-16 21:14 , Segher Boessenkool wrote:
One possible solution; take the boot-script code, and rather than interpret it directly, try compiling it.
That does not work. The OS9 bootscript contains many definitions as well. Not just colon definitions.
I thought the other definitions should work inside a colon definition. Unfortunately, I no longer have access to an openboot to test against (I retired late last year).
A trivial example is
1000 constant pagesz pagesz 1- constant pagesz-1
If you put that inside a colon def, the second pagesz will error when the colon def is compiled (the word hasn't been defined yet).
In summary, the two things you need:
-- Parse the bootscript line by line. -- Have a separate interpretation mode R stack.
The separate R stack is probably the harder issue.
It isn't hard to implement /an sich/. Both Apple OF and OFW (and thus, openboot) already have it. It will however uncover shortcomings in your text interpreter :-/
Segher
On Apr 16, 2016, at 9:52 PM, Segher Boessenkool wrote:
On Sat, Apr 16, 2016 at 09:40:04PM -0400, Tarl Neustaedter wrote:
On 2016-Apr-16 21:14 , Segher Boessenkool wrote:
One possible solution; take the boot-script code, and rather than interpret it directly, try compiling it.
That does not work. The OS9 bootscript contains many definitions as well. Not just colon definitions.
I thought the other definitions should work inside a colon definition. Unfortunately, I no longer have access to an openboot to test against (I retired late last year).
A trivial example is
1000 constant pagesz pagesz 1- constant pagesz-1
If you put that inside a colon def, the second pagesz will error when the colon def is compiled (the word hasn't been defined yet).
In summary, the two things you need:
-- Parse the bootscript line by line. -- Have a separate interpretation mode R stack.
The separate R stack is probably the harder issue.
It isn't hard to implement /an sich/. Both Apple OF and OFW (and thus, openboot) already have it. It will however uncover shortcomings in your text interpreter :-/
Apple already has this? Would you have an example program that would work in Apple's OF that would not work in OpenBIOS when using the return stack?
On Sat, Apr 16, 2016 at 09:59:52PM -0400, Programmingkid wrote:
Apple already has this?
Yes.
Would you have an example program that would work in Apple's OF that would not work in OpenBIOS when using the return stack?
How about the OS9 boot script? Heh.
Or try this perhaps:
hex 2a >r : oink [ r> ] literal ;
which is bound to give some fireworks if you don't have a separate interpret-mode return stack.
Even if it does work, try to run OINK and see if it returns what you expected.
Segher
Mac OS 9 does alter the return stack in OpenBIOS. This is perceived to be unacceptable. The solution to this issue has been stated as implementing a second return stack. This stack would just be an array that implements push and pull calls.
In the bootinfo_load.c file, in the bootinfo_init_program() function is where the Mac OS 9 boot script is read into memory, into the bootscript variable. What I want to do is replace all >r and r> calls in the bootscript buffer with other words. Something like substitute>r and substitute_r>. These words would use a separate array to push and pop data from. I think this would work because Mac OS 9's boot script only uses the return stack for data storage. Does this sound like the right thing to do?
On Sat, Apr 16, 2016 at 10:11:36PM -0400, Programmingkid wrote:
Mac OS 9 does alter the return stack in OpenBIOS. This is perceived to be unacceptable. The solution to this issue has been stated as implementing a second return stack. This stack would just be an array that implements push and pull calls.
In the bootinfo_load.c file, in the bootinfo_init_program() function is where the Mac OS 9 boot script is read into memory, into the bootscript variable. What I want to do is replace all >r and r> calls in the bootscript buffer with other words. Something like substitute>r and substitute_r>. These words would use a separate array to push and pop data from. I think this would work because Mac OS 9's boot script only uses the return stack for data storage. Does this sound like the right thing to do?
[ Please wrap your lines, this is unreadable. Thanks. ]
No; just implement a version of >R etc. that work in interpret mode. Like
: >r state @ IF postpone >r EXIT THEN ( and here, do the push to the simulated R stack ) ; IMMEDIATE
(And easier/cleaner/nicer if your system has separate compile and interpret behaviours for all words, or a separate compiler wordlist).
Segher
On Apr 16, 2016, at 10:19 PM, Segher Boessenkool wrote:
On Sat, Apr 16, 2016 at 10:11:36PM -0400, Programmingkid wrote:
Mac OS 9 does alter the return stack in OpenBIOS. This is perceived to be unacceptable. The solution to this issue has been stated as implementing a second return stack. This stack would just be an array that implements push and pull calls.
In the bootinfo_load.c file, in the bootinfo_init_program() function is where the Mac OS 9 boot script is read into memory, into the bootscript variable. What I want to do is replace all >r and r> calls in the bootscript buffer with other words. Something like substitute>r and substitute_r>. These words would use a separate array to push and pop data from. I think this would work because Mac OS 9's boot script only uses the return stack for data storage. Does this sound like the right thing to do?
[ Please wrap your lines, this is unreadable. Thanks. ]
ok.
No; just implement a version of >R etc. that work in interpret mode. Like
: >r state @ IF postpone >r EXIT THEN ( and here, do the push to the simulated R stack ) ; IMMEDIATE
(And easier/cleaner/nicer if your system has separate compile and interpret behaviours for all words, or a separate compiler wordlist).
r is implemented in forth.c in c. So what do you think about this:
/* * >r ( x -- ) (R: -- x ) */
static void tor(void) { ucell tmp = POP(); #ifdef CONFIG_DEBUG_RSTACK printk(" >R: %x\n", tmp); #endif if (interpreting_bootscript == true) pushR2(tmp); else PUSHR(tmp); }
The new code would be this part: if (interpreting_bootscript == true) pushR2(tmp); else
pushR2() would push into another array that isn't the return stack.
interpreting_bootscript would be a global variable that is set in the bootinfo_init_program() function right before the call to feval() that runs the bootscript. This variable would be set to false right after the call to feval(). Basically this:
interpreting_bootscript = true; feval(bootscript); interpreting_bootscript = false;
On Sat, Apr 16, 2016 at 10:44:47PM -0400, Programmingkid wrote:
No; just implement a version of >R etc. that work in interpret mode. Like
: >r state @ IF postpone >r EXIT THEN ( and here, do the push to the simulated R stack ) ; IMMEDIATE
(And easier/cleaner/nicer if your system has separate compile and interpret behaviours for all words, or a separate compiler wordlist).
r is implemented in forth.c in c.
The "new" (here defined, in that colon def above) >R calls the "old" version of >R (via "postpone >r"). Don't try to cram everything into the "old" definition, unles you like complexity (and that complexity can be quite enormous, look up what "state smart" words are).
Segher
On Apr 16, 2016, at 11:02 PM, Segher Boessenkool wrote:
On Sat, Apr 16, 2016 at 10:44:47PM -0400, Programmingkid wrote:
No; just implement a version of >R etc. that work in interpret mode. Like
: >r state @ IF postpone >r EXIT THEN ( and here, do the push to the simulated R stack ) ; IMMEDIATE
(And easier/cleaner/nicer if your system has separate compile and interpret behaviours for all words, or a separate compiler wordlist).
r is implemented in forth.c in c.
The "new" (here defined, in that colon def above) >R calls the "old" version of >R (via "postpone >r"). Don't try to cram everything into the "old" definition, unles you like complexity (and that complexity can be quite enormous, look up what "state smart" words are).
That code above looks a bit complex. I think implementing another word would be a lot simpler.
On Sat, Apr 16, 2016 at 11:44:11PM -0400, Programmingkid wrote:
No; just implement a version of >R etc. that work in interpret mode. Like
: >r state @ IF postpone >r EXIT THEN ( and here, do the push to the simulated R stack ) ; IMMEDIATE
(And easier/cleaner/nicer if your system has separate compile and interpret behaviours for all words, or a separate compiler wordlist).
r is implemented in forth.c in c.
The "new" (here defined, in that colon def above) >R calls the "old" version of >R (via "postpone >r"). Don't try to cram everything into the "old" definition, unles you like complexity (and that complexity can be quite enormous, look up what "state smart" words are).
That code above looks a bit complex. I think implementing another word would be a lot simpler.
But you want both words to be called >R so how will you do this? Having a separate compiler wordlist will solve this (and much bigger problems) very elegantly, but this is hard to retrofit to an existing design.
Anyway, where do you see complexity? Verbosity, sure, and doing extra work (but only in interpretation mode, so you really should not care); but complexity?
Segher
-
Programmingkid -
Segher Boessenkool -
Tarl Neustaedter