Hi - We're looking at using toke, detok in our development process. Our current setup to tokenize device drivers uses a "script" similar to this: ----- start kng_sr_script ----- \ define these words for the tokenizer tokenizer[ 0 constant ?SYSTEM-ROM 0 constant COMMENT-OUT 0 constant ?DEBUG-TX 0 constant ?DEBUG-RX h# 0110 constant ibm-Code-Revision-Level h# 17D5 constant ibm-VendorId h# 5831 constant ibm-king-DeviceId h# 020000 constant ClsCode ]tokenizer fload kng_main.of ----- end kng_sr_script ----- Then in kng_main.of, we do: ----- start kng_main.of ----- .... tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer SET-REV-LEVEL tokenizer[ hex ibm-VendorId ibm-king-DeviceId ClsCode decimal ]tokenizer PCI-HEADER FCODE-VERSION2 ... ----- start of code...----- The version of toke I downloaded doesn't like this for 2 reasons: 1. Constant declarations in TOKENIZER[ ... ]TOKENIZER blocks emit bytes/tokens to the output, so we get extra stuff before the PCI header. I don't think it should do this; code inside tokenizer blocks should not be in the output FCODE. Is this correct? 2. Previous constant declarations aren't being looked up/interpreted when in Tokenizer mode. For example, tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer SET-REV-LEVEL causes a "empty stack: error. I have to change it to tokenizer[ 0110 ]tokenizer SET-REV-LEVEL to make it work. Is this a bug? I'm a little new to this; basically what I'm asking is if this behaviour is what was intended, or should we attempt to fix it and submit a fix to you.
Hi Mark, * Mark Wenning <wenning@us.ibm.com> [050105 23:34]:
We're looking at using toke, detok in our development process.
Nice to hear. Let me know how I can help make this happen.
Our current setup to tokenize device drivers uses a "script" similar to this: ----- start kng_sr_script ----- \ define these words for the tokenizer tokenizer[ 0 constant ?SYSTEM-ROM 0 constant COMMENT-OUT [..] ]tokenizer
It's "not really a bug" but a lacking feature. ;-) The tokenizer[ directive should directly execute the forth words before ]tokenizer. Instead toke only pushes the specified values to the stack. Simply because it has no forth engine to call inside. This results in your two problems:
The version of toke I downloaded doesn't like this for 2 reasons: 1. Constant declarations in TOKENIZER[ ... ]TOKENIZER blocks emit bytes/tokens to the output, so we get extra stuff before the PCI header. I don't think it should do this; code inside tokenizer blocks should not be in the output FCODE. Is this correct?
2. Previous constant declarations aren't being looked up/interpreted when in Tokenizer mode. For example, tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer SET-REV-LEVEL causes a "empty stack: error. I have to change it to tokenizer[ 0110 ]tokenizer SET-REV-LEVEL to make it work. Is this a bug?
It is something that needs to be fixed, no doubt.
I'm a little new to this; basically what I'm asking is if this behaviour is what was intended, or should we attempt to fix it and submit a fix to you.
There are basically two possible solutions to the problem: 1) Implement parsing for the words you use in the existing framework of toke. ie. implement a "constant" and a "hex" word. This is pretty straight forward, but will break the next time someone is using a different forthish feature. 2) the clean solution: Implement future versions of toke on top of the OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth implementation that is well suited. Clearly, 2. is the long term solution. Stefan
2) the clean solution: Implement future versions of toke on top of the OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth implementation that is well suited.
3) Implement the FCode tokenizer in some standard Forth dialect, for example, in ANS Forth -- it'll run on _any_ Forth system (that implements that standard), then (this should include BeginAgain). Segher
* Segher Boessenkool <segher@kernel.crashing.org> [050109 02:34]:
2) the clean solution: Implement future versions of toke on top of the OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth implementation that is well suited.
3) Implement the FCode tokenizer in some standard Forth dialect, for example, in ANS Forth -- it'll run on _any_ Forth system (that implements that standard), then (this should include BeginAgain).
Which is 2). BeginAgain is just a light ANS forth implementation. Stefan
2) the clean solution: Implement future versions of toke on top of the OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth implementation that is well suited.
3) Implement the FCode tokenizer in some standard Forth dialect, for example, in ANS Forth -- it'll run on _any_ Forth system (that implements that standard), then (this should include BeginAgain).
Which is 2). BeginAgain is just a light ANS forth implementation.
Of course, and I already said that -- I was just emphasising the need for a _portable_ tokenizer, not something that runs _only_ on BeginAgain. Segher
participants (3)
-
Mark Wenning -
Segher Boessenkool -
Stefan Reinauer