Hi Mark,
* Mark Wenning wenning@us.ibm.com [050105 23:34]:
We're looking at using toke, detok in our development process.
Nice to hear. Let me know how I can help make this happen.
Our current setup to tokenize device drivers uses a "script" similar to this: ----- start kng_sr_script ----- \ define these words for the tokenizer tokenizer[ 0 constant ?SYSTEM-ROM 0 constant COMMENT-OUT [..] ]tokenizer
It's "not really a bug" but a lacking feature. ;-) The tokenizer[ directive should directly execute the forth words before ]tokenizer. Instead toke only pushes the specified values to the stack. Simply because it has no forth engine to call inside.
This results in your two problems:
The version of toke I downloaded doesn't like this for 2 reasons:
- Constant declarations in TOKENIZER[ ... ]TOKENIZER blocks emit
bytes/tokens to the output, so we get extra stuff before the PCI header. I don't think it should do this; code inside tokenizer blocks should not be in the output FCODE. Is this correct?
- Previous constant declarations aren't being looked up/interpreted
when in Tokenizer mode. For example, tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer SET-REV-LEVEL causes a "empty stack: error. I have to change it to tokenizer[ 0110 ]tokenizer SET-REV-LEVEL to make it work. Is this a bug?
It is something that needs to be fixed, no doubt.
I'm a little new to this; basically what I'm asking is if this behaviour is what was intended, or should we attempt to fix it and submit a fix to you.
There are basically two possible solutions to the problem: 1) Implement parsing for the words you use in the existing framework of toke. ie. implement a "constant" and a "hex" word. This is pretty straight forward, but will break the next time someone is using a different forthish feature.
2) the clean solution: Implement future versions of toke on top of the OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth implementation that is well suited.
Clearly, 2. is the long term solution.
Stefan