* Mark Wenning <wenning(a)us.ibm.com> [050105 23:34]:
We're looking at using toke, detok in our development process.
Nice to hear. Let me know how I can help make this happen.
Our current setup to tokenize device drivers uses a
"script" similar to
----- start kng_sr_script -----
\ define these words for the tokenizer
0 constant ?SYSTEM-ROM
0 constant COMMENT-OUT
It's "not really a bug" but a lacking feature. ;-) The tokenizer[
directive should directly execute the forth words before ]tokenizer.
Instead toke only pushes the specified values to the stack. Simply
because it has no forth engine to call inside.
This results in your two problems:
The version of toke I downloaded doesn't like this for 2
1. Constant declarations in TOKENIZER[ ... ]TOKENIZER blocks emit
bytes/tokens to the output,
so we get extra stuff before the PCI header. I don't think it should
do this; code inside tokenizer blocks
should not be in the output FCODE. Is this correct?
2. Previous constant declarations aren't being looked
when in Tokenizer mode. For
tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer SET-REV-LEVEL
causes a "empty stack: error. I have to change it to
tokenizer[ 0110 ]tokenizer SET-REV-LEVEL
to make it work. Is this a bug?
It is something that needs to be fixed, no doubt.
I'm a little new to this; basically what I'm asking is if
is what was intended, or should we attempt to fix it and submit a fix to
There are basically two possible solutions to the problem:
1) Implement parsing for the words you use in the existing framework of
toke. ie. implement a "constant" and a "hex" word. This is pretty
straight forward, but will break the next time someone is using a
different forthish feature.
2) the clean solution: Implement future versions of toke on top of the
OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
implementation that is well suited.
Clearly, 2. is the long term solution.