[OpenBIOS] bug in toke?

[Stefan Reinauer]

Hi Mark,

* Mark Wenning <wenning at us.ibm.com> [050105 23:34]:
> We're looking at using toke, detok in our development process.

Nice to hear. Let me know how I can help make this happen.
 
> Our current setup to tokenize device drivers uses a "script" similar to 
> this:
> -----   start kng_sr_script -----
>    \  define these words for the tokenizer
>    tokenizer[
>        0           constant ?SYSTEM-ROM
>        0           constant COMMENT-OUT
>        [..] 
>    ]tokenizer

It's "not really a bug" but a lacking feature. ;-) The tokenizer[
directive should directly execute the forth words before ]tokenizer.
Instead toke only pushes the specified values to the stack. Simply
because it has no forth engine to call inside. 

This results in your two problems:

> The version of toke I downloaded doesn't like this for 2 reasons:
> 1.  Constant declarations in TOKENIZER[    ...   ]TOKENIZER blocks  emit 
> bytes/tokens to the output,
> so we get extra stuff before the PCI header.   I don't think it should 
> do this; code inside tokenizer blocks
> should not be in the output FCODE.  Is this correct?

> 2.  Previous constant declarations aren't being looked up/interpreted 
> when in Tokenizer mode.  For
> example,
>    tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
> causes a "empty stack: error.  I have to change it to
>    tokenizer[ 0110 ]tokenizer  SET-REV-LEVEL
> to make it work.  Is this a bug? 

It is something that needs to be fixed, no doubt.

> I'm a little new to this; basically what I'm asking is if this behaviour 
> is what was intended, or should we attempt to fix it and submit a fix to 
> you.
 
There are basically two possible solutions to the problem:
 1) Implement parsing for the words you use in the existing framework of
    toke. ie. implement a "constant" and a "hex" word. This is pretty
    straight forward, but will break the next time someone is using a
    different forthish feature.

 2) the clean solution: Implement future versions of toke on top of the
    OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
    implementation that is well suited. 

Clearly, 2. is the long term solution. 

Stefan