bug in toke?

[Mark Wenning]

Hi -
We're looking at using toke, detok in our development process.

Our current setup to tokenize device drivers uses a "script" similar to 
this:
-----   start kng_sr_script -----
    \  define these words for the tokenizer
    tokenizer[
        0           constant ?SYSTEM-ROM
        0           constant COMMENT-OUT

0           constant ?DEBUG-TX
        0           constant ?DEBUG-RX

h# 0110 constant ibm-Code-Revision-Level

h# 17D5     constant ibm-VendorId
        h# 5831     constant ibm-king-DeviceId
        h# 020000   constant ClsCode
    ]tokenizer

fload kng_main.of
----- end kng_sr_script -----

Then in kng_main.of, we do:
----- start kng_main.of   -----
    ....
   tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL

tokenizer[ hex ibm-VendorId ibm-king-DeviceId  ClsCode decimal 
]tokenizer PCI-HEADER

FCODE-VERSION2

...
----- start of code...-----

The version of toke I downloaded doesn't like this for 2 reasons:
1.  Constant declarations in TOKENIZER[    ...   ]TOKENIZER blocks  emit 
bytes/tokens to the output,
so we get extra stuff before the PCI header.   I don't think it should 
do this; code inside tokenizer blocks
should not be in the output FCODE.  Is this correct?
2.  Previous constant declarations aren't being looked up/interpreted 
when in Tokenizer mode.  For
example,
    tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
causes a "empty stack: error.  I have to change it to
    tokenizer[ 0110 ]tokenizer  SET-REV-LEVEL
to make it work.  Is this a bug?

I'm a little new to this; basically what I'm asking is if this behaviour 
is what was intended, or should we attempt to fix it and submit a fix to 
you.

[Stefan Reinauer]

Hi Mark,

* Mark Wenning  [050105 23:34]:
> We're looking at using toke, detok in our development process.

Nice to hear. Let me know how I can help make this happen.
> Our current setup to tokenize device drivers uses a "script" similar to 
> this:
> -----   start kng_sr_script -----
>    \  define these words for the tokenizer
>    tokenizer[
>        0           constant ?SYSTEM-ROM
>        0           constant COMMENT-OUT
>        [..] 
>    ]tokenizer

It's "not really a bug" but a lacking feature. ;-) The tokenizer[
directive should directly execute the forth words before ]tokenizer.
Instead toke only pushes the specified values to the stack. Simply
because it has no forth engine to call inside.

This results in your two problems:
> The version of toke I downloaded doesn't like this for 2 reasons:
> Constant declarations in TOKENIZER[    ...   ]TOKENIZER blocks  emit
>
> bytes/tokens to the output,
> so we get extra stuff before the PCI header.   I don't think it should 
> do this; code inside tokenizer blocks
> should not be in the output FCODE.  Is this correct?
> Previous constant declarations aren't being looked up/interpreted
>
> when in Tokenizer mode.  For
> example,
>    tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
> causes a "empty stack: error.  I have to change it to
>    tokenizer[ 0110 ]tokenizer  SET-REV-LEVEL
> to make it work.  Is this a bug?

It is something that needs to be fixed, no doubt.
> I'm a little new to this; basically what I'm asking is if this behaviour 
> is what was intended, or should we attempt to fix it and submit a fix to 
> you.

There are basically two possible solutions to the problem:
 1) Implement parsing for the words you use in the existing framework of
    toke. ie. implement a "constant" and a "hex" word. This is pretty
    straight forward, but will break the next time someone is using a
    different forthish feature.

2) the clean solution: Implement future versions of toke on top of the
    OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
    implementation that is well suited.

Clearly, 2. is the long term solution.

Stefan

[Segher Boessenkool]

> the clean solution: Implement future versions of toke on top of the
> OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
> implementation that is well suited.

3) Implement the FCode tokenizer in some standard Forth dialect, for
example, in ANS Forth -- it'll run on _any_ Forth system (that 
implements
that standard), then (this should include BeginAgain).

Segher

[Stefan Reinauer]

* Segher Boessenkool  [050109 02:34]:
> > the clean solution: Implement future versions of toke on top of the
> > OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
> > implementation that is well suited.
> Implement the FCode tokenizer in some standard Forth dialect, for
>
> example, in ANS Forth -- it'll run on _any_ Forth system (that 
> implements
> that standard), then (this should include BeginAgain).

Which is 2). BeginAgain is just a light ANS forth implementation.

Stefan

[Segher Boessenkool]

> > > the clean solution: Implement future versions of toke on top of
> > >
> > > the
> > >    OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
> > >    implementation that is well suited.
> > Implement the FCode tokenizer in some standard Forth dialect, for
> >
> > example, in ANS Forth -- it'll run on _any_ Forth system (that
> > implements
> > that standard), then (this should include BeginAgain).
>
> Which is 2). BeginAgain is just a light ANS forth implementation.

Of course, and I already said that -- I was just emphasising the
need for a _portable_ tokenizer, not something that runs _only_ on
BeginAgain.

Segher