bug in toke?

[Mark Wenning]

Hi -
We're looking at using toke, detok in our development process.

Our current setup to tokenize device drivers uses a "script" similar to 
this:
-----   start kng_sr_script -----
    \  define these words for the tokenizer
    tokenizer[
        0           constant ?SYSTEM-ROM
        0           constant COMMENT-OUT

        0           constant ?DEBUG-TX
        0           constant ?DEBUG-RX

        h# 0110 constant ibm-Code-Revision-Level

        h# 17D5     constant ibm-VendorId
        h# 5831     constant ibm-king-DeviceId
        h# 020000   constant ClsCode
    ]tokenizer

    fload kng_main.of
----- end kng_sr_script -----

Then in kng_main.of, we do:
----- start kng_main.of   -----
    ....
   tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
 
   tokenizer[ hex ibm-VendorId ibm-king-DeviceId  ClsCode decimal 
]tokenizer PCI-HEADER
  
   FCODE-VERSION2

    ...
----- start of code...-----

The version of toke I downloaded doesn't like this for 2 reasons:
1.  Constant declarations in TOKENIZER[    ...   ]TOKENIZER blocks  emit 
bytes/tokens to the output,
so we get extra stuff before the PCI header.   I don't think it should 
do this; code inside tokenizer blocks
should not be in the output FCODE.  Is this correct?
2.  Previous constant declarations aren't being looked up/interpreted 
when in Tokenizer mode.  For
example,
    tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
causes a "empty stack: error.  I have to change it to
    tokenizer[ 0110 ]tokenizer  SET-REV-LEVEL
to make it work.  Is this a bug? 

I'm a little new to this; basically what I'm asking is if this behaviour 
is what was intended, or should we attempt to fix it and submit a fix to 
you.

[Stefan Reinauer]

Hi Mark,

* Mark Wenning <wenning(a)us.ibm.com> [050105 23:34]:
  We're looking at using toke, detok in our
development process.
Nice to hear. Let me know how I can help make this happen.
 
  Our current setup to tokenize device drivers uses a
"script" similar to 
 this:
 -----   start kng_sr_script -----
    \  define these words for the tokenizer
    tokenizer[
        0           constant ?SYSTEM-ROM
        0           constant COMMENT-OUT
        [..] 
    ]tokenizer
It's "not really a bug" but a lacking feature. ;-) The tokenizer[
directive should directly execute the forth words before ]tokenizer.
Instead toke only pushes the specified values to the stack. Simply
because it has no forth engine to call inside. 

This results in your two problems:

  The version of toke I downloaded doesn't like this
for 2 reasons:
 1.  Constant declarations in TOKENIZER[    ...   ]TOKENIZER blocks  emit 
 bytes/tokens to the output,
 so we get extra stuff before the PCI header.   I don't think it should 
 do this; code inside tokenizer blocks
 should not be in the output FCODE.  Is this correct?
  2.  Previous constant declarations aren't being
looked up/interpreted 
 when in Tokenizer mode.  For
 example,
    tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
 causes a "empty stack: error.  I have to change it to
    tokenizer[ 0110 ]tokenizer  SET-REV-LEVEL
 to make it work.  Is this a bug?
It is something that needs to be fixed, no doubt.

  I'm a little new to this; basically what I'm
asking is if this behaviour 
 is what was intended, or should we attempt to fix it and submit a fix to 
 you.
 
There are basically two possible solutions to the problem:
 1) Implement parsing for the words you use in the existing framework of
    toke. ie. implement a "constant" and a "hex" word. This is pretty
    straight forward, but will break the next time someone is using a
    different forthish feature.

 2) the clean solution: Implement future versions of toke on top of the
    OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
    implementation that is well suited. 

Clearly, 2. is the long term solution. 

Stefan

[Segher Boessenkool]

   2) the clean solution: Implement future versions of
toke on top of the
     OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
     implementation that is well suited.
3) Implement the FCode tokenizer in some standard Forth dialect, for
example, in ANS Forth -- it'll run on _any_ Forth system (that 
implements
that standard), then (this should include BeginAgain).


Segher

[Stefan Reinauer]

* Segher Boessenkool <segher(a)kernel.crashing.org> [050109 02:34]:
  2) the clean
solution: Implement future versions of toke on top of the
    OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
    implementation that is well suited.
 
 3) Implement the FCode tokenizer in some standard Forth dialect, for
 example, in ANS Forth -- it'll run on _any_ Forth system (that 
 implements
 that standard), then (this should include BeginAgain).
Which is 2). BeginAgain is just a light ANS forth implementation.

Stefan

[Segher Boessenkool]

  2) the clean solution: Implement future versions of
toke on top of 
 the
    OpenBIOS forth engine "BeginAgain". This is a pretty minimal forth
    implementation that is well suited.
 3) Implement the FCode tokenizer in some standard Forth dialect, for
 example, in ANS Forth -- it'll run on _any_ Forth system (that
 implements
 that standard), then (this should include BeginAgain).
 Which is 2). BeginAgain is just a light ANS forth implementation.
Of course, and I already said that -- I was just emphasising the
need for a _portable_ tokenizer, not something that runs _only_ on
BeginAgain.


Segher