toke core dumps

[Prasanna Kumar]

Hello All,

When I try to tokenize a large FCode source file, the
tokenizer crashes with Segmentation Fault.

Debug messages:

tst.fth:6763: debug: tokenizing control word 'endof'
tst.fth:6764: debug: read token 'endcase', length=7
tst.fth:6764: debug: matched internal opcode 0x0013
tst.fth:6764: debug: tokenizing control word 'endcase'
tst.fth:6764: debug: endcase offset 0x8401
tst.fth:6764: debug: endcase offset 0xffff83f1
                                    ??????????
Segmentation fault (core dumped)

When I looked at the code, the offending function was
in emit.c :

s16 receive_offset(void)
{
        s16 offs=0;

if (offs16) {
                offs= ((*opc)<<8)|(*(opc+1));
        } else {
                offs=(*opc);
       }
        return offs;
}

If the msb of *opc is set 1(Value >0 0x8yyy), the offs
gets changed to ffff8yyy.

Changing offs to u16 and receive_offset returning
value of u16 type would solve the problem. Change the
prototype for receive_offset in emit.h as well.

Modified code:

u16 receive_offset(void) <----This line changed
{
        u16 offs=0;      <---- This line changed

if (offs16) {
                offs= ((*opc)<<8)|(*(opc+1));
        } else {
                offs=(*opc);
       }
        return offs;
}

Steps to reproduce the problem very simply:
- Create a fcode source file with around 7000
variables.
- Use the following simple test code:

: test ( -- )
   1 to temp1
   2 to temp2
   temp1 case
     1 of
       ." One" cr
     endof
     2 of temp2 
       ." Two" cr
     endof
   endcase
;

- Run tokenizer on the file. toke will core dump.

Regards,
Prasanna

__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

[Stefan Reinauer]

Hi Prasana,

thanks a lot for your bug report. It's nice to see that toke
is widely used nowadays and I'm glad if we can get the last 
few bugs out of it.
> If the msb of *opc is set 1(Value >0 0x8yyy), the offs
> gets changed to ffff8yyy.
>
> Changing offs to u16 and receive_offset returning
> value of u16 type would solve the problem. Change the
> prototype for receive_offset in emit.h as well.

I've changed the code in a slightly different way, doing 
the same thing though.
> Create a fcode source file with around 7000
> variables.

Are you sure you mean variables? A single FCode program has a limitation
of 2048 variables due to the space reserved for them (Local FCode
numbers go from 0x800 to 0xfff).

The problem you described occurs though if your bytecode size exceeds
0x7fff bytes.

I fixed this in openbios--main--patch-21. Now you get an error message
if you exceed the maximum word count and case..endcase constructs are
working for code larger than 32KB ;-)

You can also download toke-0.6.9 in a bit.

Stefan

[Prasanna Kumar]

Hi Stefan,

Thanks a lot for your reply and fixing the problem.

--- Stefan Reinauer  wrote:
> Hi Prasana,
>
> thanks a lot for your bug report. It's nice to see
> that toke is widely used nowadays and I'm glad if we

can get
> the last few bugs out of it.

Yea sure Stefan.
> > If the msb of *opc is set 1(Value >0 0x8yyy), the
>
> offs
> > gets changed to ffff8yyy.
> >
> > Changing offs to u16 and receive_offset returning
> > value of u16 type would solve the problem. Change
>
> the
> > prototype for receive_offset in emit.h as well.
>
> I've changed the code in a slightly different way,
> doing  the same thing though.
> > Create a fcode source file with around 7000
> > variables.
>
> Are you sure you mean variables? A single FCode
> program has a limitation
> of 2048 variables due to the space reserved for them
> (Local FCode
> numbers go from 0x800 to 0xfff).

I just wanted to show that if the offset reaches >7fff
bytes, this problem occurs. I didn't check if the
local FCode numbers got repeated or not.
> The problem you described occurs though if your
> bytecode size exceeds
> 0x7fff bytes.
>
> I fixed this in openbios--main--patch-21. Now you
> get an error message
> if you exceed the maximum word count and
> case..endcase constructs are
> working for code larger than 32KB ;-)

Thank you Stefan.
> You can also download toke-0.6.9 in a bit.

Sure. Will be waiting for it.

~Prasanna

__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com