[LinuxBIOS] romcc and preprocessor problems

[Stefan Reinauer]

Hi,

since the romcc builtin preprocessor silently eats lines of code,
can we just use the gcc preprocessor (cpp) before calling romcc as
a workaround?

Is there a reason romcc has its own preprocessor?

Stefan

[Segher Boessenkool]

> since the romcc builtin preprocessor silently eats lines of code,
> can we just use the gcc preprocessor (cpp) before calling romcc as
> a workaround?

That should work (if you get the -I directives, etc., right).
But use gcc -E instead of cpp...
> Is there a reason romcc has its own preprocessor?

Probably not.

Segher

Stefan Reinauer  writes:
> Hi,
>
> since the romcc builtin preprocessor silently eats lines of code,
> can we just use the gcc preprocessor (cpp) before calling romcc as
> a workaround?

I'd love to see a reproducer of that problem.  I don't have a good feel
that it has been root caused.  The parser is fairly straight forward
so if we can reproduce this problem it shouldn't be hard to fix.

I'm not convinced the problem is actually isolated to just the c pre processor.
> Is there a reason romcc has its own preprocessor?

A couple of reasons.
- It was comparatively easy.
- It makes the dependencies much easier to get right.
- We don't have to worry about gcc polluting the compile with #defines
  that are not correct for the romcc environment.
- There are several C features that you cannot implement properly unless
  you implement some level of c preprocessor.

All of that said you can look back into the history to when romcc did not have
a preprocessor and can probably make that work.  I'd much rather fix
the stupid parsing bug.

Eric

[Ben Hewson]

Hi Eric

Please find attached some simple test cases.
These were compiled using the following

./romcc -O  -fno-eliminate-inefectual-code
-I../../../../../src/arch/i386/include/  -o testdef3.inc  testdef3.c

if you look at testdef2.c, testdef2.inc you can see that a C++ comment
has caused one line to be missed, no comment or a C comment are fine. I
checked no comments, C comments and C++ comments on both the #if and
#else lines. The one thing I did not check was a C++ comment on the
#endif to see if the following line was missed.

I may be missing needed options for romcc, but I have tried similar
tests in a proper build, with the same results.

regards
Ben
> Stefan Reinauer  writes:
> > Hi,
> >
> > since the romcc builtin preprocessor silently eats lines of code,
> > can we just use the gcc preprocessor (cpp) before calling romcc as
> > a workaround?
>
> I'd love to see a reproducer of that problem.  I don't have a good feel
> that it has been root caused.  The parser is fairly straight forward
> so if we can reproduce this problem it shouldn't be hard to fix.
>
> I'm not convinced the problem is actually isolated to just the c pre processor.
> > Is there a reason romcc has its own preprocessor?
>
> A couple of reasons.
> It was comparatively easy.
> It makes the dependencies much easier to get right.
> We don't have to worry about gcc polluting the compile with #defines
> that are not correct for the romcc environment.
> There are several C features that you cannot implement properly unless
> you implement some level of c preprocessor.
>
> All of that said you can look back into the history to when romcc did not have
> a preprocessor and can probably make that work.  I'd much rather fix
> the stupid parsing bug.
>
> Eric

.section ".rom.data"
.balign 16
L1:
.int 0xff, 0, 0, 0
L2:
.int 0xffff, 0, 0, 0
.section ".rom.text"
    /* ,(null):0.0 */
L1001:
L1002:
    /* outb,io.h:14.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* outw,io.h:19.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* outl,io.h:24.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* inb,io.h:30.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* inw,io.h:36.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* inl,io.h:41.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* outsb,io.h:85.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* outsw,io.h:94.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* outsl,io.h:103.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* insb,io.h:113.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* insw,io.h:122.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* insl,io.h:131.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* writeb,io.h:140.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* writew,io.h:145.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* writel,io.h:150.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* readb,io.h:155.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* readw,io.h:160.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* readl,io.h:165.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* main,testdef.c:12.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6d
.byte 0x61
.byte 0x69
.byte 0x6e
.byte 0x00
.section ".rom.text"
    /* ,(null):0.0 */
    /* ,:0.0 */
#ASM
    
#NOT ASM
    /* main,testdef.c:12.1 */
    /* main,testdef.c:14.16 */
    /* main,testdef.c:14.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef.c:14.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef.c:14.13
     */
    mov  $10 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef.c:14.13
     */
    /* main,testdef.c:20.16 */
    /* main,testdef.c:20.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef.c:20.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef.c:20.13
     */
    mov  $11 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef.c:20.13
     */
    /* main,testdef.c:21.16 */
    /* main,testdef.c:21.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef.c:21.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef.c:21.13
     */
    mov  $12 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef.c:21.13
     */
    /* main,testdef.c:25.16 */
    /* main,testdef.c:25.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef.c:25.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef.c:25.13
     */
    mov  $15 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef.c:25.13
     */
    /* main,testdef.c:12.1 */
    /* ,:0.0 */
 /* unknown %eax */
 /* unknown %ebx */
 /* unknown %ecx */
 /* unknown %edx */
#ASM
    
#NOT ASM
/* max inline depth 3 */

.section ".rom.data"
.balign 16
L1:
.int 0xff, 0, 0, 0
L2:
.int 0xffff, 0, 0, 0
.section ".rom.text"
    /* ,(null):0.0 */
L1001:
L1002:
    /* outb,io.h:14.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* outw,io.h:19.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* outl,io.h:24.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* inb,io.h:30.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* inw,io.h:36.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* inl,io.h:41.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* outsb,io.h:85.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* outsw,io.h:94.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* outsl,io.h:103.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* insb,io.h:113.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* insw,io.h:122.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* insl,io.h:131.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* writeb,io.h:140.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* writew,io.h:145.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* writel,io.h:150.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* readb,io.h:155.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* readw,io.h:160.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* readl,io.h:165.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* main,testdef.c:12.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6d
.byte 0x61
.byte 0x69
.byte 0x6e
.byte 0x00
.section ".rom.text"
    /* ,(null):0.0 */
    /* ,:0.0 */
#ASM
    
#NOT ASM
    /* main,testdef.c:12.1 */
    /* main,testdef.c:14.16 */
    /* main,testdef.c:14.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef.c:14.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef.c:14.13
     */
    mov  $10 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef.c:14.13
     */
    /* main,testdef.c:21.16 */
    /* main,testdef.c:21.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef.c:21.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef.c:21.13
     */
    mov  $12 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef.c:21.13
     */
    /* main,testdef.c:25.16 */
    /* main,testdef.c:25.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef.c:25.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef.c:25.13
     */
    mov  $15 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef.c:25.13
     */
    /* main,testdef.c:12.1 */
    /* ,:0.0 */
 /* unknown %eax */
 /* unknown %ebx */
 /* unknown %ecx */
 /* unknown %edx */
#ASM
    
#NOT ASM
/* max inline depth 3 */

.section ".rom.data"
.balign 16
L1:
.int 0xff, 0, 0, 0
L2:
.int 0xffff, 0, 0, 0
.section ".rom.text"
    /* ,(null):0.0 */
L1001:
L1002:
    /* outb,io.h:14.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* outw,io.h:19.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* outl,io.h:24.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* inb,io.h:30.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* inw,io.h:36.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* inl,io.h:41.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* outsb,io.h:85.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* outsw,io.h:94.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* outsl,io.h:103.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6f
.byte 0x75
.byte 0x74
.byte 0x73
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* insb,io.h:113.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* insw,io.h:122.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* insl,io.h:131.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x69
.byte 0x6e
.byte 0x73
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* writeb,io.h:140.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* writew,io.h:145.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* writel,io.h:150.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x77
.byte 0x72
.byte 0x69
.byte 0x74
.byte 0x65
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* readb,io.h:155.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x62
.byte 0x00
.section ".rom.text"
    /* readw,io.h:160.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x77
.byte 0x00
.section ".rom.text"
    /* readl,io.h:165.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x72
.byte 0x65
.byte 0x61
.byte 0x64
.byte 0x6c
.byte 0x00
.section ".rom.text"
    /* main,testdef3.c:12.1 */
.section ".rom.data"
.balign 1
L0:
.byte 0x6d
.byte 0x61
.byte 0x69
.byte 0x6e
.byte 0x00
.section ".rom.text"
    /* ,(null):0.0 */
    /* ,:0.0 */
#ASM
    
#NOT ASM
    /* main,testdef3.c:12.1 */
    /* main,testdef3.c:14.16 */
    /* main,testdef3.c:14.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef3.c:14.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef3.c:14.13
     */
    mov  $10 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef3.c:14.13
     */
    /* main,testdef3.c:20.16 */
    /* main,testdef3.c:20.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef3.c:20.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef3.c:20.13
     */
    mov  $11 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef3.c:20.13
     */
    /* main,testdef3.c:21.16 */
    /* main,testdef3.c:21.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef3.c:21.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef3.c:21.13
     */
    mov  $12 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef3.c:21.13
     */
    /* main,testdef3.c:25.16 */
    /* main,testdef3.c:25.20 */
    /* ,:0.0 */
    /*
     * outb,io.h:14.1
     * main,testdef3.c:25.13
     */
    /*
     * __builtin_outb,<built-in>:1.0
     * outb,io.h:15.23
     * main,testdef3.c:25.13
     */
    mov  $15 , %al
    outb %al, $100 
    /*
     * outb,io.h:14.1
     * main,testdef3.c:25.13
     */
    /* main,testdef3.c:12.1 */
    /* ,:0.0 */
 /* unknown %eax */
 /* unknown %ebx */
 /* unknown %ecx */
 /* unknown %edx */
#ASM
    
#NOT ASM
/* max inline depth 3 */

Ben Hewson  writes:
> Hi Eric
>
> Please find attached some simple test cases.
> These were compiled using the following
>
> ./romcc -O  -fno-eliminate-inefectual-code
> -I../../../../../src/arch/i386/include/  -o testdef3.inc  testdef3.c
>
> if you look at testdef2.c, testdef2.inc you can see that a C++ comment
> has caused one line to be missed, no comment or a C comment are fine. I
> checked no comments, C comments and C++ comments on both the #if and
> #else lines. The one thing I did not check was a C++ comment on the
> #endif to see if the following line was missed.
>
> I may be missing needed options for romcc, but I have tried similar
> tests in a proper build, with the same results.

I took a quick look and it turns out the problem was the tokenizer was
treating the newline as part of the EOL comment.  This only showed
up in the preprocessor because that really the only place that cares
about seeing the EOL token separately from a space.

Please test the attached patch.  Thanks.

Eric
--- ../romcc-0.64/romcc.c	2006-12-02 07:39:11.000000000 -0700
+++ romcc.c	2006-12-02 07:41:23.000000000 -0700
@@ -4027,10 +4027,15 @@
    	tok = TOK_SPACE;
    	tokp = next_char(file, tokp, 1);
    	while((c = get_char(file, tokp)) != -1) {
-			tokp = next_char(file, tokp, 1);
+			/* Advance to the next character only after we verify
+			 * the current character is not a newline.  
+			 * EOL is special to the preprocessor so we don't
+			 * want to loose any.
+			 */
    		if (c == '\n') {
    			break;
    		}
+			tokp = next_char(file, tokp, 1);
    	}
    }
    /* Comments */