OpenBIOS October 2006

openbios@openbios.org

14 participants
38 discussions

r113 - fcode-utils
by svn＠openbios.org 31 Oct '06

31 Oct '06

Author: stepan Date: 2006-10-31 00:06:43 +0100 (Tue, 31 Oct 2006) New Revision: 113 Modified: fcode-utils/Makefile Log: fix test log path Modified: fcode-utils/Makefile =================================================================== --- fcode-utils/Makefile 2006-10-30 22:09:59 UTC (rev 112) +++ fcode-utils/Makefile 2006-10-30 23:06:43 UTC (rev 113) @@ -51,7 +51,8 @@ cp toke/toke testsuite cp detok/detok testsuite cp romheaders/romheaders testsuite - make -C testsuite all CygTestLogs=`pwd`/testlogs + make -C testsuite all CygTestLogs=`pwd`/testlogs/testlogs-ppc-linux + #make -C testsuite all CygTestLogs=`pwd`/testlogs/testlogs-x86-cygwin # lcov required for html reports coverage:

1 0

r112 - in fcode-utils: . testsuite
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 23:09:59 +0100 (Mon, 30 Oct 2006) New Revision: 112 Added: fcode-utils/testsuite/GenCoverage Modified: fcode-utils/Makefile Log: fix automated coverage Modified: fcode-utils/Makefile =================================================================== --- fcode-utils/Makefile 2006-10-30 17:55:20 UTC (rev 111) +++ fcode-utils/Makefile 2006-10-30 22:09:59 UTC (rev 112) @@ -21,6 +21,8 @@ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA, 02110-1301 USA # +VERSION:=$(shell grep ^\#.*TOKE_VERSION < toke/toke.c |cut -f2 -d\" ) + all: make -C toke make -C detok @@ -51,6 +53,10 @@ cp romheaders/romheaders testsuite make -C testsuite all CygTestLogs=`pwd`/testlogs +# lcov required for html reports +coverage: + @testsuite/GenCoverage . fcode-suite-$(VERSION) "FCode suite $(VERSION)" + @testsuite/GenCoverage toke toke-$(VERSION) "Toke $(VERSION)" .PHONY: all clean distclean toke detok romheaders tests Added: fcode-utils/testsuite/GenCoverage =================================================================== --- fcode-utils/testsuite/GenCoverage (rev 0) +++ fcode-utils/testsuite/GenCoverage 2006-10-30 22:09:59 UTC (rev 112) @@ -0,0 +1,24 @@ +#!/bin/bash + +SRCDIR=$1 +DSTDIR=$2 +BASENAME=$DSTDIR +TITLE="$3" + +printf "Generating info file..." +geninfo -t $BASENAME -o $BASENAME.info $SRCDIR 2>/dev/null +printf "ok\n" + +printf "Filtering info file..." +sed -i "/stat\.h/ { +N +N +N +N +d +}" $BASENAME.info +printf "ok\n" + +genhtml --output-directory $DSTDIR --show-details --highlight --legend --frames --title "$TITLE" $BASENAME.info + + Property changes on: fcode-utils/testsuite/GenCoverage ___________________________________________________________________ Name: svn:executable + *

1 0

r109 - in fcode-utils: . documentation
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 17:58:53 +0100 (Mon, 30 Oct 2006) New Revision: 109 Added: fcode-utils/documentation/ fcode-utils/documentation/detok.html fcode-utils/documentation/localvalues.html fcode-utils/documentation/toke.html fcode-utils/documentation/workbook.css Modified: fcode-utils/README Log: add documentation Modified: fcode-utils/README =================================================================== --- fcode-utils/README 2006-10-30 16:29:59 UTC (rev 108) +++ fcode-utils/README 2006-10-30 16:58:53 UTC (rev 109) @@ -8,8 +8,9 @@ - localvalues NOTE: all parts of the OpenBIOS FCODE suite are GPL v2, except the localvalues -implementation which is released under the Common Public License (CPL) version -1.0 (see: http://www.opensource.org/licenses/cpl1.0.php) +implementation and parts of the documentation which are released under the +Common Public License (CPL) version 1.0 +(see: http://www.opensource.org/licenses/cpl1.0.php) For information please contact info(a)coresystems.de Added: fcode-utils/documentation/detok.html =================================================================== --- fcode-utils/documentation/detok.html (rev 0) +++ fcode-utils/documentation/detok.html 2006-10-30 16:58:53 UTC (rev 109) @@ -0,0 +1,343 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> +<head> + + <title>ESW PART-FW: OpenBIOS DeTokenizer detok (A User's Guide)</title> + <link rel="stylesheet" type="text/css" + href="workbook.css"> +</head> +<body> +<h1 style="text-align: center;" class="title"><a class="mozTocH1" + name="mozTocId94624"></a>OpenBIOS DeTokenizer detok</h1> +<h1 style="text-align: center;"><a class="mozTocH1" + name="mozTocId186871"></a>(A +User's Guide)</h1> +<div class="document-control"> +<ul> + <li>Updated Wed, 18 Oct 2006 at 12:37 PDT by David L. Paktor</li> +</ul> +</div> +Copyright © 2005 International Business +Machines®, +All Rights Reserved. +Licensed under the <a + href="http://www.opensource.org/licenses/cpl1.0.php">Common Public +License (CPL) version 1.0</a> + + +<h1><a class="mozTocH2" name="mozTocId48196"></a>Table of Contents</h1> +<ol id="mozToc"> + <li><a + href="#mozTocId734288">Overview</a></li> + <li><a href="#mozTocId101390">Output Formats</a> + <ol> + <li><a href="#mozTocId555273">Sample source file:</a></li> + <li><a href="#mozTocId805568">DeTokenizer output with no +options selected:</a></li> + <li><a href="#mozTocId621166">DeTokenizer output with the +"verbose" option selected:</a></li> + <li><a href="#mozTocId409633">DeTokenizer output with the +"offsets" option selected:</a></li> + <li><a href="#mozTocId460016">DeTokenizer output with both +the "verbose" and "offsets" options selected:</a></li> + </ol> + </li> + <li><a href="#mozTocId930428">Command-Line Format</a> + <ol> + <li><a href="#mozTocId582301">Command-Line Options</a> + <ol> + <li><a href="#mozTocId403694">Switches</a></li> + <li><a href="#mozTocId532000">The "Additional FCodes" +file</a></li> + <ol> + <li><a href="#mozTocId814200">Special Functions + </a></li> + </ol> + </ol> + </li> + </ol> + </li> + <li><a href="#mozTocId657867">End Of Document</a></li> +</ol> +<h2><a class="mozTocH2" name="mozTocId734288"></a>Overview</h2> +The DeTokenizer is an adjunct tool to the Tokenizer, and performs the +reverse function, in a fashion.  That is, it converts binary FCode +into a form that can be read for purposes of verification.  This +implementation is not, however, a complete DeTokenizer in the sense of +one whose output can be turned back through the Tokenizer to generate +the same binary.  Such programs might exist, but this is not one +of them. +<h2><a class="mozTocH2" name="mozTocId101390"></a>Output Formats</h2> +The output of this DeTokenizer basically consists of one token +(sometimes two) per line, with optional additional information, +depending on which Command-Line Options have been specified. + +<h3 style="margin-left: 40px;"><a class="mozTocH3" name="mozTocId555273"></a>Sample +source file: +</h3> +<div style="margin-left: 40px;">The easiest way to describe the +different optional output formats would +be by creating an example of a source file that has been Tokenized and +displaying the output of the DeTokenizer, applied to its resultant +FCode binary, with the various options. + +Our example source file looks like this: + +</div> +<pre style="font-family: courier new; margin-left: 40px;">\ Demo program for DeTokenizer output format display. tokenizer[ h# 17d5 \ Vendor ID: 0x17d5 h# 5417 \ Device ID: 0x5417 h# 020000 \ Class Code: 0x020000 (Ethernet) h# f2ed \ Rev-Level ]tokenizer SET-REV-LEVEL pci-header fcode-version2 headers hex : hello/goodbye ( hello? -- ) if ." Hello, you big beautiful world!" else ." Goodbye, cruel world. Gggga-a-a-ackkkkk!" then ; : I-say-hello true hello/goodbye ; : You-say-goodbye false hello/goodbye ; fcode-end</pre> +<h3 style="margin-left: 40px;"><a class="mozTocH3" name="mozTocId805568"> +</a>DeTokenizer output with no options selected:</h3> +<div style="margin-left: 40px;"> +With no options selected, the DeTokenizer output looks like this: + +</div> +<pre + style="font-family: courier new; margin-left: 40px; color: rgb(204, 0, 0);">\ PCI Header identified \ Offset to Data Structure = 0x001c (28) \ PCI Data Structure identified \ Data Structure Length = 0x0018 (24) \ Vendor ID: 0x17d5 \ Device ID: 0x5417 \ Class Code: 0x020000 (Ethernet controller) \ Image Revision: 0xf2ed \ Code Type: 0x01 (Open Firmware) \ Image Length: 0x0001 blocks (512 bytes) \ Last PCI Image. start1 ( 16-bit offsets) format: 0x08 checksum: 0x33d5 (Ok) len: 0x009e ( 158 bytes) named-token hello/goodbye 0x800 b(:) b?branch 0x0028 ( =dec 40) b(") ( len=0x1f [31 bytes] ) " Hello, you big beautiful world!" type bbranch 0x0030 ( =dec 48) b(>resolve) b(") ( len=0x29 [41 bytes] ) " Goodbye, cruel world. Gggga-a-a-ackkkkk!" type b(>resolve) b(;) named-token I-say-hello 0x801 b(:) -1 hello/goodbye b(;) named-token You-say-goodbye 0x802 b(:) 0 hello/goodbye b(;) end0 \ Detokenization finished normally after 158 bytes. \ PCI Image padded with 302 bytes of zero</pre> +<h3 style="margin-left: 40px;"><a class="mozTocH3" name="mozTocId621166"></a><a + name="with_verbose"></a>DeTokenizer output +with the "verbose" option selected:</h3> +<div style="margin-left: 40px;">The "verbose" option adds a display of +the hex value of each token +processed, (as well as a signature block), thus: + +</div> +<pre + style="font-family: courier new; margin-left: 40px; color: rgb(204, 0, 0);">\ Welcome to the OpenBIOS detokenizer v0.6.1 \ detok Copyright(c) 2001-2005 by Stefan Reinauer. \ Written by Stefan Reinauer, <stepan(a)openbios.org> \ This program is free software; you may redistribute it under the terms of \ the GNU General Public License. This program has absolutely no warranty. \ \ (C) Copyright 2005 IBM Corporation. All Rights Reserved. \ PCI Header identified \ Offset to Data Structure = 0x001c (28) \ PCI Data Structure identified \ Data Structure Length = 0x0018 (24) \ Vendor ID: 0x17d5 \ Device ID: 0x5417 \ Class Code: 0x020000 (Ethernet controller) \ Image Revision: 0xf2ed \ Code Type: 0x01 (Open Firmware) \ Image Length: 0x0001 blocks (512 bytes) \ Last PCI Image. start1 ( 0x0f1 ) ( 16-bit offsets) format: 0x08 checksum: 0x33d5 (Ok) len: 0x009e ( 158 bytes) named-token ( 0x0b6 ) hello/goodbye 0x800 b(:) ( 0x0b7 ) b?branch ( 0x014 ) 0x0028 ( =dec 40) b(") ( 0x012 ) ( len=0x1f [31 bytes] ) " Hello, you big beautiful world!" type ( 0x090 ) bbranch ( 0x013 ) 0x0030 ( =dec 48) b(>resolve) ( 0x0b2 ) b(") ( 0x012 ) ( len=0x29 [41 bytes] ) " Goodbye, cruel world. Gggga-a-a-ackkkkk!" type ( 0x090 ) b(>resolve) ( 0x0b2 ) b(;) ( 0x0c2 ) named-token ( 0x0b6 ) I-say-hello 0x801 b(:) ( 0x0b7 ) -1 ( 0x0a4 ) hello/goodbye ( 0x800 ) b(;) ( 0x0c2 ) named-token ( 0x0b6 ) You-say-goodbye 0x802 b(:) ( 0x0b7 ) 0 ( 0x0a5 ) hello/goodbye ( 0x800 ) b(;) ( 0x0c2 ) end0 ( 0x000 ) \ Detokenization finished normally after 158 bytes. \ PCI Image padded with 302 bytes of zero</pre> +<h3 style="margin-left: 40px;"><a class="mozTocH3" name="mozTocId409633"></a><a + name="with_offsets"></a>DeTokenizer output +with the "offsets" option selected:</h3> +<div style="margin-left: 40px;">The "offsets" option shows the position +of the tokens relative to the +start of the first FCode block after a PCI header (if one is present) +and the destination-offset of each +branch. If more than one FCode header follows a single PCI header, the +offset-counter will continue; if a new PCI header is encountered, the +offset-counter will be reset and will begin counting again from zero +after the end of the latest PCI header. +Without the "verbose" option, i.e., with just the +"offsets" option by itself, the DeTokenizer output looks like this: +</div> +<pre + style="font-family: courier new; margin-left: 40px; color: rgb(204, 0, 0);">\ PCI Header identified \ Offset to Data Structure = 0x001c (28) \ PCI Data Structure identified \ Data Structure Length = 0x0018 (24) \ Vendor ID: 0x17d5 \ Device ID: 0x5417 \ Class Code: 0x020000 (Ethernet controller) \ Image Revision: 0xf2ed \ Code Type: 0x01 (Open Firmware) \ Image Length: 0x0001 blocks (512 bytes) \ Last PCI Image. 0: start1 ( 16-bit offsets) 1: format: 0x08 2: checksum: 0x33d5 (Ok) 4: len: 0x009e ( 158 bytes) 8: named-token hello/goodbye 0x800 25: b(:) 26: b?branch 0x0028 ( =dec 40 dest = 67 ) 29: b(") ( len=0x1f [31 bytes] ) " Hello, you big beautiful world!" 62: type 63: bbranch 0x0030 ( =dec 48 dest = 112 ) 66: b(>resolve) 67: b(") ( len=0x29 [41 bytes] ) " Goodbye, cruel world. Gggga-a-a-ackkkkk!" 110: type 111: b(>resolve) 112: b(;) 113: named-token I-say-hello 0x801 128: b(:) 129: -1 130: hello/goodbye 132: b(;) 133: named-token You-say-goodbye 0x802 152: b(:) 153: 0 154: hello/goodbye 156: b(;) 157: end0 \ Detokenization finished normally after 158 bytes. \ PCI Image padded with 302 bytes of zero</pre> +<h3 style="margin-left: 40px;"><a class="mozTocH3" name="mozTocId460016"></a><a + name="with_verbose_and_offsets"></a>DeTokenizer output +with both the "verbose" and "offsets" options selected:</h3> +<div style="margin-left: 40px;">Combining the "verbose" and "offsets" +options results in something that +looks like this: + +</div> +<pre + style="font-family: courier new; margin-left: 40px; color: rgb(204, 0, 0);">\ Welcome to the OpenBIOS detokenizer v0.6.1 \ detok Copyright(c) 2001-2005 by Stefan Reinauer. \ Written by Stefan Reinauer, <stepan(a)openbios.org> \ This program is free software; you may redistribute it under the terms of \ the GNU General Public License. This program has absolutely no warranty. \ \ (C) Copyright 2005 IBM Corporation. All Rights Reserved. \ PCI Header identified \ Offset to Data Structure = 0x001c (28) \ PCI Data Structure identified \ Data Structure Length = 0x0018 (24) \ Vendor ID: 0x17d5 \ Device ID: 0x5417 \ Class Code: 0x020000 (Ethernet controller) \ Image Revision: 0xf2ed \ Code Type: 0x01 (Open Firmware) \ Image Length: 0x0001 blocks (512 bytes) \ Last PCI Image. 0: start1 ( 0x0f1 ) ( 16-bit offsets) 1: format: 0x08 2: checksum: 0x33d5 (Ok) 4: len: 0x009e ( 158 bytes) 8: named-token ( 0x0b6 ) hello/goodbye 0x800 25: b(:) ( 0x0b7 ) 26: b?branch ( 0x014 ) 0x0028 ( =dec 40 dest = 67 ) 29: b(") ( 0x012 ) ( len=0x1f [31 bytes] ) " Hello, you big beautiful world!" 62: type ( 0x090 ) 63: bbranch ( 0x013 ) 0x0030 ( =dec 48 dest = 112 ) 66: b(>resolve) ( 0x0b2 ) 67: b(") ( 0x012 ) ( len=0x29 [41 bytes] ) " Goodbye, cruel world. Gggga-a-a-ackkkkk!" 110: type ( 0x090 ) 111: b(>resolve) ( 0x0b2 ) 112: b(;) ( 0x0c2 ) 113: named-token ( 0x0b6 ) I-say-hello 0x801 128: b(:) ( 0x0b7 ) 129: -1 ( 0x0a4 ) 130: hello/goodbye ( 0x800 ) 132: b(;) ( 0x0c2 ) 133: named-token ( 0x0b6 ) You-say-goodbye 0x802 152: b(:) ( 0x0b7 ) 153: 0 ( 0x0a5 ) 154: hello/goodbye ( 0x800 ) 156: b(;) ( 0x0c2 ) 157: end0 ( 0x000 ) \ Detokenization finished normally after 158 bytes. \ PCI Image padded with 302 bytes of zero</pre> +<div style="margin-left: 80px;">There's another option called "line +numbers" but it only numbers the +lines of output.  It's easy enough to describe, and so needs no +illustration. +</div> + + + +<h2><a class="mozTocH2" name="mozTocId930428"></a>Command-Line Format</h2> +<div style="margin-left: 40px;">The command-line format is simply: + +</div> +<div + style="font-family: courier new,courier,monospace; margin-left: 80px;">detok +[options] fc-file [fc-file ...] +</div> + +<div style="margin-left: 40px;">The output of this DeTokenizer is +directed to STDOUT, so there is no +"Output file" option per se.  Simply redirect the output to the +file in which you wish to keep the results, using the standard Shell +conventions. +</div> +<h3 style="margin-left: 40px;"><a class="mozTocH3" name="mozTocId582301"></a>Command-Line +Options</h3> +<div style="margin-left: 40px;">Command-Line option Switches are +case-sensitive; +only one option has an applicable argument, and that one is a file +name.  Its case sensitivity is, of course, dependent on the Host +Operating System. +</div> +<h4 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId403694"></a>Switches</h4> +<ul style="margin-left: 40px;"> + <li>-h  +or  -?</li> +</ul> +<div style="margin-left: 80px;">Print a brief help message and then +exit. +</div> +<ul style="margin-left: 40px;"> + <li>-v</li> +</ul> +<div style="margin-left: 80px;">Verbose  --  display +<a href="#with_verbose">additional information</a>:  the hex value +of each token processed, as +well as a signature block. +</div> +<ul + style="font-weight: bold; font-family: courier new; margin-left: 40px;"> + <li>-o</li> +</ul> +<div style="margin-left: 80px;">Offsets  --  display the +<a href="#with_offsets">positions of the tokens</a> relative to the +start of the first FCode block after a PCI header (if one is present), +and +the destination-offset of each branch. +Note that the <a href="#with_verbose_and_offsets">combination</a> +of +the Verbose and Offsets options yields the maximum amount of useful +information.  +</div> +<ul style="margin-left: 40px;"> + <li>-n</li> +</ul> +<div style="margin-left: 80px;">Line Numbers  --  display the +sequential number of each line of output. +Note that the -n and -o options +are mutually exclusive; if both are specified, -o will be +favoured. +</div> +<ul style="margin-left: 40px;"> + <li>-a</li> +</ul> +<div style="margin-left: 80px;">Process All input.  Do not stop +when end0 +has been encountered.  This option is usually not needed, but may +be useful in cases where a file has been corrupted or when +something very strange has been Tokenized... +</div> +<ul style="margin-left: 40px;"> + <li>-f +<FCode-List-File></li> +</ul> +<div style="margin-left: 80px;">Pre-load Additional FCodes before +processing.  These might be, for instance, a set of +vendor-specific FCodes that were generated for a specific vendor's +products by a Tokenizer customized for that specific vendor.  A +detailed discussion of <a href="#Additional_FCodes_file">the +"Additional FCodes" file</a> will be presented in a separate dedicated +section. </div> + +<h4 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId532000"></a><a + name="Additional_FCodes_file"></a>The +"Additional FCodes" file</h4> +<div style="margin-left: 80px;">Some vendors' FCode drivers contain +non-standard FCode tokens.  In order to accommodate those +situations, provision is made to specify the names of the FCodes in +question.  The -f +command-line option permits the user to specify an "Additional FCodes +List" file, which will be read before detokenization begins and which +will contain the list of "Additional FCodes" to be recognized. +The format of the file is as follows: + +<ol> + <li>One entry, consisting of an FCode and its name, on a line.  +The FCode Number is given first, in the form of a hex number, preceded +by an optional 0x +or 0X  +(Thus: 0x602 +or 0X602 +or simply  602 +are all equivalent.)  At least one blank space separates the FCode +Number from the Name, which must be on the same line.  Any number +of blanks are permitted, and any text that follows the Name is +permitted and will be ignored.</li> + <li>Blank lines are permitted and will be ignored.</li> + <li>Comment lines are permitted and will be ignored.  +A comment-line starts with either a pound-sign ( # +) or a backslash ( \  +).</li> + <li>FCode Numbers are limited to the range 0x10..0x7ff  +Numbers smaller than 0x10 +are the leading-byte of a two-byte FCode, and numbers from 0x800 +and up are assigned by the tokenizer.  Lines with numbers outside +the permitted range will be ignored, and a message will be printed.</li> + <li>FCode numbers that are already assigned will not be permitted to +be overwritten.   Lines with numbers that are already assigned +will be ignored, and a message will be printed.  </li> +</ol> +If the file cannot be read, that will be regarded as an immediate +failure and cause the program to exit. + +<h5><big><a name="mozTocId814200"></a><a name="Special_Functions"></a>Special +Functions</big></h5> +In addition to non-standard FCode tokens with simple behavior, some +vendors' FCode drivers also contain non-standard FCode tokens with +complex behavior.  An example that was recently encountered is "double(lit)" which precedes a +double-length (i.e., 64-bit) literal.  This DeTokenizer is +structured to allow the creation of a list of pre-defined Special +Function names, each of which has a special behavior associated with +it.  When one of those names occurs in the "Additional FCodes +List" file, it will be recognized; the FCode Number given with it is +assigned to it.  When that FCode number is encountered, the +assigned special behavior will be exercised. +Adding to the list of Special Function names, and associating a +new behavior with the added function, requires modifying the +DeTokenizer code, but the infrastructure that is already in place +should make this a manageable task for even a modestly skilled +programmer. + +At the present writing, only one such Special Function name is +supported, and that one is, of course,  double(lit) +Its associated special behavior is to collect the next eight bytes +from the FCode input stream and display them as a double-length literal. + +If you modify the DeTokenizer to recognize additional Special +Function names, please update this document to list them and describe +their special behaviors.  Thank you.  + +</div> + +<h2><a class="mozTocH2" name="mozTocId657867"></a>End Of Document</h2> + +</body> +</html> Added: fcode-utils/documentation/localvalues.html =================================================================== --- fcode-utils/documentation/localvalues.html (rev 0) +++ fcode-utils/documentation/localvalues.html 2006-10-30 16:58:53 UTC (rev 109) @@ -0,0 +1,859 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> +<head> + + <title>ESW PART-FW: Support for "Local Values" in the OpenBIOS +Tokenizer</title> + <link rel="stylesheet" type="text/css" + href="workbook.css"> +</head> +<body> +<h1 class="title"><a class="mozTocH1" name="mozTocId810036"></a>Support +for "Local Values" in the OpenBIOS Tokenizer</h1> +<div class="document-control"> +<ul> + <li>Updated Wed, 26 Jul 2006 at 10:04 PDT by David L. Paktor + </li> +</ul> +</div> + Copyright © 2005 International Business +Machines®, All Rights Reserved. +Licensed under the <a + href="http://www.opensource.org/licenses/cpl1.0.php">Common Public +License (CPL) version 1.0</a> + + +<h2><a class="mozTocH2" name="mozTocId207724"></a>Table Of Contents</h2> +<ol id="mozToc"> +<li><a href="#mozTocId810036">Support +for "Local Values" in the OpenBIOS Tokenizer</a> + <ol> + <li><a href="#mozTocId944022">Overview</a></li> + <li><a href="#mozTocId303607">Scope of this Document</a></li> + <li><a href="#mozTocId272611">Design Objectives</a></li> + <li><a href="#mozTocId383861">Functional +Requirements</a></li> + <li><a href="#mozTocId969736">Interface +Specification</a> + <ol> + <li><a href="#mozTocId699042">Syntax:</a> + <ol> + <li><a href="#mozTocId94063">Declaration: +(Curly-braces:   {   and +  }   )</a></li> + <li><a href="#mozTocId444210">Separation +Character between Initialized and +Uninitialized Local Values</a></li> + <li><a href="#mozTocId715222">Value Extraction: +Invocation</a></li> + <li><a href="#mozTocId677769">Value Assignment: The +  ->   +Operator.</a></li> + <li><a href="#mozTocId189696">Scope of Local Values:</a></li> + </ol> + </li> + <li><a href="#mozTocId107943">Semantics:</a> + <ol> + <li><a href="#mozTocId459457">Order of +Initialization:</a></li> + <li><a href="#mozTocId908599">Note two significant +departures from the "Local Variables" +discussed in the ANSI FORTH Standard document: </a></li> + <li><a href="#mozTocId213536">Value Extraction</a></li> + <li><a href="#mozTocId563286">Value Assignment</a></li> + </ol> + </li> + </ol> + </li> + <li><a href="#mozTocId505598">Implementation</a> + <ol> + <li><a href="#mozTocId134995">Compilation -- the +Parser:</a> + <ol> + <li><a href="#mozTocId467215">Run-Time Support +Function names:</a> + <ol> + <li><a href="#mozTocId497158">Example:</a></li> + </ol> + </li> + </ol> + </li> + <li><a href="#mozTocId361894">Run-Time Support -- +the Local Values Support "Library":</a> + <ol> + <li><a href="#mozTocId191315">Operational Run-Time +Support +Functions:</a></li> + <li><a href="#mozTocId750894">Important note regarding +floading of the Local Values Support Functions file</a></li> + <li><a href="#mozTocId684194">Adjusting the size of +Local Values Storage:</a></li> + <li><a href="#mozTocId208497">Additional Support +Function(s):</a> + <ol> + <li><a href="#mozTocId781114">CATCH / THROW</a></li> + </ol> + </li> + </ol> + </li> + </ol> + </li> + </ol> + </li> +</ol> +<h2><a class="mozTocH2" name="mozTocId944022"></a>Overview</h2> + +The goal of this project is to implement support for "Local Values" +in the OpenBIOS Tokenizer. +"Local Values" are an IBM-specific extension of FORTH syntax, +currently used both by the FCode Tokenizer and Platform Firmware. +They might be considered a variant that meets the spirit, if not the +letter, of the suggestions for a "Locals word set" discussed +-- but not specified -- in the ANSI FORTH Standard, Section 13 and +Appendix A.13 + +(Please note that the ANSI document does not really specify this +feature, +because the Committee could not reach an agreement. Appendix A.13 +records the somewhat lively discussions that accompanied this topic.) +We will refer to this feature with the nomenclature "Local Values" in preference to +"Local +Variables" or "Locals" in order to (a) more accurately +characterize the behavior of these objects, and (b) further emphasize +the differences between the IBM-specific extension and those discussed +in the ANSI document. + +<h2><a class="mozTocH2" name="mozTocId303607"></a>Scope of this Document</h2> + +The sections labeled <a style="font-weight: bold;" href="#Syntax">Syntax</a> +and <a style="font-weight: bold;" href="#Semantics">Semantics</a> +describe the user's view of this feature. +The section +labeled <a style="font-weight: bold;" href="#Implementation">Implementation</a> +is a description of the +underlying parsing and support mechanisms that meet the <a + style="font-weight: bold;" href="#Design_Objectives">Design Objectives</a>. + + +<h2><a class="mozTocH2" name="mozTocId272611"></a><a + name="Design_Objectives"></a>Design Objectives</h2> +The +"Local Values" extension is intended to relieve programmers maintaining +FCode device drivers from the complexity involved in keeping track of +the +positions of the various items on the stack. The programmer can, +instead, +refer to these items by symbolic names, in a manner similar to "<big>C</big>" +syntax. + +<h2><a class="mozTocH2" name="mozTocId383861"></a>Functional +Requirements</h2> + +Tokenizing source-code that makes use of the "Local Values" syntax +shall still result in Industry Standard FCode that can be interpreted +by the +FCode interpreter of any Open Firmware-compliant Host Platform, without +imposing any IBM-specific requirements. +Also, the implementation shall support a means whereby to remain +compatible with IBM's existing code-base. + +<h2><a class="mozTocH2" name="mozTocId969736"></a>Interface +Specification</h2> + +<h3><a class="mozTocH3" name="mozTocId699042"></a><a name="Syntax"></a>Syntax:</h3> +<h4><a class="mozTocH4" name="mozTocId94063"></a>Declaration: +(Curly-braces:   <big>{</big>   and +  <big>}</big>   )</h4> + +Local Values may only be declared in connection with a colon-definition +(A "word" in FORTH parlance.) + Declaration of Local Values is triggered by an open-curly-brace +(i.e.,:   {   ) , and ends with a close-curly-brace ( +  }   +). +A further distinction is made between Initialized Local Values and +Uninitialized Local Values:  Initialized Local Values are declared +first, and +are separated by a <a href="#Separation_Character_between_Initialized">special +character</a> from Uninitialized Local Values. + +Declaration of Local Values may only occur once within the body of +the colon-definition. + +Declaration of Local Values after code has been compiled into the +body of the word is not recommended, but is permitted.  A Local +Values +Declaration that occurs inside a Flow-Control Structure will be +reported as an <a href="TokenizerNewFeatures.htm#An_ERROR">Error</a>. + +A Local Values Declaration may include comments and may continue +across +multiple lines.  See the <a href="#mozTocId497158">example</a> in +the <a href="#Implementation">Implementation</a> section. +<h4><a class="mozTocH4" name="mozTocId444210"></a><a + name="Separation_Character_between_Initialized"></a>Separation +Character between Initialized and +Uninitialized Local Values</h4> +Two symbols are accepted as the separator between +Initialized and Uninitialized Local Values, the Semicolon +( ; ) and the Vertical-Bar (   |   ). + +Since, in FORTH, Semicolon is heavily fraught with a very important +meaning, +it is preferable to use a different symbol -- one that isn't used +for anything else -- as the separator between Initialized and +Uninitialized Local Values. Better still would be a symbol that's given +at least passing mention in the discussion about the (failed) attempt +to establish an ANSI standard for Locals (see the ANSI Forth Spec., +section 13.6.2.1795). + +The Vertical-Bar +symbol +(   |   ) +fills that bill nicely. +<a name="Legacy_Separator"></a>Local Values Declarations +will accept Semicolon as an alternative ("Legacy") separator between +Initialized +and Uninitialized Local Values, and issue a Warning message to the +effect +that the +use of Semicolons in that context is deprecated in favor of the +Vertical-Bar. +The User may suppress this +message by means of a <a + href="TokenizerNewFeatures.htm#Command-Line_options_">Command-line +switch</a>, known as the +<a href="TokenizerNewFeatures.htm#Special_Feature_Flags">Special-Feature +Flag</a> named <a href="TokenizerNewFeatures.htm#SF_Flag_LV_Leg_Mssg">NoLV-Legacy-Message</a> +, which is described in the <a href="TokenizerNewFeatures.htm">Tokenizer +User's Guide</a>. +<a name="Disallow_Legacy"></a>Conversely, the User who wishes to +disallow the use of Semicolon as +an alternative separator may do so by means of the +Special-Feature Flag named <a + href="TokenizerNewFeatures.htm#SF_Flag_LV_Leg_Sep">NoLV-Legacy-Separator</a> +.  When the Legacy Local Values +Separator is thus disallowed, occurrences will be treated as an +Error. +<h4><a class="mozTocH4" name="mozTocId715222"></a>Value Extraction: +Invocation</h4> +A name declared as a Local Value may be invoked within the body of a +word in connection with which it was declared, simply by name, in a +manner similar to a name defined by the Standard FORTH defining-word, VALUE +, to mean that its associated value is to be extracted and placed onto +the Stack. +<h4><a class="mozTocH4" name="mozTocId677769"></a>Value Assignment: The +  <big>-></big>   +Operator.</h4> +The symbol   ->   (dash angle-bracket, pronounced +"dash-arrow"), may precede the name of a Local Value, and it may not +precede anything else. +No comments are permitted between the   ->   +and the Local-Value name to which it applies. +The   ->   and +the Local-Value name to which it applies must be on the same line. +The   ->   operator relates to the Local-Value +name to which it is applied in a manner similar to the way the  TO  +operator relates, when it is applied, to a name defined by  VALUE  +; it causes the numeric value on +top of the Parameter Stack to be popped and stored into -- associated +with -- the named Local Value. +<h4><a class="mozTocH4" name="mozTocId189696"></a>Scope of Local Values:</h4> +Upon completion of the definition of the word in connection with which +a set of Local Values was declared, i.e., at the semicolon, the names +of the Local Values cease to be recognized. If the same names are +declared in connection with a subsequent definition, they are only +applicable to that subsequent definition, as if they were newly +created. No warning is issued, nor do rules concerning "overloading" +apply. +<h3><a class="mozTocH3" name="mozTocId107943"></a><a name="Semantics"></a>Semantics:</h3> +<h4><a class="mozTocH4" name="mozTocId459457"></a>Order of +Initialization:</h4> +Uninitialized Local Values do not have a value until one is assigned +within the definition, by the use of the   ->  +("dash-arrow") operator. +Initialized Local Values are initialized from the stack at the start +of execution of the defined word, in the same order as the convention +for a stack-diagram, i.e., the first-named Local Value is +initialized from the stack-item whose depth corresponds to the total +number of initialized Local Values, the last-named Local Value is +initialized from the top-of-stack item, and so on in between. + +The following will serve to illustrate: + +<code> +: <word-name> ( P_x ... P_y   P_0 P_1 ... P_n-2 P_n-1 -- +??? ) +                    +                    +        { IL_0 IL_1 +... IL_n-2 IL_n-1 | UL_0 UL_1 } +              \     +  At the start of the word, +IL_0 through IL_n-1 are initialized +              \       +with P_0 thorough P_n-1, respectively, and the stack contains +                    +        +                    +                    +                    +          (   P_x ... P_y   ) + +</code> +<h4><a class="mozTocH4" name="mozTocId908599"></a>Note two significant +departures from the "Locals" +discussed in the ANSI FORTH Standard document: +</h4> +(1) The ANSI FORTH Committee discussions make no provision for +Uninitialized Locals, + +and + + (2) The order of initialization is reversed. In the ANSI document, +Locals are initialized +in the order they are declared, so that the first-declared will take +the topmost value on the stack, and the last-declared will take the +deepest value. + + The general consensus within IBM is that this scheme is confusing +at best, and does not serve the intent of the <a + href="#Design_Objectives">Design Objectives</a>. + +<h4><a class="mozTocH4" name="mozTocId213536"></a>Value Extraction</h4> +When the name of a Local Value is invoked, its associated value is +extracted +and pushed onto the Parameter Stack. +<h4><a class="mozTocH4" name="mozTocId563286"></a>Value Assignment</h4> +If the   ->   ("dash-arrow") symbol precedes the +name of a Local Value, then the numeric value on +top of the Parameter Stack is popped and stored into -- associated with +-- the named Local-Value. +Following the   ->   ("dash-arrow") symbol +with anything other than the name of a Local Value is an Error. + +<h2><a class="mozTocH2" name="mozTocId505598"></a><a + name="Implementation"></a>Implementation</h2> + + +<h3><a class="mozTocH3" name="mozTocId134995"></a>Compilation -- the +Parser:</h3> + +A separate area (a "vocabulary", in Forth parlance) must be reserved +where temporary compile-time +definitions of the new Local Value names can be created, and whence the +new Local Value names can be removed after the definition of the word +in connection with which they were declared is completed. Variables +must also be set aside to keep count of the Initialized and +Uninitialized Local Value names declared inside the curly-braces. + Each new Local Value name has an integer assigned to it. The Parser +assigns successive integers, starting with 0, to the Local Value names, +in the order that they are declared, and enters the name of each new +Local Value, together with its assigned integer, into the separate +reserved temporary area. + + After all the Local Value names have been declared, i.e., after the +close-curly-brace has been read, the Parser compiles-in the number of +Initialized Local Values, followed by the number of Uninitialized Local +Values, where they will act as arguments to the <a href="#push-locals">appropriate +function</a>, +which the Parser compiles-in immediately after. The function will be +the special one that allocates space for, and initializes, the Local +Values at the time they are about to be used. + + While the definition under construction is being compiled, the area +where the temporary compile-time definitions of the new Local Value +names have been created must be available to the scanning process, so +that the new names will be recognized when invoked. Also, it should be +scanned first, ahead of any other word-lists, so that the Local Value +names will supercede any similarly-named words, in case of a +naming-overlap. + + When a Local Value's name is invoked, the Parser compiles-in its +assigned integer as an argument to the <a href="#_local">appropriate +function</a>, which is +compiled-in immediately after. The function will be a common one that +will push onto the stack the address at which the numbered Local Value +can be accessed. The Parser will then compile-in either the "fetch" +function (   @   ) or the "store" function (   ! +  ), depending on whether the Local Value name was invoked by +itself or in conjunction with the   ->   operator. +This way the User/Programmer's view of Local Values' VALUE-style +behavior is preserved. +The FORTH functions exit and ; (semicolon) have to +be overloaded. (Section 13.3.3 of the ANSI document also mentions +  ;CODE   and   DOES>   but +these are not recognized by the Tokenizer, so we will not discuss them +here.) The overloaded definitions must take special action at +compile-time +(note that   ;   -- semicolon -- does that normally, +anyway, but exit does not) to: compile-in the total number of +Local Values as an argument to the <a href="#pop-locals">appropriate +function</a>, which is +compiled-in immediately afterwards, before completing their normal +behavior. The function in this case will be the special one that +releases the space that had been allocated for the Local Values, and +restores the state of Local Values storage to the way the calling +routine left it. Semicolon must also clear the area where the temporary +compile-time definitions of the new local-names were created, +rendering them inaccessible. +<h4><a class="mozTocH4" name="mozTocId467215"></a>Run-Time Support +Function names:</h4> +The names of the three functions that the Parser compiles-in must be +well-known and documented, so that they can be implemented and exported +correctly in the +Run-Time Support "Library" file. +The three functions' names are: +           {push-locals} ( #ilocals +#ulocals -- ) +           {pop-locals} +( total#locals +-- ) +           _{local} +( local-var# -- +addr ) + +<h4><a class="mozTocH4" name="mozTocId497158"></a>Example:</h4> +The following is an example of a word definition that declares and +makes +use of Local Values; it also shows inclusion of comments and +continuation +of the Declaration +across multiple lines: + +<tt> + +</tt> +<table style="text-align: left; width: 821px; height: 356px;" border="0" cellpadding="0" cellspacing="0"> + + + <tbody> + + + <tr> + + + <td style="text-align: left; vertical-align: top; width: 33%; white-space: nowrap;">: + faber ( m4 m3 n2 n1 n0 -- m4 m3 ) + + +    { + + +       \ These are initialized values: + + +      _otter + + +      _weasel + + +      _skunk + + + + + +      | + + + + + +      \ These are uninitialized: + + +      _muskrat + + +       _mole + + +    } + + +    _skunk 40 *        -> _muskrat + + +   _muskrat alloc-mem -> _mole + + +   base @ + + +   hex      _weasel (.) +_mole place + + +   decimal   _otter (.) _mole $cat + + +   base ! + + +   _mole count type + + +   _mole _muskrat free-mem + + + ;<tt></tt></td> + + + <td style="text-align: left; vertical-align: top; white-space: nowrap; background-color: rgb(255, 255, 255);">\ Does nothing useful. Just an example. + + +\ BEGIN the declaration of Local Values. + + + + + +\ _otter is initialized with +the value of n2 + + +\ _weasel is initialized with +the value of n1 + + +\ _skunk is +initialized +with the value of n0 + + +\ and will be used to +determine an amount of memory +to allocate. + + +\ Vertical bar ends the group of Initialized Local Values. + + +\ NOTE: m4 and m3 stay on the stack. + + + + + +\ _muskrat will take the +final size of the allocation. + + + \ _mole will hold the address of the allocated memory + + +\ END the declaration of Local Values. + + + <tt></tt></td> + + + </tr> + + + + + </tbody> +</table> + + +</tt> + The compilation of +faber +starts with   3 2 +{push-locals}   . The first invocation of _skunk +(by itself) compiles as   2 _{local} @   and the +sequence   +-> +_muskrat +  compiles as   3 _{local) !   +Finally,  faber +ends with   5 {pop-locals}   before the unnest +. After +that, the local-names are no longer accessible. +<h3><a class="mozTocH3" name="mozTocId361894"></a>Run-Time Support -- +the Local Values Support "Library":</h3> + +An FCode program that makes use of the Local Values extension will +need to incorporate an implementation of the three compiled-in +functions named above, together with a collection of functions that +support them. These support functions must be defined in a way such +that they can be tokenized into Industry Standard FCode, conformant to +the Open Firmware specification, without imposing any non-Standard +requirements on the FCode interpreter of the Host Platform. +The obvious way to deliver this package of support functions would +be to incorporate, into +the FCode source being Tokenized, a Prologue or "Library" file that +contains the definitions of the three above-named compiled-in +functions, along with all their required support. + +A file defining the Local Values Support Functions has been written +and +will be delivered as part of the implementation of this +Project.  The user/programmer will be responsible for floading +it into the FCode source program to be Tokenized. + +The user/programmer has the option of specifying the +placement of the Local Values Support Functions file within the body of +the +FCode source program, and even of making alterations to it, if needed. + +<a name="Error_handling"></a>Error handling: +If the Local Values Support Functions file is not floaded, then the +Parser, when it completes the processing of a Local Values declaration, +i.e, when it encounters the close-curly-brace, or, similarly, when it +encounters an invocation of a Local Value's name, will proceed as +normal to compile-in the call to the appropriate function. That +function's name will not be recognized, and the Tokenizer will exhibit +the normal error-behavior for an invocation of an unrecognized name. + +<h4><a class="mozTocH4" name="mozTocId191315"></a>Operational Run-Time +Support +Functions:</h4> +To make Local Values operate, we will, of course, need to reserve an +area of Backing Storage. The size of that area will be <a + href="#Adjusting_size_of_L_V_Storage">adjustable by +the programmer</a>, and we have chosen a suitable default. +We define a +  locals-base   pointer that will point to the base +-- within the reserved Local Values Storage Area -- of the set of Local +Values currently in use; it will be initialized to point just past the +end of the locals-storage area. + +<a name="_local"></a> The address to which the   <n> +  _{local} +  routine will point is calculated as the given number of cells +above the   locals-base  pointer. +<a name="push-locals"></a> The   ( #I-Ls #U-Ls -- )  +{push-locals} +  routine works in two stages: for the Unitialized Local Values, +it +simply decrements the   locals-base   pointer by the +number of cells given in the top argument. The Initialized Local Values +are +then handled one at a time: the   locals-base   +pointer is decremented by a single cell, and the data-item on top of +the parameter stack is popped and stored into the cell at which the +  locals-base   pointer now points. The result is +that the topmost stack-item is placed in the last-declared Initialized +Local, and so on down the line until the lowest stack-item is placed in +the first-declared Initialized Local Value. Neat, sweet, and petite. +<a name="pop-locals"></a> The   ( #-Ls -- )   {pop-locals} +  +routine simply increments the   locals-base   +pointer by the given number of cells, which is the total number of +Local Values used by the function in which it +occurs. + Because functions that use Local Values can call each other, (i.e., +the use of Local Values can be nested), the depth of the nesting might +be +unpredictable. Therefore, the   {push-locals}   +routine must perform error-checking: Before decrementing the  locals-base  +pointer, it must test whether doing so would put the pointer +below the start of the area reserved for Local Values Storage. Such an +error is inevitably fatal, and can only be handled by an   ABORT +  occurring in conjunction with a warning message advising the +programmer to increase the size of the Local Values Storage (and, by +implication, re-Tokenize). + + It will be the developer's responsibility to catch all such errors +during early testing. To prevent generating hidden errors of this sort, +the programmer is advised to use Local Values judiciously, and +particularly to avoid using them in functions that may be called +re-entrantly or recursively to an uncontrolled depth. Fortunately, such +routines are rare and easily identified. + +Additional help can be provided in the form of a second floadable +Local Values Support Function source file -- to be used during +development only -- that would overload the +  {push-locals}   +and +  {pop-locals}   routines with the additional action +of keeping track of  -- and, of course, displaying at will +--  +the maximum depth used in the course of a test run. Such overloading of +functions is very simple and straightforward in FORTH. + +<h4><a class="mozTocH4" name="mozTocId750894"></a><a + name="Instance_Data_in_L_V_Support"></a>Important note +regarding floading +of the Local Values Support Functions file</h4> +In order to simplify management of the allocation and de-allocation of +the area of Backing Storage, and to assure independence among instances +of a device-package, both the reserved Local Values Storage Area and +the  locals-base pointer are created as part +of the device-node's instance +data. + +The <a href="TokenizerNewFeatures.htm#INSTANCE_at_run_time">consequence</a> +of this is that, in device-drivers that are configured +with <a href="TokenizerNewFeatures.htm#Multiple_device_nodes">multiple +device-nodes</a>, the Local Values Support Functions file +must be re-floaded for +each device-node that uses Local Values.  That is to say, every +invocation of the new-device +command creates a new device node; if that new device-node will be +making use of  Local Values, then the Local Values Support +Functions file must be floaded +again. +The Tokenizer is sophisticated enough to keep a <a + href="TokenizerNewFeatures.htm#device-node_vocabularies">separate +vocabulary +for each device-node</a>, and will flag an <a href="#Error_handling">Error</a> +if Local Values are used in a device-node for which the Local Values +Support Functions file has not been floaded. +However, should the user so choose, a <a + href="TokenizerNewFeatures.htm#Example_4">means</a> is available +whereby a single floading +of the Local Values +Support Functions can become <a + href="TokenizerNewFeatures.htm#Global_Definitions">accessible to all +Device Nodes</a> in a driver, trading off economy of System-memory +for convenience of programming. +<h4><a class="mozTocH4" name="mozTocId684194"></a><a + name="Adjusting_size_of_L_V_Storage"></a>Adjusting the size of +Local Values Storage:</h4> +The user is responsible for declaring the maximum depth of the run-time +Local Values stack, in storage units (Cells). + +This may be accomplished either by: +<ul> + <li>defining a constant named   _local-storage-size_  +before floading the +Local Values Support Functions file</li> + <li>defining   _local-storage-size_  as a +Command-Line User-Symbol when the Tokenizer is run.</li> +</ul> +<div style="margin-left: 40px;">The form of the Command-Line +User-Symbol definition resembles: +<div style="text-align: center;">   -d +'_local-storage-size_=d# 42' +</div> + (Be sure to enclose it within quotes +so that the Shell treats it as a single string, and, of course, replace +the  42  +with the actual number you need...) +</div> + +If   _local-storage-size_  is defined +both ways, the Command-Line User-Symbol will prevail. + +If the   _local-storage-size_  definition +is omitted, the Local Values Support Functions file will supply a +default. +<h4><a class="mozTocH4" name="mozTocId208497"></a>Additional Support +Function(s):</h4> +<h5><a class="mozTocH5" name="mozTocId781114"></a>CATCH / THROW</h5> +Another way that a function might exit prematurely is via a call to throw + . +An FCode program that utilizes Local Values, that calls throw +, +and that has a corresponding catch to guard it, will +need to +keep its Local Values properly synchronized. + +A throw done by an FCode program that does not have a +corresponding catch to guard it will be caught outside +the scope of that FCode program, and the question of synchronizing +Local Values will be rendered irrelevant. + +An <a href="TokenizerNewFeatures.htm#The_OVERLOAD_Directive">overloaded</a> +catch in the Local Values Support +Functions file does the job. + +Constructing it was quite simple: It needs to (a) save +the   locals-base   pointer onto the return stack, +(b) do a system (generic) CATCH, and (c) restore the   locals-base +  pointer. Counterintuitive though this might be, it does not even +need to examine the result of the system (generic) CATCH ; it can +restore the   locals-base   pointer in either case. +If the result was zero (i.e., no throw occurred), the +Local Values Pointer will be the same as it was when saved and +restoring it +will be harmless... + + + +<h2><a class="mozTocH2" name="mozTocId951082"></a>End Of Document</h2> + + + + +</body> +</html> Added: fcode-utils/documentation/toke.html =================================================================== --- fcode-utils/documentation/toke.html (rev 0) +++ fcode-utils/documentation/toke.html 2006-10-30 16:58:53 UTC (rev 109) @@ -0,0 +1,3968 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> +<html> +<head> + + <title>ESW PART-FW: New Features in the OpenBIOS Tokenizer toke (A +User's Guide)</title> + <link rel="stylesheet" type="text/css" + href="workbook.css"> +</head> +<body> +<h1 style="text-align: center;" class="title"><a class="mozTocH1" + name="mozTocId449746"></a>New Features in OpenBIOS Tokenizer toke +</h1> +<h1 style="text-align: center;"><a class="mozTocH1" + name="mozTocId459488"></a>(A +User's Guide) +</h1> +<div class="document-control"> +<ul> + <li>Updated Wed, 18 Oct 2006 at 12:39 PDT by David L. Paktor </li> +</ul> +</div> +Copyright © 2005 International Business +Machines®, +All Rights Reserved. +Licensed under the <a + href="http://www.opensource.org/licenses/cpl1.0.php">Common Public +License (CPL) version 1.0</a> + + +<h2><a class="mozTocH2" name="mozTocId700910"></a><a name="TOC"></a>Table +Of Contents</h2> +<ol id="mozToc"> +<a href="#mozTocId459488">New +Features in OpenBIOS Tokenizer toke (A User's Guide)</a> + <li><a href="#mozTocId601312">Overview</a></li> + <li><a href="#mozTocId119716">Scope of this Document</a> + <ol> + <li><a href="#mozTocId206214">What this document +does not cover: </a></li> + </ol> + </li> + <li><a href="#mozTocId343012">Error Detection +and other messages:</a></li> + <li><a href="#mozTocId271949">Case Sensitivity</a> + </li> + <li><a href="#mozTocId280593">Categories of Features</a> + <ol> + <li><a href="#mozTocId460640">Directives</a></li> + <li><a href="#mozTocId930032">Command-Line options</a></li> + <li><a href="#mozTocId379181">Non-standard input, +syntaxes and behavior</a></li> + <li><a href="#mozTocId303703">“Tokenizer-Escape” Mode</a></li> + </ol> + </li> + <li><a href="#mozTocId959733">Features, by Category:</a> + <ol> + <li><a href="#mozTocId499618">Command-Line options</a> + <ol> + <li><a href="#mozTocId510152">Switches</a></li> + <li><a href="#mozTocId514350">Include-List Directories</a></li> + <li><a href="#mozTocId515750">Trace the Creation and +Invocation of FCode or +"Tokenizer Escape"-mode Definitions</a> + </li> + <li><a href="#mozTocId580387">Command-Line Symbol Definitions</a></li> + <li><a href="#mozTocId632060">Special-Feature Flags</a></li> + </ol> + </li> + <li><a href="#mozTocId445349">“Tokenizer-Escape” Mode</a></li> + <ol> + <li><a href="#mozTocId445715">The  emit-byte  +command</a></li> + <li><a href="#mozTocId446081">Other commands</a></li> + <ol> + <li><a href="#mozTocId446447">Standard Commands</a></li> + <ol> + <li><a href="#mozTocId446813">The  next-fcode  +command</a></li> + </ol> + <li><a href="#mozTocId447179">Non-Standard operations</a></li> + <ol> + <li><a href="#mozTocId447545">constant</a></li> + <li><a href="#mozTocId447911">emit-fcode</a></li> + </ol> + <li><a href="#mozTocId448277">Additional FORTH-compatible +operations</a> + </li> + </ol> + </ol> + <li><a href="#mozTocId649525">Directives</a> + <ol> + <li><a href="#mozTocId557512">Conditional Tokenization</a> + <ol> + <li><a href="#mozTocId686605">The Conditional-block +Terminator</a></li> + <li><a href="#mozTocId763728">The +"False" segment-switcher</a></li> + <li><a href="#mozTocId375310">Condition-Testers</a></li> + <ol> + <li><a href="#mozTocId104560">True/False flag +on the top of the stack</a></li> + <li><a href="#mozTocId956359">Existence or +non-existence of an +FCode or "Tokenizer Escape"-mode definition</a></li> + <li><a href="#mozTocId880808">Definition or +non-definition of a +Command-Line Symbol</a></li> + </ol> + <li><a href="#mozTocId891224">Tracing the state of +Conditional Tokenization</a> + </li> + </ol> + </li> + <li><a href="#mozTocId202222">Suspending the Duplicate-Name +Test +for one Definition ("Overloading")</a></li> + <li><a href="#mozTocId202620">Suspending the Multi-Line +Warning for one occasion</a> + </li> + <li><a href="#mozTocId203416">Controlling the Scope of +Definitions</a> + </li> + <li><a href="#mozTocId314185">Evaluating +Command-line Symbols </a></li> + <li><a href="#mozTocId655967">Outputting +Arbitrary Byte-Sequences to +the FCode Binary</a></li> + <li><a href="#mozTocId114013">Encoding +blocks of binary data taken from a file</a></li> + <li><a href="#mozTocId69109">Generating Special Text-Strings +and Literals</a> + </li> + <ol> + <li><a href="#mozTocId70133">Current date or time</a></li> + <li><a href="#mozTocId71157">Name of the function +currently being defined</a></li> + <li><a href="#mozTocId72181">Input-File Name and Line Number</a></li> + </ol> + <li><a href="#mozTocId844437">Pre-pending +PCI Headers to FCode +images </a></li> + <li><a href="#mozTocId931658">Modifying the PCI +Header</a> + <ol> + <li><a href="#mozTocId504353">"Revision +Level of the Vendor's ROM"</a></li> + <ol> + <li><a href="#mozTocId503878">Byte-Order</a></li> + </ol> + <li><a href="#mozTocId53081">"Last +Image Indicator" bit</a></li> + </ol> + </li> + <li><a href="#mozTocId410469">Changing the name of the +Binary Output File</a></li> + <li><a href="#mozTocId982642">Issuing messages at +Tokenization +time.</a></li> + <li><a href="#mozTocId550942">Changing Special-Feature +Flags +from Source</a> + <ol> + <li><a href="#mozTocId551846">Displaying +Special-Feature Flags from Source</a></li> + </ol> + </li> + <li><a href="#mozTocId573280">Manipulating the +FCode-Token-Number Assignment Counter</a></li> + <ol> + <li><a href="#mozTocId566072">Saving +and Restoring</a></li> + <li><a href="#mozTocId568474">Resetting</a> + </li> + </ol> + <li><a href="#mozTocId575686">Resetting +Symbols Defined in Either Mode</a></li> + </ol> + </li> + <li><a href="#mozTocId957252">Non-standard input, +syntaxes, and behavior </a> + <ol> + <li><a href="#mozTocId850914">Ambiguous +Conditions</a> + <ol> + <li><a href="#mozTocId162252">Tokens +expected on the same +line</a></li> + <li><a href="#mozTocId246601">The  h# + d# and + o#  +directives </a></li> + <li><a href="#mozTocId978546">The +commands  leave  + ?leave  +or  unloop  +outside of a loop-control framework</a></li> + <li><a href="#mozTocId73206">The +commands  [']  +or  '  +followed by a name that is not a valid target</a></li> + <li><a href="#mozTocId74410">The Forth word to +followed by a name that is not a valid target </a> + </li> + <li><a href="#mozTocId75614">The word instance + </a></li> + <ol> + <li><a href="#mozTocId76818">Followed by an +inapplicable defining-word</a></li> + <li><a href="#mozTocId78022">Left unresolved</a></li> + <li><a href="#mozTocId79226">Not Allowed in...</a></li> + </ol> + <li><a href="#mozTocId884280">The command fload +inside a colon-definition + </a></li> + </ol> + </li> + <li><a href="#mozTocId901662">Non-Standard +Synonyms for Standard +Functions</a></li> + <li><a href="#mozTocId732819">Non-Standard +Functions or behaviors</a> + <ol> + <li><a href="#mozTocId522750">Alias</a></li> + <li><a href="#mozTocId824541">String-Escape +characters and other +String-Gathering Features </a></li> + <ol> + <li><a href="#mozTocId945290">Quoted +String-Escape Characters</a></li> + <li><a href="#mozTocId758170">Embedded New-Lines</a></li> + <li><a href="#mozTocId42456">Hex-Sequence +Processing </a></li> + <li><a href="#mozTocId150131">C-Style +String-Escape Characters</a></li> + </ol> + <li><a href="#mozTocId40998">The ABORT" +(Abort-Quote) command</a></li> + <li><a href="#mozTocId526780">Conveniently +Convert Short +Character-Sequence to Number</a></li> + <ol> + <li><a href="#mozTocId53626">Left-Justified</a> + </li> + </ol> + <li><a href="#mozTocId54171">F[']   +("Eff-Bracket-Tick-Bracket")</a></li> + <ol> + <li><a href="#mozTocId54716">F['] in "Normal +Tokenization" Mode</a></li> + <li><a href="#mozTocId55261">F['] in "Tokenizer-Escape" +Mode</a> + </li> + </ol> + <li><a href="#mozTocId887752">Shell-Environment Variables +in File-Names</a> + </li> + </ol> + </li> + <li><a href="#mozTocId505746">Featured +behaviors, supported by this +Tokenizer, that are not mentioned in the Standard</a> + <ol> + <li><a href="#mozTocId530910">User-Defined Macros</a> + </li> + <li><a href="#mozTocId539069">Multiple +device-nodes</a></li> + <ol> + <li><a href="#mozTocId540605">Global Definitions</a> + </li> + </ol> + <li><a href="#mozTocId354426">Multiple +FCode-block Images</a></li> + <li><a href="#mozTocId232809">Multiple +PCI-Image</a></li> + </ol> + </li> + </ol> + </li> + </ol> + </li> + <li><a href="#mozTocId194505">Examples:</a> + <ol> + <li><a href="#mozTocId544211">Example #1:</a></li> + <li><a href="#mozTocId80432">Example #2: </a></li> + <li><a href="#mozTocId455242">Example #3:</a></li> + <li><a href="#mozTocId861304">Example #4:</a></li> + <li><a href="#mozTocId866180">Example #5:</a> + </li> + </ol> + </li> +</ol> +<h2><a class="mozTocH2" name="mozTocId601312"></a>Overview</h2> + +The goal of this project is to produce an FCode Tokenizer that can +both +be used in-house and be presented to third-party vendors, VARs and the +like, as a professional-quality tool for their use, without adversely +affecting IBM's +Intellectual-Property rights. +We are using, as a starting basis, the Open-Source tokenizer  +that is available from the <a href="http://openbios.org/">OpenBIOS +project</a> .   +We expect to be able to deliver this tool to our vendors by +returning our modifications to the OpenBIOS project, from whence it can +be obtained openly by anyone. +<h2><a class="mozTocH2" name="mozTocId119716"></a>Scope of this Document</h2> + +In order to achieve the first part of the goal -- making it suitable +for in-house use -- a number of features were added that are not +covered in the IEEE-1275 Standard for Boot Firmware (also referred to +herein simply as "the Standard").  +This document describes those <a href="#Categories_of_Features">features</a> +and how they are used. + +There will be a brief overview of <a + href="#Error_Detection_and_other_messages:">Error Detection</a> and +other +messages. +There will be some <a href="#Examples:">examples</a> at the end. + +<h3><a class="mozTocH3" name="mozTocId206214"></a>What this document +does not cover: +</h3> +<ul> + <li>Standard behavior, particularly conversion between ANSI / +IEEE-1275 Standard Forth and IEEE-1275 Standard FCode, and standard +Tokenizer Macros.  These are +expected and will be regarded as "presumed" by this document.</li> + <li>Changes to older versions of the OpenBIOS Tokenizer.</li> + <li>A complete list of Tokenization Source Errors detected and +reported.</li> + <li>A complete list of conditions that generate WARNING or ADVISORY +messages.</li> + <li>A complete list of FATAL conditions. + </li> + <li>Support for <a href="TokenizerLocalValues.htm">IBM-Style Local +Values</a>. That is the subject of a +separate writeup.</li> +</ul> + + +<h2><a class="mozTocH2" name="mozTocId343012"></a><a + name="Error_Detection_and_other_messages:"></a>Error Detection +and other messages:</h2> +The Tokenizer is capable of producing the following kinds of messages: +<ul> + <li>FATAL</li> + <li>ERROR</li> + <li>WARNING</li> + <li>ADVISORY</li> + <li>MESSAGE generated by the User.</li> + <li>TRACE-NOTE + </li> +</ul> +A FATAL +condition is sufficient cause to immediately stop activity.  It is +usually (but not always) a symptom of a system failure, rather than a +result of User +input. +<a name="An_ERROR"></a>An ERROR +occurs as a result of User input.  It is a condition sufficient to +make the run a failure, but not to stop activity.  Unless the -i +<a href="#Ignore_Errors_option">("Ignore Errors")</a> <a + href="#Command-Line_options_">Command-Line +option</a> has been specified, the production of a Binary +Output file +will be suppressed and theTokenizer will exit with a non-zero status, +if an Error Message has been issued. + +<a name="A_WARNING"></a>A WARNING +is issued for a condition that, while not necessarily an error, might +be something to avoid.  E.g.,  a deprecated feature, or a +feature that might be incompatible with other Standard tokenizers. +<a name="Advisories"></a>An ADVISORY +message is issued for a condition that is a response to User input, and +where processing continues unchanged, but it is nonetheless (in +this author's opinion) worthwhile to give the User a "heads-up" to make +sure that what you got is what you wanted. ADVISORY +messages are only displayed when the <a + style="font-style: italic; font-weight: bold;" href="#verbose_option">verbose</a> +<a href="#Command-Line_options_">Command-Line option</a> is selected. + +A User-generated MESSAGE +-- unsurprisingly -- is a message generated by the User via any of the +<a href="#User_Messages">directives</a> supported for that purpose. + +<a name="Trace_Notes"></a>A TRACE-NOTE is +issued if the <a href="#Trace_Symbols">Trace-Symbols</a> +feature is activated, whenever a symbol on the Trace-List is either +created or invoked. + +<a name="Message_Format"></a>Each message of the above types is +accompanied by the name of the +source file and the line number in which the condition that triggered +it was detected, as well as the current position to which data is being +generated into the Binary Output.  If a PCI Header is in effect, +the position relative to the end of that PCI Header will also be shown; +this is to maintain consistency with the <a + href="DeTokenizer.htm#with_offsets">"offsets</a>" displayed be the <a + href="DeTokenizer.htm">DeTokenizer</a> + +<a name="Errors_Ignored"></a> +The Tokenizer typically runs through to completion of the source file, +displaying an ERROR message for each error it encounters (I.e., it does +not "bail" after the first error).  If the -i +<a href="#Ignore_Errors_option">("Ignore +Errors")</a> <a href="#Command-Line_options_">Command-Line +option</a> has been specified, the Tokenizer will attempt to +produce binary output for each error, as far as is feasible, and will +produce +a Binary Output file.  While this practice is not recommended, the +author acknowledges that it +might be useful in some limited circumstances. +At the end of its run, the Tokenizer will print a tally of the number +of each type of message that was generated. +<h2><a name="mozTocId271949"></a><a name="Case_Sensitivity"></a>Case +Sensitivity</h2> +Although Command-Line option +<a href="#Command_Line_Switches">switches</a> +are case-sensitive, this Tokenizer is insensitive to the distinctions +between upper- or +lower- -case characters in all other matters: in Command-Line <a + href="#Symbol_Definitions">symbol-definitions</a> and <a + href="#Special_Feature_Flags">Special-Feature Flag</a> settings and in +the Input Source. +Character-case is preserved in string sequences and in the +assignment +of names of headered definitions, but is ignored for purposes of +name-matching.  Case sensitivity of filenames, of course, is +dependent on the Host Operating System. + +This Tokenizer supports a pair of Special-Feature Flags that will +enable the User to <a href="#Token_Name_Cases">over-ride the +preservation of character-case</a> in the +assignment +of names of headered definitions. +<h2><a class="mozTocH2" name="mozTocId280593"></a><a + name="Categories_of_Features"></a>Categories of <a + href="#Features_by_Category">Features</a></h2> + +The features described in this document fall into four categories: +<ol> + <li><a href="#Directives">Directives</a></li> + <li><a href="#Command-Line_options">Command-Line options</a></li> + <li><a href="#Non-standard_input_syntaxes">Non-standard</a> +input, syntaxes, and behavior</li> + <li>Options available within “<a href="#Tokenizer-Escape_Mode">tokenizer-escape</a>” +mode (I.e., within +the scope of a   tokenizer[  +... ]tokenizer  block).</li> +</ol> +<h3><a class="mozTocH3" name="mozTocId460640"></a><a name="Directives"></a>Directives</h3> +<a href="#Directives_">Directives</a> are commands that occur within +the body of the Tokenization +Source File, but are not part of the FCode program itself.  They +are recognized by the Tokenizer and cause it to perform special actions +that might serve such diverse functions as: +<ul> + <li>Printing a message to the Standard Output during the Tokenization +process</li> + <li>Controlling whether selected passages of  the Source are +processed or ignored (also called Conditional Compilation or +Conditional Tokenization)</li> + <li>Outputting arbitrary sequences of bytes to the FCode binary file</li> +</ul> +<h3><a class="mozTocH3" name="mozTocId930032"></a><a + name="Command-Line_options"></a>Command-Line options</h3> +<a href="#Command-Line_options_">Command-Line options</a> are, as the +name suggests, switches that can be +set on the command-line when the Tokenizer is invoked.  They +affect behaviors such as: +<ul> + <li>The verbosity of output during the Tokenization +process</li> + <li>Definition of symbols that feed into some of the Conditional +Tokenization Directives + </li> + <li>Allowing output to be produced even when errors have been +detected.</li> + <li>Selectively +enabling variants of Non-standard input, syntaxes, and behavior + </li> +</ul> +<h3><a class="mozTocH3" name="mozTocId379181"></a><a + name="Non-standard_input_syntaxes"></a><a + href="#Non-standard_input_syntaxes_">Non-standard input</a>, +syntaxes and behavior</h3> +Certain sequences of source code that are not covered by the IEEE-1275 +Standard are, nonetheless, recognized by this Tokenizer and cause it to +produce valid FCode.  Examples of these include: +<ul> + <li>"C"-style escape characters in strings</li> + <li>Sun-style versus Apple-style implementation of the command  ABORT"  + + </li> + <li>IBM-style Local-Values</li> +</ul> +In order to +accommodate the conflicting goals of standard-compliance and +convenience,  non-Standard variants will be selectively enabled by +a set of Command-Line options known as "<a href="#Special_Feature_Flags">Special-Feature +Flags</a>" +<h3><a class="mozTocH3" name="mozTocId303703"></a><a + name="Tokenizer-Escape_Mode"></a>“Tokenizer-Escape” Mode</h3> +Section C.3.1 of +the IEEE-1275 Standard describes “<a href="#Tokenizer-Escape_Mode_">tokenizer-escape</a>” +mode and its basic +command, <a href="#emit_byte">emit-byte</a>  +The OpenBIOS Tokenizer toke, while it does not have a full FORTH +capability, does support a few additional Forth-style commands that can +be used to: +<ul> + <li>Define and retrieve named constants</li> + <li>Create aliases</li> + <li>Set the value of the next FCode-token + </li> +</ul> + + +<h2><a class="mozTocH2" name="mozTocId959733"></a><a + name="Features_by_Category"></a>Features, by Category:</h2> +The following sections will supply a complete list of the new features, +organized by category. + +<h3><a class="mozTocH3" name="mozTocId499618"></a><a + name="Command-Line_options_"></a>Command-Line options:</h3> +Command-Line option Switches are <a href="#Case_Sensitivity">case-sensitive</a>; +their arguments, where applicable, are not. + +<h4><a class="mozTocH4" name="mozTocId510152"></a><a + name="Command_Line_Switches"></a>Switches</h4> +<ul> + <li>-h  +or  -?</li> +</ul> +<div style="margin-left: 40px;">Print a brief help message and then +exit. +</div> +<ul> + <li><a name="verbose_option"></a>-v</li> +</ul> +Verbose  --  print additional +messages (including <a href="#Advisories">Advisories</a>) during +tokenization. + +<ul> + <li + style="font-weight: bold; font-family: courier new,courier,monospace;"><a + name="Ignore_Errors_option"></a>-i</li> +</ul> +Ignore Errors.  Generate a Binary +Output  <a href="#Errors_Ignored">even if +errors were reported</a>. + +<ul> + <li><a name="Binary_Output_Name_option"></a>-o +<OutputFileName></li> +</ul> +Direct the Binary Output (FCode result of +Tokenization) to the named +file instead of to the <a href="#Binary_Output_File_Name_Default">default-named +file</a>.  This option is not valid when multiple input files are +named. + +<ul> + <li><a name="FLoad_List_option"></a>-l</li> +</ul> +FLoad List -- Collect the names of floaded +files into an +FLoad-List File.  The names collected are in the same form as they +were presented in the fload +statements. + +The name of the FLoad-List File is +derived +from +the name of +the Binary Output File, by replacing its extension with  .fl +, or, if the Binary Output File name had no extension, merely appending +the +extension  .fl + +The Binary Output File name used for this +purpose is either the one specified <a + href="#Binary_Output_Name_option">on the Command +Line</a>,  or the one created by <a + href="#Binary_Output_File_Name_Default">default</a>. + +<ul> + <li><a name="Depncy_List_option"></a>-P     +(Note:  Switch is upper-case) + </li> +</ul> +Dependency List -- Collect the +fully-resolved pathnames +of floaded +and <a href="#ENCODE_FILE">ENCODE</a>d files into a Dependency-List +File.  The names collected are in the form that is presented to +the Host Operating System:  <a + href="#Shell_Env_Vbles_in_File_Names">Shell Environment Variables</a> +and +related expressions will be fully expanded, and the directory within +the <a href="#Include_List">Include-List</a> in which the file was +found will +be attached. + +The name of the Dependency-List File will +be the same as that of the FLoad-List File, except that its extension +will be .P +instead of .fl + +<ul> + <li>If an FLoad List  or a Dependency List is being collected, +the +names of any files that cannot be read, for any reason, will be +collected into +a Missing-Files-List file.</li> +</ul> +The name of the Missing-Files-List file +will be the same as that of the FLoad-List File except that its +extension will be .fl.missing +instead of .fl +The Missing-Files-List file will not be +created if all of the files are read successfully. + +If the name +of the Binary Output File is <a + href="#Binary_Output_File_Name_Directive">changed +by a directive</a> +embedded within the Tokenization Source File, that will not alter the +names of the FLoad-List, Dependency List or Missing-Files-List files. +<h4><a class="mozTocH4" name="mozTocId514350"></a><a name="Include_List"></a>Include-List +Directories</h4> +<ul> + <li>-I +directory     (Note:  Switch is +upper-case)</li> +</ul> +This Tokenizer supports the notion of an +Include-List.  The User creates the Include-List by specifying a +number of -I +directory pairs on the Command-Line.  All file-reads, +whether for an fload +command or an <a href="#ENCODE_FILE">encode-file</a> +directive, will involve a search for the named file through the +directories of the Include-List, in the order they were supplied on the +Command-Line. + +If no Include-List is created, file-reads +are relative to the Current Working +Directory.  If an Include-List is +created, file-reads are +restricted to the directories within it.  For the Current Working +Directory to be included in the file-search, it must be specified +explicitly.  -I<big>.</big>  +will accomplish that quite +effectively. + +<h4><a name="mozTocId515750"></a><a name="Trace_Symbols"></a>Trace the +Creation and Invocation of FCode or +"Tokenizer Escape"-mode Definitions</h4> +<ul> + <li>-T +<symbol> +      (Note:  Switch is upper-case)</li> +</ul> +This Tokenizer supports the notion of a +"Trace-List".  The User creates the Trace-List by specifying a +number of -T +<symbol> pairs on the Command-Line. + +When a name is defined, whether as an +FCode, an alias, a Macro or anything else, either in normal +tokenization mode or "Tokenizer Escape"-mode, if it matches a symbol +that has been added to the Trace List, a <a href="#Trace_Notes">Trace +Note</a> Message will be issued indicating that a definition of that +name has been created.   Subsequent <a href="#Trace_Notes">Trace +Note</a> Messages +will be issued when the definition of that name is invoked. +This "Trace-Symbols" +feature can be helpful during maintenance of Legacy +code, for instance, when multiple symbols carry the same name. + +<h4><a class="mozTocH4" name="mozTocId580387"></a><a + name="Symbol_Definitions"></a>Command-Line Symbol +Definitions</h4> +<ul> + <li style="font-family: courier new,courier,monospace;"><a + name="Define_Command-Line_symbol"></a>-d +Symbol[=Value]</li> +</ul> +Define a Command-Line Symbol.  +Optionally, assign a value to it.  If you wish the "value" to +contain spaces or quotes, you can accomplish that using the shell +escape conventions.  This sequence may be repeated.  Once a +Symbol is defined on the command-line, it stays in effect for the +duration of the entire batch of tokenizations (i.e., if there are +multiple input files named on the command line).  Command-Line +Symbols can be <a href="#Definition_or_non-definition_of_a">tested</a> +for purposes of Conditional Tokenization, or their assigned values can +be +<a href="#Evaluating_Command-line_Symbols">Evaluated</a>. + +<h4><a class="mozTocH4" name="mozTocId632060"></a><a + name="Special_Feature_Flags"></a>Special-Feature Flags +</h4> +<ul> + <li style="font-family: courier new,courier,monospace;">-f [no]<FlagName> </li> +</ul> +The Tokenizer +recognizes a specific set of Special-Feature Flag-names; each is +associated with a +specific non-Standard variant +behavior.  Pass the Flag-name as an argument to the -f +switch to enable the behavior; to disable it, precede the Flag-name +with the +optional string No + + +The settings of the Special-Feature Flags +can also be <a href="#Changing_C-L_Flags_from_Source">changed</a> or <a + href="#Displaying_C-L_Flags_from_Source">displayed</a> from +within the Source Input File + +The Special-Feature Flags are all +initially set to be enabled, except where noted. + +The Flag-names and their associated +Special-Features are as follows: +<ul> + <ul> + <li><a name="SF_Flag_Loc_Val"></a>Local-Values</li> + </ul> +</ul> +Support IBM-style <a + href="TokenizerLocalValues.htm">Local Values</a> ("LV"s).  +Initially disabled. + +<ul> + <ul> + <li><a name="SF_Flag_LV_Leg_Sep"></a>LV-Legacy-Separator</li> + </ul> +</ul> +Allow Semicolon for Local Values +<a + href="TokenizerLocalValues.htm#Separation_Character_between_Initialized">Separator</a> +("<a href="TokenizerLocalValues.htm#Disallow_Legacy">Legacy</a>"). + +<ul> + <ul> + <li><a name="SF_Flag_LV_Leg_Mssg"></a>LV-Legacy-Message</li> + </ul> +</ul> +Display a <a + href="TokenizerLocalValues.htm#Legacy_Separator">Warning +Message</a> when +Semicolon +is used as the Local Values Separator. + +<ul> + <ul> + <li><a name="SF_Flag_AB_Q"></a>ABORT-Quote</li> + </ul> +</ul> +Allow <a href="#SFeat_AB_Q">ABORT"</a> +macro. + +<ul> + <ul> + <li><a name="SF_Flag_Sun_AB_Q"></a>Sun-ABORT-Quote</li> + </ul> +</ul> +ABORT" with <a href="#SFeat_Sun_AB_Q">implicit +IF ... THEN</a> + +<ul style="font-family: courier new,courier,monospace;"> + <ul> + <li><a + name="SF_Flag_AB_Q_Throw"></a>Abort-Quote-Throw</li> + </ul> +</ul> +Use -2 +THROW , rather than <a href="#Abort_Quote_Abort">ABORT</a>, +in an <a href="#SFeat_AB_Q">Abort"</a> phrase +<ul style="font-family: courier new,courier,monospace;"> + <ul> + <li><a name="SF_Flag_Str_Rem_Esc"></a>String-remark-escape</li> + </ul> +</ul> +Allow <a href="#SFeat_Str_Rem_Esc">"\ +(Quote-Backslash)</a> to interrupt +string parsing. + +<ul> + <ul> + <li><a name="SF_Flag_Hex_Rem_Esc"></a>Hex-remark-escape</li> + </ul> +</ul> +Allow <a href="#SFeat_Hex_Rem_Esc">\ +(Backslash)</a> to interrupt +<a href="#Hex_Seq_Procg">hex-sequence parsing</a> within a string. + +<ul> + <ul> + <li><a name="SF_Flag_C_Str_Esc"></a>C-Style-string-escape</li> + </ul> +</ul> +Allow the <a + href="#C-Style_String-Escape_Characters"><big>C</big>-style +String-Escape</a> +pairs  \n +\t +and <a href="#SFeat_BS_xx">\xx\</a> +to be treated as special characters +in string parsing. + +<ul> + <ul> + <li>Always-Headers</li> + </ul> +</ul> +Over-ride occurrences of the +Standard directive headerless +in the Source with -- effectively -- headers +to make all definitions have a header.  Occurrences of the +directive external +will continue to behave in the Standard manner.  Initially +disabled. + +<ul> + <ul> + <li>Always-External</li> + </ul> +</ul> +All definitions will be made as +though under the external +directive; occurrences of either Standard directive headerless +or headers +in the Source will be over-ridden.  This Special-Feature Flag will +also over-ride the Always-Headers +Special-Feature Flag in the event that both have been specified.  +Initially disabled. + +<ul> + <ul> + <li><a + name="Warn_if_Duplicate_Flag"></a>Warn-if-Duplicate</li> + </ul> +</ul> +<div style="margin-left: 120px;">Display a <a href="#A_WARNING">WARNING</a> +message whenever +a +definition is made +whose name duplicates that of an existing definition.  Disabling +this flag will suspend the duplicate-names test globally, until it +is re-enabled.  A Directive is supported that will <a + href="#The_OVERLOAD_Directive">suspend the +test for the duration of only a single definition</a>, without +affecting global behavior. +</div> +<ul> + <ul> + <li><a + name="Obso_FCode_Warning"></a>Obsolete-FCode-Warning</li> + </ul> +</ul> +<div style="margin-left: 120px;">Display a <a href="#A_WARNING">WARNING</a> +message whenever an FCode function is invoked that the Standard +identifies as "obsolete". +</div> +<ul> + <ul> + <li><a + name="Trace_Condls"></a>Trace-Conditionals</li> + </ul> +</ul> +<div style="margin-left: 120px;">Issue <a href="#Advisories">Advisory +Messages</a> about the <a href="#Tracing_Conditional_State">state of</a> +<a href="#Conditional_Tokenization">Conditional Tokenization</a>.  +(Remember that <a href="#Advisories">Advisory +Messages</a> are displayed only if the "verbose" option <a + href="#verbose_option">-v</a> +is set.)  +Initially disabled. +</div> +<ul> + <ul> + <li><a + name="Token_Name_Case_Upper"></a><a name="Token_Name_Cases"></a>Upper-Case-Token-Names + </li> + <li><a + name="Token_Name_Case_Lower"></a>Lower-Case-Token-Names </li> + </ul> +</ul> +<div style="margin-left: 120px;">When outputting the names of headered +functions ("Token-Names") to the Binary Output File, over-ride the +character-case in which the names appeared in the Source, and convert +them to Upper- or Lower- -Case, respectively.  (These flags do not +affect text string sequences, whose character-case is always +preserved.)  +Initially disabled. +</div> +<ul> + <ul> + <li><a + name="Big_End_PCI_Rev_Level"></a>Big-End-PCI-Rev-Level</li> + </ul> +</ul> +<div style="margin-left: 120px;">Save the "<a href="#PCI_Rev_Level">Revision +Level of the +Vendor's ROM</a>" field of the PCI Header in <a + href="#PCI_Rev_Level_Byte_Order">Big-Endian byte-order</a>, rather +than  "Little-Endian" as per the general PCI Standard +convention. (This flag does not affect any other field of the PCI +Header).  +Initially disabled. +</div> +<ul> + <ul> + <li><a + name="Ret_Stk_Interp"></a>Ret-Stk-Interp + </li> + </ul> +</ul> +<div style="margin-left: 120px;">Allow Return-Stack Operations during +Interpretation.  While the Standard specifies that usage of the +operators  >r  +r@  +and  r> +while interpreting is allowed, actual practice in the industry is +inconsistent.  Developers who wish to take a more cautious +approach to this question can disable this flag so that any attempt to +use the operators  >r  +r@  +and  r> +in the "interpreting" state will generate an <a href="#An_ERROR">ERROR</a> +Message. +</div> + +Also, the pseudo-Flag-name  help  +will cause a list of the Flag-names and their associated +Special-Features to be printed. + +The use of some of these flags is illustrated in <a href="#Example_2">Example +#2</a> + +<h3><a class="mozTocH3" name="mozTocId445349"></a><a + name="Tokenizer-Escape_Mode_"></a>“Tokenizer-Escape” Mode</h3> +The directive  tokenizer[   +behaves as specified in Section C.3.1 of the IEEE-1275 Standard: it +saves the current tokenizer numeric conversion radix, sets the radix to +sixteen (hexadecimal) and enters “tokenizer-escape” mode.  +Likewise, the directive  ]tokenizer  +restores the radix and resumes the Tokenizer’s normal behavior. + +For convenience and compatibility with IBM's source-base, the +directives  f[  +and  f]   +are synonyms for  tokenizer[  +and ]tokenizer +respectively.  In addition, the variant  ]f   +is a synonym for f]  +. + +The numeric conversion radix can be changed during  +“Tokenizer-Escape” mode by the use of the standard directives  hex +decimal  and  +octal  .  These will always change the numeric +conversion radix in “tokenizer-escape” mode; even if  +“tokenizer-escape” mode was entered in the middle of a +colon-definition, they will not issue an FCode sequence.  And, as +per the Standard, the numeric conversion radix will be restored when +the Tokenizer returns to "Normal" mode. + +<a name="Tkz_Esc_Scope"></a>Function-names that were defined during +normal tokenization mode will +not be recognized in “Tokenizer-escape” mode and vice-versa.  The +one exception is that <a href="#F_B-T-B_TkzEsc">F[']</a> , +even when invoked in “Tokenizer-escape” mode, will apply to +function-names that were defined during normal tokenization mode. +<h4 style="margin-left: 40px;"><a name="mozTocId445715"></a>The  <a name="emit_byte"></a>emit-byte  +command +</h4> +<a name="Run-time_data_stack"></a>This +Tokenizer supports the  emit-byte  +command as specified in the Section cited above.  In order to be +able to do that, the “tokenizer-escape” mode needs to be able to +support a tokenization-time data stack, and, indeed, it does. + +<h4 style="margin-left: 40px;"><a name="mozTocId446081"></a>Other +commands</h4> +In +“tokenizer-escape” mode, a string representing a number (optionally +preceded by one of the directives h# +d# or +o#  ) causes that number to be pushed to the stack, where +it is available for use by any of several other commands as follows: + +<h5 style="font-weight: bold; margin-left: 80px;"><big><a + name="mozTocId446447"></a>Standard Commands</big></h5> +<ul style="margin-left: 120px;"> + <li><a name="mozTocId446813"></a>The <a + name="next_fcode"></a>next-fcode command </li> +</ul> +<div style="margin-left: 160px;">Set the value of the +FCode-token-number +assignment counter to the +number +that was on the stack.  Subsequent fcode-functions defined in the +source will be assigned FCode token numbers starting with this +number.  +If the number supplied on the stack is outside the legal range +specified by the IEEE-1275 Standard, the attempt will be disallowed and +reported as an <a href="#An_ERROR">Error</a>. +Note that this Tokenizer supports additional directives for <a + href="#Manipulate_FCode_Number">manipulating the FCode-token-number +assignment counter</a>. + +</div> +<h5 style="margin-left: 80px;"><big><a name="mozTocId447179"></a>Non-Standard +operations +</big></h5> +<ul style="margin-left: 120px;"> + <li><a name="Tkz_Esc_Const"></a><a + name="mozTocId447545"></a>constant +<Name> + </li> +</ul> +Define <Name> +as a named constant with the value that was on the stack.  <Name> +will be known within “tokenizer-escape” mode but will not be recognized +during normal tokenization.  When <Name> +is invoked, its value will be pushed onto the stack.  Two named +constants are pre-defined: true +and +false + +<ul + style="font-family: courier new,courier,monospace; margin-left: 120px;"> + <li><a name="emit_fcode"></a><a + name="mozTocId447911"></a>emit-fcode</li> +</ul> +Emit the value that was on the stack as +an FCode token, either one or two bytes long, depending on the +value.  Report an <a href="#An_ERROR">Error</a> if the number +supplied on the stack is +outside the legal range +specified by the IEEE-1275 Standard.  Since this is sort of a +"cheat",  issue an <a href="#Advisories">Advisory</a> if the +operation is successful. + +<h4 style="margin-left: 40px;"><a name="mozTocId448277"></a>Additional +FORTH-compatible operations</h4> + +In addition to the above, “Tokenizer-escape” mode recognizes a limited +number of FORTH-compatible named constants and operations, as follows: + +<ul style="margin-left: 40px;"> + <li>TRUE     +Push  -1 onto the stack </li> + <li>FALSE     +Push  0 onto the stack</li> + <li>0=     +Invert the Boolean-flag on top of the stack:  replace 0 with TRUE +or +any non-zero number with FALSE</li> + <li>SWAP     +Exchange top two stack items</li> + <li>2SWAP     +Exchange top two pairs of stack items</li> + <li>noop  +Take no action. </li> +</ul> +These are different from the +corresponding words in "Normal" mode, +which would +compile an FCode token. In “Tokenizer-escape” mode, they initiate an +immediate +action within the Tokenization process. + +<h3><a class="mozTocH3" name="mozTocId649525"></a><a name="Directives_"></a>Directives</h3> +This Tokenizer supports a number of directives that are not +specified +by the Standard, but which serve functions as follows: + +<ul> + <li><a href="#Conditional_Tokenization">Conditional Tokenization</a></li> + <li><a href="#The_OVERLOAD_Directive">Suspending</a> the +Duplicate-Name +Test for one Definition ("Overloading")</li> + <li>Suspending the <a href="#Allow_Multi_Line">Multi-Line Warning</a> +for one occasion + </li> + <li><a href="#mozTocId203416">Controlling the +Scope of +Definitions</a></li> + <li><a href="#Evaluating_Command-line_Symbols">Evaluation</a> of +symbols defined on the command-line</li> + <li>Outputting <a href="#FLITERAL">arbitrary sequences</a> of bytes +to the FCode binary file</li> + <li><a href="#ENCODE_FILE">Encoding</a> blocks of binary data taken +from a file</li> + <li>Generating <a href="#Gen_Specl_Txt_Strs">Special Text-Strings</a> +and Literals: + </li> + <ul> + <li>The <a href="#fcode_date">current +date or time</a>.</li> + <li>The <a href="#function_name">name +of the function</a> currently +being defined</li> + <li>The <a href="#Inp_Fil_Nm_Strg">Input-File Name</a> and Line +Number + </li> + </ul> + <li>Pre-pending <a href="#Pre_pending_PCI_Headers_to_FCode_images_">PCI +headers</a> to FCode +images</li> + <li><a href="#Modifying_PCI_Header">Modifying</a> the PCI Header + </li> + <li>Changing the name of the <a href="#Binary_Output_File_Name">Binary +Output File</a></li> + <li>Issuing a <a href="#User_Messages">message</a> at tokenization +time</li> + <li><a href="#Changing_C-L_Flags_from_Source">Over-riding</a> the +setting of a <a href="#Special_Feature_Flags">Special-Feature Flag</a></li> + <li><a href="#Displaying_C-L_Flags_from_Source">Displaying</a> the +settings of all the <a href="#Special_Feature_Flags">Special-Feature +Flags</a> </li> + <li><a href="#Manipulate_FCode_Number">Saving, Restoring and +Resetting the +FCode-Token-Number Assignment Counter</a> </li> + <li><a href="#Resetting_Symbols_Defined_in_Either_Mode">Resetting the +symbols</a> defined either in Normal Tokenization Mode or in "Tokenizer +Escape" mode. + </li> +</ul> +<h4><a class="mozTocH4" name="mozTocId557512"></a><a + name="Conditional_Tokenization"></a>Conditional Tokenization</h4> +The +OpenBIOS Tokenizer toke supports a fully-nestable Conditional +Tokenization capability comparable to the one in the "<big>C</big>" +pre-processor.  A +Conditional Tokenization block consists of either one or two segments +-- a "True" segment and an optional "False" segment -- delimited by <a + name="Conditional_Operators"></a>Conditional Operators of three types: +<ul> + <li>the <a href="#Condition_Testers">Condition-Tester</a></li> + <li>the optional <a href="#The_False_segment-switcher">"False" +segment switcher</a></li> + <li>the <a href="#The_Conditional-block_Terminator">Conditional-block +terminator</a></li> +</ul> +The "True" segment  immediately follows the Condition-Tester, +and +is ended either by the "False" segment switcher or by the +Conditional-block terminator.  If the "False" segment +switcher  is present, it introduces the "False" segment, which +is ended by the +Conditional-block terminator.  The three delimiters (in reverse +order) are: + +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId686605"></a><a + name="The_Conditional-block_Terminator"></a><big>The Conditional-block +Terminator</big></h5> +<div style="margin-left: 40px;">Exit the current level of Conditional +Tokenization. +If the +Conditional-block in question was nested within another +one, resume conditional processing at the +level of the enclosing segment.  When the outermost +Conditional-block is exited, resume normal processing. + +While it is a requirement that  the +Conditional-block terminator be contained within the same Input File as +the Condition-Tester and its optional "False" segment switcher, the +body of a Conditional Segment may contain separate fload +directives, which will be processed or ignored in accordance with the +prevailing Condition.  An <a href="#Example_4_Condl_FLoad">illustration</a> +can be seen in <a href="#Example_4">Example #4</a>. +There are several synonyms for the Conditional-block terminator: +</div> +<ul style="margin-left: 40px;"> + <big> </big><li>[then]</li> + <big> </big><li>#then</li> + <big> </big><li>[#then]</li> + <big> </big><li>[endif]</li> + <big> </big><li>#endif</li> + <big> </big><li>[#endif]</li> +</ul> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId763728"></a><a + name="The_False_segment-switcher"></a><big>The +"False" segment-switcher</big></h5> +Reverses the sense of the condition and +introduces the "False" +segment. +If the Condition-Test resulted in "TRUE", +then the first segment -- the +"True" segment -- was processed and the "False" segment will be +ignored.  Conversely, if the Condition-Test resulted in "FALSE", +then the first segment -- the "True" segment -- was ignored and the +"False" segment will be processed. +There are three synonyms for this +function: + +<ul style="margin-left: 40px;"> + <li>[else]</li> + <li>#else</li> + <li>[#else]</li> +</ul> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId375310"></a><a + name="Condition_Testers"></a><big>Condition-Testers</big></h5> +<div style="margin-left: 40px;">Tokenization can be controlled +according to the following conditions: +</div> +<ul style="margin-left: 40px;"> + <li>A <a href="#TrueFalse_flag_on_the_top_of_the_stack">True/False +flag</a> on the top of the stack</li> + <li><a href="#Existence_or_non-existence_of_an_FCode">Existence</a> +or non-existence of an FCode or "Tokenizer Escape"-mode +definition</li> + <li><a href="#Definition_or_non-definition_of_a">Definition</a> or +non-definition of a Command-Line symbol.</li> +</ul> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId104560"></a><a + name="TrueFalse_flag_on_the_top_of_the_stack"></a><big>True/False flag +on the top of the +stack</big></h5> +<div style="margin-left: 80px;"> +A number that was placed on the stack +in “Tokenizer-Escape” mode will be consumed and tested.  If the +number was +non-zero, the Condition-Test will result in "TRUE", and the first +segment -- the "True" segment -- will be processed and the "False" +segment, if present, will be ignored.  Conversely, if the number +was zero, the Condition-Test will result in "FALSE", and the first +segment -- the "True" segment -- will be ignored and the "False" +segment, if present, will be processed. +There is only one word for this +function: + +<ul> + <li>[if]</li> +</ul> +The [if] +Operator can be invoked from either "Tokenizer Escape" mode or normal +tokenization mode, but the number to be tested must have been put on +top of the Stack during "Tokenizer Escape" mode. +</div> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId956359"></a><a + name="Existence_or_non-existence_of_an_FCode"></a><big>Existence or +non-existence of an +FCode or "Tokenizer Escape"-mode definition</big></h5> +A name-string must follow the directive +<a href="#Expected_on_same_line">on the same line</a>.  A +search for that name is conducted through the word-list of the mode -- +i.e., +"Tokenizer Escape" mode versus "Normal" mode -- and the <a + href="#Scope_Directives">Scope</a> -- i.e., "<a + href="#Global_Definitions">Global</a>" versus "<a + href="#device-node_vocabularies">Current Device-Node</a>" -- in which +the +Tokenizer is currently operating. +If the directive is for the existence of +the definition, and the name +is found, or if the directive is for the non-existence of the +definition, and the name is not found, the Condition-Test will result +in "TRUE".  Otherwise, the Condition-Test results in "FALSE". +<div style="margin-left: 80px;">The synonyms for the test for existence of a +definition are: +</div> +<ul style="margin-left: 80px;"> + <li>[ifexist]</li> + <li>#ifexist</li> + <li>[#ifexist]</li> + <li>[ifexists]</li> + <li>#ifexists</li> + <li>[#ifexists]</li> +</ul> +(Note the variants with +and without a final 's') + +<div style="margin-left: 80px;">The synonyms for the test for non-existence of +a definition are: +</div> +<ul style="margin-left: 80px;"> + <li>[ifnexist]</li> + <li>#ifnexist</li> + <li>[#ifnexist]</li> +</ul> +<div style="margin-left: 120px;">(Note that variants with a final 's' +didn't make sense here.) +</div> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId880808"></a><a + name="Definition_or_non-definition_of_a"></a><big>Definition or +non-definition of a +Command-Line Symbol</big></h5> +<div style="margin-left: 80px;">As noted <a + href="#Define_Command-Line_symbol">above</a>, the Tokenizer +recognizes the syntax +</div> +<div style="margin-left: 120px;">-d +Symbol[=Value] + +</div> +<div style="margin-left: 80px;">as the means by which to define a +Command-Line Symbol and optionally assign a value to it. +</div> +<div style="margin-left: 80px;"> +A pair of Condition-Tester directives +are supported that will test, respectively, for definition or +non-definition of a named Symbol.  Their operation is similar to +that of the Tests for existence of non-existence of a Definition, in +that a name-string must follow the directive +<a href="#Expected_on_same_line">on the same line</a>, but they are +different in the matter of where the +search for the name string is conducted:  these directives search +the list of Command-Line Symbols.  The relation between the type +of directive and the result of the search is also similar to that of +the Tests for existence of non-existence of a Definition. +</div> +<div style="margin-left: 80px;">The synonyms for the test for definition of a +Command-Line Symbol are: +<ul> + <li>[ifdef]</li> + <li>#ifdef</li> + <li>[#ifdef]</li> +</ul> +</div> +<div style="margin-left: 80px;">The synonyms for the test +for non-definition +of a Command-Line Symbol are: +<ul> + <li>[ifndef]</li> + <li>#ifndef</li> + <li>[#ifndef] + </li> +</ul> +</div> +<div style="margin-left: 80px;">The test is simply to verify the +existence of a Symbol; it is not sensitive to +whether or not the optional value was assigned.  +</div> +<h5 style="margin-left: 40px;"><a name="mozTocId891224"></a><a + name="Tracing_Conditional_State"></a><big>Tracing the state of +Conditional +Tokenization</big></h5> +<div style="margin-left: 40px;">The nesting and un-nesting of +<a href="#Conditional_Tokenization">Conditional Tokenization</a> can +become complicated; an floaded +file may contain a segment-switcher or Conditional-block terminator +that interacts with the prevailing Condition-Tester that originated in +another Source file, which might lead to unexpected results.  To +help trace the status of Conditional Tokenization, the User can invoke +the <a href="#Trace_Condls">Trace-Conditionals</a>  +Special-Feature Flag together with the"verbose" option ( <a + href="#verbose_option">-v</a> +).  This will cause an <a href="#Advisories">Advisory +Message</a> to be displayed whenever a <a href="#Conditional_Operators">Conditional +Operator</a> is +encountered, even if it is in a segment that is already being ignored. +The message will indicate the location of the associated <a + href="#Condition_Testers">Condition-Tester</a> (or the state of the +condition, if the Conditional Operator is a Condition-Tester) and whether +the segment is being Processed or being Ignored. </div> + +<h4><a class="mozTocH4" name="mozTocId202222"></a><a + name="The_OVERLOAD_Directive"></a><big>Suspending +the Duplicate-Name Test for one Definition ("Overloading")</big></h4> +When a definition is created whose name duplicates that of an +existing definition, default behavior is to issue a <a + href="#A_WARNING">WARNING</a> +notification. + +Intentionally creating such a +duplicately-named definition is called "overloading" the name.  +This may be required, for instance, to supplant future invocations of +the named function with a version that incorporates the earlier +instance of the function of +the same name and supplies additional behavior. + +When this is intentional, no warning should be issued.  This +Tokenizer +supports a directive called overload +which the User may invoke to bypass the duplicate-name test for the +duration of only one definition that follows +it (as contrasted with <a href="#Warn_if_Duplicate_Flag">suspending +the test globally</a>).  An illustration of its use may be seen in +<a href="#Example_4">Example 4</a>. + +<h4><a class="mozTocH4" name="mozTocId202620"></a><a + name="Allow_Multi_Line"></a><big>Suspending the Multi-Line Warning +for one occasion</big></h4> +The Standard allows Comments and Strings that extend across multiple +lines, terminated by their specific delimiter.  In addition, this +Tokenizer allows User-generated Messages and Local-Values Declarations +that, likewise, extend across multiple lines.  Because of the +potential for a cascade of errors that can be caused by a missing +delimiter, this Tokenizer issues a <a href="#A_WARNING">WARNING</a> +notification whenever such an entity is encountered.  <a + href="#Example_5_func_nam">Example #5</a> shows an occasion where this +might be helpful. + +When this is intentional, no warning should be issued.  This +Tokenizer +supports a directive called multi-line +which the User may invoke to suppress the notification for the next +multi-line item that follows +it. +<h4><a class="mozTocH4" name="mozTocId203416"></a><a + name="Scope_Directives"></a><big>Controlling +the Scope of Definitions</big></h4> +The User may wish to create a collection of utility functions +that will be available to all the device-nodes in a <a + href="#Multiple_device_nodes">multiple-node +driver</a>.  Directly accessing such functions across device-nodes +can +be risky if any component uses instance +data.  However, if the use of instance +data is precluded, such access can be performed safely. +This Tokenizer +supports a pair of directives for setting the Scope of +Definitions.  The one called global-definitions +will cause all subsequent definitions -- Aliases and Macros as well as +FCode -- to be entered into the same vocabulary -- referred to as the +"core" -- from which all the Standard functions are drawn.  The +other, called device-definitions +, will cause the Scope of Definitions to revert to the <a + href="#current_device-node">current +device-node</a>.  The use of these directives is +illustrated in +<a href="#Example_4">Example #4</a>. + +While Global Scope is in effect, the use of the word <a + href="#Instance_Not_Allowed">instance</a> +will not be allowed, nor will definitions made in any device-node be +recognized.  Conversely, an attempt to enter Global Scope while instance +is in effect will be reported as an <a href="#An_ERROR">Error</a> +condition. +<h4><a class="mozTocH4" name="mozTocId314185"></a><a + name="Evaluating_Command-line_Symbols"></a><big>Evaluating +Command-line Symbols</big></h4> +A Directive is supported that can be used to extract the <a + href="#Define_Command-Line_symbol">optional +value</a> that was assigned to a Command-Line Symbol. +The Symbol name must appear <a href="#Expected_on_same_line">on the +same line</a> as the directive. +If the Symbol name is not found or no value has been assigned to it, +a <a href="#A_WARNING">WARNING</a> will +be issued. +Otherwise, the associated value will be interpreted as though it were +source-code. +The synonyms for the directive to evaluate a Command-Line Symbol's +assigned value are: + +<ul> + <li>[defined]</li> + <li>#defined</li> + <li>[#defined]</li> +</ul> +<h4><a class="mozTocH4" name="mozTocId655967"></a><big><a + name="FLITERAL"></a>Outputting +Arbitrary Byte-Sequences to +the FCode Binary</big></h4> +In addition to the Standard emit-byte +command, this Tokenizer supports a directive called FLITERAL +, which will emit the number popped from the top of the Data Stack as a +literal to the FCode Binary, as though the number were invoked during +normal operation. +The FLITERAL +directive can be invoked from either "Tokenizer Escape" mode or normal +tokenization mode, but the number to be emitted must have been put on +top of the Stack during "Tokenizer Escape" mode. +<h4><a class="mozTocH4" name="mozTocId114013"></a><big><a + name="ENCODE_FILE"></a>Encoding blocks of binary data taken from a +file +</big></h4> +The directive ENCODE-FILE  +takes a file-name as an argument <a href="#Expected_on_same_line">on +the same line</a>.  It translates +the contents of the file into a series of strings interspersed +with  encode-bytes  +and  encode+  +commands, resulting in a block of byte-encoded binary data. +<h4><a name="mozTocId69109"></a><big><a name="Gen_Specl_Txt_Strs"></a>Generating +Special Text-Strings and Literals</big></h4> +This Tokenizer supports a set of directives that will present a string +or a literal +for processing as though it were part of the Tokenization Source +File.  Except as otherwise noted, these directives may be invoked +in either "Normal" or +"Tokenizer-Escape" mode:  in "Normal" mode, the string produced +will be +compiled-in as an in-line text +string; in "Tokenizer-Escape" mode, it will be displayed as a +User-generated <a href="#User_Messages">MESSAGE</a>. + +<h5 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId70133"></a><big><a + name="fcode_date"></a>Current date or time</big></h5> +<div style="margin-left: 40px;">The [fcode-date] +directive produces a string  that consists of the current date, +formatted as  mm/dd/ccyy +. +</div> +The [fcode-time] +directive produces a string  that consists of the current time, +formatted as  hh:mm:ss ZZZ  +(where ZZZ +represents the local Time-Zone). + +An <a href="#timestamp_prop">illustration</a> +is included in <a href="#Example_4">Example 4</a>. + +<h5 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId71157"></a><big><a + name="function_name"></a>Name of the Function currently +being defined</big></h5> +<div style="margin-left: 40px;"> +The [function-name] +directive, when invoked in "Normal" mode, produces an in-line string +that consists of the name of the function (colon-definition) most +recently -- or currently +being -- +defined.  It will persist until a new colon-definition is +started.  +This can be useful for embedding interpetation-time or run-time +debugging-statements into or after +functions' definitions.  In "Tokenizer-Escape" mode, it will +display the +function name in more detail as a +User-generated <a href="#User_Messages">MESSAGE</a>.  +<a href="#Example_5">Example 5</a> illustrates its +use in "Normal" mode, and has a small <a href="#Example_5_func_nam">illustration</a> +of its use in "Tokenizer-Escape" mode. + +</div> +<h5 style="margin-left: 40px;"><a name="mozTocId72181"></a><big><a + name="Inp_Fil_Nm_Strg"></a>Input-File Name and Line Number</big></h5> +<div style="margin-left: 40px;">The [input-file-name]directive +produces an in-line string that +consists of the name of the current Input File. +The [line-number]directive +produces an in-line numeric literal giving the Line-Number within the +current Input File. + +These directives +can only be invoked in "Normal" mode; they are not supported -- or +needed -- in +"Tokenizer-Escape" mode,  because any User-generated <a + href="#User_Messages">MESSAGE</a> will +print that information. An <a href="#Example_5_func_nam">illustration</a> +is included in <a href="#Example_5">Example 5</a>. + +</div> +<h4><a class="mozTocH4" name="mozTocId844437"></a><big><a + name="Pre_pending_PCI_Headers_to_FCode_images_"></a>Pre-pending +PCI Headers to FCode +images</big> +</h4> +The directive pci-header +, which can be invoked from either "Tokenizer Escape" mode or normal +tokenization mode, will cause a  PCI Header to be generated to the +Binary Output file, in front of the FCode Binary image.  It takes +three parameters from the Data Stack:  Vendor ID as the third item +on the Stack, Device ID as the second, and Class Code as the top item +on the Data Stack.  These must be put on the Stack during +"Tokenizer Escape" mode. +An attempt to issue the pci-header +directive after FCode output has begun will be reported as an <a + href="#An_ERROR">Error</a> condition. + +The PCI Header cannot be completed until after the FCode Binary image +has been completely tokenized.  The directive to complete the PCI +Header should be issued at the end of the process.  Synonyms for +this directive are: + +<ul> + <li>pci-header-end</li> + <li>pci-end</li> +</ul> +<h4 style="font-weight: bold;"><a class="mozTocH4" name="mozTocId931658"></a><a + name="Modifying_PCI_Header"></a><big>Modifying the PCI +Header</big></h4> +Directives are supported that will alter specific fields of the PCI +Header.  They can be issued at any time before the pci-end  +directive to specify the contents of certain fields of the PCI Data +Structure as follows: +<ul> + <li>"Revision Level of the Vendor's ROM"  (2 bytes at +offset 0x12)</li> + <li>"Last Image Indicator" bit (High-order bit of 1 byte at offset +0x15)</li> +</ul> +The syntax for these functions is: +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId504353"></a><big><a + name="PCI_Rev_Level"></a>"Revision +Level of the Vendor's ROM"</big></h5> +<div style="margin-left: 40px;">To set the "Revision Level of the +Vendor's ROM" field of the PCI Header to the number popped from the top +of the Data Stack, use any of these synonyms: +<ul> + <li>set-rev-level</li> + <li>pci-revision</li> + <li>pci-code-revision</li> +</ul> +These directives can be invoked from either "Tokenizer Escape" mode or +normal tokenization mode, but the number to be used must have been put +on top of the Stack during "Tokenizer Escape" mode. +The default setting for this field is 1. + +<h6 style="font-weight: bold; margin-left: 40px;"><a + name="mozTocId503878"></a><big><a name="PCI_Rev_Level_Byte_Order"></a>Byte-Order</big></h6> +Ordinarily, the "Revision Level of the +Vendor's ROM" field of the PCI Header follows the general PCI Standard +convention of "Little-Endian" byte-order.  However, some +organizations' legacy code saves this field in Big-Endian order.  +To +accommodate the need for compatibility, this +Tokenizer supports a <a href="#Special_Feature_Flags">Special-Feature +Flag</a> called <a href="#Big_End_PCI_Rev_Level">Big-End-PCI-Rev-Level</a>.  +Its default is no, +i.e., to follow the PCI Standard.  + +</div> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId53081"></a><big><a + name="Last_Image_Indicator_bit"></a>"Last +Image Indicator" bit</big></h5> +<div style="margin-left: 40px;">The "Last Image Indicator" bit can be +specified in either of three ways: +<ul> + <li>Force it to be set to TRUE.  Synonyms for this are:</li> + <ul> + <li>last-image</li> + <li>last-img</li> + </ul> +</ul> +<ul> + <li>Force it to be reset to FALSE.  Synonyms for this are:</li> + <ul> + <li>not-last-image</li> + <li>not-last-img</li> + </ul> +</ul> +<ul> + <li>Set or reset it according to the number popped from the top +of the Data Stack.  Synonyms for this are:</li> + <ul> + <li>set-last-image</li> + <li>set-last-img</li> + </ul> +</ul> +</div> +<div style="margin-left: 80px;">If the number was zero, the "Last Image +Indicator" bit will be reset to FALSE; if it was non-zero, the bit will +be set to TRUE.</div> +These directives can be invoked from +either "Tokenizer Escape" +mode or +normal tokenization mode, but the number to be examined must have been +put +on top of the Stack during "Tokenizer Escape" mode. + +The default setting for this field is +TRUE. +<h4><a class="mozTocH4" name="mozTocId410469"></a><big><a + name="Binary_Output_File_Name"></a>Changing the name of the +Binary Output File</big></h4> +<a name="Binary_Output_File_Name_Default"></a>By default, the name +of the Binary Output File is derived from the +name +of the Source Input File, by replacing its extension with  .fc +, or, if  the Input File name had no extension, merely appending +the +extension  .fc + +The name of the Binary Output File may be specified on the Command +Line,  as noted <a href="#Binary_Output_Name_option">above</a>. +<a name="Binary_Output_File_Name_Directive"></a> +The name of the Binary Output File may also be specified by a directive +embedded within the Tokenization Source File.  Synonyms for the +directive are  save-image  +and  save-img +; the name follows after the directive <a href="#Expected_on_same_line">on +the same line</a>. + +The directive can be invoked from either "Tokenizer Escape" mode or +normal tokenization mode. It does not cause an imediate action (i.e., +saving the Binary Image), but merely alters the name of the file to +which the image will be saved when tokenization is completed. + +Note that this directive does not change the name of the <a + href="#FLoad_List_option">FLoad-List File</a>, if one has been +specified. + +<h4><a class="mozTocH4" name="mozTocId982642"></a><big><a + name="User_Messages"></a>Issuing messages at Tokenization +time.</big></h4> +Directives are supported that will gather text and output it as a +User-generated MESSAGE +during +Tokenization. +These directives can be invoked from either "Tokenizer +Escape" mode or normal tokenization mode: +<ul> + <li>#MESSAGE"  +will parse text in <a href="#String_Gathering_Features">string-collection +mode</a>, i.e., with +sensitivity to the special escaped-characters, until a concluding +Quote-followed-by-whitespace,</li> +</ul> +<div style="margin-left: 40px;">and the synonyms: +</div> +<ul> + <li>[MESSAGE]</li> + <li>#MESSAGE</li> + <li>[#MESSAGE]</li> +</ul> +<div style="margin-left: 40px;">will gather the text remaining on the +line. +</div> +In addition,  the commands  ."  +(Dot-Quote) and  .(  +(Dot-Paren), which have Standard meanings in normal tokenization +mode, will, when invoked in +"Tokenizer Escape" mode, +collect text in their Standard manner and output it as a User-generated +MESSAGE: +<ul> + <li>Dot-Quote will parse text in <a href="#String_Gathering_Features">string-collection +mode</a>,</li> + <li>Dot-Paren will collect text until a +balancing Close-Paren --  )  +-- character.</li> +</ul> +Also, as noted earlier, the <a href="#fcode_date">[fcode-date]</a> +and <a href="#fcode_date">[fcode-time]</a> +directives, when invoked in +"Tokenizer Escape" mode, will display the current date and time, +respectively, as a User-generated +MESSAGE: +<h4><a class="mozTocH4" name="mozTocId550942"></a><big><a + name="Changing_C-L_Flags_from_Source"></a>Changing Special-Feature +Flags +from Source</big></h4> +A directive is supported that can be used from within the Tokenization +Source File to over-ride the setting of any one of the <a + href="#Special_Feature_Flags">Special-Feature +Flags</a> that might have been made from the Command Line.  The +name of the Flag -- with, of course, an optional leading No +-- must follow after the directive <a href="#Expected_on_same_line">on +the same line</a>.  It can be +invoked from either "Tokenizer Escape" mode or normal tokenization +mode.  Its synonyms are: +<ul> + <li>[FLAG]</li> + <li>#FLAG</li> + <li>[#FLAG]</li> +</ul> +The User is hereby admonished to exercise caution when using this +directive.  Not all combinations are meaningful, and automated +error-checking is not feasible.  An <a href="#Advisories">Advisory +Message</a> will +be issued to remind the User of the change. + +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId551846"></a><big><a + name="Displaying_C-L_Flags_from_Source"></a>Displaying +Special-Feature Flags from Source</big></h5> +<div style="margin-left: 40px;">A directive is supported that can be +used from within the Tokenization +Source File to display a <a href="#User_Messages">MESSAGE</a> with +a list of the settings of all of the Special-Feature Flags.  Its +synonyms are: +</div> +<div style="margin-left: 40px;"> +<ul> + <li>[FLAGS]</li> + <li>#FLAGS</li> + <li>[#FLAGS]</li> + <li>SHOW-FLAGS</li> +</ul> +</div> +<h4><a class="mozTocH4" name="mozTocId573280"></a><big><a + name="Manipulate_FCode_Number"></a>Manipulating the +FCode-Token-Number Assignment Counter</big></h4> +In addition to the Standard  <a href="#next_fcode">next-fcode</a>  +command, this Tokenizer supports additional directives that can be used +to +alter the normal sequence of FCode-token-number assignment. +In order to protect against unintended collisions in FCode-token +numbers, which can cause severe +errors at run-time, this Tokenizer will report an <a href="#An_ERROR">Error</a> +the first time +an FCode-number assignment overlaps FCodes that were assigned in a +different range. + +Because the programmer may choose to "recycle" FCode-token-numbers +intentionally, a directive is supported that clears the records of +FCode-number assignments and resets the FCode-token-number assignment +counter to its inital value. +<h5 style="margin-left: 40px;"><big><a name="mozTocId566072"></a>Saving +and Restoring</big></h5> +<div style="margin-left: 40px;"> +The directive FCode-Push +can be used to save the value of the FCode-token-number assignment +counter on the <a href="#Run-time_data_stack">tokenization-time +data +stack</a> .  Later, the saved value can be restored by invoking +the FCode-Pop +directive.  Note that FCode-Pop  +is functionally equivalent to <a href="#next_fcode">next-fcode</a> +except that  FCode-Push + and  FCode-Pop  +can be invoked from either "Tokenizer Escape" mode or "Normal" mode.</div> +<h5 style="margin-left: 40px;"><big><a name="mozTocId568474"></a>Resetting</big></h5> +<div style="margin-left: 40px;"> +The directive FCode-Reset +will re-initalize the value of the FCode-token-number assignment +counter and clear the records of previous FCode-number +assignments.  Note that it is not equivalent to: +F[ +   h# +800 next-fcode +  F] +which merely re-initalizes the value of the FCode-token-number +assignment counter, but does not prevent the first assignment of this +FCode-number from being regarded as overlapping an earlier +assignment  and therefore being reported as an <a href="#An_ERROR">Error</a>. +The FCode-token-number assignment +counter and the records of previous FCode-number assignments will also +be automatically re-initalized when a <a href="#Multiple_PCI_Image">new +PCI-Image</a> block is started. + +</div> +<h4><a class="mozTocH4" name="mozTocId575686"></a><big>Resetting +Symbols Defined in Either Mode</big></h4> +Symbol names defined in "Normal" mode persist until a <a + href="#Multiple_device_nodes">device-node is finished</a>, or until +the end of an FCode-block or of a PCI Image. +Symbol names defined in "Tokenizer Escape" mode persist throughout +the entire run of the Tokenizer. +The directive Reset-Symbols +enables the User to delete symbol-definitions at any other time, should +that be required. Reset-Symbols  +is sensitive to the the mode in which the Tokenizer is operating:  +invoked during "Tokenizer Escape" mode, it will cause only the +definitions made under "Tokenizer Escape" mode to be deleted; during +"Normal" mode, only those made under "Normal" mode.  <a + href="#Example_1">Example +#1</a> contains an invocation in "Tokenizer Escape" mode. + +<h3><a class="mozTocH3" name="mozTocId957252"></a><a + name="Non-standard_input_syntaxes_"></a>Non-standard input, +syntaxes, and behavior +</h3> +In this section, we will discuss: + +<ul> + <li>This Tokenizer's handling of conditions the Standard calls +"ambiguous"</li> + <li>Non-Standard synonyms for Standard functions</li> + <li>Non-Standard functions or behaviors that are supported by this +Tokenizer.</li> + <li>Featured behaviors, supported by this Tokenizer, that are not +mentioned in the Standard. </li> +</ul> +<h4 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId850914"></a>Ambiguous +Conditions</h4> +<div style="margin-left: 80px;">The Standard only identifies a few +questionable combinations as "ambiguous conditions" or "behavior ... +unspecified" and leaves their +handling open to interpretation of the implementor.  Here's how +this Tokenizer handles them: +</div> +<div style="margin-left: 40px;"> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId162252"></a><big><a + name="Expected_on_same_line"></a>Tokens +expected on the same +line</big></h5> +Certain commands and directives expect a +token to appear on the same line, and will report an <a + href="#An_ERROR">Error</a> condition if no +token appears on the same line: +<ul> + <ul> + <li>The  <a href="#The_h_d_and_o_directives_">h#  + d#  +and  o#</a>  +directives, and the <a href="#Char_Seq_To_Num">a#</a> and <a + href="#Char_to_Num_Left_Justified">al#</a> +directives. + </li> + <li>The    <a + href="#Existence_or_non-existence_of_an_FCode">[ifexist]  + [ifnexist]</a>   + <a href="#Definition_or_non-definition_of_a">[ifdef]   + [ifndef]</a>  +and  <a href="#Evaluating_Command-line_Symbols">[defined]</a>   +directives and all their synonyms</li> + <li>The commands   [']  +and  ' + + </li> + <li>The  <a href="#F_Eff-Bracket-Tick-Bracket">F[']</a>  +directive</li> + <li>The <a href="#Alias">ALIAS</a> +directive  (expects two tokens on the same line)</li> + <li>The <a href="#Macros">[macro]</a> +directive  (expects at least two tokens on the same line) </li> + <li>The directives  control +ascii char + and +[char]</li> + <li>The  FLOAD  +command</li> + <li>The  <a href="#Binary_Output_File_Name">SAVE-IMAGE</a> +directive</li> + <li>The  <a href="#Changing_C-L_Flags_from_Source">[FLAG]</a> +directive and its synonyms + </li> + <li>The  <a href="#ENCODE_FILE">ENCODE-FILE</a> +directive </li> + </ul> +</ul> +</div> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId246601"></a><a + name="The_h_d_and_o_directives_"></a><big>The  +h# +d# and +o# +directives</big></h5> +<div style="margin-left: 40px;">The directives h# +d# and o#   +convert the token immediately following to a +number, in hex, decimal or +octal, respectively. +If the token is present, but cannot be converted to a number using +the +appropriate radix, i.e., if the conversion fails, the +Tokenizer will issue a <a href="#A_WARNING">WARNING</a>, +ignore the directive and attempt to +process the token +as ordinary input.  If the name is not +known, the "ordinary processing" will catch the error. +</div> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId978546"></a><big>The +commands  leave  +?leave  +or  unloop  +outside of a loop-control framework</big></h5> +<div style="margin-left: 40px;">The Standard only speaks briefly about +the execution consequences of this condition, and says nothing about +whether a Tokenizer should permit such a sequence. This Tokenizer will +report it as an <a href="#An_ERROR">Error</a> +condition. +</div> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId73206"></a><big>The +commands  [']  +or  '  +followed by a name that is not a valid target</big></h5> +<div style="margin-left: 40px;">Valid targets for the commands   [']  +and  '  +are function-names that were defined during normal tokenization +mode.  +This Tokenizer reports an <a href="#An_ERROR">Error</a> +if the name following the  [']  +or  '  +is not a valid target (e.g., is the name of a macro).  It will +then +attempt to +process the name in the ordinary manner.</div> +<h5 style="margin-left: 40px;"><big><a class="mozTocH5" + name="mozTocId74410"></a>The Forth word to +followed by a name that is not a valid target</big></h5> +<div style="margin-left: 40px;">The Standard specifies the Forth word to +as the means for setting the value of a word defined with either the value +or defer +defining-words, but says nothing about how a Tokenizer should respond +to attempts to apply it to other  targets.    This +Tokenizer reports an <a href="#An_ERROR">Error</a> +if the name following the to +is not a valid target (e.g., is the name of a macro or a +colon-definition). +An exception is made for a word defined as a variable +:  Although such words are not a valid target for to, +many platforms' firmware will execute the sequence correctly; this +Tokenizer will issue a <a href="A_WARNING">WARNING</a>. +</div> +<h5 style="margin-left: 40px;"><big><a class="mozTocH5" + name="mozTocId75614"></a><a name="Instance"></a>The +word instance</big> +</h5> +<h6 style="font-weight: bold; margin-left: 80px;"><big><a + name="mozTocId76818"></a>Followed by an inapplicable +defining-word</big></h6> +<div style="margin-left: 80px;">The Standard implies (but does not +state) that, +once instance +has been executed, it remains in effect until an applicable +defining-word ( value +, variable +, defer +or buffer: +) +is encountered. For example, in the sequence: +instance : nonsense dup swap +drop ; +variable boombah +the colon-definition of nonsense is +unaffected by the occurrence of instance +, which will, instead, be applied to the defining-word variable +(which defines boombah ) +on a later line. + +Since instance +would typically be followed immediately by the defining-word +to which it is intended to apply, a sequence like the above can +reasonably be presumed to be a likely error.  This Tokenizer will +issue +a <a href="A_WARNING">WARNING</a> when an +inapplicable +defining-word is encountered while instance +is in effect, and another WARNING when the dangling instance +is finally applied to a valid definer. + +</div> +<h6 style="margin-left: 80px;"><big><a name="mozTocId78022"></a>Left +unresolved</big></h6> +<div style="margin-left: 80px;">Multiple occurrences of instance +are not cumulative; if a second instance +is encountered before the first is applied, only one occurrence of an +applicable defining-word will be modified.  This Tokenizer will +issue a <a href="#A_WARNING">WARNING</a> when +a second occurrence of instance +is encountered before the first has been applied. </div> +<div style="margin-left: 80px;"> +The Standard says nothing about what +should occur +if instance +has not been applied by the time a device-node is "finish"ed +or a new device-node +is started.  In most Platforms' implementations of the FCode +Interpreter, the instance +state will remain in effect.  This Tokenizer will issue a WARNING +the first time a device-node +is changed while an instance +remains unresolved. + +</div> +<h6 style="margin-left: 80px;"><a name="mozTocId79226"></a><big><a + name="Instance_Not_Allowed"></a>Not +Allowed in...</big></h6> +<div style="margin-left: 80px;"> +Although the Standard says nothing about the subject, an inclusion +of instance +inside +a Colon-definition would serve no purpose and is <a + href="#INSTANCE_at_run_time">potentially harmful</a>; instance +should not be allowed in a Colon-definition. This Tokenizer will +detect an attempt to compile instance +into +a Colon-definition and report it as an <a href="#An_ERROR">Error</a> +condition. +Also, instance +can not be allowed when <a href="#Global_Definitions">Global Scope</a> +is in effect, as <a href="#INSTANCE_at_run_time">explained elsewhere</a>. + +</div> +<h5 style="margin-left: 40px;"><a class="mozTocH5" name="mozTocId884280"></a><big>The +command fload +inside a colon-definition</big></h5> +<div style="margin-left: 40px;">If the command fload +is encountered while a colon-definition +is in progress, this Tokenizer will process it immediately, and read +the contents of +the floaded +file as a continuation of Source input.   </div> +<h4 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId901662"></a>Non-Standard +Synonyms for Standard +Functions</h4> +<div style="margin-left: 40px;">This Tokenizer accepts the following +non-Standard words as synonyms for the Standard words, as listed: +<ul> + <li>ENDIF +as a synonym for THEN</li> + <li>NAME +as a synonym for DEVICE-NAME</li> +</ul> +</div> +<h4 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId732819"></a>Non-Standard +Functions or behaviors</h4> +<div style="margin-left: 40px;">This section will discuss features +supported by this Tokenizer that extend capabilities regarded by the +Standard as unsupportable, or that are left only vaguely specified as +"implementation-dependent".  +It will also present a few minor items that are introduced as +conveniences for the User. +</div> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId522750"></a><a + name="Alias"></a><big>Alias</big></h5> +In normal tokenization mode the alias +directive behaves as specified in the Standard, i.e., it creates a new +command with the exact behavior of an existing command. Any occurrence +of the new command will cause the assigned FCode of the old command to +be generated, with all the implications that follow from that. +The Standard, further, states that:  +"In FCode source, alias +cannot be called from within a colon definition."  However, this +Tokenizer can handle that, and issues a <a href="#A_WARNING">WARNING</a> +message when it +does so. +The Standard does not specify the +behavior of  alias  +in  "Tokenizer Escape" mode.  This +Tokenizer will allow alias +commands issued in "Tokenizer Escape" mode to take effect in that mode, +and, furthermore, aliases to words that are recognized in either mode +may be created in either mode and will be recognized in either mode. + +Generally speaking, an alias definition +will take on the <a href="#Scope_Directives">Scope</a> that is current +at the time it is made:  If Device Scope is in effect, the new +name -- even if it is an alias to a word that has <a + href="#Global_Definitions">Global scope</a> -- will only be accessible +from the current +device-node.   An <a href="#Advisories">ADVISORY</a> +message will be issued for this condition. + +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId824541"></a><a + name="String_Gathering_Features"></a> +<big>String-Escape characters and other String-Gathering Features</big></h5> +<div style="margin-left: 80px;">The Standard specifies, in effect, that +a string is terminated by a double-quote (the " +character) when it is immediately followed by whitespace; that if the " +is followed by an open-parenthesis, hex-sequence parsing begins until a +close-parenthesis is found; and that if the " +is followed by any other character, "the result is +implementation-dependent". +<h6 style="margin-left: 40px;"><big><a class="mozTocH5" + name="mozTocId945290"></a>Quoted +String-Escape Characters</big></h6> +Common practice -- widely enough used as +to merit consideration as an undocumented de-facto standard -- has been +to recognize a set of letters and to translate them into special +characters when they immediately follow the " +. + +This +Tokenizer translates String-Escape Quoted-pairs as follows: + + +<table style="text-align: left; width: 100%;" border="0" cellpadding="1" + cellspacing="4"> + <tbody> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"n + </td> + <td style="vertical-align: top;">New-Line </td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"l</td> + <td style="vertical-align: top;">New-Line</td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"r</td> + <td style="vertical-align: top;">Carriage-Return</td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"t</td> + <td style="vertical-align: top;">Horizontal Tab</td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"f</td> + <td style="vertical-align: top;">Form-Feed</td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"b</td> + <td style="vertical-align: top;">Backspace</td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"!</td> + <td style="vertical-align: top;">Bell</td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"^L</td> + <td style="vertical-align: top;">Quote-Caret followed by a letter +is +translated as "Control"-the-letter.</td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"Other</td> + <td style="vertical-align: top;">Any unrecognized character +following the  " +  +is taken verbatim. + </td> + </tr> + <tr> + <td style="vertical-align: top;">""</td> + <td style="vertical-align: top;">The way to embed a double-quote +into a string is to +escape it with itself.  (This is a special instance of the +preceding rule) + </td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"(</td> + <td style="vertical-align: top;">As was mentioned above, +Quote-Open-Parenthesis begins parsing a hex-sequence as per Section A.2 +of the +Standard under the description of the " +operator.  Details will be discussed <a href="#Hex_Seq_Procg">below</a>. + </td> + </tr> + <tr> + <td style="vertical-align: top; text-align: left; width: 105px;">"\<a + name="SFeat_Str_Rem_Esc"></a></td> + <td style="vertical-align: top;">Quote-Backslash permits +insertion of remarks into the middle of a string definition; it will +interrupt +string parsing, causing the remainder of +the line, together with any whitespace that might begin the new line, +to be ignored. + Because this feature is not in usual +practice, the User can disable it by invoking the <a + href="#SF_Flag_Str_Rem_Esc">noString-remark-escape</a> <a + href="#Special_Feature_Flags"> Special-Feature +Flag</a>. + If this feature is disabled, the Backslash following the  " +  +will be taken verbatim; the Backslash, the text following it on the +remainder of the line, the new-line and the whitespace on the next line +-- all of which would otherwise have been ignored -- will be included +in the +string parsing and incorporated into the result. + </td> + </tr> + </tbody> +</table> + +</div> +<div style="margin-left: 120px;"> +<h6><big><a class="mozTocH5" name="mozTocId758170"></a>Embedded +New-Lines</big></h6> +The Standard makes no mention about what to do when a string reaches +a new-line before its termination.  This +Tokenizer continues parsing and includes the new-line, together with +any whitespace that might begin the new line, in the result. + +<h6><big><a class="mozTocH5" name="mozTocId42456"></a><a + name="Hex_Seq_Procg"></a>Hex-Sequence +Processing</big></h6> +The Standard mentions parsing hexadecimal characters in pairs in the +substring between  "(  +and  )  +but makes no mention of what to do when a single hexadecimal character +appears between nonhexadecimal characters.  The usual practice is +to treat it as representing the low-order digit of a single byte, and +this Tokenizer follows that convention. +<a name="SFeat_Hex_Rem_Esc"></a>This Tokenizer  also supports +an option -- not in usual practice, +hence listed separately here -- that permits insertion of remarks in +the middle of a hex-sequence in a string.  The occurrence of a +single \ +(backslash) character will interrupt hex-sequence parsing, causing the +remainder of the line, +together with any whitespace that might begin the next line, to be +ignored.  The User can disable this feature by invoking the <a + href="#SF_Flag_Hex_Rem_Esc">noHex-remark-escape</a> <a + href="#Special_Feature_Flags"> Special-Feature +Flag</a>. + +If the Hex-Remark-Escape feature is disabled, the Backslash will be +treated as an ordinary nonhexadecimal character, and Hex-Sequence +parsing will proceed.  Any hexadecimal characters on the +remainder of the line  -- which would otherwise have been ignored +-- will be recognized and incorporated into the result.  +<h6><big><a class="mozTocH5" name="mozTocId150131"></a><a + name="C-Style_String-Escape_Characters"></a><big>C</big>-Style +String-Escape Characters</big></h6> +In addition to the Quoted String-Escape sequences, this +Tokenizer recognizes a small subset of the Backslash-based +String-Escape +pairs that are used in <big>C</big> and translates them +similarly. +Specifically, \n  +and  \t +are translated into New-Line and Horizontal Tab respectively. +<a name="SFeat_BS_xx"></a>If the Backslash is followed by other +characters, the +Tokenizer will attempt to read them as a digit-string, using the +current base, and create a numeric byte. The numeric sequence ends with +the first non-numeric character (which is treated as a delimiter and +consumed, unless it's a double-quote, in which case it's allowed to +terminate the string or apply whatever action is triggered by the +character following it).  If the value represented by the numeric +sequence exceeds +the size of a byte, its low-order byte will be used and a <a + href="#A_WARNING">WARNING</a> will +be issued. + +If the first character after the backslash was non-numeric, the +character will be used literally (and a WARNING will be issued).  +As in <big>C</big>, the backslash can be used to escape itself. + +</div> +The User can disable this feature by +invoking the <a href="#SF_Flag_C_Str_Esc">noC-Style-string-escape</a> +<a href="#Special_Feature_Flags">Special-Feature +Flag</a>. + +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId40998"></a><big><a + name="SFeat_AB_Q"></a>The <big>ABORT"</big> +(Abort-Quote) command +</big></h5> +<div style="margin-left: 80px;">The Standard is not very clear about +the use of this FORTH command, and does not assign it an FCode.  +Nonetheless, it has historically been treated as a macro, and this +Tokenizer will support that treatment. +<a name="SFeat_Sun_AB_Q"></a>To complicate matters, there are two +distinct styles in which this +macro is used in FCode drivers, "Apple" style and "Sun" style: + +In Sun Style, the sequence, in the Source, would look like this: + +<Condition> ABORT" Message text" +Semantically, it would mean that if the <Condition> +is true, the Message +text would be printed and a -2 +THROW would be performed; conversely, if the <Condition> +is false, the Message +text  would be bypassed and execution would continue with +the next token after. + +The sequence could be translated into FCode as a macro like this: +<div style="margin-left: 40px;"><Condition> IF " +Message text" TYPE  -2 THROW +THEN +</div> +In Apple Style, the Source supplies the surrounding IF +... THEN +.  The action of the ABORT" +command is to leave +the Message +text on the stack and perform the -2 +THROW unconditionally, with the expectation that the system CATCH +will print the string +it finds on the stack. + +The Source sequence would look like this: + +<Condition> IF ABORT" +Message text" THEN + +The ABORT" +... +" +portion of the +sequence would be translated into FCode as a macro like this: +<div style="margin-left: 40px;">" Message +text" -2 THROW + +</div> +Because the ABORT" +command is not specified in the Standard, the User can disable it by +invoking the <a href="#SF_Flag_AB_Q">noABORT-Quote</a>  +<a href="#Special_Feature_Flags">Special-Feature +Flag</a>. + +The User who chooses to enable this feature, can, further, select to +disable "Sun" style in favor of "Apple" style by invoking the <a + href="#SF_Flag_Sun_AB_Q">noSun-ABORT-Quote</a> +<a href="#Special_Feature_Flags">Special-Feature +Flag</a>. + +<a name="Abort_Quote_Abort"></a>And to complicate matters even +further, some Legacy  applications prefer to use the ABORT  +command (note there's no quote) in place of the -2 +THROW  sequence.   Although the ABORT +command is not recommended, it is +a legitimate FCode function, and this Tokenizer supports a <a + href="#Special_Feature_Flags">Special-Feature +Flag</a>, called <a href="#SF_Flag_AB_Q_Throw">Abort-Quote-Throw</a>, +which controls whether an ABORT" +(Abort-Quote) phrase will be tokenized with the -2 +THROW  sequence or with the ABORT +function.  The User who chooses to have  ABORT" +(Abort-Quote) phrases tokenized with the ABORT +function can do so by invoking <a href="#SF_Flag_AB_Q_Throw">noAbort-Quote-Throw</a> + +</div> +<h5 style="margin-left: 80px;"><a name="mozTocId526780"></a><big><a + name="Char_Seq_To_Num"></a>Conveniently Convert Short +Character-Sequence to Number</big> +</h5> +Occasionally, a User needs to create a +numeric constant whose value corresponds to a short sequence of +characters.  For instance, PCIR +will get coded as h# +50434952.  This tokenizer supports a convenient directive +called  a# +, syntactically similar to  <a href="#The_h_d_and_o_directives_">h# +d# and o#</a> +, which makes the conversion directly.  The above example can be +written:  a# +PCIR , sparing the programmer -- and the maintainer -- from +needing to translate ASCII on the fly. + +The  a#   +operator expects +its target argument <a href="#Expected_on_same_line">on the same line</a>. + +If the target-sequence contains more than +four characters,  the last four will become the number; if the +target-sequence contains fewer than four characters,  they will +fill the low-order part of the number.  (I.e.,  the operation +of  a#  +is right-justified.)  Thus: + +<div style="margin-left: 160px;">a# +CPU                +         is equivalent to +                    +        h# +00435055 +</div> +<div style="margin-left: 120px;"> and +</div> +<div style="margin-left: 160px;">a# +LotsOfStuff     is equivalent to  a# +tuff   or   h# +74756666</div> +Also, the conversion is case-sensitive: +  a# +cpu   is equivalent to   h# +00637075 +<h6 style="margin-left: 120px;"><a class="mozTocH5" name="mozTocId53626"></a><big><a + name="Char_to_Num_Left_Justified"></a>Left-Justified</big></h6> +<div style="margin-left: 120px;">To accommodate situations that call +for +the characters to occupy the high-order part of the number, this +Tokenizer supports an +"Ascii-Left" conversion directive called al# +, which is equivalent to  a#  +in all respects except when the +target-sequence contains fewer than four characters:  al# +CPU  is equivalent to   h# +43505500</div> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId54171"></a><a + name="F_Eff-Bracket-Tick-Bracket"></a><big>F[']   +("Eff-Bracket-Tick-Bracket")</big></h5> +Syntactically similar to the Standard ['] +("Bracket-Tick-Bracket").  Valid targets for F['] +are the same as for <a href="#mozTocId73206">[']  +or  '</a> +.  Attempts to apply F['] +to an invalid target will be handled similarly. + +This directive acquires the given word's +FCode-Token number, +which is then used according to whether the directive is invoked during +"Normal Tokenization" or "Tokenizer-Escape" mode: +<h6 style="margin-left: 120px;"><big><big><a name="mozTocId54716"></a><a + name="F_B-T-B_Normal"></a>F['] in "Normal Tokenization" Mode:</big></big></h6> +The given word's FCode-Token number is +tokenized as a literal, which can be used, for instance, as the +argument to a  get-token  +or  set-token  +command. + +<div style="margin-left: 120px;">For example, the sequence:<big> </big> +</div> +<div style="margin-left: 200px;">F['] +rl@ get-token +</div> +<div style="margin-left: 160px;">is equivalent to:<big> </big> +</div> +<div style="margin-left: 200px;"> h# +234  get-token +</div> +<h6 style="margin-left: 120px;"><big><big><a name="mozTocId55261"></a><a + name="F_B-T-B_TkzEsc"></a>F['] in "Tokenizer-Escape" Mode:</big></big></h6> +This function is the one <a + href="#Tkz_Esc_Scope">exception to the general rule</a> about the <a + href="#Scope_Directives">scope</a> of words recognized in +"Tokenizer-Escape" mode; it will recognize function-names that +were defined during normal tokenization mode and that were <a + href="#current_device-node">current at the time</a> "Tokenizer-Escape" +mode was entered. + +The given word's FCode-Token number is +pushed +onto the data-stack, from whence it can be used, for instance, as the +numeric argument to a <a href="#Tkz_Esc_Const">constant</a> +definition. + +<h5 style="margin-left: 80px;"><a name="mozTocId887752"></a><big><a + name="Shell_Env_Vbles_in_File_Names"></a>Shell-Environment +Variables in +File-Names</big></h5> +The filename that follows the fload +command or the <a href="#ENCODE_FILE">encode-file</a> +directive may be an +absolute path, a path relative to the Current Working +Directory, or a path relative to one of the +directories in the <a href="#Include_List">Include-List</a>.  It +may also contain Shell Environment Variables and +related expressions recognized by the Host Operating System environment +in which the +Tokenizer is running.  These will all be expanded before loading +the +file.  An illustration may be +seen in <a href="#Example_4">Example #4</a>. +An <a href="#Advisories">ADVISORY</a> +message showing the expanded value will be printed if the <a + style="font-style: italic; font-weight: bold;" href="#verbose_option">verbose</a> +<a href="#Command-Line_options_">option</a> has +been selected or in the event of a failure to read the file. + +<h4 style="margin-left: 40px;"><a class="mozTocH4" name="mozTocId505746"></a>Featured +behaviors, supported by this +Tokenizer, that are not mentioned in the Standard</h4> +<h5 style="margin-left: 80px;"><a name="mozTocId530910"></a><big><a + name="Macros"></a>User-Defined Macros</big></h5> +<div style="margin-left: 80px;">The Standard mentions built-in +tokenizer macros (such as 3drop for +example) but makes no provision for the User to define any additional +macros. +This Tokenizer supports a directive that allows the User to define +additional macros.  Its syntax is: + +[macro]  +<macroname>  cmnd1 cmnd2 cmnd3 ... cmndN +The entire body of the macro definition must be contained on a +single line.  The linefeed at the end of the line will be included +as part of the macro definition. + +In this Tokenizer,  macros are implemented as simple +string +substitutions, interpreted at the time they are invoked.  If a +component of the macro should change its meaning -- i.e., be redefined +-- then, on subsequent invocations of the macro, the new meaning will +take effect.  (Note that this is different from an <a + href="#Alias">alias</a>).  It is also (eminently) possible to +define a macro +that uses a name that has not been defined at the time the macro is +defined, to define that name later, and to invoke the macro after +the name has been defined.  This is legitimate and will work. + +For the same reason, macros may be nested; i.e., one macro may be +defined in such a way as to invoke another.  However, if a macro +-- or a series of nested macros -- were to invoke a macro that is +already running, that will be detected and reported as an <a + href="#An_ERROR">Error</a> condition.  +For a simple example, a User who needed to identify all occurrences of +the word 2drop +might attempt to write: +<div + style="margin-left: 40px; font-family: courier new,courier,monospace;"> overload [macro]  2drop  message" 2DROP called here" 2drop +</div> + in the expectation that the second 2drop  +would be interpreted as the generic one, which outputs the +corresponding token. +However, because the macro definition is not "compiled" in the same +way +as a colon-definition, but is, instead, interpreted at run-time, the 2drop +that would be executed would, in fact, be the macro itself, leading to +an infinite loop of messages (if the condition were not +detected...).  In order to protect against this condition, the +User should, instead, do something like this: +<div + style="margin-left: 40px; font-family: courier new,courier,monospace;">\    +Keep a "generic" 2DROP handy. +alias +generic-2drop  2drop +overload [macro] 2drop message" 2DROP called here" generic-2drop +</div> + +This has the added advantage that, when the passage in which the +notification is needed comes to an end, the User can restore 2drop +to its Standard behavior with: + +<div + style="margin-left: 40px; font-family: courier new,courier,monospace;">overload  alias  2drop  generic-2drop</div> +A User-Defined Macro takes on the <a href="#Scope_Directives">Scope</a> +that is current at the time it is created.  An illustration may be +seen in <a href="#Example_4">Example #4</a>. +</div> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId539069"></a><big><a + name="Multiple_device_nodes"></a>Multiple +device-nodes</big></h5> +<div style="margin-left: 80px;">The typical use of a tokenizer is to +write a driver for a single-node device.  For such an application, +the commands new-device +and finish-device +are not required.  However, many newer devices consist of +complexes of subordinate and peer devices within a single assembly; +some additional caution is required when developing drivers for such +configurations. +<a name="INSTANCE_at_run_time"></a>In particular, an attempt within +one device-node to access directly +a method defined in another device-node must be flagged as an +error.  Consider +what would happen at run-time if it were allowed:  the called +method +would be expecting the instance-pointer to be pointing to the instance +data of the device-node in which that method was defined, but it would, +instead, +be pointing to the instance data of the device-node that made the +call.  +This is an invitation to havoc that would be -- to put it politely -- +somewhat difficult to trace. + +The correct way to invoke a method across device-node boundaries is +via $call-parent +or $call-method +or the like. + +<a name="device-node_vocabularies"></a>In order to detect such +errors early on, this Tokenizer keeps track +of separate but linked "vocabularies" +associated with device-nodes. When the command new-device +is encountered in interpretation mode, a new device-node vocabulary is +opened and new +definitions are entered into it.  Definition-names created in the +preceding device-node -- presumably the parent of the newly started +device -- are suspended from accessiblity. + +Correspondingly, when the finish-device +command is encountered in interpretation mode, the "vocabulary" of the +device +being ended is emptied ("forgotten" in classic Forth parlance) and the +"vocabulary" of the parent-device is resumed. + +<a name="current_device-node"></a>The device-node vocabulary to +which definitions are being entered at +any given time, and from which definitions are accessible, may be +referred to as the current +device-node for purposes of discussion. + +Note that the Tokenizer does not switch vocabularies when the new-device  +or  finish-device +commands are encountered in compilation mode (i.e., when they are being +compiled-in to a method); they are treated as ordinary tokens, since +the shift to a new device-node will not occur until run-time. + +The commands new-device +and finish-device +must remain in balance.  If a finish-device  +is encountered without a prior corresponding new-device, +or if the end of FCode (or a <a + href="#Resetting_Symbols_Defined_in_Either_Mode">Reset-Symbols</a> +directive issued in "normal" mode) is reached and not all occurrences +of new-device +are balanced by a call to finish-device, +it will be reported as an <a href="#An_ERROR">Error</a> condition. +Definitions made in "Tokenizer-Escape" mode, however, are +independent of device-node vocabularies and remain accessible until +they +are explicitly reset by a <a + href="#Resetting_Symbols_Defined_in_Either_Mode">Reset-Symbols</a> +directive issued in "Tokenizer-Escape" mode. + +</div> +<h6 style="margin-left: 120px;"><a name="mozTocId540605"></a><big><big><a + name="Global_Definitions"></a>Global +Definitions</big></big></h6> +<div style="margin-left: 120px;">Occasionally, the User might need to +create definitions of methods that are +intended to be directly accessible to all the device-nodes in a +driver, even though they reside in one device +node.  Such +definitions are, effectively, added to the "core" vocabulary, and are +Global in scope.  <a href="#Scope_Directives">Directives</a> to +control the creation of Global Definitions are supported by this +Tokenizer.</div> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId354426"></a><big><a + name="Multiple_FCode-block_Images"></a>Multiple +FCode-block Images</big></h5> +<div style="margin-left: 80px;">In addition to supporting complexes of +subordinate and peer devices within a single assembly, this Tokenizer +supports configurations in which subordinate devices' drivers are kept +in separate but related FCode blocks.  This is accomplished simply +by invoking multiple bodies of code bracketed by  fcode-version<n>  +and fcode-end +(or one of its synonyms). +When the fcode-end +(or equivalent) that ends one body of code is processed, and before +the fcode-version<n> +that begins the next, the definitions that had been created are +forgotten, but assignment of FCode-token numbers will continue in +sequence.  Likewise, definitions made in "Tokenizer-Escape" mode +will persist. + +The User who desires to reset one or the other of these, or both, +can do so by issuing the directives: + +<div style="margin-left: 40px;">FCode-Reset + +<div style="margin-left: 40px;">or +</div> +</div> +<div style="margin-left: 40px;">Reset-Symbols +</div> +</div> +<div style="margin-left: 40px;"> +<div style="margin-left: 40px;"> +respectively, in "Tokenizer-Escape" mode.  This is illustrated in <a + href="#Example_3">Example #3</a>. +</div> +</div> +<h5 style="margin-left: 80px;"><a class="mozTocH5" name="mozTocId232809"></a><big><a + name="Multiple_PCI_Image"></a>Multiple +PCI-Image</big></h5> +<div style="margin-left: 80px;">The PCI Standard allows for a chain of +PCI Images to be bundled together in a single binary (as indicated by +the <a href="#Last_Image_Indicator_bit">"Last Image" bit</a>).  +This Tokenizer supports such configurations by correctly processing +Source that contains multiple occurrences of blocks of code bracketed +by pci-header +... pci-header-end  +directives.  An illustration can be seen in <a href="#Example_1">Example +#1</a>, below. +</div> +<h2><a class="mozTocH2" name="mozTocId194505"></a><a name="Examples:"></a>Examples:</h2> +<h3><a class="mozTocH3" name="mozTocId544211"></a><a name="Example_1"></a>Example +#1:</h3> +In the first example, the "Inner Body" file has some sections that +compile +different code, depending on the value of a named-constant +switch.  The "Outer Shell " file controls the floading +of the +"Inner Body":  It reminds the User to specify the setting of the +switch, then produces an image with two PCI-Images, based on opposite +settings of the switch.  Note that the use of Reset-Symbols +in "Tokenizer-Escape" mode before the redefinition of function-switch, +to avoid a "Duplicate definition" condition, may be omitted if the User +is willing to tolerate the warning. +<h5><a class="mozTocH5" name="mozTocId563670"></a>File:  +InnerBody.fth</h5> +fcode-version2 +... Common code <obligatory +sneeze> ... +F[   +function-switch   F]   +[if] +    True-conditional +code +      +[message]   The true +path was wisely chosen +[else] +    +False-conditional code +      +[message]   You have +foolishly chosen the false path. +[then] +...  +More common code  <now cough>. +end0 +<h5><a class="mozTocH5" name="mozTocId570875"></a>File:  +OuterShell.fth</h5> +[ifndef]  first-path +  +F[ + ."  Add a +command-line switch: +-d ""first-path=<true|false>"" "n"\ +     "tthen run +this again." +    +F] +[else] +  F[ +   [defined] +first-path  constant +function-switch +<div style="margin-left: 40px;">vendor-id  +device1-id  class-code1   pci-header +     +F] +not-last-image +fload InnerBody.fth +pci-header-end + +</div> +  F[  reset-symbols +   [defined] +first-path  0=  constant function-switch +<div style="margin-left: 40px;">vendor-id  +device2-id  class-code2   pci-header +     +F] +last-image +fload InnerBody.fth +pci-header-end +</div> +[then] +<h5><a class="mozTocH5" name="mozTocId953372"></a>Command-Line +invocation:</h5> +toke +-v -d "first-path=true" +OuterShell.fth + +<h3><a class="mozTocH3" name="mozTocId80432"></a><a name="Example_2"></a>Example +#2: +</h3> +The second example illustrates the use of +<a href="#Special_Feature_Flags">Special-Feature +Flags</a> to select or de-select +specific non-standard +features. + +The User is developing code that will run across all platforms, and +therefore must +must be neutral with regard to "Sun"- or "Apple"- -style usage of ABORT" +; this can be best achieved by disallowing the use of ABORT" +altogether. + +There is no concern, however, about compatibility of the Source with +other Tokenizers, so the User need not forgo the conveniences of Local +Values and String-remark-escapes. + +Furthermore, the Source contains many passages taken from IBM Legacy +sources, and the User does not wish to see a WARNING message when the +Legacy Locals Separator is used. + +The command-line for these conditions would include the following: +toke +-f NOabort-quote -f local-values -f NOlv-legacy-warning +Alternatively, these flags may be set from <a + href="#Changing_C-L_Flags_from_Source">within the Source code</a> +thus: + +<div + style="margin-left: 40px; color: rgb(68, 34, 0); font-family: courier new,courier,monospace; font-weight: bold;">[flag] +NOabort-quote +[flag] local-values +[flag] NOlv-legacy-warning +</div> +Note that the invocation of  -f +local-values is necessary, as its default state is to be +disabled.  +Also, because  -f +NOabort-quote  is invoked, the setting of the  Sun-ABORT-Quote  +flag is irrelevant. + +<h3><a class="mozTocH3" name="mozTocId455242"></a><a name="Example_3"></a>Example +#3:</h3> +The third example illustrates two situations where two or three +<a href="#Multiple_FCode-block_Images">FCode blocks</a> +are incorporated into a single PCI Image. + +In the first situation, the "Outer" block will byte-load +the code of +the "Inner" block before the "Outer" block has been completely +interpreted.  It is therefore important to avoid collisions in +FCode-token numeric assignments. + +<div + style="color: rgb(68, 34, 0); font-family: courier new,courier,monospace; margin-left: 40px;"> +F[ +<div style="margin-left: 40px;">h# 5afe  set-rev-level + +h# beef        +\ Vendor +h# c0de        +\ Device ID +h# 90210       \ +Class Code   (A "classy" ZIP Code... ;-) +</div> +F]     pci-header + +fcode-version2 + +fload +outer-block.fth + +fcode-end + +\  +At this point, the last definition's assigned FCode number +is approximately 0Xac0 + +fcode-version2 + +\  The next +definitions' assigned FCode numbers continue from +0Xac1 + +fload +inner-block.fth + +fcode-end + +pci-header-end + +</div> +In the second situation, the "Outer" block will have been completely +interpreted before it begins to byte-load +the code of the two "Inner" blocks.  It can safely discard the +token-table of the interpreter, and allow its assigned FCode-token +numbers to be re-cycled.  Furthermore, the large number of +definitions presents a real risk that the full range of usable +FCode-token numbers will be exhausted.  For these reasons, the +User finds it necessary to reset the FCode-token numeric assignments to +their initial state before tokenizing the two "Inner" blocks.  + +<div + style="margin-left: 40px; color: rgb(68, 34, 0); font-family: courier new,courier,monospace;"> +F[ +<div style="margin-left: 40px;"> +h# face   +   set-rev-level +h# cafe   +            \ +Vendor +h# d00d   +            \ +Device ID +h# 95014   +           \ Another Class(y +ZIP) Code</div> +F]     pci-header + +fcode-version2 + +fload outer-block.fth + +fcode-end + +\  At this +point, the last definition's assigned FCode number +is approximately 0Xae0 + +fcode-version2 + +\  +Reset the assigned FCode numbers for the first "inner" block +  +FCode-Reset +  +fload inner-block_01.fth + +fcode-end + +\  Load the second "inner" block. + +fcode-version2 + +\  Reset +the assigned FCode numbers for the second "inner" block + +FCode-Reset + +fload inner-block_02.fth + +fcode-end + +pci-header-end +</div> + +<h3><a class="mozTocH3" name="mozTocId861304"></a><a name="Example_4"></a>Example +#4</h3> +This example serves to illustrate the use of the directives that +control the +<a href="#Scope_Directives">Scope of Definitions</a>, and also to show +a means whereby the IBM-Style +Local Values +Support File can be incorporated at a <a href="#Global_Definitions">Global</a> +level.  Normally, <a href="#Multiple_device_nodes">that would be +problematical</a> because +the <a href="TokenizerLocalValues.htm#Instance_Data_in_L_V_Support">Local +Values +Support functions</a> are written to use instance +data, in order to conserve use of System memory.  By temporarily +over-riding the definition of instance +in the <a href="#over_ride_instance">manner shown</a>, the User has +traded-off economy of System-memory +for convenience of programming. + +This example also offers an illustration of the use of <a + href="#Shell_Env_Vbles_in_File_Names">Shell-Environment Variables in +File-Names</a>:  Let us suppose that the directory in which the +main file for the controller of the assembly whose driver being +compiled here resides in a directory-tree, several layers under the +root.  Immediately below the root is a sub-directory called shared +that contains shared functions and the Local Values support, and +elsewhere under the tree are sharable bodies of driver code for +device-modules that can be incorporated into various assemblies.  +Let us also suppose that the Makefile that governs this Tokenization +process sets Environment Variables; the root of the tree is called DevRoot, +and the directory in which the sharable bodies reside are called SCZ +and SLZ +respectively.  And let us further suppose that the <a + href="#Example_4_Condl_FLoad">inclusion of these two subsidiary devices</a> +is +optional, controlled by command-line symbol definitions. +<div + style="margin-left: 40px; color: rgb(68, 34, 0); font-family: courier new,courier,monospace;"> +\ Define a few macros and functions that will become available +\ to all subordinate device-nodes in this bundle. + +\ Some of our functions are written using Local Values. +\ We can make the Local Values Support accessible Globally +\     without incurring any Warning messages, and +without +\     altering the LocalValuesSupport file, if we +temporarily +\     disable the definition of INSTANCE + +fcode-version2 + +headers + +global-definitions + +<a name="over_ride_instance"></a>\  Bypass warning about  +Instance +\      without altering +LocalValuesSupport file + +\  Leave ourselves a way to recover +alias generic-instance  +instance + +\  Here's where we disable it.  During development, give +confirmation +\      that the macro has been activated by +printing a message.  Also, +\      insert a token in the place where +the  Instance  would have +\      gone so that our detokenizations line +up +when we inspect.  We +\      do this by defining a comand-line +symbol called  proto-code +\  For the final production run, we can make the macro completely +\      silent by leaving  proto-code  +undefined. +[ifdef] proto-code +    overload [macro] +instance  message" +Bypassing Instance"  noop +[else] +       \  Make it completely silent +    overload [macro] +instance f[  noop  ]f +[endif] + +fload ${DevRoot}/shared/LocalValuesSupport.fth + +\  Restore normal meaning of  Instance , also in Global scope. +overload alias instance +generic-instance + +\  Here's a global definition that uses Local Values +: $CAT   ( _max _str1 _len1 _str2 _len2 -- _max _str1 _len1' ) +   { _max _str1 +_len1 _str2 _len2 } +   _len1 _max < +if                    +\ there is room +      _str2 _str1 _len1 + _len2 _max _len1 - +min move +   then +   _max _str1 _len1 _len2 + _max min   \ always leave +total length +; + +\  Here is a global macro +[macro] 4DUP   2over 2over + +\  And another +[macro] 4DROP  2drop +2drop + +\  And yet another +[macro] (.h)  base @ +swap hex (.) rot base ! + +\  Other shared functions are in a file in the Shared Code +directory: +fload +${DevRoot}/shared/sharedfuncts.fth + +\  Definitions in all subsequent device-nodes will be able to +access +\      the $CAT function and the macros  +4DUP  and  +4DROP, as well as +\      to make use of IBM-Style Local Values +without re-floading +\      the Support File + +\  Now let's get back into the primary device-node. +device-definitions + +\  Use instance data to create a large temporary buffer +d# 40 dup instance buffer:  temp-buf      +( max ) + +\  Convolutions to create a name on the fly... +s" controller_" tuck   +                 +( max len adr len ) +temp-buf +swap                       +    ( max len adr buf len ) +move                                +    ( max len ) +temp-buf +swap                       +    ( max buf len ) +my-self +(.h)                            +( max buf len adr2 len2 ) +$cat                                    +( max buf len ) +device-name drop + +\  Make a macro of that.  Define it in parts. +\  Requires the core of the name on the stack. +\  Macro will create its own  temp-buf  in the current +device. + +global-definitions + +\  First part of the operation:   ( adr len -- +max adr len ) +\  Note that "instance" is allowed here because it is not +\      invoked -- or even parsed --  +until the macro is invoked. +[macro]  +create-buffr  d# 40 dup instance buffer: temp-buf  -rot + +\  Second part of the operation:  ( max adr len -- max buf +len ) +[macro]  +name-to-buffr   tuck temp-buf swap move temp-buf swap + +\  Last part of the operation:  ( max buf len -- ) +\  We're using  my-self  as part of the device-name +\      for no particularly good reason other than +\      that it makes for an interesting example...  +;-} +[macro]  +buffr-to-dev-name  my-self (.h) $cat device-name drop + +\  Combine the parts: +[macro]  make-my-dev  +create-buffr  name-to-buffr  buffr-to-dev-name + +\  Back to the primary device-node. +device-definitions + +\  Load its methods from the Current Directory +fload controller_methods.fth + +<a name="timestamp_prop"></a>\  Create timestamp property based on +actual tokenization date and time +\  Combine into a single string (with a space between). +\      No trailing null-byte until end. +[fcode-date] encode-bytes   "  " encode-bytes encode+ +[fcode-time] encode-string encode+ " release-time" property + +\  Log timestamp to the Audit Trail +tokenizer[   [fcode-date] [fcode-time]  ]tokenizer + +<a name="Example_4_Condl_FLoad"></a>\  First optional subsidiary +device.  Inclusion controlled by command-line symbol. +[ifdef] scuzzy +<div style="margin-left: 40px;">new-device + +" scuzzy_"  make-my-dev + +\  Load its methods +fload ${SCZ}/scuzzy_methods.fth + +finish-device +</div> +[endif] + +\  Second optional subsidiary device, controlled by command-line +symbol. +[ifdef] sleazy +<div style="margin-left: 40px;">new-device + +" sleazy_"  make-my-dev + +\  Load its methods +fload ${SLZ}sleazy_methods.fth + +finish-device +</div> +[endif] + +\  That's enough for now... +fcode-end +</div> +<h3><a class="mozTocH3" name="mozTocId866180"></a><a name="Example_5"></a>Example +#5</h3> +This example serves to illustrate the use of the <a + href="#function_name">[function-name]</a> +directive to create a series of "you are here" debugging messages. + +We will create a pair of macros whose names we can cut'n'paste at +the beginning and end of every function we want to give a "you are +here" message, switchable by a local debug-flag.  The macros will +be globally defined, but will make reference to locally-defined names +for the debug-flag and the device-node name string. +<div + style="margin-left: 40px; color: rgb(68, 34, 0); font-family: courier new,courier,monospace;"> + +fcode-version2 + +global-definitions +\  Each dev-node will create its own debug-flag and alias it +to  debug-me? +\  Each dev-node creates a macro called my-dev-name giving its +device-name + +[macro] .fname&dev  [function-name] type ."  in " my-dev-name type +    +[macro] name-my-dev   +my-dev-name device-name + +[macro] .dbg-enter  +debug-me? @ if ." +Entering " .fname&dev  +cr then +[macro] .dbg-leave  +debug-me? @ if ." +Leaving "  +.fname&dev  cr then +device-definitions + +headers +\  Top-most device, named billy +[macro] my-dev-name  " +billy" +name-my-dev + +\  debug-billy?  is a flag the user can turn "on" to get +debug-messages +\  from the methods of the device we call "billy" +variable debug-billy?  +debug-billy? off +\  Set it up for the macro +alias debug-me? debug-billy? + +: bill +    +.dbg-enter            +\  This will display  Entering bill in billy +    [char] G dup +    control G 3drop +    +.dbg-leave            +\  This will display  Leaving bill in billy + +; + +: factl recursive  ( n -- n! ) +type     ." Entering First vers. of " .fname&dev cr +    ?dup 0= if +1 +    else  dup 1-  +factl * +    then +    ." Leaving +First vers. of " +.fname&dev type cr +; + +: factl ( n -- n! ) +type     ." Entering Second vers. of " [function-name] +cr +    ?dup 0= if +1 factl +    else  dup 1- +recurse * +    then +type     ." Leaving Second vers. of " [function-name] +cr +; + +variable naught +defer  do-nothing +20 value twenty +30 value thirty +40 buffer: forty +50 constant fifty +create three 0 , 00 , h# +000 , +struct +4 field >four +constant /four + +: peril +    +.dbg-enter            +\  This will display  Entering peril in billy +    ['] noop to do-nothing +    100 to +thirty +    5 to +naught              +\  Generates a WARNING +    thirty dup - abort" +Never Happen" +    +.dbg-leave            +\  You get the idea... +; + +: thirty ( new-val -- ) +    .dbg-enter +    dup to thirty +    alias .dec .d              +\  Generates a WARNING and an ADVISORY +    ." Dirty"  .dec +    .dbg-leave +; + +\  First subsidiary device, "child" of billy +new-device +    instance variable cheryl +    [macro]  +my-dev-name  " cheryl" +    name-my-dev + +    \  Third-level device, "grandchild" of billy +    new-device +        [macro]  my-dev-name  " meryl" +        name-my-dev +        variable debug-meryl?  +debug-meryl? off +        alias debug-me? debug-meryl? + +        : merle +        +.dbg-enter           +\  This will display  Entering merle in meryl +            +cheryl                 +\  ERROR.  Is in a different dev-node. +        .dbg-leave +        ; +    variable +beryl +    finish-device + +    \  Now we're back to "cheryl" +    +    variable +debug-cheryl?  debug-cheryl? off +    alias debug-me? debug-cheryl? + +     : queryl +    +.dbg-enter           +\  This will display  Entering queryl in cheryl +    over +rot dup nip drop swap   \  Not the most useful +code...  ;-} +    .dbg-leave +     ; +finish-device +\  And we're back to billy. +: droop ( -- ) +    +.dbg-enter            +\  This will display  Entering droop in billy +    twenty +    0 ?do i .h loop +    +.dbg-leave            +\  You get the idea.... +; + +<a name="Example_5_func_nam"></a> +\  The following will generate some unexpected errors: +: quack ( maybe not -- oops!  I-forgot-the-close-paren +    +.dbg-enter            +\  This should display  Entering quack in billy +                          +\  But it doesn't display anything and I don't know why. +    h# 30  .d +    +.dbg-leave            +\  and I'm still baffled... +;      \  Oh, well, soldier on... +: cluck  ( don't count -- before hatched )  \  Now, there's a close-paren.... +</div> + +<div + style="color: black; font-family: helvetica,arial,sans-serif; font-weight: bold;">Note:  At this point, +the Tokenizer has issued a <a href="#Allow_Multi_Line">Multi-Line +Warning</a> about a Comment that +began six lines earlier, which gives the User a good indication as to +just what +went wrong... </div> + +<div + style="margin-left: 40px; color: rgb(68, 34, 0); font-family: courier new,courier,monospace;">    +.dbg-enter            +\  This does display, but not what I expected. +    ." Coming from" <a href="#Inp_Fil_Nm_Strg">[input-file-name]</a> +type ."  line +" <a href="#Inp_Fil_Nm_Strg">[line-number]</a> .d cr +tokenizer[  <a href="#function_name">[function-name]</a>   +]tokenizer    \   Let's not wait for run-time +    +.dbg-leave            +\  Now I get it... +; +fcode-end + +</div> + +<h2><a class="mozTocH2" name="mozTocId856440"></a>End Of Document</h2> + +</body> +</html> Added: fcode-utils/documentation/workbook.css =================================================================== --- fcode-utils/documentation/workbook.css (rev 0) +++ fcode-utils/documentation/workbook.css 2006-10-30 16:58:53 UTC (rev 109) @@ -0,0 +1,426 @@ +/************ This page modified on March 14, 2001 by Sharon Rempert ***************/ +.title +{ + text-align: right; +} +/*table*/ +table +{ + padding: 3px; + border: 1px; + margin-left: 20px; +} +.small +{ + font-size: 10pt; +} + +/* body */ +body +{ + margin: 1em 2em 2em 1em; + font-family: sans-serif; + color: black; + background: white; +} + +.draft +{ + margin: 1em 2em 2em 2em; + color: black; + background: white url(/:/projects/esw/process/doc/draft.gif) no-repeat fixed top left; +} + +p, a,ul,dt,li,dd,ol,b,i,blockquote,font +{ + font-family: sans-serif; +} + +address +{ + font-size: 10pt; + text-align: center; + font-family: sans-serif; +} + +.logo +{ + text-align: center; +} + +/*text*/ + + +p.adoption +{ + color: #0050B2; + background: white; + font-size: large; +} + +p.copyright {font-size: small;} + +p.warning +{ + color: #901010; + background: white; + margin: 1em 4em 1em 4em; + border: solid red; +} + +.note1 +{ + width: 80%; + border: solid #800080; +} +.note +{ + color: green; + background: white; +} + +pre +{ + white-space: pre; + font-family: monospace; + margin-left: 2em; + color: maroon; + background: white; +} + +pre.example +{ + white-space: pre; + color: #203838; + background: #B0E0E0; + margin: 1em 4em 1em 4em; + padding: 0.5em 0.5em 0.5em 0.5em; + font-family: monospace; + border: none; +} + +pre.code +{ + white-space: pre; + color: maroon; + background: #eed6b4; + margin: 1em 4em 1em 0em; + padding: 1em 1em 1em 1em; + font-family: monospace; + border: none; +} + +h1, h2, h3, h4, h5, h6 +{ + font-family: sans-serif; + text-align: left; + font-weight: bold; + color: #0050B2; + background: white; +} + +.special +{ + color: #FF0000; +} +.spec_head +{ + color: #FF0000; + text-decoration: underline; + font-weight: bold; +} +.spec_pre +{ + font-style: oblique; +} + + +h1.title +{ + text-align: left; +} + +ul.toc +{ + list-style: none; +} + +dl { margin-top: 1em; } + +th, td +{ + font-family: sans-serif; + vertical-align:top; +} +.command +{ + font-weight: 700; + color: #0050B2; + background-color: white; +} + +.description +{ + margin: 40px 0px 0px 0px; + font-family: sans-serif; +} + +/*media*/ + +@media screen +{ + a:hover + { + background: #FFFFAA; + color: blue; + } +} + +@media aural +{ + h1, h2, h3 + { + stress: 20; + richness: 90; + } + + .hide {speak: none;} + + p.copyright + { + volume: x-soft; + speech-rate: x-fast; + } + + dt {pause-before: 20%;} + pre {speak-punctuation: code;} +} + +/* inline markup */ + +em {font-style: italic;} + +strong {font-weight: bold;} + +.emp +{ + font-weight: bold; + text-decoration: underline; +} + + +.strongerEmp +{ + font-weight: 900; + font-style: italic; +} + +cite {font-style: italic;} + +dfn +{ + font-weight: 700; + font-style: italic; +} + +var +{ + font-style: italic; + font-family: monospace; + color: #0000CD; + background: white; +} + +acronym +{ + font-variant: small-caps; + font-style: normal; +} + +abbr {font-style: normal;} + +code +{ + font-style: normal; + font-family: monospace; + color: #a72ea0; + background: white; +} + +samp +{ + font-style: normal; + font-family: monospace; + color: #a723a0; + background: white; +} + +kbd +{ + font-style: italic; + font-family: monospace; + color: #0000CD; + background: white; +} + +/*hide*/ + +.hide {display: none} + +/*span*/ + +span.doc-string +{ + /* purple */ + color: purple; + background-color: white; +} + +span.reference +{ + /* red3 */ + color: rgb(205,0,0); + background-color: white; +} + +span.type +{ + /* MediumPurple3 */ + color: rgb(137,104,205); + background-color: white; +} + +span.function-name +{ + /* forestgreen */ + color: rgb(34,139,34); + font-weight: bold; + background-color: white; +} + +span.string +{ + /* blue4 */ + color: rgb(0,0,139); + background-color: white; +} + +span.comment +{ + /* red */ + color: red; + font-style:italic; + background-color: white; +} + +span.variable-name +{ + /* blue3 */ + color: rgb(0,0,205); + background-color: white; +} + +span.preprocessor +{ + /* blue3 */ + color: rgb(0,0,205); + background-color: white; +} + +span.keyword +{ + /* goldenrod3 */ + color: rgb(205,155,29); + background-color: white; +} + +.lists +{ + font-size: 14pt; + font-family: sans-serif; + text-align: left; + font-weight: bold; + color: #0050B2; + background: white; +} + + +.text +{ + font-family: sans-serif; +} + +/*div*/ + +.subtoc +{ + /*padding: 1em;*/ + border: solid thin; + /*margin: 1em 0;*/ + color: white; + background: #d0d0d0; +} + + +div.head {margin-bottom: 1em;} + +div.head h1 +{ + margin-top: 2em; + clear: both; +} + +div.head table +{ + margin-left: 2em; + margin-top: 2em; +} + +div.head img +{ + color: black; + background: white; + border: none; +} + +.document-control_2 +{ + border: solid #404040; + color: black; + background: #e0e0e0; +} + +.document-control +{ + margin-top: 1em; + margin-left: 10%; + margin-bottom: 1em; + border: solid #404040; + color: black; + background: #e0e0e0; +} + +/* why doesn't dt work here? */ +div.document-control dl dt +{ + font-weight: bold; +} + + +/*links*/ + +a.normref +{ + color: red; + background: white; +} + +a.informref +{ + color: green; + background: white; +} +.no_bullet +{ + list-style-type: none; +}

1 0

r108 - in fcode-utils: . romheaders
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 17:29:59 +0100 (Mon, 30 Oct 2006) New Revision: 108 Added: fcode-utils/COPYING fcode-utils/README fcode-utils/TODO fcode-utils/romheaders/COPYING Log: some more infrastructure Added: fcode-utils/COPYING =================================================================== --- fcode-utils/COPYING (rev 0) +++ fcode-utils/COPYING 2006-10-30 16:29:59 UTC (rev 108) @@ -0,0 +1,341 @@ + + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 51 Franklin St, Fifth Floor, Boston, MA, 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) 19yy <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA, 02110-1301 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) 19yy name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. Added: fcode-utils/README =================================================================== --- fcode-utils/README (rev 0) +++ fcode-utils/README 2006-10-30 16:29:59 UTC (rev 108) @@ -0,0 +1,16 @@ +This is the OpenBIOS FCODE suite. + +It consists of: + +- toke +- detok +- romheaders +- localvalues + +NOTE: all parts of the OpenBIOS FCODE suite are GPL v2, except the localvalues +implementation which is released under the Common Public License (CPL) version +1.0 (see: http://www.opensource.org/licenses/cpl1.0.php) + + +For information please contact info(a)coresystems.de + Added: fcode-utils/TODO =================================================================== --- fcode-utils/TODO (rev 0) +++ fcode-utils/TODO 2006-10-30 16:29:59 UTC (rev 108) @@ -0,0 +1,298 @@ +fcode-utils/detok/decode.c + Line 46: + * + * Still to be done: + * Handling of indent-level is not correct. Branches should + * balance with their resolvers; constructs like do..loop + * case/of/endof/endcase are a few major examples. + * This will be tricky; the rules need to be carefully thought + * out, and the implementation might be more complex than + * at first meets the eye... + * + + Line 280: + * + * Still to be done: + * More sophisticated error-checking for invalid offsets. Look + * at the type of branch, and what should be expected in the + * vicinity of the destination. (This might be best served + * by a separate routine). + * This might also help with handling the indent-level correctly... + * Also, if indentation were to be handled in this routine, + * there would be no need to return the value of the offset. + * + + +fcode-utils/detok/pcihdr.c + Line 67: + * + * Still to be done: + * Print (as remarks) full descriptions of headers' fields + * Error check for wrong "Format" + * Skip past non-FCode blocks, thru multiple data-blocks + * Recognize PCI header in unexpected place or out-of-place + * + + Line 130: + * + * Still to be done: + * Error-check; look for inconsistencies: + * Return a Negative Number if data-stream appears to be a PCI + * header, but has erroneous or inconsistent sub-field contents. + * Value and meaning of the Negative Number yet to be defined. + * + + Line 188: + * + * Still to be done: + * Error-check; look for wrong "Code Type" or other inconsistencies: + * Return a Negative Number if data-stream appears to be a + * valid PCI Data Structure, but has erroneous or inconsistent + * sub-field contents. + * Value and meaning of the Negative Number yet to be defined. + * Skip past non-FCode data-blocks, even multiple blocks + * + + Line 364: + * + * Still to be done: + * Handle error cases. At present, neither is_pci_header() + * nor is_pci_data_struct() returns a negative number, + * but when they are modified to do so, we must handle it. + * + + Line 442: + * + * Still to be done: + * Come up with something more elegant for non-zero filler. + * + + +fcode-utils/detok/printformats.c + Line 74: + * + * Still to be done: + * Define a routine, call it PrintComment , to print the given + * string surrounded by open-paren-space ... space-close-paren + * Define a single central routine, call it safe_malloc , + * to do the test for null and print "No Memory" and exit. + * Define a single central routine, call it PrintError , to: + * Print the given error message + * Show the input file and line number + * Collect error-flags for failure-exit at end of operation. + * + + +fcode-utils/toke/devnode.c + Line 67: + * + * Still to be done: + * Add a pair of fields to the data-structure for the Input File and + * Line Number where "finish-device" occurred. When a device-node + * is "finish"ed, do not delete it, but instead fill in those + * fields and the move the node to a separate linked-list. + * When looking whether a word exists in an ancestor-node, also + * check whether it was in a device-node that was finished and + * print both where it was started and where it was finished. + * + + +fcode-utils/toke/emit.c + Line 34: + * + * Still to be done: + * Re-arrange routine and variable locations to clarify the + * functions of this file and its companion, stream.c + * This file should be concerned primarily with management + * of the Outputs; stream.c should be primarily concerned + * with management of the Inputs. + * Hard to justify, pragmatically, but will make for easier + * maintainability down the proverbial road... + * + + +fcode-utils/toke/flowcontrol.c + Line 75: + * + * Still to be done: + * Correct analysis of Return-Stack usage around Flow-Control + * constructs, including within Do-Loops or before Loop + * Elements like I and J or UNLOOP or LEAVE. + * Similarly, Return-Stack usage around IF ... ELSE ... THEN + * statements needs analysis. For instance, the following: + * + * blablabla >R yadayada IF R> gubble ELSE flubble R> THEN + * + * is, in fact, correct, while something like: + * + * blablabla >R yadayada IF R> gubble THEN + * + * is an error. + * + * Implementing an analysis that would be sufficiently accurate + * to justify reporting an ERROR with certainty (rather than + * a mere WARNING speculatively) would probably require full + * coordination with management of Flow-Control constructs, + * and so is noted here. + * + + +fcode-utils/toke/macros.c + Line 166: + * + * Still to be done: + * If an error is encountered during Macro evaluation, display + * supplemental information giving the name of the Macro + * being run, and the file and line number in which it was + * defined. + * This will require changes to the way user Macros are added + * and retained, and to the way error messages are displayed. + * + + +fcode-utils/toke/nextfcode.c + Line 90: + * + * Still to be done: + * Detect and report when the Current Range stops overlapping + * one particular Range and starts overlapping another. + * + + +fcode-utils/toke/scanner.c + Line 1046: + * + * Still to be done: + * Better protection against PC pointer-over-run past END. + * Currently, this works, but it's held together by threads: + * Because init_stream forces a null-byte at the end of + * the input buffer, parse_number() exits immediately upon + * encountering it. This situation could be covered more + * robustly... + * + + Line 2192: + * + * Still to be done: + * Full detection of whether the Return-Stack has been cleared + * when required, including analysis of Return-Stack usage + * within Flow-Control constructs, and before Loop elements... + * + + Line 2259: + * + * Still to be done: + * Correct analysis of Return-Stack usage around Flow-Control + * constructs. Consider, for instance, the following: + * + * blablabla >R yadayada IF R> gubble ELSE flubble R> THEN + * + * It is, in fact, correct, but the present scheme would + * tag it as a possible error. Conversely, something like: + * + * blablabla >R yadayada IF R> gubble THEN + * + * would not get tagged, even though it is actually an error. + * + * The current simple scheme also does not cover Return-Stack + * usage within Do-Loops or before Loop elements like I and + * J or UNLOOP or LEAVE. Implementing something like that + * would probably need to be integrated in with Flow-Control + * constructs, and will be noted in flowcontrol.c + * + + Line 2323: + * + * Still to be done: + * Correct analysis of Return-Stack usage... + * + + Line 3299: + * + * Still to be done: + * If the definer-name is not found, we might still look up + * the target name in the various vocabularies and use + * a phrase for those. E.g., if it is a valid token, + * we could say it's defined as a "primitive". (I'm + * not sure what we'd say about an FWord...) + * + + Line 4617: + * + * Still to be done: + * Correct analysis of Return-Stack usage within Do-Loops + * or before Loop Elements like I and J or UNLOOP or LEAVE. + * + + +fcode-utils/toke/stream.c + Line 60: + * + * Still to be done: + * Re-arrange routine and variable locations to clarify the + * functions of this file and its companion, emit.c + * This file should be concerned primarily with management + * of the Inputs; emit.c should be primarily concerned + * with management of the Outputs. + * Hard to justify, pragmatically, but will make for easier + * maintainability down the proverbial road... + * + + Line 1070: + * + * Still to be done: + * Set a flag when carr-ret has been replaced by space; + * when a string crosses a line, if this flag is set, + * issue a warning that an extra space has been inserted. + * + + +fcode-utils/toke/toke.c + Line 363: + * + * Still to be done: + * Devise a syntax to allow the command-line to specify multiple + * input files together with an output file name for each. + * Currently, the syntax allows only one output file name to be + * specified; when multiple input file names are specified, + * the specification of an output file name is disallowed, + * and only the default output file names are permitted. + * While this works around the immediate problem, a more + * elegant solution could be devised... + * + + +fcode-utils/toke/usersymbols.c + Line 78: + * + * Still to be done: + * Convert the handling of user-defined symbols to the T.I.C. + * data-structure and its support routines. This should + * eliminate any further need of String-Substitution-type + * vocabularies. User-defined symbols will, however, still + * need to be a separate vocabulary from the Global, because + * they are required to stay in effect for the duration of + * the entire batch of tokenizations... + * (Afterthought: This is only true for user-defined symbols that + * were created on the command-line; if we ever allow symbols + * to be defined in the Source file, they should be as volatile + * as anything else that comes from a source file... + * Appending source-file-derived user-defined symbols to the Global + * Vocabulary could be a quasi-simple way to accomplish this.) + * + + Line 246: + * + * Still to be done: + * Hook-in this routine to the processing of: ['] F['] H# FLOAD + * etc., and wherever else it might be needed or useful. + * + + Line 316: + * + * Still to be done: + * Space the duplicate-name notation evenly; line it up past + * the longest name-with-value. + * + + Added: fcode-utils/romheaders/COPYING =================================================================== --- fcode-utils/romheaders/COPYING (rev 0) +++ fcode-utils/romheaders/COPYING 2006-10-30 16:29:59 UTC (rev 108) @@ -0,0 +1,341 @@ + + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 51 Franklin St, Fifth Floor, Boston, MA, 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) 19yy <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA, 02110-1301 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) 19yy name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License.

1 0

r107 - in fcode-utils/testsuite: TokBrack TokCondl TokMisc TokeErrs TokoLoco
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 17:25:35 +0100 (Mon, 30 Oct 2006) New Revision: 107 Added: fcode-utils/testsuite/TokBrack/TokBrkTst_04.fth fcode-utils/testsuite/TokMisc/ContErr0.fth fcode-utils/testsuite/TokMisc/DtkVSFcTst.scr fcode-utils/testsuite/TokMisc/VSFCtest.fth fcode-utils/testsuite/TokMisc/VendSpecFCodes fcode-utils/testsuite/TokeErrs/DecodProp.fth fcode-utils/testsuite/TokoLoco/GlobalLocalValuesDevel.fth fcode-utils/testsuite/TokoLoco/LocalValuesDevelSupport.fth fcode-utils/testsuite/TokoLoco/LocalValuesSupport.fth fcode-utils/testsuite/TokoLoco/TotalLocalValuesSupport.fth Removed: fcode-utils/testsuite/TokMisc/OldTkzr/ Modified: fcode-utils/testsuite/TokBrack/TestArgs fcode-utils/testsuite/TokCondl/CondMacAlias_01.fth fcode-utils/testsuite/TokCondl/TestArgs fcode-utils/testsuite/TokMisc/AllBiFCTypes.fth fcode-utils/testsuite/TokMisc/ContErr3.fth fcode-utils/testsuite/TokMisc/MiscFeatures.DOS.fth fcode-utils/testsuite/TokMisc/MiscFeatures.fth fcode-utils/testsuite/TokMisc/StringsGenl.fth fcode-utils/testsuite/TokMisc/TestArgs fcode-utils/testsuite/TokeErrs/DevImbal.fth fcode-utils/testsuite/TokeErrs/DupNams.fth fcode-utils/testsuite/TokeErrs/MiscFeatErrs.fth fcode-utils/testsuite/TokeErrs/TestArgs fcode-utils/testsuite/TokoLoco/GlobalLocalValues.fth fcode-utils/testsuite/TokoLoco/TestArgs Log: update testsuite to 1.0.2 Modified: fcode-utils/testsuite/TokBrack/TestArgs =================================================================== --- fcode-utils/testsuite/TokBrack/TestArgs 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokBrack/TestArgs 2006-10-30 16:25:35 UTC (rev 107) @@ -10,7 +10,7 @@ # Lines that have no label, no switches, and no script # do not need any commas. -# Updated Wed, 08 Mar 2006 at 19:11 PST by David L. Paktor +# Updated Wed, 11 Oct 2006 at 24:44 PDT by David L. Paktor # Might as well get the date-and-time one out of the way first... TokBrkTst_02 , , -i -l @@ -18,3 +18,4 @@ TokBrkTst_01 , alwyshdr , -i -f ALWays-heADers TokBrkTst_03 TokBrkErrTst_01 , , -i +TokBrkTst_04 , , -i -T pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_aretz -T whatsis Added: fcode-utils/testsuite/TokBrack/TokBrkTst_04.fth =================================================================== --- fcode-utils/testsuite/TokBrack/TokBrkTst_04.fth (rev 0) +++ fcode-utils/testsuite/TokBrack/TokBrkTst_04.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,76 @@ +\ Test of long names and duplication of names and maybe Tracing +\ Updated Wed, 18 Oct 2006 at 13:34 PDT by David L. Paktor + +fcode-version2 + +global-definitions + +true constant flunky? + +alias whoosis whatsis +alias whatsis whoosis + +external +decimal + +100 constant pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_aretz + +device-definitions + +headers + +h# 3760 constant whatsis + +#message" Sync Up Diffs w/ prev. release."n" +alias whoosis pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_aretz +#message" +Sync Up again."n" +headerless +: pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_aretz + 100 +; +pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_aretz 2 * constant clone + +\ Make sure the matching goes all the way... +instance +: pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_oilum + clone + ['] whoosis execute +; + +new-device +80 constant this_name_has_a_whole_lot_of_syllables_and_so_would_not_be_a_good_ingredient_in_ice_cream_but_at_least_you_know_what_it_means + +headers + f[ 1cec6ea3 constant a_name_with_too_many_letters_should_not_matter_in_tokenizer_escape_mode + ]f + +: brand-x-ice-cream \ Because it uses ingredients with too many syllables + pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_aretz dup 20 - do i . loop cr + ." Nameless one..." this_name_has_a_whole_lot_of_syllables_and_so_would_not_be_a_good_ingredient_in_ice_cream_but_at_least_you_know_what_it_means ." equals " . cr + ." But everyone likes an " + f[ a_name_with_too_many_letters_should_not_matter_in_tokenizer_escape_mode + f] fliteral . + clone . cr +; + +flunky? if + d# 3760 constant whatsis +else + d# 374 constant whatsis +then + + +finish-device + +overload : clone + pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_aretz + pele_yoetz_ne-ema-an_anochi_hu_ha-omer_v-oseh_v-ain_c-moni_bchol_ha_oilum + whatsis + whoosis +; + +end0 + + + Modified: fcode-utils/testsuite/TokCondl/CondMacAlias_01.fth =================================================================== --- fcode-utils/testsuite/TokCondl/CondMacAlias_01.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokCondl/CondMacAlias_01.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -76,7 +76,7 @@ f[ ." Let's try some spurious [else] action, whaddya say?" ]f : [else] ." Don't do it" ; [message] Fake [else] got through create [else] [message] Fake [else] got through - h# defeca8e constant [else] [message] Fake [else] got through + h# DeFeCA8e constant [else] [message] Fake [else] got through h# -41100132 value [else] [message] Fake [else] got through d# 64 buffer: [else] [message] Fake [else] got through struct Modified: fcode-utils/testsuite/TokCondl/TestArgs =================================================================== --- fcode-utils/testsuite/TokCondl/TestArgs 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokCondl/TestArgs 2006-10-30 16:25:35 UTC (rev 107) @@ -10,7 +10,7 @@ # Lines that have no label, no switches, and no script # do not need any commas. -# Updated Thu, 27 Jul 2006 at 15:20 PDT by David L. Paktor +# Updated Fri, 13 Oct 2006 at 16:19 PDT by David L. Paktor TokExstCondTstY , , -l TokExstCondTstN , , -l Modified: fcode-utils/testsuite/TokMisc/AllBiFCTypes.fth =================================================================== --- fcode-utils/testsuite/TokMisc/AllBiFCTypes.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokMisc/AllBiFCTypes.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -1,7 +1,7 @@ \ Test all the known Built-In FCode tokens \ that have specific definition Types -\ Updated Wed, 16 Aug 2006 at 14:30 PDT by David L. Paktor +\ Updated Tue, 10 Oct 2006 at 11:00 PDT by David L. Paktor \ Applying "TO" to them ought to generate errors \ except for the ones that legitimately take "TO", @@ -9,7 +9,6 @@ fcode-version2 - [message] Constants. Should generate errors d# 10 to -1 ( CONST ) d# 10 to -1 ( CONST ) @@ -61,4 +60,12 @@ h# 12 to span ( VRBLE ) h# 12 to state ( VRBLE ) +multi-line #message" Using ['] on words that are both FWords and FCodes "\ + should generate no errors" +['] new-device drop +['] finish-device drop +['] offset16 drop +['] instance drop +['] end0 drop + fcode-end Added: fcode-utils/testsuite/TokMisc/ContErr0.fth =================================================================== --- fcode-utils/testsuite/TokMisc/ContErr0.fth (rev 0) +++ fcode-utils/testsuite/TokMisc/ContErr0.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,73 @@ +\ Test Erroneous Control Constructs, cunningly contrived +\ to pass the "Old" tokenizer, which doesn't have the +\ check for Control-Structure matching. + +\ Updated Thu, 29 Jun 2006 at 14:52 PDT by David L. Paktor + + +fcode-version2 + +headers + +: garfield + ." if" if + ." begin" begin + \ \ Leave this out because Old tokenizer duzzent dew it rite.... + \ ." Question-Leave?" ?leave + ." 0 if unloop exit then" 0 if unloop exit then + ." Would you be leave..." leave + ." loop?" loop + ." Done with garfield" +; + +: odie + ." 0 0 ?do" 0 0 ?do + ." i drop" i drop + ." zero if unloop exit then" 0 if unloop exit then + ." Who would be leave..." leave + ." again" again + ." then" then + ." Done with odie" +; + +." Outside of colon" +." 1 0 do" 1 0 do + i constant what? ." This is actually supposed to be legit..." +." again" again +." then" then +." Was that awful or what?" + +\ Snippet similar to something in Firmworks manual + +h# 5000 constant /DHCP-SCRATCH + +/DHCP-SCRATCH ( size ) ['] alloc-mem +." dhcp-scratch alloc-mem" cr .s cr +catch +." catch dhcp-scratch alloc-mem" cr .s cr +?dup if + ." alloc-mem Failed!!!" cr .s cr + throw + ." This is also worng..." exit +else + ." alloc-mem okay." .s cr + ( vaddr ) + ( vaddr ) constant DHCP-SCRATCH +then + +\ A CASE statement where the ENDOFs are missing +\ still passes the "Old" tokenizer. + +: crazy-aces ( n -- ) + case + 0 of ." And a-nutt'n'" + 1 of ." And a-won" + 2 of ." And a-too" + 3 of ." And a-tree" + 4 of ." and afford" + 5 of ." Dat's enuff" + ( default ) ." It's not my default!" + endcase ." Just in case you end up here..." +; + +fcode-end Modified: fcode-utils/testsuite/TokMisc/ContErr3.fth =================================================================== --- fcode-utils/testsuite/TokMisc/ContErr3.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokMisc/ContErr3.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -132,8 +132,8 @@ then loop -#message" Same line as a loop" ." ...Same line as a loop" - +#message" + Same line as a loop" ." ...Same line as a loop" 4 0 do i test_something if to targ : runny_stuff begin 1 0 do targ u. #message" Error here, too..." j #message" But not here either" 1 0 do j . loop Added: fcode-utils/testsuite/TokMisc/DtkVSFcTst.scr =================================================================== --- fcode-utils/testsuite/TokMisc/DtkVSFcTst.scr (rev 0) +++ fcode-utils/testsuite/TokMisc/DtkVSFcTst.scr 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,41 @@ +#! /bin/csh -f +# +# Script to run detokenizer with Vendor-Specific FCodes. +# This worked just fine on the TestArgs command line, in the "script" field +# on CygWin and GNU_Linux, but gets jammed-up on AI +# So we have to pump it into a separate script. Fooey! +# +# Well, in for a penny and all that...X +# Let's make it a bit more general. +# First param is the base-name of both the input .fc file +# and the output .DeTok file +# Second param is the name of the Vendor-Specific FCodes file + +# If it's already a script, might as well error-check +alias onecr 'echo "" ; alias onecr true' +if ( $#argv < 1 ) then + onecr + echo $0 Missing First arg, Base-name of input .fc and output .DeTok files + set ERROR +endif +if ( $#argv < 2 ) then + onecr + echo $0 Missing Second arg, Name of the Vendor-Specific FCodes file + set ERROR +endif +if ( $?ERROR ) exit 1 + +if ( ! -r $1.fc ) then + onecr + echo $0 Cannot read input file $1.fc + set ERROR +endif +if ( ! -r $2 ) then + onecr + echo $0 Cannot read Vendor-Specific FCodes file $2 + set ERROR +endif +if ( $?ERROR ) exit 2 + + +../detok -v -o -f $2 $1.fc > $1.DeTok Property changes on: fcode-utils/testsuite/TokMisc/DtkVSFcTst.scr ___________________________________________________________________ Name: svn:executable + * Modified: fcode-utils/testsuite/TokMisc/MiscFeatures.DOS.fth =================================================================== --- fcode-utils/testsuite/TokMisc/MiscFeatures.DOS.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokMisc/MiscFeatures.DOS.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -1,7 +1,6 @@ \ Obvious pun intended... -\ Updated Tue, 31 May 2005 at 12:07 by David L. Paktor +\ Updated Tue, 17 Oct 2006 at 12:57 PDT by David L. Paktor - alias // \ fcode-version2 @@ -12,21 +11,41 @@ char G emit control G emit control [ emit +global-definitions +\ Each dev-node will create its own debug-flag and alias it to debug-me? +\ Each dev-node will create a macro called my-dev-name giving its device-name + [macro] .fname&dev [function-name] type ." in " my-dev-name type + [macro] name-my-dev my-dev-name device-name + [macro] .dbg-enter debug-me? @ if ." Entering " .fname&dev cr then + [macro] .dbg-leave debug-me? @ if ." Leaving " .fname&dev cr then +device-definitions + +\ Top-most device, named billy +[macro] my-dev-name " billy" +name-my-dev + +variable debug-bell? debug-bell? off alias debug-me? debug-bell? : bell + .dbg-enter [char] G dup control G 3drop + .dbg-leave ; : factl recursive ( n -- n! ) + ." Entering First vers. of " [function-name] type cr ?dup 0= if 1 - else dup 1- * factl + else dup 1- factl * then + ." Leaving First vers. of " [function-name] type cr ; : factl ( n -- n! ) + ." Entering Second vers. of " [function-name] type cr ?dup 0= if 1 factl else dup 1- recurse * then + ." Leaving Second vers. of " [function-name] type cr ; variable naught @@ -41,31 +60,88 @@ constant /four : peril + .dbg-enter ['] noop is do-nothing 100 is thirty 5 is naught thirty dup - abort" Never Happen" + .dbg-leave ; : thirty ( new-val -- ) + .dbg-enter dup to thirty alias .dec .d \ Should this be allowed? ." Dirty" .dec + .dbg-leave ; tokenizer[ alias fliteral1 fliteral // This should be a harmless remark. h# deadc0de ]tokenizer fliteral1 +\ First subsidiary device, "child" of billy +new-device + instance variable cheryl + [macro] my-dev-name " cheryl" + name-my-dev + + instance + \ Third-level device, "grandchild" of billy + new-device + [macro] my-dev-name " meryl" + name-my-dev + + variable beryl + + variable debug-meryl? debug-meryl? off + alias debug-me? debug-meryl? + : meryl + .dbg-enter + cheryl + alias .deck .dec + alias feral cheryl + alias .heck .h + .dbg-leave + ; + finish-device + + \ Now we're back to "cheryl" + + variable debug-cheryl? debug-cheryl? off + alias debug-me? debug-cheryl? + : queryl + .dbg-enter + over rot dup nip drop swap \ Not the most useful code... ;-} + .dbg-leave + ; +finish-device + +\ Some interpretation-time after the fact markers... +alias colon : +overload [macro] : ." Cleared " [input-file-name] type ." line " [line-number] .d cr colon + +alias semicolon ; +overload [macro] ; semicolon ." Finished defining " [function-name] type cr + +\ And we're back to billy. : droop ( -- ) + .dbg-enter \ This will display Entering droop in billy twenty tokenizer[ alias .x .h \ Should this generate a warning? + [function-name] ]tokenizer 0 ?do i .x loop -; + .dbg-leave +; f[ [function-name] ]f +headerless : ploop ( -- ) + .dbg-enter fifty 0 do i drop 2 +loop + .dbg-leave ; +overload alias : colon +overload alias ; semicolon fcode-end Modified: fcode-utils/testsuite/TokMisc/MiscFeatures.fth =================================================================== --- fcode-utils/testsuite/TokMisc/MiscFeatures.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokMisc/MiscFeatures.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -1,5 +1,5 @@ \ Obvious pun intended... -\ Updated Fri, 10 Feb 2006 at 15:53 PST by David L. Paktor +\ Updated Tue, 17 Oct 2006 at 12:57 PDT by David L. Paktor alias // \ fcode-version2 @@ -116,22 +116,32 @@ ; finish-device +\ Some interpretation-time after the fact markers... +alias colon : +overload [macro] : ." Cleared " [input-file-name] type ." line " [line-number] .d cr colon + +alias semicolon ; +overload [macro] ; semicolon ." Finished defining " [function-name] type cr + \ And we're back to billy. : droop ( -- ) .dbg-enter \ This will display Entering droop in billy twenty tokenizer[ alias .x .h \ Should this generate a warning? + [function-name] ]tokenizer 0 ?do i .x loop .dbg-leave -; +; f[ [function-name] ]f headerless : ploop ( -- ) .dbg-enter fifty 0 do i drop 2 +loop .dbg-leave ; +overload alias : colon +overload alias ; semicolon fcode-end Modified: fcode-utils/testsuite/TokMisc/StringsGenl.fth =================================================================== --- fcode-utils/testsuite/TokMisc/StringsGenl.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokMisc/StringsGenl.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -1,6 +1,6 @@ \ Test of various formats of strings -\ Updated Mon, 06 Mar 2006 at 17:39 PST by David L. Paktor +\ Updated Tue, 10 Oct 2006 at 11:46 PDT by David L. Paktor fcode-version2 @@ -9,13 +9,14 @@ ." Empty string next." ." " -." BSlashes: \t\1fea9\abdc\n\1f\\fece" +." BSlashes#1: \t\1fea9\abdc\n\1f\\fece" decimal -." BSlashes: \t\a7\c01a" +." BSlashes#2: \t\a7\c01a" +." Dec 1193176 = 0x1234D8 Dec 176 = 0xB0 \1193176\" hex -." BSlashes: \n\a7\c01a" -." BSlashes: \t\a7\\c0\\1a" -." BSlashes: \t\a7\\c0\\1a"( feedface)" +." BSlashes#3: \n\a7\c01a" +." BSlashes#4: \t\a7\\c0\\1a" +." BSlashes#5: \t\a7\\c0\\1a"( feedface)" ." 3 BSlashes, then QOpen. \t\Q\n"(090abcdefeca8e beeffece b020)Zoh. "(1 23 4 567 8 9 0 1 2 3 0 a b c 30)" .( Dot-Paren-NoSpace)cr cr .( Dot-Paren Space) cr cr Modified: fcode-utils/testsuite/TokMisc/TestArgs =================================================================== --- fcode-utils/testsuite/TokMisc/TestArgs 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokMisc/TestArgs 2006-10-30 16:25:35 UTC (rev 107) @@ -7,8 +7,9 @@ # A script-command with arguments (but no commas). # Lines that have no label, no switches, and no script # do not need any commas. +# A leading +V in the script-command field will suppress -v -# Updated Thu, 07 Sep 2006 at 11:49 PDT by David L. Paktor +# Updated Mon, 16 Oct 2006 at 17:22 PDT by David L. Paktor MiscFeatures , , -i MiscFeatures , applabq , -i -f NOSUN-ABORT-Quote @@ -23,15 +24,15 @@ StringsGenl , noqbsrmk , -f NOSTRING-REMARK-ESCAPE BasicCont ExtCont -OldTkzr/ContErr0 , , -i +ContErr0 , , -i ContErr1 , , -i ContErr2 , , -i ContErr3 , , -i MulDev , , -i MulDev_01 , , -i MulDev_02 , , -i -T jeeky -T kinky +MulDev_02 , NoV , -T jeeky -T kinky , +V MinPCIHtst -OldTkzr/SimplStrEscs StrEscs StrEscs_01 FlagSet , , -i @@ -72,3 +73,7 @@ BatchTst , Help , -I. -I../TokeCommon -l -h , rm *_cpy* CaseTkns + +# Vendor-Specific FCodes, just a little... +VSFCtest +VSFCtest , VSfc , , DtkVSFcTst.scr VSFCtest.VSfc VendSpecFCodes Added: fcode-utils/testsuite/TokMisc/VSFCtest.fth =================================================================== --- fcode-utils/testsuite/TokMisc/VSFCtest.fth (rev 0) +++ fcode-utils/testsuite/TokMisc/VSFCtest.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,62 @@ +\ Generate something that looks like FCode generated by a Very Custom Tokenizer + + +fcode-version1 + +headers +hex + +F[ + \ If we want to test this against the previous release, + \ we can't do this: 341 emit-fcode + \ Instead, we have to do it in two parts: + + 03 emit-byte 41 emit-byte \ 0341 = "Simple" Vendor-specific FCode" + ]F +-1 +dup 0> if +F[ + \ Likewise here 369 emit-fcode + 03 emit-byte 69 emit-byte \ 0369 = "double(lit)" + ff emit-byte + ff emit-byte + ff emit-byte + ff emit-byte + 00 emit-byte + 00 emit-byte + 00 emit-byte + 00 emit-byte + ]F +then +dup 0= if +F[ + \ Here, too 369 emit-fcode + 03 emit-byte 69 emit-byte \ 0369 = "double(lit)" + ff emit-byte + ff emit-byte + ff emit-byte + f0 emit-byte + 00 emit-byte + 00 emit-byte + 00 emit-byte + 00 emit-byte + ]F +then + +0< if +F[ + \ Last one 369 emit-fcode + 03 emit-byte 69 emit-byte \ 0369 = "double(lit)" + de emit-byte + fe emit-byte + ca emit-byte + 8e emit-byte + be emit-byte + ef emit-byte + fe emit-byte + ce emit-byte + ]F +then +constant moochie + +fcode-end Added: fcode-utils/testsuite/TokMisc/VendSpecFCodes =================================================================== --- fcode-utils/testsuite/TokMisc/VendSpecFCodes (rev 0) +++ fcode-utils/testsuite/TokMisc/VendSpecFCodes 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,4 @@ +# Sample of Vendor-Specific FCodes for test + + 0x341 Three-for-One,aBargain! + 0x369 double(lit) Special-Function pre-defined name Added: fcode-utils/testsuite/TokeErrs/DecodProp.fth =================================================================== --- fcode-utils/testsuite/TokeErrs/DecodProp.fth (rev 0) +++ fcode-utils/testsuite/TokeErrs/DecodProp.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,26 @@ +\ Updated Tue, 17 Oct 2006 at 15:32 PDT by David L. Paktor + + +fcode-version2 + +\ The classic definition, using the return-stack +[macro] decade-bites >r over r@ + swap r@ - rot r> + +" grubby-nit-picker" device-name +" bloated lackey of the capitalist toy-mongers" + encode-bytes " santa-claus" property + +" name" get-my-property if ." Sorry, Charlie!" +else + 6 decade-bites ." Starts with: " type + 2drop +then cr + + +" santa-claus" get-my-property if ." Eat flaming death!" +else + 7 decode-bytes ." Starts with: " type + 2drop +then cr + +fcode-end Modified: fcode-utils/testsuite/TokeErrs/DevImbal.fth =================================================================== --- fcode-utils/testsuite/TokeErrs/DevImbal.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokeErrs/DevImbal.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -22,7 +22,12 @@ new-device create eek! 18 c, 17 c, 80 c, 79 c, : freek eek! 4 bounds ?do i c@ . 1 +loop ; -: greek -1 if freek then ; +: greek + recursive -1 if ." By name" greek + ." other name" freek + else ." Re-Curse you!" recurse + then +; [message] About to access method from parent node : hierareek eek! Modified: fcode-utils/testsuite/TokeErrs/DupNams.fth =================================================================== --- fcode-utils/testsuite/TokeErrs/DupNams.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokeErrs/DupNams.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -1,10 +1,39 @@ \ Let's make a batch of duplicate definitions in various scopes... -\ Updated Fri, 02 Jun 2006 at 09:48 PDT by David L. Paktor +\ Updated Thu, 12 Oct 2006 at 13:17 PDT by David L. Paktor +\ Tracing: alley-oop boop croop doop drop dup foop +\ gloop _harpo hoop koop loop noop poop +\ shtoop zoop +\ fontbytes blink-screen base bell + +\ alley-oop +\ boop +\ croop Global Macro +\ doop +\ drop Built-in word, aliased, invoked +\ dup Built-in word +\ foop Global alias to dup +\ floop Global Macro, Alias to flop (alias to drop), +\ redefined in subordinate device +\ gloop Undefined, invoked +\ _harpo Local, in subordinate device +\ hoop +\ koop +\ loop Built-in word +\ noop Built-in word, redefined in second FCode block +\ poop +\ shtoop +\ zoop +\ fontbytes Built-in VALUE +\ blink-screen Built-in DEFER +\ base Built-in VARIABLE +\ bell Built-in CONSTANT + + alias foop dup -[macro] croop foop message" Using FOOP " +[macro] croop foop #message" Using FOOP " alias dup croop @@ -22,7 +51,8 @@ global-definitions alias flop drop -[macro] floop flop message" Using FLOOP " +#message" Sync Up Diffs w/ prev. Release."n"n" +[macro] floop flop #message" Using FLOOP " alias drop floop device-definitions @@ -32,13 +62,15 @@ fcode-version2 : noop ." Op? No!" ; -[macro] zoop noop message" I Care." +[macro] zoop noop #message" I Care." : poop h# -21013572 ; new-device : zoop ." Nothing like the other zoop" croop ; : croop ." Sort of like F-Troop with a higher GPA..." foop + drop + floop ; : foop ." Shop bop-a-looma, a-lop bam boom!" ; : floop ." Oh, Jiggly!" ; @@ -46,6 +78,7 @@ f[ 127 constant _harpo ]f poop -> _cheeko f[ _cheeko constant a__gent ]f + floop ; alias droop drop @@ -59,6 +92,8 @@ _harpo droop drupe + boop + floop ; alias shoop encode-int @@ -73,6 +108,18 @@ finish-device + : stoop + floop + gloop + shoop + ; + alias coop floop + : troop + shoop + coop + poop + ; + finish-device start4 \ Let's just stick in an extra, and another error besides... fcode-end Modified: fcode-utils/testsuite/TokeErrs/MiscFeatErrs.fth =================================================================== --- fcode-utils/testsuite/TokeErrs/MiscFeatErrs.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokeErrs/MiscFeatErrs.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -1,11 +1,17 @@ \ Obvious pun intended... -\ Updated Fri, 14 Jul 2006 at 12:50 PDT by David L. Paktor +\ Updated Mon, 09 Oct 2006 at 09:57 PDT by David L. Paktor [flag] Local-Values f[ ." This is a test" ]f fcode-version1 -fload LocalValuesSupport.fth +global-definitions + headers + h# 130 constant _local-storage-size_ + headerless +device-definitions + +fload TotalLocalValuesSupport.fth noop noop noop headers @@ -59,10 +65,16 @@ _a _b + i * dup -> d _c * to _e j . loop + ['] _a + f['] _e + f[ f['] _b + f['] dup emit-fcode + h# 0f emit-fcode ]f + _a _b + _c * ['] + factl catch if ." Run in circles, scream and shout!" then ; - : DMA-ALLOC ( n -- vaddr ) " dma-alloc" $call-parent ; : HOOBARTH ( n -- vaddr ) " hoobarth" $call-parent ; : MY-END0 ( -- n ) ['] end0 ; Modified: fcode-utils/testsuite/TokeErrs/TestArgs =================================================================== --- fcode-utils/testsuite/TokeErrs/TestArgs 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokeErrs/TestArgs 2006-10-30 16:25:35 UTC (rev 107) @@ -13,21 +13,26 @@ # Most files in this category need the -i switch # Some may have additional variants. -# Updated Thu, 07 Sep 2006 at 11:49 PDT by David L. Paktor +# Updated Wed, 18 Oct 2006 at 11:44 PDT by David L. Paktor MiscFeatErrs , , -i -I. -I.. -LocalTest , NoLvSupp , -i -f Local-Values -I. -I.. -LocalTest , , -i -f Local-Values -I. -I.. -f Always-Headers -d dont_omit_support -LocalTest1 , , -f Local-Values -f noLV-Legacy-Message -I. -I.. +MiscFeatErrs , GLoc , -i -I. -I.. -d Global-Locals +MiscFeatErrs , LocR , -i -I. -I.. -d Locals-Release +MiscFeatErrs , GlLr , -i -I. -I.. -d Global-Locals -d Locals-Release +MiscFeatErrs , Lss , -i -I. -I.. -d _local-storage-size_=3022 +MiscFeatErrs , NoLss , -i -I. -I.. -d _local-storage-size_ +LocalTest , NoLvSupp , -i -f Local-Values -I. -I../TokoLoco +LocalTest , , -i -f Local-Values -I. -I../TokoLoco -f Always-Headers -d dont_omit_support +LocalTest1 , , -f Local-Values -f noLV-Legacy-Message -I. -I../TokoLoco ControlMismatch , , -i -SuppLocErrTest4 , , -i -f Local-Values -l -I. -I.. -testLocDevSupG , , -i -f Local-Values -I. -I.. -testLocDevSupG , SymSiz , -i -f Local-Values -d "_local-storage-size_= d# 120" -I. -I.. +SuppLocErrTest4 , , -i -f Local-Values -l -I. -I../TokoLoco +testLocDevSupG , , -i -f Local-Values -I. -I../TokoLoco +testLocDevSupG , SymSiz , -i -f Local-Values -d "_local-storage-size_= d# 120" -I. -I../TokoLoco PCIhdrErrs , , -i FCShdrErrs , , -i PCI_FCShdrErrs , , -i CommOvrRun , , -i -l -LocOvrRun , , -i -f Local-Values -l -I. -I.. +LocOvrRun , , -i -f Local-Values -l -I. -I../TokoLoco SplitImbal , , -i -l TokCondUnconcF , , -i -l TokCondUnconcT , , -i -l @@ -46,13 +51,13 @@ UserMacroErrors , , -i CStrEscOvflw , , -i -l DevImbal , , -i -DevNodAli , , -i -l -I. -I.. -DevNodAli_01 , , -i -l -I. -I.. -GlobScopErrTst , NoI , -f Local-Values -I. -I.. -GlobScopErrTst , , -i -f Local-Values -I. -I.. -GlobScopErrTst_01 , , -i -f Local-Values -I. -I.. -GlobScopErrTst_02 , , -i -f Local-Values -I. -I.. -GlobScopErrTst_03 , , -i -f Local-Values -I. -I.. +DevNodAli , , -i -l -I. -I../TokoLoco +DevNodAli_01 , , -i -l -I. -I../TokoLoco +GlobScopErrTst , NoI , -f Local-Values -I. -I../TokoLoco +GlobScopErrTst , , -i -f Local-Values -I. -I../TokoLoco +GlobScopErrTst_01 , , -i -f Local-Values -I. -I../TokoLoco +GlobScopErrTst_02 , , -i -f Local-Values -I. -I../TokoLoco +GlobScopErrTst_03 , , -i -f Local-Values -I. -I../TokoLoco # The script-fields below prepare for the ExpPath InclLst and DePList tests Frinstnce , , -i , touch ExpPath.NoV.{P,f{c,l{,.missing}}} Frinstnce , NoI , , chmod a-rw ../TokeCommon/{MyBeerAndYouCannotHaveIt.fth,NoRead.bin} ExpPath.NoV.{P,f{c,l{,.missing}}} @@ -69,7 +74,7 @@ # Done with the ExpPath and InclLst tests ToAtEnd , , -i -ToAtEnd , WLocSup , -i -I . -I .. +ToAtEnd , WLocSup , -i -I . -I ../TokoLoco ObsoFCodes , , -i ObsoFCodes , NoWarn , -f NoObsolete-FCode-Warning @@ -84,7 +89,7 @@ AbrtWQuotStr , , -i # The new "Trace" feature, and handling of duplicate and mis-scoped names... -DupNams , , -f Local-Values -T croop -T drop -T _harpo -T dup -T noop -T zoop -T foop +DupNams , , -f Local-Values -T alley-oop -T boop -T croop -T doop -T drop -T dup -T foop -T floop -T gloop -T _harpo -T koop -T loop -T noop -T poop -T shtoop -T zoop -T fontbytes -T blink-screen -T base -T bell TooManyFCodes , , -l -o TooManyFCodes.fl TooManyFCodes , Reg , -i @@ -98,3 +103,6 @@ RetStkDep , , -i CondlDefn , , -i + +DecodProp +DecodProp , nrsi , -f noRet-Stk-Interp Modified: fcode-utils/testsuite/TokoLoco/GlobalLocalValues.fth =================================================================== --- fcode-utils/testsuite/TokoLoco/GlobalLocalValues.fth 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokoLoco/GlobalLocalValues.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -1,7 +1,13 @@ +\ %Z%%M% %I% %W% %G% %U% +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + \ Load Locals Support under Global-Definitions. Bypass Instance warning -\ Updated Fri, 10 Mar 2006 at 14:47 PST by David L. Paktor - \ Make sure this option is turned on. [flag] Local-Values Added: fcode-utils/testsuite/TokoLoco/GlobalLocalValuesDevel.fth =================================================================== --- fcode-utils/testsuite/TokoLoco/GlobalLocalValuesDevel.fth (rev 0) +++ fcode-utils/testsuite/TokoLoco/GlobalLocalValuesDevel.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,31 @@ +\ %Z%%M% %I% %W% %G% %U% +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Load Support file for development of FCode that uses Local Values +\ under Global-Definitions. Bypass Instance warning. +\ Replace this with GlobalLocalValues.fth in your final product. + +\ Make sure this option is turned on. +[flag] Local-Values + +global-definitions + +\ Bypass warning about Instance without altering LocalValuesSupport file +alias generic-instance instance +[macro] bypass-instance f[ noop .( Bypassed instance!) f] + +overload alias instance bypass-instance + +fload LocalValuesSupport.fth +fload LocalValuesDevelSupport.fth + +\ Replace normal meaning of Instance, still in Global scope. +overload alias instance generic-instance + +\ Restore Device-Definitions scope. +device-definitions Added: fcode-utils/testsuite/TokoLoco/LocalValuesDevelSupport.fth =================================================================== --- fcode-utils/testsuite/TokoLoco/LocalValuesDevelSupport.fth (rev 0) +++ fcode-utils/testsuite/TokoLoco/LocalValuesDevelSupport.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,46 @@ +\ %Z%%M% %I% %W% %G% %U% +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Support file for development of FCode that uses Local Values +\ FLoad this right after LocalValuesSupport.fth +\ Remove it from your final product. + +\ Exported Function: max-local-storage-size ( -- n ) +\ Returns the measured maximum size of storage for Local Values +\ used by any given test run. This number can be used to guide +\ the declaration of _local-storage-size_ +\ +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Count the current depth on a per-instance basis, +\ but collect the maximum depth over all instances. + +headers +0 instance value local-storage-depth + +external +0 value max-local-storage-size +headers + +\ Overload the {push-locals} and {pop-locals} routines to do this. +\ Do not suppress the overload warnings; they'll serve as a reminder. +: {pop-locals} ( #locals -- ) + local-storage-depth over - to local-storage-depth + {pop-locals} +; + +: {push-locals} ( #ilocals #ulocals -- ) + 2dup + local-storage-depth + + dup to local-storage-depth + max-local-storage-size max + to max-local-storage-size + {push-locals} +; + + Added: fcode-utils/testsuite/TokoLoco/LocalValuesSupport.fth =================================================================== --- fcode-utils/testsuite/TokoLoco/LocalValuesSupport.fth (rev 0) +++ fcode-utils/testsuite/TokoLoco/LocalValuesSupport.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,134 @@ +\ %Z%%M% %I% %W% %G% %U% +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ The support routines for Local Values in FCode. + +\ Function imported +\ _local-storage-size_ \ Size, in cells, of backing store for locals +\ \ A constant. If not supplied, default value of d# 64 will be used. +\ +\ Functions exported: +\ {push-locals} ( #ilocals #ulocals -- ) +\ {pop-locals} ( #locals -- ) +\ _{local} ( local-var# -- addr ) +\ +\ Additional overloaded function: +\ catch \ Restore Locals after a throw + +\ The user is responsible for declaring the maximum depth of the +\ run-time Locals stack, in storage units, by defining the +\ constant _local-storage-size_ before floading this file. +\ The definition may be created either by defining it as a constant +\ in the startup-file that FLOADs this and other files in the +\ source program, or via a command-line user-symbol definition +\ of a form resembling: -d '_local-storage-size_=d# 42' +\ (be sure to enclose it within quotes so that the shell treats +\ it as a single string, and, of course, replace the "42" with +\ the actual number you need...) +\ If both forms are present, the command-line user-symbol value will +\ be used to create a duplicate definition of the named constant, +\ which will prevail over the earlier definition, and will remain +\ available for examination during development and testing. The +\ duplicate-name warning, which will not be suppressed, will also +\ act to alert the developer of this condition. +\ To measure the actual usage (in a test run), use the separate tool +\ found in the file LocalValuesDevelSupport.fth . +\ If the user omits defining _local-storage-size_ the following +\ ten-line sequence will supply a default: + +[ifdef] _local-storage-size_ + f[ [defined] _local-storage-size_ true ]f +[else] + [ifexist] _local-storage-size_ + f[ false ]f + [else] + f[ d# 64 true ]f + [then] +[then] ( Compile-time: size true | false ) +[if] fliteral constant _local-storage-size_ [then] + +_local-storage-size_ \ The number of storage units to allocate + cells \ Convert to address units + dup \ Keep a copy around... + ( n ) instance buffer: locals-storage \ Use one of the copies + +\ The Locals Pointer, added to the base address of locals-storage +\ points to the base-address of the currently active set of Locals. +\ Locals will be accessed as a positive offset from there. +\ Start the Locals Pointer at end of the buffer. +\ A copy of ( N ), the number of address units that were allocated +\ for the buffer, is still on the stack. Use it here. + ( n ) instance value locals-pointer + +\ Support for {push-locals} + +\ Error-check. +: not-enough-locals? ( #ilocals #ulocals -- error? ) + + cells locals-pointer swap - 0< +; + +\ Error message. +: .not-enough-locals ( -- ) + cr ." FATAL ERROR: Local Values Usage exceeds allocation." cr +; + +\ Detect, announce and handle error. +: check-enough-locals ( #ilocals #ulocals -- | <ABORT> ) + not-enough-locals? if + .not-enough-locals + abort + then +; + +\ The uninitialized locals can be allocated in a single batch +: push-uninitted-locals ( #ulocals -- ) + cells locals-pointer swap - to locals-pointer +; + +\ The Initialized locals are initted from the items on top of the stack +\ at the start of the routine. If we allocate them one at a time, +\ we get them into the right order. I.e., the last-one named gets +\ the top item, the earlier ones get successively lower items. +: push-one-initted-local ( pstack-item -- ) + locals-pointer 1 cells - + dup to locals-pointer + locals-storage + ! +; + +\ Push all the Initialized locals. +: push-initted-locals ( N_#ilocals-1 ... N_0 #ilocals -- ) + 0 ?do push-one-initted-local loop +; + +: {push-locals} ( N_#ilocals ... N_1 #ilocals #ulocals -- ) + 2dup check-enough-locals + push-uninitted-locals ( ..... #i ) + push-initted-locals ( ) +; + +\ Pop all the locals. +\ The param is the number to pop. +: {pop-locals} ( total#locals -- ) + cells locals-pointer + to locals-pointer +; + +\ The address from/to which values will be moved, given the local-var# +: _{local} ( local-var# -- addr ) + cells locals-pointer + locals-storage + +; + +\ We need to overload catch such that the state of the Locals Pointer +\ will be preserved and restored after a throw . +overload : catch ( ??? xt -- ???' false | ???'' throw-code ) + locals-pointer >r ( ??? xt ) ( R: old-locals-ptr ) + catch ( ???' false | ???'' throw-code ) ( R: old-locals-ptr ) + \ No need to inspect the throw-code. + \ If catch returned a zero, the Locals Pointer + \ is valid anyway, so restoring it is harmless. + r> to locals-pointer +; Modified: fcode-utils/testsuite/TokoLoco/TestArgs =================================================================== --- fcode-utils/testsuite/TokoLoco/TestArgs 2006-10-30 13:26:11 UTC (rev 106) +++ fcode-utils/testsuite/TokoLoco/TestArgs 2006-10-30 16:25:35 UTC (rev 107) @@ -12,18 +12,18 @@ # All files in this category need the -f Local-Values switch # Some will have additional variants. -# Updated Mon, 20 Mar 2006 at 11:18 PST by David L. Paktor +# Updated Fri, 13 Oct 2006 at 16:19 PDT by David L. Paktor -testNest , , -f Local-Values -l -I. -I.. -SupportedLocalTest , , -f LOCAL-Values -l -I. -I.. -SupportedLocalTest3 , , -f Local-VALUES -l -I. -I.. -SupportedLocalTest4 , , -f Local-Values -l -I. -I.. -testDevelSupport , , -f local-values -I. -I.. +testNest , , -f Local-Values -l +SupportedLocalTest , , -f LOCAL-Values -l +SupportedLocalTest3 , , -f Local-VALUES -l +SupportedLocalTest4 , , -f Local-Values -l +testDevelSupport , , -f local-values # Additional variants: -SupportedLocalTest , nodupwrn , -f Local-VALUES -f NOWarn-if-DUPLicate -l -I. -I.. -SupportedLocalTest , nolegacy , -f LoCAL-VALUes -f NoLV-Legacy-Separator -l -I. -I.. -SupportedLocalTest , nolgcmsg , -f Local-ValUES -f NOLV-LEGAcy-Message -l -I. -I.. -SupportedLocalTest , nolocals , -i -f noLocal-Values -l -I. -I.. -SupportedLocalTest3 , DefLocStgSiz , -f Local-VALUes -d '_loCAL-STORagE-SIZe_=h# 42' -l -I. -I.. +SupportedLocalTest , nodupwrn , -f Local-VALUES -f NOWarn-if-DUPLicate -l +SupportedLocalTest , nolegacy , -f LoCAL-VALUes -f NoLV-Legacy-Separator -l +SupportedLocalTest , nolgcmsg , -f Local-ValUES -f NOLV-LEGAcy-Message -l +SupportedLocalTest , nolocals , -i -f noLocal-Values -l +SupportedLocalTest3 , DefLocStgSiz , -f Local-VALUes -d '_loCAL-STORagE-SIZe_=h# 42' -l Added: fcode-utils/testsuite/TokoLoco/TotalLocalValuesSupport.fth =================================================================== --- fcode-utils/testsuite/TokoLoco/TotalLocalValuesSupport.fth (rev 0) +++ fcode-utils/testsuite/TokoLoco/TotalLocalValuesSupport.fth 2006-10-30 16:25:35 UTC (rev 107) @@ -0,0 +1,52 @@ +\ %Z%%M% %I% %W% %G% %U% +\ (C) Copyright 2005-2006 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Control file for loading of Local Values Support file with variants. +\ Command-line Symbol-definitions select whether the support will +\ be under Global-Definitions, and whether to include the extra +\ Development-time support features. +\ +\ The command-line symbols are: +\ Global-Locals +\ and +\ Locals-Release +\ +\ The default is device-node-specific support in a Development-time setting. +\ +\ If Global-Locals is defined, support will be under Global-Definitions +\ If Locals-Release is defined, this is a final production release run, +\ and the Development-time support features will be removed. + +\ Make sure this option is turned on. +[flag] Local-Values + +[ifdef] Global-Locals + \ Load Support file under Global-Definitions. + global-definitions + + \ Bypass warning about Instance without altering LocalValuesSupport file + alias generic-instance instance + [macro] bypass-instance f[ noop .( Bypassed instance!) f] + + overload alias instance bypass-instance +[endif] \ Global-Locals + +fload LocalValuesSupport.fth + +[ifndef] Locals-Release + \ Load Development-time support features + fload LocalValuesDevelSupport.fth +[endif] \ not Locals-Release + +[ifdef] Global-Locals + \ Replace normal meaning of Instance, still in Global scope. + overload alias instance generic-instance + + \ Restore Device-Definitions scope. + device-definitions +[endif] \ Global-Locals

1 0

r106 - fcode-utils/testsuite
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 14:26:11 +0100 (Mon, 30 Oct 2006) New Revision: 106 Modified: fcode-utils/testsuite/GlobalLocalValues.fth fcode-utils/testsuite/GlobalLocalValuesDevel.fth fcode-utils/testsuite/LocalValuesDevelSupport.fth fcode-utils/testsuite/LocalValuesSupport.fth fcode-utils/testsuite/TotalLocalValuesSupport.fth Log: fix links to use our own copy. Modified: fcode-utils/testsuite/GlobalLocalValues.fth =================================================================== --- fcode-utils/testsuite/GlobalLocalValues.fth 2006-10-30 11:59:18 UTC (rev 105) +++ fcode-utils/testsuite/GlobalLocalValues.fth 2006-10-30 13:26:11 UTC (rev 106) @@ -1 +1 @@ -link ../../LocalValues/GlobalLocalValues.fth \ No newline at end of file +link ../localvalues/GlobalLocalValues.fth \ No newline at end of file Modified: fcode-utils/testsuite/GlobalLocalValuesDevel.fth =================================================================== --- fcode-utils/testsuite/GlobalLocalValuesDevel.fth 2006-10-30 11:59:18 UTC (rev 105) +++ fcode-utils/testsuite/GlobalLocalValuesDevel.fth 2006-10-30 13:26:11 UTC (rev 106) @@ -1 +1 @@ -link ../../LocalValues/GlobalLocalValuesDevel.fth \ No newline at end of file +link ../localvalues/GlobalLocalValuesDevel.fth \ No newline at end of file Modified: fcode-utils/testsuite/LocalValuesDevelSupport.fth =================================================================== --- fcode-utils/testsuite/LocalValuesDevelSupport.fth 2006-10-30 11:59:18 UTC (rev 105) +++ fcode-utils/testsuite/LocalValuesDevelSupport.fth 2006-10-30 13:26:11 UTC (rev 106) @@ -1 +1 @@ -link ../../LocalValues/LocalValuesDevelSupport.fth \ No newline at end of file +link ../localvalues/LocalValuesDevelSupport.fth \ No newline at end of file Modified: fcode-utils/testsuite/LocalValuesSupport.fth =================================================================== --- fcode-utils/testsuite/LocalValuesSupport.fth 2006-10-30 11:59:18 UTC (rev 105) +++ fcode-utils/testsuite/LocalValuesSupport.fth 2006-10-30 13:26:11 UTC (rev 106) @@ -1 +1 @@ -link ../../LocalValues/LocalValuesSupport.fth \ No newline at end of file +link ../localvalues/LocalValuesSupport.fth \ No newline at end of file Modified: fcode-utils/testsuite/TotalLocalValuesSupport.fth =================================================================== --- fcode-utils/testsuite/TotalLocalValuesSupport.fth 2006-10-30 11:59:18 UTC (rev 105) +++ fcode-utils/testsuite/TotalLocalValuesSupport.fth 2006-10-30 13:26:11 UTC (rev 106) @@ -1 +1 @@ -link ../../LocalValues/TotalLocalValuesSupport.fth \ No newline at end of file +link ../localvalues/TotalLocalValuesSupport.fth \ No newline at end of file

1 0

r105 - fcode-utils/detok
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 12:59:18 +0100 (Mon, 30 Oct 2006) New Revision: 105 Modified: fcode-utils/detok/addfcodes.c fcode-utils/detok/dictionary.c Log: detok 1.0.2 merge part 3 Modified: fcode-utils/detok/addfcodes.c =================================================================== --- fcode-utils/detok/addfcodes.c 2006-10-30 11:31:16 UTC (rev 104) +++ fcode-utils/detok/addfcodes.c 2006-10-30 11:59:18 UTC (rev 105) @@ -46,6 +46,10 @@ * Identified this need when working with in-house code, * which uses some custom functions. This solution * is (hoped to be) general enough to cover all cases. + * Mon, 16 Oct 2006 by David L. Paktor + * Add "special function" words. So far, only one added: + * double-literal Infrastructure will support + * adding others as needed. * **************************************************************************** */ @@ -58,6 +62,21 @@ * **************************************************************************** */ + +/* ************************************************************************** + * + * Global Variables Exported : + * + * For "special function" identification, we will need to + * export Global Variables, each of which is a pointer + * to the address of the FCode field of the entry in + * the Special Functions List for that function. + * + * Variable Associated Name + * double_lit_code double-literal + * + **************************************************************************** */ + #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -73,6 +92,8 @@ * vfc_remainder Remainder of Vendor-FCodes buffer to be scanned * vfc_line_no Number of current line in Vendor-FCodes buffer * vfc_buf_end Pointer to end of Vendor-FCodes buffer + * spcl_func_list List of reserved Special Function names + * spcl_func_count Number of reserved Special Function names * **************************************************************************** */ @@ -81,6 +102,22 @@ static int vfc_line_no = 0; static char *vfc_buf_end; +/* Special Functions List */ +/* Initial fcode-field value of -1 guarantees they won't be used */ +token_t spcl_func_list[] = { + TOKEN_ENTRY( -1, "double(lit)" ), /* Entry [0] */ +}; + +static const int spcl_func_count = (sizeof(spcl_func_list)/sizeof(token_t)) ; + +/* Global Variables for "special function" identification */ +/* Each is a pointer to the FCode field of the entry in + * the Special Functions List for that function. + */ + +u16 *double_lit_code = &spcl_func_list[0].fcode; + + /* ************************************************************************** * * Function name: skip_whitespace @@ -231,11 +268,19 @@ * after the name will be ignored, so an extra "comment" * is permitted. The FCode number must be in hex, with * an optional leading 0x or 0X For example: 0X407 - * The valid range is 0x010 to 0x7ff. Numbers above 0x800 - * infringe upon the are reserved for FCodes generated - * by the tokenization process. + * The valid range for the FCode numbers is 0x010 to 0x7ff. + * Numbers above 0x800 infringe upon the area reserved + * for FCodes generated by the tokenization process. * Numbers already in use will be ignored. A Message will be * printed even if the name matches the one on the line. + * Names may not be longer than 31 characters. + * Certain names will be reserved for special functions. + * Those names will be entered in the detok_table + * with a value of -1 and again in the static + * table associated with this function, below, to + * supply the variable that will be used to match + * the name with the special function. + * * **************************************************************************** */ @@ -267,6 +312,13 @@ char *lookup_result; char *fc_name_cpy; + /* For each line of input, we need to check that we have + * two strings, one of which is the number and the + * second of which is the name. We will check for + * the various formats allowed for the number + */ + + /* Start with a lower-case 0x */ scan_result = sscanf(current_vfc_line, "0x%x %32s", &vs_fc_number, vs_fc_name); @@ -315,6 +367,28 @@ continue; } + /* Check if the name is on the "Special Functions List" */ + { + bool found_spf = FALSE; + int indx; + for (indx = 0; indx < spcl_func_count; indx++) { + if ( strcmp( vs_fc_name, spcl_func_list[indx].name) == 0 ) { + char strbuf[64]; + found_spf = TRUE; + spcl_func_list[indx].fcode = vs_fc_number; + link_token( &spcl_func_list[indx]); + added_fc_count++; + sprintf( strbuf, "Added Special Function FCode " + "number 0x%03x, name %s\n", vs_fc_number, vs_fc_name); + printremark( strbuf); + break; + } + } + + if (found_spf) + continue; + } + /* We've passed all the tests! */ fc_name_cpy = strdup(vs_fc_name); add_token((u16) vs_fc_number, fc_name_cpy); @@ -323,11 +397,10 @@ } if (verbose) { - char *strbfr = malloc(85); + char strbfr[32]; sprintf(strbfr, "Added %d FCode numbers\n", added_fc_count); printremark(strbfr); - free(strbfr); } close_stream(); Modified: fcode-utils/detok/dictionary.c =================================================================== --- fcode-utils/detok/dictionary.c 2006-10-30 11:31:16 UTC (rev 104) +++ fcode-utils/detok/dictionary.c 2006-10-30 11:59:18 UTC (rev 105) @@ -37,13 +37,6 @@ #include "detok.h" bool check_tok_seq = TRUE; - -typedef struct token { - char *name; - u16 fcode; - struct token *next; -} token_t; -#define TOKEN_ENTRY(num, name) { name, (u16)num, (token_t *)NULL } static token_t *dictionary; /* Initialize dynamically to accommodate AIX */ static char *fcerror = "ferror"; @@ -52,7 +45,7 @@ { token_t *curr; - for (curr = dictionary; curr != NULL; curr = curr->next) + for (curr = dictionary; curr != NULL; curr = curr->prev) if (curr->fcode == number) break; @@ -64,6 +57,33 @@ /* ************************************************************************** * + * Function name: link_token + * Synopsis: Simply link a ready-made token-table entry to + * the dictionary, without side-effects. + * + * Inputs: + * Parameters: + * curr_token The token-table entry to link + * Local Static Variables: + * dictionary Pointer to the "tail" of the + * FCode-Tokens vocabulary. + * + * Outputs: + * Returned Value: NONE + * Local Static Variables: + * dictionary Updated to point to the new entry. + * + **************************************************************************** */ + +void link_token( token_t *curr_token) +{ + curr_token->prev = dictionary; + + dictionary = curr_token; +} + +/* ************************************************************************** + * * Function name: add_token * Synopsis: Add an entry to the FCode-Tokens vocabulary. * @@ -72,8 +92,6 @@ * number Numeric value of the FCode token * name Name of the function to display * Global/Static Variables: - * dictionary Pointer to the "tail" of the - * FCode-Tokens vocabulary. * check_tok_seq TRUE = "Check Token Sequence" * A retro-fit to accommodate * adding Vendor FCodes @@ -81,7 +99,6 @@ * Outputs: * Returned Value: NONE * Global/Static Variables: - * dictionary Updated to point to the new entry. * last_defined_token Updated to the given FCode token * Memory Allocated * For the new entry. @@ -116,11 +133,10 @@ exit(-ENOMEM); } - curr->next = dictionary; - curr->fcode = number; curr->name = name; + curr->fcode=number; - dictionary = curr; + link_token( curr); if (check_tok_seq) { /* Error-check, but not for first time. */ @@ -543,7 +559,7 @@ dictionary_reset_position = dictionary; for (indx = 1; indx < dictionary_indx_max; indx++) { - detok_table[indx].next = &detok_table[indx - 1]; + detok_table[indx].prev = &detok_table[indx - 1]; } } @@ -553,7 +569,7 @@ next_t = dictionary; while (next_t != dictionary_reset_position) { - next_t = dictionary->next; + next_t = dictionary->prev; free(dictionary->name); free(dictionary); dictionary = next_t;

1 0

r104 - fcode-utils/detok
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 12:31:16 +0100 (Mon, 30 Oct 2006) New Revision: 104 Modified: fcode-utils/detok/decode.c fcode-utils/detok/detok.c Log: detok 1.0.2 merge commit II Modified: fcode-utils/detok/decode.c =================================================================== --- fcode-utils/detok/decode.c 2006-10-30 10:04:23 UTC (rev 103) +++ fcode-utils/detok/decode.c 2006-10-30 11:31:16 UTC (rev 104) @@ -38,6 +38,7 @@ #include "detok.h" #include "stream.h" +#include "addfcodes.h" static int indent; /* Current level of indentation */ @@ -125,8 +126,7 @@ static void decode_lines(void) { if (show_linenumbers) { - printf("%6d: ", - show_offsets ? token_streampos : linenum++); + printf("%6d: ",show_offsets ? token_streampos : linenum++); } } @@ -384,6 +384,18 @@ printf("0x%x\n", lit); } +static void double_length_literal(void) +{ + u16 quadhh, quadhl, quadlh, quadll; + + output_token(); + quadhh = get_num16(); + quadhl = get_num16(); + quadlh = get_num16(); + quadll = get_num16(); + printf("0x%04x.%04x.%04x.%04x\n", quadhh, quadhl, quadlh, quadll); +} + static void offset16(void) { decode_default(); @@ -536,10 +548,28 @@ end_found = TRUE; decode_default(); break; + +#if 0 /* Fooey on C's petty limitations! */ + /* I'd like to be able to do this: */ + /* Special Functions */ + case *double_lit_code: + double_length_literal(); + break; +#endif /* Fooey on C's petty limitations! */ + default: + { + /* Have to do this clumsy thing instead */ + if ( token == *double_lit_code ) + { + double_length_literal(); + break; + } + decode_default(); } } +} /* ************************************************************************** @@ -556,15 +586,18 @@ * Outputs: * Returned Value: NONE * Global/Static Variables: - * offs16 FALSE if Starter was version1, else TRUE + * offs16 FALSE if Starter was version1, TRUE for all + * other valid Starters, otherwise unchanged. * fclen On error, gets set to reach end of input stream * Otherwise, gets set by decode_start() * Printout: + * On error, print a new-line to finish previous token's line. * * Error Detection: * First byte not a valid FCode Start: Print message, restore * input pointer to initial value, set fclen to [(end of - * input stream) - (input pointer)], return FALSE. + * input stream) - (input pointer)], leave offs16 unchanged. + * Return FALSE. * * Process Explanation: * This routine error-checks and dispatches to the routine that @@ -578,18 +611,20 @@ { long err_pos; u16 token; + bool new_offs16 = TRUE; err_pos = get_streampos(); indent = 0; token = next_token(); - offs16 = TRUE; + switch (token) { case 0x0fd: /* version1 */ - offs16 = FALSE; + new_offs16 = FALSE; case 0x0f0: /* start0 */ case 0x0f1: /* start1 */ case 0x0f2: /* start2 */ case 0x0f3: /* start4 */ + offs16 = new_offs16; decode_start(); break; default: Modified: fcode-utils/detok/detok.c =================================================================== --- fcode-utils/detok/detok.c 2006-10-30 10:04:23 UTC (rev 103) +++ fcode-utils/detok/detok.c 2006-10-30 11:31:16 UTC (rev 104) @@ -45,7 +45,7 @@ #include "stream.h" #include "addfcodes.h" -#define DETOK_VERSION "1.0.0" +#define DETOK_VERSION "1.0.2" #define CORE_COPYR "(C) Copyright 2001-2006 Stefan Reinauer.\n" \ "(C) Copyright 2006 coresystems GmbH <info(a)coresystems.de>" @@ -69,8 +69,8 @@ CORE_COPYR "\n" IBM_COPYR "\n" "Written by Stefan Reinauer, <stepan(a)openbios.org>\n" "This program is free software; you may redistribute it " - "under the terms of\nthe GNU General Public License v2. This " - "program has absolutely no warranty.\n\n"); + "under the terms of\nthe GNU General Public License v2. " + "This program has absolutely no warranty.\n\n"); pfunct = (is_error ? (vfunct) printf : printremark); @@ -190,3 +190,4 @@ return 0; } +

1 0

r103 - fcode-utils/detok
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 11:04:23 +0100 (Mon, 30 Oct 2006) New Revision: 103 Modified: fcode-utils/detok/addfcodes.h fcode-utils/detok/detok.h Log: detok 1.0.2 merge Modified: fcode-utils/detok/addfcodes.h =================================================================== --- fcode-utils/detok/addfcodes.h 2006-10-30 09:50:41 UTC (rev 102) +++ fcode-utils/detok/addfcodes.h 2006-10-30 10:04:23 UTC (rev 103) @@ -39,6 +39,16 @@ /* ************************************************************************** * * + * Global Variables Exported + * + **************************************************************************** */ + +/* For "special function" identification */ +extern u16 *double_lit_code; + + +/* ************************************************************************** * + * * Function Prototypes / Functions Exported: * **************************************************************************** */ Modified: fcode-utils/detok/detok.h =================================================================== --- fcode-utils/detok/detok.h 2006-10-30 09:50:41 UTC (rev 102) +++ fcode-utils/detok/detok.h 2006-10-30 10:04:23 UTC (rev 103) @@ -35,10 +35,28 @@ #include "types.h" +/* Structure of an entry in a token-table + * Consists of: + * (1) Name of the token + * (2) FCode of the token + * (3) Link-pointer to previous entry. + */ + +typedef struct token { + char *name; + u16 fcode; + struct token *prev; +} token_t; + +/* Macro for creating an entry in a token-table data-array */ +#define TOKEN_ENTRY(num, name) { name, (u16)num, (token_t *)NULL } + + /* Prototypes for functions exported from * detok.c decode.c printformats.c pcihdr.c and dictionary.c */ +void link_token(token_t *curr_token); void add_token(u16 number, char *name); void init_dictionary(void); void reset_dictionary(void);

1 0

r102 - in fcode-utils: . localvalues
by svn＠openbios.org 30 Oct '06

30 Oct '06

Author: stepan Date: 2006-10-30 10:50:41 +0100 (Mon, 30 Oct 2006) New Revision: 102 Added: fcode-utils/localvalues/ fcode-utils/localvalues/GlobalLocalValues.fth fcode-utils/localvalues/GlobalLocalValuesDevel.fth fcode-utils/localvalues/LocalValuesDevelSupport.fth fcode-utils/localvalues/LocalValuesSupport.fth fcode-utils/localvalues/TotalLocalValuesSupport.fth Log: localvalues support, contributed by David Paktor <dlpaktor(a)netscape.net> Added: fcode-utils/localvalues/GlobalLocalValues.fth =================================================================== --- fcode-utils/localvalues/GlobalLocalValues.fth (rev 0) +++ fcode-utils/localvalues/GlobalLocalValues.fth 2006-10-30 09:50:41 UTC (rev 102) @@ -0,0 +1,27 @@ +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Load Locals Support under Global-Definitions. Bypass Instance warning + +\ Make sure this option is turned on. +[flag] Local-Values + +global-definitions + +\ Bypass warning about Instance without altering LocalValuesSupport file +alias generic-instance instance +[macro] bypass-instance f[ noop .( Bypassed instance!) f] + +overload alias instance bypass-instance + +fload LocalValuesSupport.fth + +\ Replace normal meaning of Instance, still in Global scope. +overload alias instance generic-instance + +\ Restore Device-Definitions scope. +device-definitions Added: fcode-utils/localvalues/GlobalLocalValuesDevel.fth =================================================================== --- fcode-utils/localvalues/GlobalLocalValuesDevel.fth (rev 0) +++ fcode-utils/localvalues/GlobalLocalValuesDevel.fth 2006-10-30 09:50:41 UTC (rev 102) @@ -0,0 +1,30 @@ +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Load Support file for development of FCode that uses Local Values +\ under Global-Definitions. Bypass Instance warning. +\ Replace this with GlobalLocalValues.fth in your final product. + +\ Make sure this option is turned on. +[flag] Local-Values + +global-definitions + +\ Bypass warning about Instance without altering LocalValuesSupport file +alias generic-instance instance +[macro] bypass-instance f[ noop .( Bypassed instance!) f] + +overload alias instance bypass-instance + +fload LocalValuesSupport.fth +fload LocalValuesDevelSupport.fth + +\ Replace normal meaning of Instance, still in Global scope. +overload alias instance generic-instance + +\ Restore Device-Definitions scope. +device-definitions Added: fcode-utils/localvalues/LocalValuesDevelSupport.fth =================================================================== --- fcode-utils/localvalues/LocalValuesDevelSupport.fth (rev 0) +++ fcode-utils/localvalues/LocalValuesDevelSupport.fth 2006-10-30 09:50:41 UTC (rev 102) @@ -0,0 +1,45 @@ +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Support file for development of FCode that uses Local Values +\ FLoad this right after LocalValuesSupport.fth +\ Remove it from your final product. + +\ Exported Function: max-local-storage-size ( -- n ) +\ Returns the measured maximum size of storage for Local Values +\ used by any given test run. This number can be used to guide +\ the declaration of _local-storage-size_ +\ +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Count the current depth on a per-instance basis, +\ but collect the maximum depth over all instances. + +headers +0 instance value local-storage-depth + +external +0 value max-local-storage-size +headers + +\ Overload the {push-locals} and {pop-locals} routines to do this. +\ Do not suppress the overload warnings; they'll serve as a reminder. +: {pop-locals} ( #locals -- ) + local-storage-depth over - to local-storage-depth + {pop-locals} +; + +: {push-locals} ( #ilocals #ulocals -- ) + 2dup + local-storage-depth + + dup to local-storage-depth + max-local-storage-size max + to max-local-storage-size + {push-locals} +; + + Added: fcode-utils/localvalues/LocalValuesSupport.fth =================================================================== --- fcode-utils/localvalues/LocalValuesSupport.fth (rev 0) +++ fcode-utils/localvalues/LocalValuesSupport.fth 2006-10-30 09:50:41 UTC (rev 102) @@ -0,0 +1,133 @@ +\ (C) Copyright 2005 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ The support routines for Local Values in FCode. + +\ Function imported +\ _local-storage-size_ \ Size, in cells, of backing store for locals +\ \ A constant. If not supplied, default value of d# 64 will be used. +\ +\ Functions exported: +\ {push-locals} ( #ilocals #ulocals -- ) +\ {pop-locals} ( #locals -- ) +\ _{local} ( local-var# -- addr ) +\ +\ Additional overloaded function: +\ catch \ Restore Locals after a throw + +\ The user is responsible for declaring the maximum depth of the +\ run-time Locals stack, in storage units, by defining the +\ constant _local-storage-size_ before floading this file. +\ The definition may be created either by defining it as a constant +\ in the startup-file that FLOADs this and other files in the +\ source program, or via a command-line user-symbol definition +\ of a form resembling: -d '_local-storage-size_=d# 42' +\ (be sure to enclose it within quotes so that the shell treats +\ it as a single string, and, of course, replace the "42" with +\ the actual number you need...) +\ If both forms are present, the command-line user-symbol value will +\ be used to create a duplicate definition of the named constant, +\ which will prevail over the earlier definition, and will remain +\ available for examination during development and testing. The +\ duplicate-name warning, which will not be suppressed, will also +\ act to alert the developer of this condition. +\ To measure the actual usage (in a test run), use the separate tool +\ found in the file LocalValuesDevelSupport.fth . +\ If the user omits defining _local-storage-size_ the following +\ ten-line sequence will supply a default: + +[ifdef] _local-storage-size_ + f[ [defined] _local-storage-size_ true ]f +[else] + [ifexist] _local-storage-size_ + f[ false ]f + [else] + f[ d# 64 true ]f + [then] +[then] ( Compile-time: size true | false ) +[if] fliteral constant _local-storage-size_ [then] + +_local-storage-size_ \ The number of storage units to allocate + cells \ Convert to address units + dup \ Keep a copy around... + ( n ) instance buffer: locals-storage \ Use one of the copies + +\ The Locals Pointer, added to the base address of locals-storage +\ points to the base-address of the currently active set of Locals. +\ Locals will be accessed as a positive offset from there. +\ Start the Locals Pointer at end of the buffer. +\ A copy of ( N ), the number of address units that were allocated +\ for the buffer, is still on the stack. Use it here. + ( n ) instance value locals-pointer + +\ Support for {push-locals} + +\ Error-check. +: not-enough-locals? ( #ilocals #ulocals -- error? ) + + cells locals-pointer swap - 0< +; + +\ Error message. +: .not-enough-locals ( -- ) + cr ." FATAL ERROR: Local Values Usage exceeds allocation." cr +; + +\ Detect, announce and handle error. +: check-enough-locals ( #ilocals #ulocals -- | <ABORT> ) + not-enough-locals? if + .not-enough-locals + abort + then +; + +\ The uninitialized locals can be allocated in a single batch +: push-uninitted-locals ( #ulocals -- ) + cells locals-pointer swap - to locals-pointer +; + +\ The Initialized locals are initted from the items on top of the stack +\ at the start of the routine. If we allocate them one at a time, +\ we get them into the right order. I.e., the last-one named gets +\ the top item, the earlier ones get successively lower items. +: push-one-initted-local ( pstack-item -- ) + locals-pointer 1 cells - + dup to locals-pointer + locals-storage + ! +; + +\ Push all the Initialized locals. +: push-initted-locals ( N_#ilocals-1 ... N_0 #ilocals -- ) + 0 ?do push-one-initted-local loop +; + +: {push-locals} ( N_#ilocals ... N_1 #ilocals #ulocals -- ) + 2dup check-enough-locals + push-uninitted-locals ( ..... #i ) + push-initted-locals ( ) +; + +\ Pop all the locals. +\ The param is the number to pop. +: {pop-locals} ( total#locals -- ) + cells locals-pointer + to locals-pointer +; + +\ The address from/to which values will be moved, given the local-var# +: _{local} ( local-var# -- addr ) + cells locals-pointer + locals-storage + +; + +\ We need to overload catch such that the state of the Locals Pointer +\ will be preserved and restored after a throw . +overload : catch ( ??? xt -- ???' false | ???'' throw-code ) + locals-pointer >r ( ??? xt ) ( R: old-locals-ptr ) + catch ( ???' false | ???'' throw-code ) ( R: old-locals-ptr ) + \ No need to inspect the throw-code. + \ If catch returned a zero, the Locals Pointer + \ is valid anyway, so restoring it is harmless. + r> to locals-pointer +; Added: fcode-utils/localvalues/TotalLocalValuesSupport.fth =================================================================== --- fcode-utils/localvalues/TotalLocalValuesSupport.fth (rev 0) +++ fcode-utils/localvalues/TotalLocalValuesSupport.fth 2006-10-30 09:50:41 UTC (rev 102) @@ -0,0 +1,51 @@ +\ (C) Copyright 2005-2006 IBM Corporation. All Rights Reserved. +\ Licensed under the Common Public License (CPL) version 1.0 +\ for full details see: +\ http://www.opensource.org/licenses/cpl1.0.php +\ +\ Module Author: David L. Paktor dlpaktor(a)us.ibm.com + +\ Control file for loading of Local Values Support file with variants. +\ Command-line Symbol-definitions select whether the support will +\ be under Global-Definitions, and whether to include the extra +\ Development-time support features. +\ +\ The command-line symbols are: +\ Global-Locals +\ and +\ Locals-Release +\ +\ The default is device-node-specific support in a Development-time setting. +\ +\ If Global-Locals is defined, support will be under Global-Definitions +\ If Locals-Release is defined, this is a final production release run, +\ and the Development-time support features will be removed. + +\ Make sure this option is turned on. +[flag] Local-Values + +[ifdef] Global-Locals + \ Load Support file under Global-Definitions. + global-definitions + + \ Bypass warning about Instance without altering LocalValuesSupport file + alias generic-instance instance + [macro] bypass-instance f[ noop .( Bypassed instance!) f] + + overload alias instance bypass-instance +[endif] \ Global-Locals + +fload LocalValuesSupport.fth + +[ifndef] Locals-Release + \ Load Development-time support features + fload LocalValuesDevelSupport.fth +[endif] \ not Locals-Release + +[ifdef] Global-Locals + \ Replace normal meaning of Instance, still in Global scope. + overload alias instance generic-instance + + \ Restore Device-Definitions scope. + device-definitions +[endif] \ Global-Locals

1 0