% This is a file written using noweb
%
% $Id: ftparse.nw,v 1.7 1996/09/17 13:45:01 gaertner Exp $
%

\documentclass{article}
\usepackage{a4,version,pretzeldoc,noweb}
\begin{document}

\includeversion{bigspecs}
\includeversion{impls}
%\excludeversion{bigspecs}% without elaborate specification
%\excludeversion{impls}% without implementation



\title{A Generator for the Prettyprinting Scanner}
\author{Felix G\"artner}
\maketitle


\begin{abstract}
    This file describes the interface \begin{impls}and
    implementation\end{impls} of a new \CPP\ class that can be used to
    build a prettyprinting scanner from a formatted token file.
    A formatted token file is a file containing a description of
    ``formatted tokens'' for prettyprinting a certain programming
    language. It is used within the framework of the ``pretzel''
    project. Pretzel is a universal prettyprinter generator.
\end{abstract}


\tableofcontents


\section{Introduction}

A {\em prettyprinting scanner} is a scanner that not only returns a
token code for the next token from the input but also returns an
indication ({\em attribute}) of how this token is to be
prettyprinted. Thus a prettyprinting scanner doesn't merely return
simple tokens, but {\em formatted tokens}. Such a scanner is a
prerequisite for a {\em prettyprinting parser}, that in turn takes the
formatted tokens and puts them together according to a {\em formatted
grammar}, i.e.~a grammar that is enhanced with formatting rules.
Such a parser can be regarded as a normal prettyprinter.

This file describes the interface of a new \CPP\ class that can be
used to generate such a prettyprinting scanner from rules that are
specified in a file, the so called {\em formatted token file}. About
the precise format of this file confer the main documentation of the
``pretzel'' project, of which this program is a part. I will assume
that you know this format (or at least have seen an example file) and
that you are familiar with the prettyprinting method that is used by
all components. The program of concern here is the actual generator
for the prettyprinting scanner.

@ This is an overview of the header file for the new class.

<<header>>=
#ifndef FTPARSE_H
#define FTPARSE_H

<<Header files to include in the specification>>
<<Declaration of the scanner generator class>>

#endif

@

\begin{impls}
  The implementation of this class will be done using the standard
  \UNIX\ tool Bison, which is a parser generator. The idea is to parse
  the formatted token file and convert it into a valid flex file, from
  which a scanner then can be build (using flex of course).

  So here's the frame of the Bison file.

<<bison-implementation>>=
%{
#include <stdlib.h>
#include "ftparse.h"
#include <assert.h>
%}

<<bison definitions>>
%%
<<bison rules>>
%%
<<bison user code>>

@

\end{impls}



\section{Interface (Overview)}


The thing of interest here is the generator for the prettyprinting
scanner. The generator is a new \CPP\ class called [[Pscan_builder]].
It has a single member function called [[build]] which does all the
work of building. It takes two arguments:
 %
\begin{enumerate}

\item The first one is a pointer to an input stream where it expects to
read the formatted token file from.

\item The second one is a pointer to an output stream, where it writes
the generated code to.

\end{enumerate}
 %
The function returns an integer value to indicate if everything
worked well (0) or if an error occured (1). For a description on how
the generated prettyprinting scanner looks, see section
\ref{sec:scannerinterface} on page \pageref{sec:scannerinterface} below.

There are two more functions for error diagnosis. The first is called
[[error_text]] which returns a more elaborate description of why the
[[build]] function might have returned a 1 as a string. The second is
called [[error_line]] which returns the line of the input file
where this error has occured. The [[debug_on]] and [[debug_off]]
functions allow to watch the generation process in details.

<<Declaration of the scanner generator class>>=
class Pscan_builder {
public:
  Pscan_builder();
  ~Pscan_builder();
  int build(istream*, ostream*);

  // for error diagnosis and debugging:
  char* error_text();
  int error_line();
  void debug_on();      // write what the generator is doing to cerr
  void debug_off();

private:
  <<Private members of the generator class>>
};

@

These are the header files needed.

<<Header files to include in the specification>>=
#include <iostream.h>
@



\section{Interface (Details)}

In this section I will present a complete reference of the new
[[Pscan_builder]] class\begin{impls} together with it's
  implementation\end{impls}.

\begin{impls}

  \subsection{Ideas behind the Implementation}

  The idea behind the implementation is to use bison to generate a
  parser and do the translation from formatted token file to flex file
  on the fly whilst parsing. The format of the formatted token file is
  so similar to a flex file, that this is possible. The parser will
  use the new scanner class [[Ftscanner]] described in the document
  {\tt ftscan.nw}.

  By default bison generates a global parser function [[yyparse]]. We
  want to shield this and all the other globals within the new class.
  So we declare the parser to be pure and redefine the prefixes of the
  globals using the {\tt -p} command line option of bison. We'll use
  the same prefix as used in generating the scanner
  (``[[Pretzel_ft]]'').

<<bison definitions>>=
%pure_parser

%{
#include "ftscan.h"       /* include scanner */
%}
@

  The scanner function expected by [[yyparse]] is called
  [[Pretzel_ftscan]] if we use the {\tt -p} option. But actually it's
  a member function of the scanner class [[Ftscanner]]. How deal with
  this problem?

  We'll define a private member of the [[Pscan_builder]] class to hold
  an instance of the scanner class and use this to scan the input
  file.

<<Private members of the generator class>>=
  Ftscanner* scanner;
@

  We'll also need to include the specification for this class then.

<<Header files to include in the specification>>=
#include "ftscan.h"
@

  So a call to the scanner is like calling the [[scan]] member
  function of the [[scanner]] itself. We need a way to simply trick
  bison into using [[scanner->scan]] everytime it wants to
  call [[Pretzel_ftlex]]. This is more difficult than it seems,
  because [[yyparse]] will be called from inside a member function
  with no idea what scanner object to use.

  We'll get around this problem by using a static variable again for
  communication. It will always hold a pointer to the current scanner
  to be used by [[yyparse]]. This is not very nice but necessary,
  since bison doesn't allow us to redefine the interface to
  [[yyparse]] at all.  This means that the [[build]] member function
  will work well with multiple instances of the [[Pscan_builder]]
  class, but {\em will not be reentrant}. (See also page
  \pageref{yyparseinterface2} for another problem concerning the
  interface of [[yyparse]].)

  \label{yyparseinterface1}

<<bison definitions>>=
%{
static Ftscanner* static_scanner = NULL;
%}
@

  So every time Bison calls [[yylex]], it should call the following
  small dummy function. It is declared static so that multiple parser
  instances don't get upset. Also, since there is a second definition
  of a [[yylex]]-type function in the scanner implementation file, we
  must take care that we don't get multiple versions of the same
  function in the linked code. That's why we have to [[#undef]] the
  [[yylex]] definition that is automatically inserted into the code
  when we use the [[-P]]-option. Also, a few macros have to be
  redefined to take care that this static version of [[yylex]] is the
  only one declared.

<<bison user code>>=
static int yylex(Attribute* *rval)
{
    return(static_scanner->scan(rval));
}
@

  Some versions of Bison (the newer ones) have a [[YYLEX_DECL]] macro
  that can be used to declare [[yylex]]. We'll declare it ourself and
  make sure it isn't declared again.

<<bison definitions>>=
%{
    /* make sure yylex is declared with name ``yylex'' */
#undef yylex
static int yylex(Attribute* *rvalue);
    /* make sure that it's only declared once */
#define YYLEX_DECL
%}
@

  The semantic value of the tokens manipulated within bison are simple
  strings that are joint together while parsing. Joining together
  strings in the right way will result in a giant string, which is the
  valid flex file, when the parsing has finished.

  As we don't want to get into any hassle copying strings from one
  place to another, we'll use the [[Attribute]] class to do this. We
  won't use the [[Format_attributes]] though, but only the
  [[Text_attributes]]. They weren't written for this task, but they
  are certainly good candidates to be used.

<<bison definitions>>=
%{
#include "attr.h"

#define YYSTYPE Attribute*
%}
@

  So now, when parsing the formatted token file and generating the
  flex file, the parser will finally arrive at the end of its work and
  have a huge attribute that contains the contents of the flex file.
  Now what?  Well, we can't output it to the output stream specified
  in the [[build]] function, since there is no way to tell [[yyparse]]
  what output stream to take. Also, we can't use a private member of
  the builder to hold this attribute temporaily, because the
  [[yyparse]] function doesn't have access to it, even if we declare
  it friend to the builder class. (This is because the [[yyparse]]
  function has no means of knowing, which object it was invoked by.)
  So the best thing to do is to store this huge
  attribute in a static variable called [[rest]] and output it later,
  when [[yyparse]] has retuned. (See also page
  \pageref{yyparseinterface} for another problem concerning the
  interface of [[yyparse]].)

  \label{yyparseinterface2}

<<bison definitions>>=
%{
static Attribute* rest = NULL;
%}
@

  A static variable within this module will hold the current text
  describing the error message encountered. Normally this will be a
  syntax error and this is the default entry. But while parsing, we
  might be able to specify more precisely what's gone wrong. In such
  cases we simply assign a new text to this variable.

<<bison definitions>>=
%{
static char* error_message = "syntax error";
%}
@

  To be able to debug the grammar, I've put in the [[debug_on]] and
  [[debug_off]] member functions into the class. They'll simply switch
  on (and off) the bison debugging mechanism. The semantic values of
  the tokens will be output to [[cerr]] too, and so we need to include
  the header of the [[Output]] class too.

<<bison definitions>>=
%{
#define YYDEBUG 1
#include "output.h"
%}
@

\end{impls}


\subsection{Creating and Destroying the Generator}

The constructor and destructor of the [[Pscan_builder]] class do
nothing much. They simply have to be there before you call [[build]].

\begin{impls}

  The implementation is straightforward too.

<<bison user code>>=
Pscan_builder::Pscan_builder()
{
    scanner = NULL;
};
Pscan_builder::~Pscan_builder()
{
    if (scanner != NULL) delete scanner;
};
@

\end{impls}

\subsection{Activation and Results of the Generator}
\label{sec:scannerinterface}

The actual work is performed when you call [[build]]. It takes a
pointer to an input stream [[is]] and a pointer to an output stream
[[os]]. It then starts to read a formatted token file from [[is]] and
outputs code for the prettyprinting scanner to [[os]].

The resulting code is flex code that contains the complete definition
of a new \CPP\ prettyprinting scanner class. This new class is a
subclass of an abstract base class [[Pscan]], that offers one virtual
function [[scan]] that is redefined within the newly generated
subclass.

The exact interface can be found in the file {\tt Pscan.h}. It is:

\label{pretscanspecs}

<<abstract base class for [[Pscan]]>>=
#include<iostream.h>
#include"attr.h"

class Pscan {
public:
  Pscan(istream*) {};
  ~Pscan() {};
  virtual int scan(Attribute**) = 0;
};
@

The interface of the generated prettyprinting scanner is:

<<Interface of the generated scanner>>=
class PSCAN_NAME : public Pscan {
public:
  PSCAN_NAME(istream*);
  ~PSCAN_NAME();
  int scan(Attribute**);
};
@

The name of the generated class is [[PSCAN_NAME]] by default and has
to be redefined using preprocessor macros, such as for example
 %
\begin{quote}
  [[#define PSCAN_NAME Pscan_for_pascal]]
\end{quote}
 %
By default the name of the derived subclass will be [[Ppscan]].

A header file containing the abstract base class would look like this:

<<header file for abstract base class of [[Pscan]]>>=
#ifndef PSCAN_H
#define PSCAN_H

<<abstract base class for [[Pscan]]>>

#endif
@

A header file containing the derived scanner class could look like
this (by default the header file is called {\tt Ppscan.h}). Note that
there is no precaution against double inclusion.

<<header file for prettyprinting scanner>>=
/* header file for a prettyprinting scanner */

/* NB: This file is NOT protected against double inclusion! */

#include "Pscan.h" // include abstract base class

#ifndef PSCAN_NAME
#define PSCAN_NAME Ppscan
#endif

<<Interface of the generated scanner>>
@

\begin{impls}

  The implementation of the [[build]] function is possibly the
  trickiest thing in this entire class. It uses the internal
  [[Ftscanner]] class object and then calls [[yyparse]].

  The assignment [[static_scanner = scanner]] before calling
  [[yyparse]] ensures that the right scanner is used. (See also page
  \pageref{yyparseinterface1} for a description of this problem.)

  We'll have to use a nonstandard [[Output]] for outputting the file
  contents, as the abstract base class can't be used. You can use any
  output subclass that doesn't upset the strings (and
  [[Latex_cweb_output]] is a good candidate for this, especially
  because it comes free with the abstract base class).

<<bison user code>>=
int Pscan_builder::build(istream* is, ostream* os)
{
  scanner = new Ftscanner(is);
  assert(scanner != NULL);
  <<prepare additional text for the generated file>>
  static_scanner = scanner;
  int ret_val = yyparse();
  if (ret_val != 0) {
    return(1);
  } else {
    Latex_cweb_output tmp_output(*os);   // create an output for [[os]]
    rest->print(tmp_output);  // and print the contents there.
    return(0);
  }
}

@

  To ensure that the resulting flex code gets right, a lot of
  declarations and additional \CPP\ code have to be added to the file.
  This is done by declaring some additional strings that get appended
  to the front and back of the flex file when it is produced. These
  additional lines essentially contain the rest of the \CPP\
  implementation of the interface specified above.

  There are three static variables holding text:

  \begin{itemize}
  \item [[initial_lines]] will hold text that is put to the very
    beginning of the resulting flex file.
  \item [[before_lines]] will hold text that is put just before the
    \verb+%%+ line that starts the rules part of the flex file.
  \item [[after_lines]] will be put behind a second \verb+%%+ line and
    will contain additional \CPP\ code.
  \end{itemize}

  The corresponding integer variables will be used to turn these
  string arrays into [[Attribute]]s, which in turn are held in the
  corresponding attribute variables.

<<bison definitions>>=
%{
static
char *initial_lines[] = {
        "/************************************************\n",
        " * This is a flex file generated by pretzel     *\n",
        " *                                              *\n",
        " ************************************************/\n\n",
  <<contents of [[initial_lines]]>>
  "\n"
  };
static
char *before_lines[] = {
        "/************************************************\n",
        " * some pretzel definitions start here:         *\n",
        " ***********************************************/\n\n",
  <<contents of [[before_lines]]>>
  "\n"
  };
static
char *after_lines[] = {
        "/************************************************\n",
        " * additional pretzel C code follows here:      *\n",
        " ***********************************************/\n",
        <<contents of [[after_lines]]>>
        "/************************************************\n",
        " * additional user defined C code follows here: *\n",
        " ***********************************************/\n",
        "\n"
  };

static
const int initial = sizeof(initial_lines)/sizeof(initial_lines[0]);
static
const int before = sizeof(before_lines)/sizeof(before_lines[0]);
static
const int after = sizeof(after_lines)/sizeof(after_lines[0]);

static Attribute *initial_text = create(null);
static Attribute *before_text = create(null);
static Attribute *after_text = create(null);
%}
@

All these lines are accumulated into the three static variables and
are inserted into the file within the grammar specified in section
\ref{sec:grammarrules} on page \pageref{sec:grammarrules}. Using the
constants like [[after]] spares us from counting the line numbers
every time we change the text. Thanks to Andreas Kirschbaum for
suggesting this hack.

<<prepare additional text for the generated file>>=
        int i=0;
        for (i=0; i<initial; i++) {
                initial_text =
                        join(initial_text,
                                create(initial_lines[i])
                        );
        }
        for (i=0; i<before; i++) {
                before_text =
                        join(before_text,
                                create(before_lines[i])
                        );
        }
        for (i=0; i<after; i++) {
                after_text =
                        join(after_text,
                                create(after_lines[i])
                        );
        }
@

The contents of the line arrays contain the actual definitions
(i.e.~the code) of the prettyprinting scanner. This code has to obey
the specifications from above (see page \pageref{pretscanspecs}).

The [[initia_lines]] contain the overall includes of headers,
redefinitions of [[YY_INPUT]] and static variables. Mainly, a
prettyprinting scanner will work in a way like the scanner described
in the file {\tt ftscan.nw}.



<<contents of [[initial_lines]]>>=
        "%{                                                     \n\n",
        " /* some needed headers: */                            \n",
        "#include<iostream.h>                                   \n",
        "#include\"attr.h\"                                     \n\n",
        " /* we'll define yywrap() which is needed for the scanner\n",
        "  * here, so we won't have to bother about it if we're \n",
        "  * doing the standard case. You can define your own version\n",
        "  * by #undef'ing PSCAN_YYWRAP and supplying your own definition\n",
        "  * in the definitions section.                        \n",
        "  */                                                   \n",
        "#define PSCAN_YYWRAP                                   \n\n",
        " /* input will come from this istream */               \n",
        "static istream * yyin_stream;                          \n\n",
        " /* redefine YYINPUT to read from input streams */     \n",
        "#undef YY_INPUT                                        \n",
        "#define YY_INPUT(buf, result, max_size) {         \\\n",
        "  if (yyin_stream->eof()) result=YY_NULL;         \\\n",
        "  else {                                          \\\n",
        "    yyin_stream->read(buf, max_size);             \\\n",
        "    result=yyin_stream->gcount();                 \\\n",
        "  }                                               \\\n",
        "}                                                      \n\n",
        "%}                                                     \n\n",
@

Within the [[before_lines]] we have to put everything that can be
redefined by the user and has to be mentioned before the body of
[[yylex]] comes along.

<<contents of [[before_lines]]>>=
        "%{                                                     \n\n",
        " /* the default name of the include file with the token\n",
        "  * code definitions is stored in PTOKDEFS_NAME and    \n",
        "  * defaults to `ptokdefs'.                            \n",
        "  */                                                   \n",
        "#ifndef PTOKDEFS_NAME                                  \n",
        "#define PTOKDEFS_NAME \"ptokdefs.h\"                   \n",
        "#endif                                                 \n\n",
        " /* now we have to include the token definitions */    \n",
        "#include PTOKDEFS_NAME                                 \n\n",
        " /* allow user to choose generated scanner class name*/\n",
        "#ifndef PSCAN_NAME                                     \n",
        "#define PSCAN_NAME Ppscan                              \n",
        "#include \"Ppscan.h\"                                  \n",
        "#endif                                                 \n\n",
        " /* redefine the interface of the scanner */           \n",
        "#undef YY_DECL                                         \n",
        "#define YY_DECL int yylex (Attribute**rvalue)          \n\n",
        " /* here's the standard version of yywrap() */         \n",
        "#ifdef PSCAN_YYWRAP                                    \n",
        "extern \"C\" int yywrap() {                            \n",
        "    return(1);                                         \n",
        "}                                                      \n",
        "#endif                                                 \n\n",
        "%}                                                     \n\n",
@

The [[after_lines]] contain the rest of the definition of the
implemented class.

<<contents of [[after_lines]]>>=
        "PSCAN_NAME::PSCAN_NAME (istream* is ) : Pscan(is)\n",
        "{                                              \n",
        "    yyin_stream = is;                          \n",
        "}                                              \n",
        "int PSCAN_NAME::scan(Attribute* *rvalue)       \n",
        "{                                              \n",
        "    return(yylex(rvalue));                     \n",
        "}                                              \n",
        "PSCAN_NAME::~PSCAN_NAME ()                     \n",
        "{                                              \n",
        "    yyrestart(NULL);                           \n",
        "}                                              \n",
@


\end{impls}

\subsection{Error Diagnosis and Debugging}

The error diagnosis functions can be used, if the [[build]] function
has returned an error code (1). Calling [[error_text]] will result in
a more elaborate explanation of the error cause (mostly ``syntax
error'') and [[error_line]] will reveal the line number of the
formatted token file, where the generator choked.

\begin{impls}

  The implementation is straighforward again. The [[error_text]]
  function simply returns the internal error message and the
  [[error_line]] function askes the scanner and returns its line
  value.

<<bison user code>>=
char *Pscan_builder::error_text()
{
  return(error_message);
}
int Pscan_builder::error_line()
{
  return(scanner->line());
}
@

The debugging member functions \ldots % XXX

\end{impls}






\begin{impls}


\section{Bison Implementation of the Generator}
\label{sec:grammarrules}

This section contains the complete bison rules for the generator
function [[yyparse]]. It is the main part of the implementation. All
this was written for GNU bison version 1.19.

\subsection{Token Declarations}

Bison will expect the scanner to return the following tokens:

<<bison definitions>>=
%token REGEXPR
%token ID
%token STRING
%token FMTCMD
%token TEXT
%token A_LINE
%token PERCENT_PERCENT
%token UNKNOWN
%token BEG_CODE
%token CODE
%token END_CODE
@

\subsection{The Grammar Rules}

Now we'll go through the grammar rules one by one in a top down
fashion.

A formatted token file consists of lots of top lines, a final top line
(which is `\verb+%%+') and a list of definitions possibly followed by
again a `\verb+%%+' line and another row of lines.

If we want to insert stuff into the definitions section of the output,
we have to do this between [[$1]] and [[$2]] of the [[join]] command
below. The [[preflex_file]] is the final token to which everything
should be reduced. So we save the semantic value of this final token
in the [[contents]] member function of the generator class.

<<bison rules>>=
preflex_file    : lines PERCENT_PERCENT def_list        {
                        rest=$$=join(initial_text,$1,
                                before_text,$2,$3,
                                create("\n%%\n"),after_text);}
                | lines PERCENT_PERCENT def_list PERCENT_PERCENT lines {
                        rest=$$=join(initial_text,$1,
                                before_text,$2,$3,$4,after_text,$5);}
;

lines           : /* empty */           {rest=$$=create(null);}
                | lines A_LINE          {rest=$$=join($1,$2);}
;

@

A definition list collects definitions of formatted grammar rules.  A
definition always starts with a regular expression. Definitions can
have an empty body. This is good for specifying rules for example to
eat up whitespace. They will come out like this in the output file:

\begin{quote}
  {\it regular-expression} \hspace{1cm} \verb+/* empty */+
\end{quote}

<<bison rules>>=
def_list        : /* empty */           {rest=$$=create("\n");}
                | def_list def          {
                        rest=$$=join($1,$2,create("\n"));}
;

def     : REGEXPR               {
                rest=$$=join(
                        create("\n"),
                        $1,
                        create("\t\t/* empty */\n")
                        );}
;

@

The next case sees a regular expression with a token name following it
but no attribute definition. Here we need the default mechanism.

The generated line in the output file will look like this:
 %
\begin{quote}
  {\it regular-expression}\hspace{1cm}[[{*rvalue=create(yytext);return(ID);}]]
\end{quote}

<<bison rules>>=
def     : REGEXPR ID            {
                rest=$$=join(
                        create("\n"),
                        $1,
                        create("\t\t{*rvalue=create(yytext);return ("),
                        $2,
                        create(");}\n")
                        );}
;

@

Of course you can influence the attribute by stating an attribute
definition after the token code. From version 2.0 of pretzel onwards
we'll have to deal with code fragments within the attribute definition
as well. This will be a little tricky. We'll allow the user to
explicitly state starting code (which is inserted before the attribute
is built) and ending code (which is put behind the attribute building
process but before the final [[return]]). Code within attribute
definitions is expected to be an [[Attribute*]] and can thus be
handled as if it were a string for format command.

Conventions: Starting code and ending code are writte outside of the
attribute definition. Any one of the three things (starting code,
attribute definition, ending code) can be omitted. The simple case
that sees none of them has been handled above. Here are the more
complex cases.

To enhance readability of the output file, every created attribute is
put onto a new line and indented a little to the right.  Also note,
that the [[join]] function is only called with two parameters at most.
This should be fixed soon. Code will be put onto lines by itself and
indented to make sure it follows flex conventions. So you will have
something like this in the output file:
 %
\begin{verbatim}
   REGEXPR      { ...possible starting code...
                *rvalue=join(
                ...translated attribute definition...
                );
                ...possible ending code...
                return(ID);}
\end{verbatim}
 %
We'll start off by saying, what [[code]] is. Code is encircled by
[[BEG_CODE]] and [[END_CODE]] delimiters which are stripped off in the
output file. Code lines following code are indented a little.

<<bison rules>>=
code    : BEG_CODE code_lines END_CODE     { rest=$$=$2; }
;

code_lines : CODE                          { rest=$$=$1; }
        | code_lines CODE                  {
                rest=$$=join($1, create("\t\t\t"), $2);
                }
;

opt_code : code                            { rest=$$=$1; }
         | /* empty */                     { rest=$$=create(null); }
;

@

In the rules for definitions, we might have optional code before the
attribute definition or behind it. have to look out for code. Starting
and ending code are optional code. Lookout here that we don't use
[[join]] with more arguments than allowed.

<<bison rules>>=
def     : REGEXPR ID opt_code '{' attr_def '}' opt_code   {
                rest=$$=join(
                        create("\n"),
                        $1,
                        create("\t\t{"),
                        $3,
                        create(" *rvalue="),
                        $5,
                        create("\n\t\t"),
                        join( $7,
                          create(" return("),
                          $2,
                          create(");}\n"))
                        );}
;

@

Here is the case for subsequent definitions joined together
with the `\verb+|+' sign.

A common error is to use lowercase in the token identifiers. This is
tried to fight here.

<<bison rules>>=
def     : REGEXPR '|' def       {
                rest=$$=join(create("\n"),$1,create("\t\t"),$2,$3);}

        | REGEXPR UNKNOWN       {
            error_message = "illegal identifier (USE UPPERCASE)"; return(3); }
;

@

An attribute definition is enclosed by curled brackets that have been
spotted already above.  As the `[[*rvalue=]]' part too has already
been generated above, we only need to produce a `[[join]]' stuff here.

<<bison rules>>=
attr_def : attr_list    {rest=$$=join(create("join(\n\t\t\t"),
                                        $1,
                                        create(");\n\t\t\t")
                                        );}
;

attr_list : attr                {rest=$$=$1;}
;

@

Here we ensure that only instances of join will be called with at most
two arguments. We'll also allow to omit the [[+]]. This is an
enhancement of Pretzel version 2.0.

<<bison rules>>=
attr_list       : attr_list '+' attr    {rest=$$=join(
                                        create("join(\n\t\t\t\t"),
                                        $1,
                                        create(",\n\t\t\t\t"),
                                        $3,
                                        create("\n\t\t\t)")
                                        );}
                | attr_list attr        {rest=$$=join(
                                        create("join(\n\t\t\t\t"),
                                        $1,
                                        create(",\n\t\t\t\t"),
                                        $2,
                                        create("\n\t\t\t)")
                                        );}
;

@

Finally we say, what an [[attr]] actually is. From version 2.0 of
\pretzel{} we also allow code to be an attribute.

<<bison rules>>=
attr    : STRING                {rest=$$=join(create("create("),
                $1,create(")"));}
        | FMTCMD                {rest=$$=join(create("create("),
                $1,create(")"));}
        | TEXT                  {rest=$$=create("create(yytext)");}
        | code                  {rest=$$=$1;}
        | UNKNOWN               {error_message = "illegal attribute";
                return(2);}
;

@



\section{Additional bison Functions}

Here's the definition of the [[yyerror]] routine needed by the
generated parser. It isn't supplied automatically, so we have to cater
for it. We don't want to output any bison error messages so we simply
return 0. Its name has to contain the same prefix as specified on the
bison command line.

<<bison user code>>=
int Pretzel_fterror (const char *s)
{
        return 0;
}
@

This line is needed so that Bison doesn't declare the supposed
[[yyerror]] routine extern.

<<bison definitions>>=
%{
int Pretzel_fterror (const char *s);
#define YYERROR_DECL
%}
@



\end{impls}



\begin{bigspecs}

\section{Examples}

This section contains some examples of how to use the [[Pscan]] class
and subclasses.

\subsection{A simple and standard example}

Here's an example for a formatted token file for a normal (default)
prettyprinting scanner. It's an example that can be used to
prettyprint simple equations using \TeX's math mode.

<<simple formatted token file example>>=
ALPH    [a-zA-Z]
ALPHNUM [a-zA-Z0-9]
NUM     [0-9]

%%

{ALPH}{ALPHNUM}*        ID              { "\\hbox{" + ** + "}" }
{NUM}+                  NUMBER
"+"                     PLUS
"-"                     MINUS
"/"                     DIV
"*"                     MULT            { "\\cdot" }
"="                     EQUAL

@

If you run this file through the generator, you'll get a class called
[[Ppscan]] that looks like this:

<<simple example class interface>>=
class Ppscan : public Pscan {
  Ppscan(istream*);
  ~Ppscan();
  int scan(Attribute**);
};
@

The header file for this class is in the file {\tt Ppscan.h}. So to
use the scanner, your program should look something like this:

\begin{verbatim}
  /* program using my new prettyprinting scanner */

  #include"Ppscan.h"

  int main() {
  ...
  Ppscan mypscanner(&cin);
  Attribute* look;
  int tokval;
  ...
  while ( (tokval = mypscanner.scan(look)) != 0 ) {
    ... /* do something with it */
    }
  ...
  return(0);
  }
\end{verbatim}

\subsection{A more complex example}

Say, you are using multiple scanners in your program. You'll need to
rename different instances of the generated scanner class to be able
to do so. For example, you want to have two scanner classes, one for
\C\ and one for \PASCAL{}. Let's look at one of them, the one for
\PASCAL. We want to call the class [[Pascalpscanner]].

To rename the class, you have to put a [[#define]] into the formatted
token file.

<<complicated formatted token file example>>=
%{
/* redefine name of generated scanner class */

#define PSCAN_NAME Pascalpscanner
%}

ALPH     [a-zA-Z]
ALPHNUM  [a-zA-Z0-9]

%%

{ALPH}{ALPHNUM}*      ID

   // more token definitions follow ...

@

Now you'll get a subclass that looks like this:

<<complex example class interface>>=
class Pascalpscanner : public Pscan {
  Pascalpscanner(istream*);
  ~Pascalpscanner();
  int scan(Attribute**);
};
@

You'll have to rename the header file {\tt Ppscan.h} into something
like {\tt Pascalpscan.h}. It is important that the [[#define]] is
issued {\em before} you include the header file in your program that
uses the scanner. So it should look something like this:

\begin{verbatim}
  /* program using the new Pascal prettyprinting scanner */

  #define PSCAN_NAME Pascalpscanner
  #include"Pascalpscan.h"

  int main() {
  ...
  Pascalpscanner pscanner(&cin);
  ... /* etc. */
\end{verbatim}

Of course it could be better to change the name of the scanner class
right inside the file {\tt Pascalpscanner} itself. This would get rid
of the nasty [[#define]]. But this is one possibility you can choose
from.



\end{bigspecs}



\end{document}


%
% $Log: ftparse.nw,v $
% Revision 1.7  1996/09/17  13:45:01  gaertner
% Added the `extern "C"' to the generated scanner too.
%
% Revision 1.6  1996/09/12  18:42:44  gaertner
% Refined declaration handling of yylex and yyerror. Now it
% works on sun and HP.
%
% Revision 1.5  1996/09/12  17:58:24  gaertner
% Changed yylex to static, so that it won't magle with the
% other global yylex. It seems that the ITI compiler fixes this
% on the fly!
%
% Revision 1.4  1996/04/12  15:30:31  gaertner
% Caters for code fragments within attribute definitions now.
%
% Revision 1.3  1996/04/12  08:26:48  gaertner
% Allowed to omit the `+' in attribute definitions.
%
% Revision 1.2  1996/03/21  11:02:27  gaertner
% Added empty implementation of constructor and destructor of
% abstract base class Pscan.
% Added an #include line for the scanner to the generated output.
% Added call :Pscan(is) to constructor of Ppscan.
%
% Revision 1.1  1996/03/19  16:02:41  gaertner
% Initial revision
%
%
