% This is a file written using noweb
%
% $Id: ftscan.nw,v 1.6 1996/09/12 18:44:56 gaertner Exp $
%

\documentclass{article}
\usepackage{a4,version,pretzeldoc,noweb}
\begin{document}

\includeversion{bigspecs}
%\includeversion{impls}
%\excludeversion{bigspecs}% without elaborate specification
\excludeversion{impls}% without implementation



\title{A Scanner for Formatted Token Files}
\author{Felix G\"artner}
\maketitle


\begin{abstract}
    This document describes the interface \begin{impls}and
    implementation\end{impls} of a scanner for formatted token files.
    A formatted token file is a file containing a description of
    ``formatted tokens'' for prettyprinting a certain programming
    language. It is used within the framework of the ``pretzel''
    project. Pretzel is a universal prettyprinter generator.
\end{abstract}


\tableofcontents


\section{Introduction}

A {\em prettyprinting scanner} is a scanner that not only returns a
token code for the next token from the input but also returns an
indication ({\em attribute}) of how this token is to be
prettyprinted. Thus a prettyprinting scanner doesn't merely return
simple tokens, but {\em formatted tokens}. Such a scanner is a
prerequisite for a {\em prettyprinting parser}, that in turn takes the
formatted tokens and puts them together according to a {\em formatted
grammar}, i.e.~a grammar that is enhanced with formatting rules.
Such a parser can be regarded as a normal prettyprinter.

This file describes the interface \begin{impls}and flex
implementation\end{impls} of a new \CPP\ class used to generate such a
prettyprinting scanner from rules that are specified in a file, the so
called {\em formatted token file}. About the precise format of this
file confer the main documentation of the ``pretzel'' project, of
which this program is a part. I will assume that you know this format
(or at least have seen an example file) and that you are familiar with
the prettyprinting method that is used by all components.

Here's an overview of the frame of the header file:

<<header>>=
#ifndef FTSCAN_H
#define FTSCAN_H

<<Header files to include in the specification>>
<<Declaration of the formatted token file scanner class>>

#endif

@

\begin{impls}

  The implementation will use the flex scanner generator (installed on
  the development site is version 2.4.6). We won't use the feature of
  flex to create a \CPP\ scanner class, because we want to stick with
  stuff that has been tested and that will probably work. A flex file
  has the following frame.

<<flex-implementation>>=
%{
#include"ftscan.h"    /* include own header */
%}

<<flex definitions>>
%%
<<flex rules>>
%%
<<flex user code>>

@

\end{impls}




\section{Interface (Overview)}

This section describes the interface of the scanner that scans
formatted token files. Formatted token files contain token definitions
and descriptions of associated attributes for a prettyprinter. This
scanner is used by a parser that checks the syntax of a given
formatted token file.

There are basicly three things you have and want to do with the scanner:

\begin{itemize}
\item Create a new scanner and initialize it to tell it where to take
  the input characters from.
\item Call it, to know, which token is next in the input stream (this
  is called ({\em scanning}).
\item Destroy the scanner explicitly (to free memory, clean up or whatever).
\end{itemize}

Here's the declaration of the new scanner class.

<<Declaration of the formatted token file scanner class>>=
class Ftscanner {
public:
  Ftscanner(istream*);    /* create and initialize new scanner */
  ~Ftscanner();           /* remove a scanner */
  int scan(Attribute**);  /* scan for a new token */

  int line();             /* returns line number of last token scanned */

  <<private members of the new scanner class>>
};

@ The function [[line]] is an additional member function aimed to help
debugging.

The formatted look of a newly scanned token is returned by a call by
reference argument from the [[Attribute]] class. See the {\tt attr.nw}
file for a description of this class. Anyway, we have to include
these headers.

<<Header files to include in the specification>>=
#include"attr.h"       /* the Attribute class */
#include<iostream.h>
@




\section{Interface (Details)}

In this section I will present every member function of the new
scanner class in a detailed fashing for a complete
reference\begin{impls} and will explain the ideas and details behind
  the implementation\end{impls}.

\begin{impls}

  A scanner generated by flex is mainly a function called [[yylex]]
  that scans from a given [[FILE*]] and returns token codes as
  integers. We'll use this function to scan the formatted token file
  and to return an additional string with the token code that describe
  the token in more detail and can be used to piece together the
  target file implementing the prettyprinting scanner. We'll use the
  [[Attribute]] class fo this purpose. (Though it wasn't designed for
  it, we can still use it here.) We also have to redefine the
  interface for [[yylex]] to suit these requirements.

<<flex definitions>>=
%{
#undef YY_DECL
#define YY_DECL int yylex(Attribute* *rvalue)
%}
@

  By default, the name of the scanner function is [[yylex]]. By
  specifying the {\tt -P} option at the command line you can redefine
  the ``yy'' prefix and thus change the name of the scanner function.
  The implementation expects that the scanner is generated using the
  option ``{tt -PPretzel_ft}'' and thus will expect the scanner
  function to be called:
   %
  \begin{quote}
    [[Pretzel_ftlex]]
  \end{quote}

\end{impls}

A short word about using multiple scanners within one program: Because
of drawbacks of the underlying algorithm \begin{impls}(the interface
of flex)\end{impls} the resulting scanner is not reentrant and using
multiple instances of the scanner class in the same program will be
extremely hazardrous. This might seem a severe drawback, but isn't too
bad in the end, because it was built to be used in only one instance.
\begin{impls}This is also the reason why I shouldn't have used a class
to implement it, but the interface was designed before learning about
flex' deficiencies.\end{impls}

\begin{impls}

  Because of the above note, we'll use static variables instead of
  private members to handle the private information of the class.

<<private members of the new scanner class>>=
/* none */
@

  We want the scanner to read from an input stream and not from a
  [[FILE*]], so we have to take care about this. We'll define a
  static variable that holds the [[istream]] object pointer and will
  be used by the scanner.

<<flex definitions>>=
%{
static istream* yyin_stream;
%}
@

  Also, we have to modify the way, in which [[yylex]] reads it's
  data. This is done by redefining the [[YY_INPUT]] macro. Thanks to
  Christine Detig for the hint.

<<flex definitions>>=
%{
#undef YY_INPUT
#define YY_INPUT(buf, result, max_size) {               \
        if (yyin_stream->eof()) result=YY_NULL;         \
        else {                                          \
                yyin_stream->read(buf, max_size);       \
                result=yyin_stream->gcount();           \
        }                                               \
}
%}
@

  The current line number has to be stored somewhere too. This is done
  within the static variable [[line]].

<<flex definitions>>=
%{
static int line_number = 1;
%}
@

\end{impls}



\subsection{Creating and Initializing a new Scanner}

Here's the constructor of the class [[Ftscanner]]. It takes a pointer
to an input stream (\CPP\ [[istream]] object) as argument and reset
the scanner's internals.

\begin{impls}

  The implementation of the constructor is pretty easy. It simply
  stores the given input stream pointer and resets the line number
  count. The line number is initially set to 1 since we have no line 0.

<<flex user code>>=
Ftscanner::Ftscanner(istream* is)
{
  line_number = 1;
  yyin_stream = is;
}

@


\end{impls}



\subsection{Destroying a Scanner}

The [[~Ftscanner]] simply destroys the specified scanner again.

\begin{impls}
  This is implemented by resetting the internal scanner in case we
  want to scan a new formatted token file. Note that although the
  scanner object dissappears, the internal scanner state still is
  valid.

<<flex user code>>=
Ftscanner::~Ftscanner()
{
  yyrestart(NULL);
}

@

\end{impls}



\subsection{Scanning}

The next function is the normal scanner member function called
[[scan]]. It returns token codes as integer values together with the
corresponding attribute (as a call by reference argument). Table
\ref{tab:returnvalues} shows all the possible values that [[scan]]
might return. They are partly defined as \C\ macros, and partly simply
the character values of the given symbols. The macros are defined in a
header file that is generated by the parser for formatted token files
(the file {\tt ftokdefs.h}). See the specifications in {\tt
ftparse.nw} for details.

\begin{impls}
  The implementation has to include this header file.

<<flex definitions>>=
%{
#include "ftokdefs.h"
%}

@

\end{impls}

From version 2.0 of \pretzel{} onwards it is allowed to put C code
into attribute definitions. The `new' tokens [[BEG_CODE]], [[CODE]]
and [[END_CODE]] reflect this change. It contains a line of code
(possibly followed by more) to be inserted into the attribute
definition.


\begin{table}[htbp]
  \begin{center}
    \leavevmode
    \begin{tabular}{ll}
symbolic name & meaning \\
\hline
[[{]]        & opening curly bracket \\
[[}]]        & closing curling bracket \\
[[+]]	     & a `plus' sign \\
[[|]]        & a vertical line \\
[[REGEXPR]]  & a regular expression \\
[[ID]]       & an identifier \\
[[STRING]]   & a string \\
[[FMTCMD]]   & a format command \\
[[TEXT]]     & the `[[**]]' symbol \\
[[BEG_CODE]] & starting code delimiter in attribute definitions \\
[[CODE]]     & a line of code in attribute definitions \\
[[END_CODE]] & ending code delimiter in attribute definitions \\
[[A_LINE]]   & a line of text (code, etc.) \\
[[PERCENT_PERCENT]]  & the `[[%%]]' line \\
[[UNKNOWN]]  & a token that couldn't be matched \\
    \end{tabular}
    \caption{(symbolic) return values of [[scan]] member function.}
    \label{tab:returnvalues}
  \end{center}
\end{table}

Basicly a formatted token file cosists of a `limbo' of \C\ code that
is followed by a line containing `\verb+%%+'.  After that come a long
series of formatted token definitions that basicly consist of a
regular expression, a symbolic name for this token and an optional
formatting instruction for this token. I'll have a short example ready
below.

The [[scan]] member function has a call by reference argument of type
[[Attribute*]] to return the semantic value of the token, whose token code
is returned as return value.

\begin{impls}

  The [[scan]] member function is a delayed call to [[yylex]].

<<flex user code>>=
int Ftscanner::scan(Attribute* *rvalue)
{
    return(yylex(rvalue));
}

@ The exact behaviour, i.e.~a complete specification of the flex
regular expressions can be found in section \ref{sec:flexbody} on page
\pageref{sec:flexbody}.

\end{impls}



\subsection{Debugging}

There is an additional member function that can be used for debugging
purposes. The value of [[line()]] returns the line number that the token
was in that was last returned.

\begin{impls}

  Looking at the implementation we have to note that the internal
  variable has to be incremented every time we scan a newline in the
  input file. This has to be checked in the definitions of section
  \ref{sec:flexbody} and is a possible bug source!

<<flex user code>>=
int Ftscanner::line()
{
  return(line_number);
}
@

\end{impls}






\begin{impls}

\section{The flex Token descriptions}
\label{sec:flexbody}

This section contains the complete flex implementation of the scanner
class. Note that we have to increment the current line number every
time we encounter and scan a newline.


The really important things in the formatted token file are the lines
inbetween the \verb+%%+ tokens (or between the first and the end of
the file if no user code is given). So we'll have the scanner ignore
everything from the beginning of the file up to the first two percents
and then start it's real work. We'll distinguish between these two
states using an exclusive start condition [[R]] (for ``rules'').

<<flex definitions>>=
%x R
@

Now lets have a look on how the scanner reads the top lines and gets
into this start condition.

When the scanner finds the first `\verb+%%+', he'll return a
[[PERCENT_PERCENT]] token and switch into ``rules'' mode. Until he
finds such a line, he'll keep on returning the prelude of the file
line by line.

<<flex rules>>=
^"%%"\n         {
                *rvalue=create(yytext);
                BEGIN(R); line_number++;
                return(PERCENT_PERCENT);
                }

        /* Otherwise we simply return one line at a time */
.*\n            {
                *rvalue=create(yytext);
                line_number++;
                return A_LINE;
                }

@

So, assume that we're in `rules mode', we'll have to look out for
unindented regular expressions, token identifiers and attribute
definitions.

<<flex rules>>=
        /* The following regular expression is a regular expression
         * for regular expressions!
         */
<R>^(([^ \t\n])|(\\\ ))+        *rvalue=create(yytext); return REGEXPR;

        /* scan an identifier (only uppercase allowed */
<R>[A-Z][A-Z0-9_]*      *rvalue=create(yytext); return ID;

@

Now for the attribute definitions. They consist of a series of strings
and formatting instructions within curled brackets. Additionally,
there might be the code delimiters enclosing one or more code lines.
Because code is somewhat more difficult to scan as formatted token
files, we'll introduce a new start condition [[AC]] for ``attribute
code''. It's exclusive too.

<<flex definitions>>=
%x AC
@

Here are the things we might encounter.

<<flex rules>>=
<R>"{"          *rvalue=create(yytext); return yytext[0];

<R>"}"          *rvalue=create(yytext); return yytext[0];

<R>"["		*rvalue=NULL; BEGIN(AC); return BEG_CODE;

@ Before we continue, a short thought: Say, we've seen an opening code
delimiter and we're in [[AC]] mode. Now all lines must be returned up
to the point where we find a matching closing delimiter. Closing
delimiters (usually an angled bracket) must be escaped if they are
used within the code. The escape character is stripped off when the
string is returned using the [[unescaped]] function.

<<flex rules>>=
<AC>(([^\]\n])|(\\\]))*	        *rvalue=create(unescaped(yytext)); return CODE;

<AC>(([^\]\n])|(\\\]))*\n	*rvalue=create(unescaped(yytext)); line_number++; return CODE;

<AC>"]"	       	*rvalue=NULL; BEGIN(R); return END_CODE;

@

The [[unescaped]] function removes backslashes from a given string if
they are followed by a closing delimiter (currently a `{\tt ]}'). The
string length shrinks by the number of backslashes removed. [It's
declared [[static]] because an identical function occurs in the file
{\tt fgscan.nw}. A better solution would be to add it to the pretzel
library.]

The implementation has two pointers into the array. The first points
to the position where the next unescaped character should be placed,
the second one holds the position of the next character to be
inspected. We only shorten the given string. That's why we don't
[[malloc]] storage and work on a copy.

<<flex definitions>>=
%{
static char* unescaped(char* s)
{
    int left = 0;   /* next position to place [[s[right]]] */
    int right = 0;
    while (s[right] != 0) { // should be `!= NULL', shouldn,t it, but
                            // some compilers choke on it
        if ((s[right] == '\\') && (s[right+1] == ']')) {
            right++;
        }
        s[left] = s[right];
        left++; right++;
    }
    s[left] = s[right];  /* terminate string */
    return(s);
}
%}
@

Here's what else we can expect in attribute definitions.

<<flex rules>>=
<R>\"[^\"]*\"   *rvalue=create(yytext); return STRING;

<R>"+"          *rvalue=create(yytext); return yytext[0];

<R>"**"         *rvalue=create("yytext"); return TEXT;

<R>null                 |
<R>indent               |
<R>outdent              |
<R>opt0                 |
<R>opt1                 |
<R>opt2                 |
<R>opt3                 |
<R>opt4                 |
<R>opt5                 |
<R>opt6                 |
<R>opt7                 |
<R>opt8                 |
<R>opt9                 |
<R>backup               |
<R>break_space          |
<R>force                |
<R>big_force            |
<R>no_indent            |
<R>cancel               *rvalue=create(yytext); return FMTCMD;

@

We'll allow the following abbreviations:

<<flex rules>>=
<R>bsp          *rvalue=create("break_space"); return FMTCMD;
<R>b_force              *rvalue=create("big_force"); return FMTCMD;

@

Tokens with the same ID and attribute definition may be concatenated
by a vertical line.

<<flex rules>>=
<R>"|"          *rvalue=create(yytext); return yytext[0];
@

Here are a few extras. We'll eat up whitespace, but are carefull
enough to increment the line number. Also, we can simply ignore the
comments.

<<flex rules>>=
<R>[\ \t]       /* eat up whitespace */
<R>\n           line_number++;
<R>"//".*       /* ignore comments */

@

To finish the rules and allow the user to add some code at the end of
the file, this is the rule to get out of the `rules mode' again.

<<flex rules>>=
<R>^"%%"\n      {
                *rvalue=create(yytext);
                BEGIN(INITIAL);
                line_number++;
                return PERCENT_PERCENT;
                }

@

In case we meet something that we can't match, we'll return an
[[UNKNOWN]] token.

<<flex rules>>=
<R>.            return(UNKNOWN);

@


\section{Additional flex functions}

This section contains the definition of the [[yywrap]] function needed
to complete a flex scanner. Returning 1 means that the scanner can
exit when receiving an end of file indication.

The [[extern "C"]] is due to Lee Wittenberg. It gets around a linkage
error with some versions of gcc.

<<flex definitions>>=
%{
extern "C" int yywrap() {
    return(1);
}
%}
@



\end{impls}





\end{document}


%
% $Log: ftscan.nw,v $
% Revision 1.6  1996/09/12  18:44:56  gaertner
% Added extern "C" (thanks Lee).
%
% Revision 1.5  1996/09/02  16:02:48  gaertner
% Changed the initialization routines and fixed `unescaped' to work
% with g++
%
% Revision 1.4  1996/04/12  15:29:45  gaertner
% Caters for code fragments within attribute definitions.
%
% Revision 1.3  1996/03/19  15:55:38  gaertner
% Tested and working version using new interface.
%
% Revision 1.3  1996/03/19  15:55:38  gaertner
% Tested and working version using new interface.
%
% Revision 1.2  1996/03/19  13:28:23  gaertner
% Version using the -+ option of flex to produce the scanner. Seems to
% upset the Output class (see diary). Saving it so I might change back
% to it again.
%
% Revision 1.1  1996/03/18  09:02:13  gaertner
% Initial revision
%
%


