Perfection is attained
not when there is nothing left to add
but when there is nothing left to take away
(Antoine de Saint-Exupéry)
(c) Software Lab. Alexander Burger
This document describes the concepts, data types, and kernel functions of the PicoLisp system.
This is not a Lisp tutorial. For an introduction to Lisp, a traditional Lisp book like "Lisp" by Winston/Horn (Addison-Wesley 1981) is recommended. Note, however, that there are significant differences between PicoLisp and Maclisp (and even greater differences to Common Lisp).
Please take a look at the PicoLisp Tutorial for an explanation of some aspects of PicoLisp, and scan through the list of Frequently Asked Questions (FAQ).
PicoLisp is the result of a language design study, trying to answer the question "What is a minimal but useful architecture for a virtual machine?". Because opinions differ about what is meant by "minimal" and "useful", there are many answers to that question, and people might consider other solutions more "minimal" or more "useful". But from a practical point of view, PicoLisp has proven to be a valuable answer to that question.
First of all, PicoLisp is a virtual machine architecture, and then a programming language. It was designed in a "bottom up" way, and "bottom up" is also the most natural way to understand and to use it: Form Follows Function.
PicoLisp has been used in several commercial and research programming projects since 1988. Its internal structures are simple enough, allowing an experienced programmer always to fully understand what's going on under the hood, and its language features, efficiency and extensibility make it suitable for almost any practical programming task.
In a nutshell, emphasis was put on four design objectives. The PicoLisp system should be
An important point in the PicoLisp philosophy is the knowledge about the architecture and data structures of the internal machinery. The high-level constructs of the programming language directly map to that machinery, making the whole system both understandable and predictable.
This is similar to assembly language programming, where the programmer has complete control over the machine.
The PicoLisp virtual machine is both simpler and more powerful than most current (hardware) processors. At the lowest level, it is constructed from a single data structure called "cell":
+-----+-----+
| CAR | CDR |
+-----+-----+
A cell is a pair of machine words, which traditionally are called CAR and CDR in the Lisp terminology. These words can represent either a numeric value (scalar) or the address of another cell (pointer). All higher level data structures are built out of cells.
The type information of higher level data is contained in the pointers to these data. Assuming the implementation on a byte-addressed physical machine and a pointer size of typically 8 bytes, each cell has a size of 16 bytes. Therefore, the pointer to a cell must point to a 16-byte boundary (a number which is a multiple of 16), and its bit-representation will look like:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0000
(the 'x' means "don't care"). For the individual data types, the
pointer is adjusted to point to other parts of a cell, in effect setting some of
the lower three bits to non-zero values. These bits are then used by the
interpreter to determine the data type.
In any case, bit(0) - the least significant of these bits - is reserved as a mark bit for garbage collection.
Initially, all cells in the memory are unused (free), and linked together to form a "free list". To create higher level data types at runtime, cells are taken from that free list, and returned by the garbage collector when they are no longer needed. All memory management is done via that free list; there are no additional buffers, string spaces or special memory areas, with two exceptions:
On the virtual machine level, PicoLisp supports
NIL.
They are all built from the single cell data structure, and all runtime data cannot consist of any other types than these three.
The following diagram shows the complete data type hierarchy, consisting of the three base types and the symbol variations:
cell
|
+-----------+-----------+
| | |
Number Symbol Pair
|
|
+--------+-----------+-----------+
| | | |
NIL Internal Transient External
A number can represent a signed integral value of arbitrary size. Internally, numeric values of up to 60 bits are stored in "short" numbers,
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxS010
Numbers larger than that are "big" numbers, stored in heap cells. The CARs of one or more cells hold the number's "digits" (64 bits each), with the least significant digit first, while the CDRs point to the remaining digits.
Bignum
|
V
+-----+-----+
| DIG | | |
+-----+--+--+
|
V
+-----+-----+
| DIG | | |
+-----+--+--+
|
V
+-----+-----+
| DIG | CNT |
+-----+-----+
The pointer to a big number points into the middle of the CAR, with an offset of 4 from the cell's start address, and the sign bit in bit(3):
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxS100
Thus, a number is recognized by the interpreter when either bit(1) is non-zero (a short number) or bit(2) is non-zero (a big number).
A symbol is more complex than a number. Each symbol has a value, and optionally a name and an arbitrary number of properties. The CDR of a symbol cell is also called VAL, and the CAR points to the symbol's tail. As a minimum, a symbol consists of a single cell, and has no name or properties:
Symbol
|
V
+-----+-----+
| / | VAL |
+-----+-----+
That is, the symbol's tail is empty (points to NIL, as indicated
by the '/' character).
The pointer to a symbol points to the CDR of the cell, with an offset of 8 bytes from the cell's start address. Therefore, the bit pattern of a symbol will be:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1000
Thus, a symbol is recognized by the interpreter when bit(3) is non-zero. (It should also be understood that both bit(2) and bit(1) must be zero, thus avoiding confusion with the number types.)
A property is a key-value pair, represented by a cons pair in the symbol's
tail. This is called a "property list". The property list may be terminated by a
number (short or big) representing the symbol's name. In the following example,
a symbol with the name "abcdefghijklmno" has three properties: A
KEY/VAL pair, a cell with only a KEY, and another KEY/VAL pair.
Symbol
|
V
+-----+-----+ +----------+---------+
| | | VAL | |'hgfedcba'|'onmlkji'|
+--+--+-----+ +----------+---------+
| tail ^
| |
V | name
+-----+-----+ +-----+-----+ +-----+--+--+
| | | ---+---> | KEY | ---+---> | | | | |
+--+--+-----+ +-----+-----+ +--+--+-----+
| |
V V
+-----+-----+ +-----+-----+
| VAL | KEY | | VAL | KEY |
+-----+-----+ +-----+-----+
Each property in a symbol's tail is either a symbol (like the single KEY
above, then it represents the boolean value T) or a cons pair with
the property key in its CDR and the property value in its CAR. In both cases,
the key should be a symbol, because searches in the property list are performed
using pointer comparisons.
The name of a symbol is stored as a number at the end of the tail. It contains the characters of the name in UTF-8 encoding, using between one and seven bytes in a short number, or eight bytes in a bignum cell. The first byte of the first character, for example, is stored in the lowest 8 bits of the number.
All symbols have the above structure, but depending on scope and
accessibility there are actually four types of symbols: NIL, internal, transient and external symbols.
NIL is a special symbol which exists exactly once in the whole
system. It is used
For that, NIL has a special structure:
NIL: /
|
V
+-----+-----+-----+-----+
|'LIN'| / | / | / |
+-----+--+--+-----+-----+
The reason for that structure is NIL's dual nature both as a
symbol and as a list:
NIL for its VAL, and be without
properties
NIL should give NIL both for
its CAR and for its CDR
These requirements are fulfilled by the above structure.
Internal symbols are all those "normal" symbols, as they are used for function definitions and variable names. They are "interned" into an index structure, so that it is possible to find an internal symbol by searching for its name.
There cannot be two different internal symbols with the same name.
Initially, a new internal symbol's VAL is NIL.
Transient symbols are only interned into an index structure for a certain time (e.g. while reading the current source file), and are released after that. That means, a transient symbol cannot be accessed then by its name, and there may be several transient symbols in the system having the same name.
Transient symbols are used
static identifiers in the C language family)
Initially, a new transient symbol's VAL is that symbol itself.
A transient symbol without a name can be created with the box or new functions.
External symbols reside in a database file (or a similar resources, see
*Ext), and are loaded into memory -
and written back to the file - dynamically as needed, and transparently to the
programmer. They are kept in memory ("cached") as long as they are accessible
("referred to") from other parts of the program, or when they were modified but
not yet written to the database file (by commit).
The interpreter recognizes external symbols internally by an additional tag bit in the tail structure.
There cannot be two different external symbols with the same name. External symbols are maintained in index structures while they are loaded into memory, and have their external location (disk file and block offset) directly coded into their names (more details here).
Initially, a new external symbol's VAL is NIL, unless otherwise
specified at creation time.
A list is a sequence of one or more cells (cons pairs), holding numbers, symbols, or cons pairs.
|
V
+-----+-----+
| any | | |
+-----+--+--+
|
V
+-----+-----+
| any | | |
+-----+--+--+
|
V
...
Lists are used in PicoLisp to emulate composite data structures like arrays, trees, stacks or queues.
In contrast to lists, numbers and symbols are collectively called "Atoms".
Typically, the CDR of each cell in a list points to the following cell,
except for the last cell which points to NIL. If, however, the CDR of
the last cell points to an atom, that cell is called a "dotted pair" (because of
its I/O syntax with a dot '.' between the two values).
The PicoLisp interpreter has complete knowledge of all data in the system, due to the type information associated with every pointer. Therefore, an efficient garbage collector mechanism can easily be implemented. PicoLisp employs a simple but fast mark-and-sweep garbage collector.
As the collection process is very fast (in the order of milliseconds per megabyte), it was not necessary to develop more complicated, time-consuming and error-prone garbage collection algorithms (e.g. incremental collection). A compacting garbage collector is also not necessary, because the single cell data type cannot cause heap fragmentation.
Lisp was chosen as the programming language, because of its clear and simple structure.
In some previous versions, a Forth-like syntax was also implemented on top of a similar virtual machine (Lifo). Though that language was more flexible and expressive, the traditional Lisp syntax proved easier to handle, and the virtual machine can be kept considerably simpler. PicoLisp inherits the major advantages of classical Lisp systems like
In the following, some concepts and peculiarities of the PicoLisp language and environment are described.
PicoLisp supports two installation strategies: Local and Global.
Normally, if you didn't build PicoLisp yourself but installed it with your operating system's package manager, you will have a global installation. This allows system-wide access to the executable and library/documentation files.
To get a local installation, you can directly download the PicoLisp tarball, and follow the instructions in the INSTALL file.
A local installation will not interfere in any way with the world outside its directory. There is no need to touch any system locations, and you don't have to be root to install it. Many different versions - or local modifications - of PicoLisp can co-exist on a single machine.
Note that you are still free to have local installations along with a global installation, and invoke them explicitly as desired.
Most examples in the following apply to a global installation.
When PicoLisp is invoked from the command line, an arbitrary number of arguments may follow the command name.
By default, each argument is the name of a file to be executed by the
interpreter. If, however, the argument's first character is a hyphen
'-', then the rest of that argument is taken as a Lisp function
call (without the surrounding parentheses), and a hyphen by itself as an
argument stops evaluation of the rest of the command line (it may be processed
later using the argv and opt functions). This whole mechanism corresponds
to calling (load T).
A special case is if the last argument is a single '+'. This
will switch on debug mode (the *Dbg
global variable) and discard the '+'.
As a convention, PicoLisp source files have the extension ".l".
Note that the PicoLisp executable itself does not expect or accept any
command line flags or options (except the '+', see above). They are
reserved for application programs.
The simplest and shortest invocation of PicoLisp does nothing, and exits
immediately by calling bye:
$ picolisp -bye
$
In interactive mode, the PicoLisp interpreter (see load) will also exit when Ctrl-D
is entered:
$ picolisp
: $ # Typed Ctrl-D
To start up the standard PicoLisp environment, several files should be loaded. The most commonly used things are in "lib.l" and in a bunch of other files, which are in turn loaded by "ext.l". Thus, a typical call would be:
$ picolisp lib.l ext.l
The recommended way, however, is to call the "pil" shell script, which
includes "lib.l" and "ext.l". Given that your current project is loaded by some
file "myProject.l" and your startup function is main, your
invocation would look like:
$ pil myProject.l -main
For interactive development it is recommended to enable debugging mode, to get the vi-style line editor, single-stepping, tracing and other debugging utilities.
$ pil myProject.l -main +
This is - in a local installation - equivalent to
$ ./pil myProject.l -main +
In any case, the directory part of the first file name supplied (normally,
the path to "lib.l" as called by 'pil') is remembered internally as the
PicoLisp Home Directory. This path is later automatically substituted for
any leading "@" character in file name arguments to I/O functions
(see path).
Instead of the default vi-style line editor, an emacs-style editor can be
used. It can be switched on permanently by calling the function
(em) (i.e. without arguments), or by passing -em on
the command line:
$ pil -em +
:
A single call is enough, because the style will be remembered in a file "~/.pil/editor", and used in all subsequent PicoLisp sessions.
To switch back to 'vi' style, call (vi), use the
-vi command line option, or simply remove "~/.pil/editor".
In Lisp, each internal data structure has a well-defined external representation in human-readable format. All kinds of data can be written to a file, and restored later to their original form by reading that file.
In normal operation, the PicoLisp interpreter continually executes an infinite "read-eval-print loop". It reads one expression at a time, evaluates it, and prints the result to the console. Any input into the system, like data structures and function definitions, is done in a consistent way no matter whether it is entered at the console or read from a file.
Comments can be embedded in the input stream with the hash #
character. Everything up to the end of that line will be ignored by the reader.
: (* 1 2 3) # This is a comment
-> 6
A comment spanning several lines may be enclosed between #{ and
}#.
Here is the I/O syntax for the individual PicoLisp data types (numbers, symbols and lists) and for read-macros:
A number consists of an arbitrary number of digits ('0' through
'9'), optionally preceded by a sign character ('+' or
'-'). Legal number input is:
: 7
-> 7
: -12345678901245678901234567890
-> -12345678901245678901234567890
Fixpoint numbers can be input by embedding a decimal point '.',
and setting the global variable *Scl
appropriately:
: *Scl
-> 0
: 123.45
-> 123
: 456.78
-> 457
: (setq *Scl 3)
-> 3
: 123.45
-> 123450
: 456.78
-> 456780
Thus, fixpoint input simply scales the number to an integer value
corresponding to the number of digits in *Scl.
Formatted output of scaled fixpoint values can be done with the format and round functions:
: (format 1234567890 2)
-> "12345678.90"
: (format 1234567890 2 "." ",")
-> "12,345,678.90"
The reader is able to recognize the individual symbol types from their syntactic form. A symbol name should - of course - not look like a legal number (see above).
In general, symbol names are case-sensitive. car is not the same
as CAR.
Besides the standard form, NIL is also recognized as
(), [] or "".
: NIL
-> NIL
: ()
-> NIL
: ""
-> NIL
Output will always appear as NIL.
Internal symbol names can consist of any printable (non-whitespace) character, except for the following meta characters:
" ' ( ) , [ ] ` ~ { }
It is possible, though, to include these special characters into symbol names
by escaping them with a backslash '\'.
The dot '.' has a dual nature. It is a meta character when
standing alone, denoting a dotted pair, but can otherwise
be used in symbol names.
As a rule, anything not recognized by the reader as another data type will be returned as an internal symbol.
A transient symbol is anything surrounded by double quotes '"'.
With that, it looks like - and can be used as - a string constant in other
languages. However, it is a real symbol, and may be assigned a value or a
function definition, and properties.
Initially, a transient symbol's value is that symbol itself, so that it does not need to be quoted for evaluation:
: "This is a string"
-> "This is a string"
However, care must be taken when assigning a value to a transient symbol. This may cause unexpected behavior:
: (setq "This is a string" 12345)
-> 12345
: "This is a string"
-> 12345
The name of a transient symbol can contain any character except the
null-byte. Control characters can be written with a preceding hat
'^' character. A hat or a double quote character can be escaped
with a backslash '\', and a backslash itself has to be escaped with
another backslash.
: "We^Ird\\Str\"ing"
-> "We^Ird\\Str\"ing"
: (chop @)
-> ("W" "e" "^I" "r" "d" "\\" "S" "t" "r" "\"" "i" "n" "g")
The combination of a backslash followed by 'n', 'r' or 't' is replaced with newline ("^J"), return ("^M") or TAB ("^I"), respectively.
: "abc\tdef\r"
-> "abc^Idef^M"
A decimal number between two backslashes can be used to specify any unicode character directly.
: "äöü\8364\xyz"
-> "äöü€xyz"
A backslash in a transient symbol name at the end of a line discards the newline, and continues the name in the next line. In that case, all leading spaces and tabs in that line are discarded, to allow proper source code indentation.
: "abc\
def"
-> "abcdef"
: "x \
y \
z"
-> "x y z"
The index for transient symbols is cleared automatically before and after
loading a source file, or it can be
reset explicitly with the ====
function. With that mechanism, it is possible to create symbols with a local
access scope, not accessible from other parts of the program.
A special case of transient symbols are anonymous symbols. These are
symbols without name (see box, box? or new). They print as a dollar sign
($) followed by a decimal digit string (actually their machine
address).
External symbol names are surrounded by braces ('{' and
'}'). The characters of the symbol's name itself identify the
physical location of the external object. This is
0' through '9',
':', ';', 'A' through 'Z'
and 'a' through 'z').
@' is zero,
'A' is 1 and 'O' is 15 (from "alpha" to "omega")),
immediately followed (without a hyphen) the starting block in octal
('0' through '7').
In both cases, the database file (and possibly the hyphen) are omitted for the first (default) file.
Lists are surrounded by parentheses ('(' and ')').
(A) is a list consisting of a single cell, with the symbol
A in its CAR, and NIL in its CDR.
(A B C) is a list consisting of three cells, with the symbols
A, B and C respectively in their CAR, and
NIL in the last cell's CDR.
(A . B) is a "dotted pair", a list
consisting of a single cell, with the symbol A in its CAR, and
B in its CDR.
PicoLisp has built-in support for reading and printing simple circular lists. If the dot in a dotted-pair notation is immediately followed by a closing parenthesis, it indicates that the CDR of the last cell points back to the beginning of that list.
: (let L '(a b c) (conc L L))
-> (a b c .)
: (cdr '(a b c .))
-> (b c a .)
: (cddddr '(a b c .))
-> (b c a .)
A similar result can be achieved with the function circ. Such lists must be used with care,
because many functions won't terminate or will crash when given such a list.
Read-macros in PicoLisp are special forms that are recognized by the reader,
and modify its behavior. Note that they take effect immediately while reading an expression, and are not seen by the
eval in the main loop.
The most prominent read-macro in Lisp is the single quote character
"'", which expands to a call of the quote function. Note that the single quote
character is also printed instead of the full function name.
: '(a b c)
-> (a b c)
: '(quote . a)
-> 'a
: (cons 'quote 'a) # (quote . a)
-> 'a
: (list 'quote 'a) # (quote a)
-> '(a)
A comma (,) will cause the reader to collect the following data
item into an idx tree in the global
variable *Uni, and to return a
previously inserted equal item if present. This makes it possible to create a
unique list of references to data which do normally not follow the rules of
pointer equality. If the value of *Uni is T, the
comma read macro mechanism is disabled.
A single backquote character "`" will cause the reader to
evaluate the following expression, and return the result.
: '(a `(+ 1 2 3) z)
-> (a 6 z)
A tilde character ~ inside a list will cause the reader to
evaluate the following expression, and (destructively) splice the result into
the list.
: '(a b c ~(list 'd 'e 'f) g h i)
-> (a b c d e f g h i)
When a tilde character is used to separate two symbol names (without surrounding whitespace), the first is taken as a namespace to look up the second (64-bit version only).
: 'libA~foo # Look up 'foo' in namespace 'libA'
-> "foo" # "foo" is not interned in the current namespace
Reading libA~foo is equivalent to switching the current
namespace to libA (with symbols), reading the symbol
foo, and then switching back to the original namespace.
Brackets ('[' and ']') can be used as super
parentheses. A closing bracket will match the innermost opening bracket, or all
currently open parentheses.
: '(a (b (c (d]
-> (a (b (c (d))))
: '(a (b [c (d]))
-> (a (b (c (d))))
Finally, reading the sequence '{}' will result in a new
anonymous symbol with value NIL, equivalent to a call to box without arguments.
: '({} {} {})
-> ($134599965 $134599967 $134599969)
: (mapcar val @)
-> (NIL NIL NIL)
PicoLisp tries to evaluate any expression encountered in the read-eval-print loop. Basically, it does so by applying the following three rules:
: 1234
-> 1234 # Number evaluates to itself
: *Pid
-> 22972 # Symbol evaluates to its VAL
: (+ 1 2 3)
-> 6 # List is evaluated as a function call
For the third rule, however, things get a bit more involved. First - as a special case - if the CAR of the list is a number, the whole list is returned as it