When programming in newLISP, certain functions and usage patterns occur repeatedly. For some problems, an optimal way to solve them evolves over time. The following chapters present example code and explanations for the solution of specific problems when programming in newLISP.
Some content is overlapping with material covered in the newLISP Users Manual and Reference or presented here with a different slant.
Only a subset of newLISP's total function repertoire is used here. Some functions demonstrated have additional calling patterns or applications not mentioned on these pages.
This collection of patterns and solutions is a work in progress. Over time, material will be added or existing material improved.
On Linux/Unix, put the following in the first line of the script/program file:
#!/usr/bin/newlisp
specifying a bigger stack:
#!/usr/bin/newlisp -s 100000
or
#!/usr/bin/newlisp -s100000
Operating systems' shells behave differently when parsing the first line and extracting parameters. newLISP takes both attached or detached parameters. Put the following lines in small script to test the behavior of the underlying OS and platform. The script changes the stack size allocated to 100,000 and limits newLISP cell memory to about 10 M bytes.
#!/usr/bin/newlisp -s 100000 -m 10
(println (main-args))
(println (sys-info))
(exit) ; important
A typical output executing the script from the system shell would be:
./arg-test
("/usr/bin/newlisp" "-s" "100000" "-m" "10" "./arg-test")
(308 655360 299 2 0 100000 8410 2)
Note that few programs in newLISP need a bigger stack configured; most programs run on the internal default of 2048. Each stack position takes an average of 80 bytes. Other options are available to start newLISP. See the Users Manual for details.
The following example shows how a file can be piped into a newLISP script.
#!/usr/bin/newlisp
#
# uppercase - demo filter script as pipe
#
# usage:
# ./uppercase < file-spec
#
# example:
# ./uppercase < my-text
#
#
(while (read-line) (println (upper-case (current-line))))
(exit)
The file will be printed to std-out translated to uppercase.
The following program would also work with binary non-textual information containing 0's :
#!/usr/bin/newlisp
;
; inout - demo binary pipe
;
; read from stdin into buffer
; then write to stdout
;
; usage: ./inout < inputfile > outputfile
;
(while (read 0 buffer 1024)
(write 1 buffer 1024))
(exit)
Set buffersize to best performance.
The following script works like a Unix grep utility iterating through files and filtering each line in a file using a regular expression pattern.
#!/usr/bin/newlisp
#
# nlgrep - grep utility on newLISP
#
# usage:
# ./nlgrep "regex-pattern" file-spec
#
# file spec can contain globbing characters
#
# example:
# ./nlgrep "this|that" *.c
#
# will print all lines containing 'this' or 'that' in *.c files
#
(dolist (fname (3 (main-args)))
(set 'file (open fname "read"))
(println "file ---> " fname)
(while (read-line file)
(if (find (main-args 2) (current-line) 0)
(write-line)))
(close file))
(exit)
The expression:
(3 (main-args))
is a short form of writing:
(rest (rest (rest (main-args))))
It returns a list of all the filenames. This form of specifying indexes for rest is called implicit indexing. See the Users Manual for implicit indexing with other functions. The expression (main-args 2) extracts the 3rd argument from the command line containing the regular expression pattern.
Pipe one-liners directly into the executable for evaluation of short expressions:
~> echo '(+ 1 2 3)' | newlisp 6 ~>
When writing bigger applications or when several programmers are working on the same code base, it is necessary to divide the code base into modules. Modules in newLISP are implemented using contexts, which are namespaces. Namespaces allow lexical isolation between modules. Variables of the same name in one module cannot clash with variables of the same name in another module.
Typically, modules are organized in one context per file. One file module may contain database access routines.
; database.lsp ; (context 'db) (define (update x y z) ... ) (define (erase x y z) ... )
Another module may contain various utilities
; auxiliary.lsp ; (context 'aux) (define (getval a b) ... )
Typically, there will be one MAIN module that loads and controls all others:
; application.lsp
;
(load "auxiliary.lsp")
(load "database.lsp")
(define (run)
(db:update ....)
(aux:putval ...)
...
...
)
(run)
When using more than one context per file, each context section should be closed with a (context MAIN) statement:
; myapp.lsp
;
(context 'A)
(define (foo ...) ...)
(context MAIN)
(context 'B)
(define (bar ...) ...)
(context MAIN)
(define (main-func)
(A:foo ...)
(B:bar ...)
)
Note that in the namespace statements for contexts A and B that the context names are quoted because they are newly created, but MAIN can stay unquoted because it already exists when newLISP starts up. However, quoting it does not present a problem.
The line (context MAIN) that closes a context can be omitted by using the following technique:
; myapp.lsp
;
(context 'A)
(define (foo ...) ...)
(context 'MAIN:B)
(define (bar ...) ...)
(context 'MAIN)
(define (main-func)
(A:foo ...)
(B:bar ...)
)
The line (context 'MAIN:B) switches back to MAIN then opens the new context B.
A function in a context may have the same name as the host context itself. This function has special characteristics:
(context 'foo) (define (foo:foo a b c) ... )
The function foo:foo is called the default function, because when using the context name foo like a function, it will default to foo:foo
(foo x y z) ; same as (foo:foo x y z)
The default function makes it possible to write functions which look like normal functions but carry their own lexical namespace. We can use this to write functions which keep state:
(context 'generator)
(define (generator:generator)
(inc acc)) ; when acc is nil, assumes 0
(context MAIN)
(generator) → 1
(generator) → 2
(generator) → 3
The following is a more complex example for a function generating a Fibonacci sequence:
(define (fibo:fibo)
(if (not fibo:mem) (set 'fibo:mem '(0 1)))
(last (push (+ (fibo:mem -1) (fibo:mem -2)) fibo:mem -1)))
(fibo) → 1
(fibo) → 2
(fibo) → 3
(fibo) → 5
(fibo) → 8
...
This example also shows how a default function is defined on-the-fly without the need of explicit context statements. As an alternative, the function could also have been written so that the context is created explicitly:
(context 'fibo)
(define (fibo:fibo)
(if (not mem) (set 'mem '(0 1)))
(last (push (+ (mem -1) (mem -2)) mem -1)))
(context MAIN)
(fibo) → 1
(fibo) → 2
(fibo) → 3
(fibo) → 5
(fibo) → 8
Although the first form is shorter, the second form is more readable.
The previous examples already presented functions packaged with data in a namespace. In the generator example the acc variable kept state. In the fibo example the variable mem kept a growing list. In both cases, functions and data are living together in a namespace. The following example shows how a namespace holds only data in a default functor:
(set 'db:db '(a "b" (c d) 1 2 3 x y z))
Just like we used the default function to refer to fibo and generator we can refer to the list in db:db by only using db. This will work in all situations where we do list indexing:
(db 0) → a (db 1) → "b" (db 2 1) → d (db -1) → z (db -3) → x (3 db) → (1 2 3 x y z) (2 1 db) → ((c d)) (-6 2 db) → (1 2)
When the default functor is used as an argument in a user defined function, the default functor is passed by reference. This means that a reference to the original contents is passed, not a copy of the list or string. This is useful when handling large lists or strings:
(define (update data idx expr)
(if (not (or (lambda? expr) (primitive? expr)))
(setf (data idx) expr)
(setf (data idx) (expr $it))))
(update db 0 99) → a
db:db → (99 "b" (c d) 1 2 3 x y z)
(update db 1 upper-case) → "b"
db:db → (99 "B" (c d) 1 2 3 x y z)
(update db 4 (fn (x) (mul 1.1 x))) →
db:db → (99 "B" (c d) 1 2.2 3 x y z)
The data in db:db is passed via the update function parameter data, which now holds a reference to the context db. The expr parameter passed is checked to determine if it is a built-in function, operator or a user defined lambda expression and then works on $it, the anaphoric system variable containing the old content referenced by (data idx).
Whenever a function in newLISP asks for a string or list in a parameter, a default functor can be passed by its context symbol. Another example:
(define (pop-last data) (pop data -1)) (pop-last db) → z db:db → (99 "B" (c d) 1 2.2 3 x y)
The function update is also a good example of how to pass operators or functions as a function argument (upper-case working on $it). Read more about this in the chapter Functions as data.
All looping functions like doargs, dolist, dostring, dotimes, dotree and for use local variables. During loop execution, the variable takes different values. But after leaving the looping function, the variable regains its old value. let, define, and lambda expressions are another method for making variables local:
let is the usual way in newLISP to declare symbols as local to a block.
(define (sum-sq a b)
(let ((x (* a a)) (y (* b b)))
(+ x y)))
(sum-sq 3 4) → 25
; alternative syntax
(define (sum-sq a b)
(let (x (* a a) y (* b b))
(+ x y)))
The variables x and y are initialized, then the expression (+ x y) is evaluated. The let form is just an optimized version and syntactic convenience for writing:
((lambda (sym1 [sym2 ...]) exp-body ) exp-init1 [ exp-init2 ...])
When initializing several parameters, a nested let, letn can be used to reference previously initialized variables in subsequent initializer expressions:
(letn ((x 1) (y (+ x 1)))
(list x y)) → (1 2)
local works the same way but variables are initialized to nil
(local (a b c) ... ; expressions using the locale variables a b c )
letex works similar to let but variables are expanded in the body to values assigned.
; assign to local variable and expand in body (letex ( (x 1) (y '(a b c)) (z "hello") ) '(x y z)) → (1 (a b c) "hello") ; as in let, parentheses around the initializers can be omitted (letex (x 1 y 2 z 3) '(x y z)) → (1 2 3)
After exiting any of the let, letn, local or letex expressions, the variable symbols used as locals get their old values back.
In newLISP, all parameters in user defined functions are optional. Unused parameters are filled with nil and are of local scope to the dynamic scope of the function. Defining a user function with more parameters than required is a convenient method to create local variable symbols:
(define (sum-sq a b , x y)
(set 'x (* a a))
(set 'y (* b b))
(+ x y))
The comma is not a special syntax feature but only a visual helper to separate normal parameters from local variable symbols. (Technically, the comma, like x and y, is a local variable and is set to nil.)
In the definition of a function default values can be specified:
(define (foo (a 1) (b 2))
(list a b))
(foo) → (1 2)
(foo 3) → (3 2)
(foo 3 4) → (3 4)
Using the args function no parameter symbols need to be used at all and args returns a list of all parameters passed but not taken by declared parameters:
(define (foo)
(args))
(foo 1 2 3) → (1 2 3)
(define (foo a b)
(args))
(foo 1 2 3 4 5) → (3 4 5)
The second example shows how args only contains the list of arguments not bound by the variable symbols a and b.
Indices can be used to access members of the (args) list:
(define (foo)
(+ (args 0) (args 1)))
(foo 3 4) → 7
(define-macro (foo)
(local (len width height)
(bind (args) true)
(println "len:" len " width:" width " height:" height)
))
(foo (width 20) (height 30) (len 10))
len:10 width:20 height:30
local will shadow / protect the values of the variables len, width and height at higher dynamic scoping levels.
Although recursion is a powerful feature to express many algorithms in a readable form, it can also be inefficient in some instances. newLISP has many iterative constructs and high level functions like flat or the built-in XML functions, which use recursion internally. In many cases this makes defining a recursive algorithm unnecessary.
Some times a non-recursive solution can be much faster and lighter on system resources.
; classic recursion
; slow and resource hungry
(define (fib n)
(if (< n 2) 1
(+ (fib (- n 1))
(fib (- n 2)))))
The recursive solution is slow because of the frequent calling overhead. Also, the recursive solution uses a lot of memory for holding intermediate and frequently redundant results.
; iteration
; fast and also returns the whole list
(define (fibo n , f)
(set 'f '(1 0))
(dotimes (i n)
(push (+ (f 0) (f 1)) f)) )
The iterative solution is fast and uses very little memory.
A memoizing function caches results for faster retrieval when called with the same parameters again. The following function makes a memoizing function from any built-in or user defined function with an arbitrary number of arguments. A namespace is created for the memoizing function as a data cache.
; speed up a recursive function using memoization
(define-macro (memoize mem-func func)
(set (sym mem-func mem-func)
(letex (f func c mem-func)
(lambda ()
(or (context c (string (args)))
(context c (string (args)) (apply f (args))))))))
(define (fibo n)
(if (< n 2) 1
(+ (fibo (- n 1))
(fibo (- n 2)))))
(memoize fibo-m fibo)
(time (fibo-m 25)) → 148
(time (fibo-m 25)) → 0
The function creates a context and default function for the original function with a new name and stores all results in symbols in the same context.
When memoizing recursive functions, include the raw lambda specification of the function so recursive calls are memoized too:
(memoize fibo
(lambda (n)
(if(< n 2) 1
(+ (fibo (- n 1))
(fibo (- n 2))))))
(time (fibo 100)) → 1
(fibo 80) → 37889062373143906
The fibo function in the last example would take hours to calculate without memoization. The memoized version takes only about a milli-second for an argument of 100.
Tree walks are a typical pattern in traditional LISP and in newLISP as well for walking through a nested list. But many times a tree walk is only used to iterate through all elements of an existing tree or nested list. In this case the built-in flat function is much faster than using recursion:
(set 'L '(a b c (d e (f g) h i) j k))
; classic car/cdr and recursion
;
(define (walk-tree tree)
(cond ((= tree '()) true)
((atom? (first tree))
(println (first tree))
(walk-tree (rest tree)))
(true
(walk-tree (first tree))
(walk-tree (rest tree)))))
; classic recursion
; 3 times faster
;
(define (walk-tree tree)
(dolist (elmnt tree)
(if (list? elmnt)
(walk-tree elmnt)
(println elmnt))))
(walk-tree L) →
a
b
c
d
e
...
Using the built-in flat in newLISP a nested list can be transformed into a flat list. Now the list can be processed with a dolist or map:
; fast and short using 'flat' ; 30 times faster with map ; (map println (flat L)) ; same as (dolist (item (flat L)) (println item))
Walking a directory tree is a task where recursion works well:
; walks a disk directory and prints all path-file names
;
(define (show-tree dir)
(when (directory? dir)
(dolist (nde (directory dir))
(if (and (directory? (append dir "/" nde))
(!= nde ".") (!= nde ".."))
(show-tree (append dir "/" nde))
(println (append dir "/" nde))))))
In this example recursion is the only solution, because the entire nested list of files is not available when the function is called but gets created recursively during function execution.
newLISP has facilities for multidimensional indexing into nested lists. There are destructive functions like push, pop, setf, set-ref, set-ref-all, sort and reverse and many others for non-destructive operations, like nth, ref, ref-all, first, last and rest etc..