SciTE Script Lexer

A lexer may be written as a script in the Lua language instead of in C++. This is a little simpler and allows lexers to be developed without using a C++ compiler.

A script lexer is attached by setting the file lexer to be a name that starts with "script_". Styles and other properties can then be assigned using this name. For example,

-- -*- coding: utf-8 -*-

function OnStyle(styler)
S_DEFAULT = 0
S_IDENTIFIER = 1
S_KEYWORD = 2
S_UNICODECOMMENT = 3
identifierCharacters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"

styler:StartStyling(styler.startPos, styler.lengthDoc, styler.initStyle)
while styler:More() do

-- Exit state if needed
if styler:State() == S_IDENTIFIER then
if not identifierCharacters:find(styler:Current(), 1, true) then
identifier = styler:Token()
if identifier == "if" or identifier == "end" then
styler:ChangeState(S_KEYWORD)
end
styler:SetState(S_DEFAULT)
end
elseif styler:State() == S_UNICODECOMMENT then
if styler:Match("»") then
styler:ForwardSetState(S_DEFAULT)
end
end

-- Enter state if needed
if styler:State() == S_DEFAULT then
if styler:Match("«") then
styler:SetState(S_UNICODECOMMENT)
elseif identifierCharacters:find(styler:Current(), 1, true) then
styler:SetState(S_IDENTIFIER)
end
end

styler:Forward()
end
styler:EndStyling()
end

proc clip(int a)
« Clip into the positive zone »
if (a > 0) a
0
end

Code Structure

Document Loop

The lexer loops through the part of the document indicated assigning a style to each character.

styler:StartStyling(styler.startPos, styler.lengthDoc, styler.initStyle)
while styler:More() do
-- Code that examines the text and sets lexical states
styler:Forward()
end
styler:EndStyling()

There are many different ways to structure the code that examines the text and sets lexical states. A structure that has proven useful in C++ lexers is to write two blocks of code as shown in the example. The first block checks if the current state should end and if so sets the state to the default 0. The second block is responsible for detecting whether a new state should be entered from the default state. This structure means everything is dealt with as switching from or to the default state and avoids having to consider many combinations of states.

Encodings

The styler iterates over whole characters rather than bytes. Thus if the document is encoded in UTF-8, styler:Current() may be a multibyte string. If the script is also encoded in UTF-8, then it is easy to check against Unicode characters with code like

if styler:Current() == "«" then

If using an encoding like Latin-1 and the script is also encoded in the same encoding then literals can be used as above.

If the language can be encoded in different ways then more complex code may be needed along with encoding-specific code.

Checking Before

Sometimes a lexer needs to see some information earlier in the file, perhaps a declaration changes the syntax or the particular form of quote at the start of a string must be matched at its end. Since the standard loop only goes forward from the starting position, different calls must be used like CharAt and StyleAt. These use byte positions and do not treat multi-byte characters as single entities.

Performance

The lexer above can lex approximately 90K per second on a 2.4 GHz Athlon 64. For most situations, this will feel completely fluid.

More complex lexers will be slower. If a lexer is so slow that the application becomes unresponsive then the lexer can choose to split up each request. It can do so by deciding upon a range of whole lines and using this range as the arguments to StartStyling. This allows the user's keystrokes and mouse moves to be processed. The lexer will automatically be called again to lex more of the document.

Writing lexers in Lua

Code Structure

Document Loop

Encodings

Checking Before

Performance

API