% language=us runpath=texruns:manuals/cld

% swaphash

\environment cld-environment

\startcomponent cld-luafunctions

\startchapter[title=Lua Functions]

\startsection[title={Introduction}]

When you run \CONTEXT\ you have some libraries preloaded. If you look into the
\LUA\ files you will find more than is discussed here, but keep in mind that what
is not documented, might be gone or done different one day. Some extensions live
in the same namespace as those provided by stock \LUA\ and \LUATEX, others have
their own. There are many more functions and the more obscure (or never being
used) ones will go away.

The \LUA\ code in \CONTEXT\ is organized in quite some modules. Those with names
like \type {l-*.lua} are rather generic and are automatically available when you
use \type {mtxrun} to run a \LUA\ file. These are discusses in this chapter. A
few more modules have generic properties, like some in the categories \type
{util-*.lua}, \type {trac-*.lua}, \type {luat-*.lua}, \type {data-*.lua} and
\type {lxml-*.lua}. They contain more specialized functions and are discussed
elsewhere, if at all.

Before we move on the the real code, let's introduce a handy helper:

\starttyping
inspect(somevar)
\stoptyping

Whenever you feel the need to see what value a variable has you can insert this
function to get some insight. It knows how to deal with several data types.

Some of the functions here are rather specific to \CONTEXT, even if we load them
in our plain \TEX\ reference format. Don't depend on them outside the perspective
of \CONTEXT.

There are more modules than discussed here. You can find more about them in the
\LUAMETATEX\ manual. Some have additional functions that are implemented on top
of low level ones. There are also modules that are (kind of) specific to for
instance \LUAMETAFUN; these are wrapped into \METAPOST\ interfaces.

\stopsection

\startsection[title={Tables}]

\startsummary[title={[lua] concat}]

These functions come with \LUA\ itself and are discussed in detail in the \LUA\
reference manual so we stick to some examples. The \type {concat} function
stitches table entries in an indexed table into one string, with an optional
separator in between. If can also handle a slice of the table

\starttyping
local str = table.concat(t)
local str = table.concat(t,separator)
local str = table.concat(t,separator,first)
local str = table.concat(t,separator,first,last)
\stoptyping

Only strings and numbers can be concatenated.

\ShowLuaExampleThree {table} {concat} {{"a","b","c","d","e"}}
\ShowLuaExampleThree {table} {concat} {{"a","b","c","d","e"},"+"}
\ShowLuaExampleThree {table} {concat} {{"a","b","c","d","e"},"+",2,3}

\stopsummary

\startsummary[title={[lua] insert remove}]

You can use \type {insert} and \type {remove} for adding or replacing entries in
an indexed table.

\starttyping
table.insert(t,position,value)
value = table.remove(t,position)
\stoptyping

The position is optional and defaults to the last entry in the table. For
instance a stack is built this way:

\starttyping
table.insert(stack,"top")
local top = table.remove(stack)
\stoptyping

Beware, the \type {insert} function returns nothing. You can provide an
additional position:

\starttyping
table.insert(list,"injected in slot 2",2)
local thiswastwo = table.remove(list,2)
\stoptyping

\stopsummary

\startsummary[title={[lua] pack unpack}]

You can access entries in an indexed table as follows:

\starttyping
local a, b, c = t[1], t[2], t[3]
\stoptyping

but this does the same:

\starttyping
local a, b, c = table.unpack(t)
\stoptyping

\ShowLuaExampleZero {table} {unpack} {{"a","b","c"}}
\ShowLuaExampleZero {table} {unpack} {{"a",nil,"b"}}
\ShowLuaExampleZero {table} {unpack} {{"a","b","c"},2}
\ShowLuaExampleZero {table} {unpack} {{"a","b","c"},1,2}

This is less efficient but there are situations where \type {unpack}
comes in handy. The reverse is done with \type {pack}:

\starttyping
local t = table.pack(1,2,3)
\stoptyping

You'll notice that the resulting table has a field \type {n} that holds the
number of entries (the last index).

\ShowLuaExampleTwo {table} {pack} {"a","b","c"}
\ShowLuaExampleTwo {table} {pack} {"a",nil,"c"}

\stopsummary

\startsummary[title={[lua] sort}]

Sorting is done with \type {sort}, a function that does not return a value but
operates on the given table.

\starttyping
table.sort(t)
table.sort(t,comparefunction)
\stoptyping

The compare function has to return a consistent equivalent of \type {true} or
\type {false}. For sorting more complex data structures there is a specialized
sort module available.

\ShowLuaExampleFour {table} {sort} {{"a","b","c"}} {}
\ShowLuaExampleFour {table} {sort} {{"a","b","c"}} {,function(x,y) return x > y end}
\ShowLuaExampleFour {table} {sort} {{"a","b","c"}} {,function(x,y) return x < y end}

\stopsummary

\startsummary[title={sorted}]

The built|-|in \type {sort} function does not return a value but sometimes it can be
if the (sorted) table is returned. This is why we have:

\starttyping
local a = table.sorted(b)
\stoptyping

\stopsummary

% table.strip

\startsummary[title={keys sortedkeys sortedhashkeys sortedhash}]

The \type {keys} function returns an indexed list of keys. The order is undefined
as it depends on how the table was constructed. A sorted list is provided by
\type {sortedkeys}. This function is rather liberal with respect to the keys. If
the keys are strings you can use the faster alternative \type {sortedhashkeys}.

\starttyping
local s = table.keys (t)
local s = table.sortedkeys (t)
local s = table.sortedhashkeys (t)
\stoptyping

Because a sorted list is often processed there is also an iterator:

\starttyping
for key, value in table.sortedhash(t) do
    print(key,value)
end
\stoptyping

There is also a synonym \type {sortedpairs} which sometimes looks more natural
when used alongside the \type {pairs} and \type {ipairs} iterators.

\ShowLuaExampleTwo {table} {keys}           {{ [1] = 2, c = 3, [true] = 1 }}
\ShowLuaExampleTwo {table} {sortedkeys}     {{ [1] = 2, c = 3, [true] = 1 }}
\ShowLuaExampleTwo {table} {sortedhashkeys} {{ a = 2, c = 3, b = 1 }}

\stopsummary

\startsummary[title={sortedhashonly sortedindexonly}]

These are more selective variants of the previous discussed collectors.

\ShowLuaExampleTwo {table} {sortedhashonly}  {{ [1] = 1, a = 2, c = 3}}
\ShowLuaExampleTwo {table} {sortedindexonly} {{ [1] = 1, a = 2, c = 1}}

\stopsummary

\startsummary[title={serialize print tohandle tofile}]

The \type {serialize} function converts a table into a verbose representation.
The \type {print} function does the same but prints the result to the console
which is handy for tracing. The \type {tofile} function writes the table to a
file, using reasonable chunks so that less memory is used. The fourth variant
\type {tohandle} takes a handle so that you can do whatever you like with the
result.

\starttyping
table.serialize (root, name, reduce, noquotes, hexify)
table.print (root, name, reduce, noquotes, hexify)
table.tofile (filename, root, name, reduce, noquotes, hexify)
table.tohandle (handle, root, name, reduce, noquotes, hexify)
\stoptyping

The serialization can be controlled in several ways. Often only the first two
options makes sense:

\ShowLuaExampleOne {table} {serialize} {{ a = 2 }}
\ShowLuaExampleOne {table} {serialize} {{ a = 2 }, "name"}
\ShowLuaExampleOne {table} {serialize} {{ a = 2 }, true}
\ShowLuaExampleOne {table} {serialize} {{ a = 2 }, false}
\ShowLuaExampleOne {table} {serialize} {{ a = 2 }, "return"}
\ShowLuaExampleOne {table} {serialize} {{ a = 2 }, 12}

\ShowLuaExampleOne {table} {serialize} {{ a = 2, [3] = "b", [true] = "6" }, nil, true}
\ShowLuaExampleOne {table} {serialize} {{ a = 2, [3] = "b", [true] = "6" }, nil, true, true}
\ShowLuaExampleOne {table} {serialize} {{ a = 2, [3] = "b", [true] = "6" }, nil, true, true, true}

In \CONTEXT\ there is also a \type {tocontext} function that typesets the table
verbose. This is handy for manuals and tracing.

\stopsummary

\startsummary[title={identical are_equal}]

These two function compare two tables that have a similar structure. The \type
{identical} variant operates on a hash while \type {are_equal} assumes an indexed
table.

\starttyping
local b = table.identical (one, two)
local b = table.are_equal (one, two)
\stoptyping

\ShowLuaExampleThree {table} {identical} {{ a = { x = 2 } }, { a = { x = 3 } }}
\ShowLuaExampleThree {table} {identical} {{ a = { x = 2 } }, { a = { x = 2 } }}

\ShowLuaExampleThree {table} {are_equal} {{ a = { x = 2 } }, { a = { x = 3 } }}
\ShowLuaExampleThree {table} {are_equal} {{ a = { x = 2 } }, { a = { x = 2 } }}

\ShowLuaExampleThree {table} {identical} {{ "one", "two" }, { "one", "two" }}
\ShowLuaExampleThree {table} {identical} {{ "one", "two" }, { "two", "one" }}

\ShowLuaExampleThree {table} {are_equal} {{ "one", "two" }, { "one", "two" }}
\ShowLuaExampleThree {table} {are_equal} {{ "one", "two" }, { "two", "one" }}

\stopsummary

\startsummary[title={tohash fromhash swapped swaphash reversed reverse mirrored}]

We use \type {tohash} quite a lot in \CONTEXT. It converts a list into a hash so
that we can easily check if (a string) is in a given set. The \type {fromhash}
function does the opposite: it creates a list of keys from a hashed table where
each value that is not \type {false} or \type {nil} is present.

\starttyping
local hashed  = table.tohash  (indexed)
local indexed = table.fromhash(hashed)
\stoptyping

The function \type {swapped} turns keys into values vise versa while the \type
{reversed} and \type {reverse} reverses the values in an indexed table. The last
one reverses the table itself (in|-|place).

\starttyping
local swapped  = table.swapped  (indexedtable)
local reversed = table.reversed (indexedtable)
local reverse  = table.reverse  (indexedtable)
local mirrored = table.mirrored (hashedtable)
\stoptyping

\ShowLuaExampleTwo {table} {tohash}   {{ "a", "b", "c" }}
\ShowLuaExampleTwo {table} {fromhash} {{ a = true, b = false, c = true }}
\ShowLuaExampleTwo {table} {swapped}  {{ "a", "b", "c" }}
\ShowLuaExampleTwo {table} {reversed} {{ "a", "b", "c" }}
\ShowLuaExampleTwo {table} {reverse}  {{ 1, 2, 3, 4 }}
\ShowLuaExampleTwo {table} {mirrored} {{ a = "x", b = "y", c = "z" }}

\stopsummary

\startsummary[title={append prepend}]

These two functions operate on a pair of indexed tables. The first table gets
appended or prepended by the second. The first table is returned as well.

\starttyping
table.append (one, two)
table.prepend(one, two)
\stoptyping

The functions are similar to loops using \type {insert}.

\ShowLuaExampleTwo {table} {append}  {{ "a", "b", "c" }, { "d", "e" }}
\ShowLuaExampleTwo {table} {prepend} {{ "a", "b", "c" }, { "d", "e" }}

\stopsummary

\startsummary[title={merge merged imerge imerged}]

You can merge multiple hashes with \type {merge} and indexed tables with \type
{imerge}. The first table is the target and is returned.

\starttyping
table.merge   (one, two, ...)
table.imerge  (one, two, ...)
\stoptyping

The variants ending with a \type {d} merge the given list of tables and return
the result leaving the first argument untouched.

\starttyping
local merged = table.merged  (one, two, ...)
local merged = table.imerged (one, two, ...)
\stoptyping

\ShowLuaExampleTwo {table} {merge}  {{ a = 1, b = 2, c = 3 }, { d = 1 }, { a = 0 }}
\ShowLuaExampleTwo {table} {imerge} {{ "a", "b", "c" }, { "d", "e" }, { "f", "g" }}

% \ShowLuaExampleTwo {table} {merged}   {{ a = 1, b = 2, c = 3 }, { d = 1 }, { a = 0 }}
% \ShowLuaExampleTwo {table} {imerged}  {{ "a", "b", "c" }, { "d", "e" }, { "f", "g" }}

\stopsummary

\startsummary[title={copy fastcopy}]

When copying a table we need to make a real and deep copy. The \type {copy}
function is an adapted version from the \LUA\ wiki. The \type {fastcopy} is faster
because it does not check for circular references and does not share tables when
possible. In practice using the fast variant is okay.

\starttyping
local copy = table.copy    (t)
local copy = table.fastcopy(t)
\stoptyping

\stopsummary

\startsummary[title={flattened}]

A nested table can be unnested using \type {flattened}. Normally you will only
use this function if the content is somewhat predictable. Often using one of the
merge functions does a similar job.

\starttyping
local flattened = table.flatten(t)
\stoptyping

\ShowLuaExampleTwo {table} {flattened} {{ a = 1, b = 2, { c = 3 }, d = 4}}
\ShowLuaExampleTwo {table} {flattened} {{ 1, 2, { 3, { 4 } }, 5}}
\ShowLuaExampleTwo {table} {flattened} {{ 1, 2, { 3, { 4 } }, 5}, 1}
\ShowLuaExampleTwo {table} {flattened} {{ a = 1, b = 2, { c = 3 }, d = 4}}
\ShowLuaExampleTwo {table} {flattened} {{ 1, 2, { 3, { c = 4 } }, 5}}
\ShowLuaExampleTwo {table} {flattened} {{ 1, 2, { 3, { c = 4 } }, 5}, 1}

\stopsummary

\startsummary[title={loweredkeys}]

The name says it all: this function returns a new table with the keys being lower
case. This is handy in cases where the keys have a change to be inconsistent, as
can be the case when users input keys and values in less controlled ways.

\starttyping
local normalized = table.loweredkeys { a = "a", A = "b", b = "c" }
\stoptyping

\ShowLuaExampleTwo {table} {loweredkeys} {{ a = 1, b = 2, C = 3}}

\stopsummary

\startsummary[title={contains}]

This function works with indexed tables. Watch out, when you look for a match,
the number \type {1} is not the same as string \type {"1"}. The function returns
the index or \type {false}.

\starttyping
if table.contains(t, 5 ) then ... else ... end
if table.contains(t,"5") then ... else ... end
\stoptyping

\ShowLuaExampleThree {table} {contains} {{ "a", 2, true, "1"}, 1}
\ShowLuaExampleThree {table} {contains} {{ "a", 2, true, "1"}, "1"}

\stopsummary

\startsummary[title={unique}]

When a table (can) contain duplicate entries you can get rid of them by using the
\type {unique} helper:

\starttyping
local t = table.unique { 1, 2, 3, 4, 3, 2, 5, 6 }
\stoptyping

\ShowLuaExampleTwo {table} {unique} { { "a", "b", "c", "a", "d" } }

\stopsummary

\startsummary[title={count}]

The name speaks for itself: this function counts the number of entries in the
given table. For an indexed table \type {#t} is faster.

\starttyping
local n = table.count(t)
\stoptyping

\ShowLuaExampleThree {table} {count} {{ 1, 2, [4] = 4, a = "a" }}

\stopsummary

\startsummary[title={sequenced}]

Normally, when you trace a table, printing the serialized version is quite
convenient. However, when it concerns a simple table, a more compact variant is:

\starttyping
print(table.sequenced(t, separator))
\stoptyping

% beware: by default sequences has | as separator

\ShowLuaExampleThree {table} {sequenced} {{ 1, 2, 3, 4}}
\ShowLuaExampleThree {table} {sequenced} {{ 1, 2, [4] = 4, a = "a" }, ", "}

\stopsummary

\startsummary[title={[lua] create}]

This is a relative new native \LUA\ table function that does what the name says:

\starttyping
local indexed = table.create(10)
local hashed  = table.create(0,10)
local hybrid  = table.create(10,10)
\stoptyping

In \LUAMETATEX\ we already introduced this creator that expects two arguments:

\starttyping
local mytable = lua.newtable(10,0)
\stoptyping

which has a variant:

\starttyping
local myarray = lua.newindex(10,2)
\stoptyping

where the values are preset (in this case to the integer value~2)..

\stopsummary

\startsummary[title={[lua] move}]

This is now a built-in helper:

\starttyping
local result = table.move(input,first,last,target)
local result = table.move(input,first,last,target,output)
\stoptyping

The last argument is optional.

\ShowLuaExampleTwo {table} {move} {{ 1, 2, 3, 4, 5, 6 }, 1, 2, 3}
\ShowLuaExampleTwo {table} {move} {{ 1, 2, 3, 4, 5, 6 }, 1, 2, 5}
\ShowLuaExampleTwo {table} {move} {{ 1, 2, 3, 4, 5, 6 }, 4, 5, 2}
\ShowLuaExampleTwo {table} {move} {{ 1, 2, 3, 4 }, 1, 2, 3, { "a", "b", "c", "d" }}

\stopsummary

\startsummary[title={sub}]

This helper does the same as \type {move} with an empty target table starting at
one. Apart from the table the arguments are optional.

\ShowLuaExampleTwo {table} {sub} {{ 1, 2, 3, 4, 5, 6 }, 1, 2 }
\ShowLuaExampleTwo {table} {sub} {{ 1, 2, 3, 4, 5, 6 }, 3, 5 }
\ShowLuaExampleTwo {table} {sub} {{ 1, 2, 3, 4, 5, 6 }, 3 }
\ShowLuaExampleTwo {table} {sub} {{ 1, 2, 3, 4, 5, 6 } }

\stopsummary

% we might only show these in some extended version

\startsummary[title={save load}]

You can use these helpers to save data and later (maybe in a second run) load
it.

\starttyping
local t = { a = 10, c = 30, d = { b = 20 } }
table.save("mytable.lua",t)

local t = table.load("mytable.lua")
\stoptyping

In \CONTEXT\ we have various ways to store data for two-pass usage which has the
advantage that one can trigger on a change. This is just a convenient variant
that you need to manage yourself.

\stopsummary

% \startsummary[title={combine}][category=context]
\startsummary[title={combine}]

This is a resolver: we feed it a source and a target

\startbuffer
\startluacode
local list  = {
    ["one"] = { preset = "alpha" },
    ["two"] = { preset = "beta"  },
}

local presets  = {
    ["alpha"] = { name = "this", slot = 123 },
    ["beta"]  = { name = "that", slot = 345 },
}

list.one = table.combine(presets[list.one.preset],list.one)

table.tocontext(list.one,false)
\stopluacode
\stopbuffer

\typebuffer

Here the preset, given as string, gets replaced by a table. We don't make
a deep copy, so you should not rewrite the result.

\getbuffer

As a side effect of the implementation you can actually do this:

\startbuffer
\startluacode
local target = { }

table.combine(target, { 4,  9,    6,   2 })
table.combine(target, { 3, nil,   1, nil })
table.combine(target, { 8, nil, nil,   7 })

table.tocontext(target,false)
\stopluacode
\stopbuffer

\typebuffer

Which gives you the set of values most recently seen:

\getbuffer

\stopsummary

\startsummary[title={fastserialize deserialize}]

These are for instance used when we store a \LUA\ table in an \SQL\ database
field.

\starttyping
local t = { "a", "a", "", "b", "c", e = "f" }
local s = table.fastserialize(t)

local s = [[return{[1]="a",[2]="a",[3]="",[4]="b",[5]="c",["e"]="f",}]]
local t = table.deserialize(s)
\stoptyping

Of course \quote {fast} is subjective because we have to construct the string
piecewise.

\ShowLuaExampleZero {table} {fastserialize} {{ "a", "a", "", "b", "c", e = "f" }}
\ShowLuaExampleTwo  {table} {deserialize}   {[[return{[1]="a",[2]="a",[3]="",[4]="b",[5]="c",["e"]="f",}]]}

\stopsummary

\startsummary[title={has_one_entry is_empty}]

Often one will inline these but we keep them around:

\ShowLuaExampleThree {table} {has_one_entry} {{ 1, 2, 3 }}
\ShowLuaExampleThree {table} {has_one_entry} {{ 1 }}
\ShowLuaExampleThree {table} {has_one_entry} {{  k = 1, v = 2 }}
\ShowLuaExampleThree {table} {has_one_entry} {{  k = 1 }}
\ShowLuaExampleThree {table} {is_empty}      {{ 1, 2, 3 }}
\ShowLuaExampleThree {table} {is_empty}      {{ }}

\stopsummary

\startsummary[title={is_simple_table}]

This is a helper that happens to be exposed so let's see what it does.

\ShowLuaExampleTwo   {table} {is_simple_table} {{ 1, 2, 3 }}
\ShowLuaExampleTwo   {table} {is_simple_table} {{ 1, "a", true }}
\ShowLuaExampleThree {table} {is_simple_table} {{ 1, "a", { 1 } }}

The return value is either a table with the elements or \type {nil}. We use this
helper in the serializer.

\stopsummary

\startsummary[title={hashed}]

This helper adds values as keys to a table, which can be handy if you want to do
a reverse lookup. It mostly makes sense for a list of strings:

\ShowLuaExampleTwo {table} {hashed} {{ "foo", "oof", "ofo" }}
\ShowLuaExampleTwo {table} {hashed} {{ 1, "a", true }}

\stopsummary

\startsummary[title={makeweak}]

Say that we have this:

\starttyping
local t = { }
for i=1,9 do
    t[i] = function() end
end
\stoptyping

The table has nine entries and it will keep them until the whole table is
collected or a value is set to \type {nil}. When we say:

\starttyping
local t = table.makeweak()
for i=1,9 do
    t[i] = function() print (i) end
end
\stoptyping

you can never be sure what the entries are but you can at least test if a value
is set. If so, you can reuse that value, if not, you can redefine the function
and reassign it. It is a way to prevent the same (defined by index) function
again and again. The value has to be an collectable object.

\stopsummary

% -- not used:
%
% \startsummary[title={setmetatablekey getmetatablekey}]
%
% \starttyping
% local t = { }
% table.setmetatablekey(t,"key","value")
% local value = table.getmetatablekey(t,"key")
% \stoptyping
%
% \stopsummary

\startsummary[title={toxml}]

This feature can best be shown with examples. The second argument is a
specification. When a \type {result} table is specified the code gets appended to
that table.

\ShowLuaExampleEight {table} {toxml} {{ "foo", "oof", { "ofo" } }, { spaces = 4 }}
\ShowLuaExampleEight {table} {toxml} {{ "foo", "oof", { "ofo" } }, { indent = 3 }}
\ShowLuaExampleEight {table} {toxml} {{ "foo", "oof", { "ofo" } }, { name = "root" }}
\ShowLuaExampleEight {table} {toxml} {{ "foo", "oof", { "ofo" } }, { name = false }}

\stopsummary

\startsummary[title={tocsv}]

This is a quick-and-dirty converter from a table of tables to a (normally) comma
separated list:

\ShowLuaExampleEight {table} {tocsv} {{ { "foo", "oof" } }, {
    preamble = true,
    fields   = { "a", "b" }
}}

The table is either index or a hash, in the later case a field list is mandate:

\ShowLuaExampleEight {table} {tocsv} {{ { b = "X", a = "Y" } }, {
    preamble = true,
    fields   = { "a", "b" }
}}

You can set the separator:

\ShowLuaExampleEight {table} {tocsv} {{ { "foo", "oof", "ofo" } }, {
    separator = ";"
}}

\stopsummary

\startsummary[title={setmetatablecall setmetatableindex setmetatablenewindex}]

These functions define (or patch) a table with metatable driven behavior;
the table is optional.

\starttyping
local t = table.setmetatableindex({ }, function(t,k) end)
local t = table.setmetatableindex({ }, sometable)
local t = table.setmetatableindex({ }, "empty")
local t = table.setmetatableindex({ }, "self")
local t = table.setmetatableindex({ }, "table")
local t = table.setmetatableindex({ }, "number")
\stoptyping

When a table slot is accessed and no value has been set, \type {empty} makes sure
that an empty string is returned, \type {self} returns the key, \type {table} an
empty table and \type {number} a zero. These values are also assigned. When a
function is passed that one has to do the assignment and return a value.

\starttyping
local t = table.setmetatablenewindex({ }, function(t,k,v) end)
local t = table.setmetatablenewindex({ }, "ignore")
\stoptyping

Here setting a table goes via the function call. Setting a value then has to be
done with \type {rawset}.

\starttyping
local t = table.setmetatablecall({ }, function(t,...) end)
\stoptyping

Calling a table triggers the set function where optional arguments are passed.

\stopsummary

\startsummary[title={setmetatablenewindices}]

You can set all three plugins in one go, so you pass three functions.

\starttyping
local t = table.setmetatableindices  ({ }, f_index, f_newindex, f_call)
\stoptyping

\stopsummary

\startsummary[title={derive}]

A shortcut to define a table that has a table as metatable index is the
following:

\starttyping
local t = table.derive { a = 1, b = 2, c = 3 }
\stoptyping

\stopsummary

\startsummary[title={sparse compact}]

You can remove entries that store an empty strings or \type {false} value from a
table:

\ShowLuaExampleTwo {table} {sparse} {{ a = 1, b = 2, c = false, d = "" }}

The \type {compact} variant calls sparse with two extra \type {true} values, one
that signals that we want to nest, and one that ensures that we keep tables.

\stopsummary

\startsummary[title={filtered}]

This helper started as an experiment, was never used but for now we keep it
around.

\startbuffer
\startluacode
for k, v in table.filtered({ foo = 1, bar = 2, oof = 3 },"f") do
    context("[%s = %s]",k,v)
end

for k, v in table.filtered({ foo = 1, bar = 2, oof = 3 },"f",true) do
    context("(%s = %s)",k,v)
end
\stopluacode
\stopbuffer

\typebuffer

The results are: \inlinebuffer. As fourth optional argument a sort compare
function can be passed.

\stopsummary

\startsummary[title={strip}]

This function removes leading and trailing spaces from string entries and omits
empty strings.

\ShowLuaExampleTwo {table} {strip} {{ "a", "b", "", " d e ", "", "f" }}

\stopsummary

% \startsummary[title={getn}]
% \stopsummary

\stopsection

\startsection[title=Math]

The \LUA\ interpreter comes with a default \type {math} library. In \LUAMETATEX\
we also have \type {xmath}, \type {xcomplex}, \type {xdecimal}, \type {posit} and
\type {vector}. The math library has a few more helpers but if we use a \LUA\
version or a built-in one depends on the engine.

In addition to the built-in math function we provide: \type {round}, \type {odd},
\type {even}, \type {div}, \type {mod}, \type {sind}, \type {cosd} and \type
{tand}, \type {cosh}, \type {sinh}, \type {tanh}, \type {pow}, \type {atan2} (an
alias), \type {ceiling} (an alias), \type {ldexp}, \type {log10}, \type {type},
\type {tointeger} and \type {ult}.

At the \TEX\ end we have a helper \type {luaexpr} that you can use to do
calculations:

\startbuffer
  \luaexpr{1 + 2.3 * 4.5 + math.pi} = \cldcontext{1 + 2.3 * 4.5 + math.pi}
\stopbuffer

\typebuffer

Both calls return the same result, but the first one is normally faster than the
\type {context} command which has quite some overhead.

\blank \getbuffer \blank

The \type {\luaexpr} command can also better deal with for instance conditions,
where it returns \type {true} or \type {false}, while \type {\cldcontext} would
interpret the boolean value as a special signal.

\stopsection

\startsection[title=Booleans]

\startsummary[title={tonumber}]

This function returns the number one or zero. You will seldom need this function.

\starttyping
local state = boolean.tonumber(str)
\stoptyping

\ShowLuaExampleThree {boolean} {tonumber} {true}

\stopsummary

\startsummary[title={toboolean}]

When dealing with configuration files or tables a bit flexibility in setting a
state makes sense, if only because in some cases it's better to say \type {yes}
than \type {true}.

\starttyping
local b = toboolean(str)
local b = toboolean(str,tolerant)
\stoptyping

When the second argument is true, the strings \type {true}, \type {yes}, \type
{on}, \type {1}, \type {t} and the number \type {1} all turn into \type {true}.
Otherwise only \type {true} is honoured. This function is also defined in the
global namespace.

\ShowLuaExampleThree {string} {toboolean} {"true"}
\ShowLuaExampleThree {string} {toboolean} {"yes"}
\ShowLuaExampleThree {string} {toboolean} {"yes",true}

\stopsummary

\startsummary[title={is_boolean}]

This function is somewhat similar to the previous one. It interprets the strings
\type {true}, \type {yes}, \type {on} and \type {t} as \type {true} and
\type{false}, \type {no}, \type {off} and \type {f} as \type {false}. Otherwise
\type {nil} is returned, unless a default value is given, in which case that is
returned.

\starttyping
if is_boolean(str)                then ... end
if is_boolean(str,default)        then ... end
if is_boolean(str,default,struct) then ... end
\stoptyping

\ShowLuaExampleThree {string} {is_boolean} {"true"}
\ShowLuaExampleThree {string} {is_boolean} {"off"}
\ShowLuaExampleThree {string} {is_boolean} {"crap",true}

Like the other boolean converters this one related pretty much to the way we
interface in \CONTEXT. For instance the type {strict} options makes that we don't
test zero and one as string.

\stopsummary

\startsummary[title={booleanstring}]

This function takes one argument and returns a boolean. Relevant values are:

\def\ShowIt#1%
  {\NC \type {#1} \NC \cldcontext{tostring(string.booleanstring(#1))} \NC \NR}

\startcolumns[n=3]
\starttabulate[|T|T|]
\ShowIt {"0"}
\ShowIt {"1"}
\ShowIt {""}
\ShowIt {"false"}
\ShowIt {"true"}
\ShowIt {0}
\ShowIt {1}
\ShowIt {yes}
\ShowIt {on}
\ShowIt {t}
\ShowIt {f}
\stoptabulate
\stopcolumns

\stopsummary

\stopsection

\startsection[title=Strings]

\LUA\ strings are simply sequences of bytes. Of course in some places special
treatment takes place. For instance \type {\n} expands to one or more characters
representing a newline, depending on the operating system, but normally, as long
as you manipulate strings in the perspective of \LUATEX, you don't need to worry
about such issues too much. As \LUATEX\ is a \UTF-8 engine, strings normally are
in that encoding but again, it does not matter much as \LUA\ is quite agnostic
about the content of strings: it does not care about three characters reflecting
one \UNICODE\ character or not. This means that when you use for instance the
functions discussed here, or use libraries like \type {lpeg} behave as you
expect.

Although we now live in \LUAMETATEX\ times a bit of history doesn't hurt. When we
started with \LUATEX, we included the \type {slunicode} library but that one
never got updated. That was no big deal because we (needed and therefore) wrote
\UNICODE\ helpers for \CONTEXT\ that were more flexible. We also didn't need some
of its features. So, instead we used string manipulators, some provided by the
engine, as well as \LPEG\ magic. Over time the repertoire evolved. At some point
\LUA\ itself got some support for \UTF8 built in. As a consequence string
functions are implemented on various modules, often depending on where other
helpers reside. In \LUAMETATEX\ a few critical converters were added to the
engine and as we never used \type {slunicode} it was just removed.

Not all of the helpers discussed here are of general use because after all they
were often made for \CONTEXT. This means that we can adapt details as we go
forward. It's not like \CONTEXT\ users have to deal with conversions when they
use the system, most is hidden. The repertoire presented here is a mix of
functions that come with \LUA, functions that are implemented in \LUA, and
functions that were added to \LUATEX\ and later \LUAMETATEX. When the later are
available, we use them instead of the \LUA\ variants.

\startsummary[title={[lua] byte char}]

As long as we're dealing with \ASCII\ characters we can use these two functions to
go from numbers to characters and vise versa.

\ShowLuaExampleSeven {string} {byte} {"luatex"}
\ShowLuaExampleSeven {string} {byte} {"luatex",1,3}
\ShowLuaExampleSeven {string} {byte} {"luatex",-3,-1}

\ShowLuaExampleSeven {string} {char} {65}
\ShowLuaExampleSeven {string} {char} {65,66,67}

\stopsummary

\startsummary[title={[lua] sub}]

You cannot directly access a character in a string but you can take any slice you
want using \type {sub}. You need to provide a start position and negative values
will count backwards from the end.

\starttyping
local slice = string.sub(str,first,last)
\stoptyping

\ShowLuaExampleThree {string} {sub} {"abcdef",2}
\ShowLuaExampleThree {string} {sub} {"abcdef",2,3}
\ShowLuaExampleThree {string} {sub} {"abcdef",-3,-2}

\stopsummary

\startsummary[title={[lua] gsub}]

There are two ways of analyzing the content of a string. The more modern and
flexible approach is to use \type {lpeg}. The other one uses some functions in
the \type {string} namespace that accept so called patterns for matching. While
\type {lpeg} is more powerfull than regular expressions, the pattern matching is
less powerfull but sometimes faster and also easier to specify. In many cases it
can do the job quite well.

\starttyping
local new, count = string.gsub(old,pattern,replacement)
\stoptyping

The replacement can be a function. Often you don't want the number
of matches, and the way to avoid this is either to store the result
in a variable:

\starttyping
local new = string.gsub(old,"lua","LUA")
print(new)
\stoptyping

or to use parentheses to signal the interpreter that only one value
is return.

\starttyping
print((string.gsub(old,"lua","LUA"))
\stoptyping

Patterns can be more complex so you'd better read the \LUA\ manual if you want to
know more about them.

\ShowLuaExampleThree {string} {gsub} {"abcdef","b","B"}
\ShowLuaExampleThree {string} {gsub} {"abcdef","[bc]",string.upper}

An optional fourth argument specifies how often the replacement has to happen

\ShowLuaExampleThree {string} {gsub} {"textextextex","tex","abc"}
\ShowLuaExampleThree {string} {gsub} {"textextextex","tex","abc",1}
\ShowLuaExampleThree {string} {gsub} {"textextextex","tex","abc",2}

You can also pass a table as replacement, as in:

\ShowLuaExampleThree {string} {gsub} {"we use tex",".",{t="T",x="X"}}

So to summarize, we can use strings, tables and functions as replacement values,
but in the later two cases when there is no return value or match, no replacement
takes place.

See the \type {match} section below for more about pattern matching in \type
{gsub}.

\stopsummary

\startsummary[title={[lua] find}]

The \type {find} function returns the first and last position of the match:

\starttyping
local first, last = find(str,pattern)
\stoptyping

If you're only interested if there is a match at all, it's enough to know that
there is a first position. No match returns \type {nil}. So,

\starttyping
if find("luatex","tex") then ... end
\stoptyping

works out okay. You can pass an extra argument to \type {find} that indicates the
start position. So you can use this function to loop over all matches: just start
again at the end of the last match.

A fourth optional argument is a boolean that signals not to interpret the pattern
but use it as|-|is.

\ShowLuaExampleThree {string} {find} {"abc.def","c%.d",1,false}
\ShowLuaExampleThree {string} {find} {"abc.def","c%.d",1,true}
\ShowLuaExampleThree {string} {find} {"abc%.def","c%.d",1,false}
\ShowLuaExampleThree {string} {find} {"abc%.def","c%.d",1,true}

See the \type {match} section below for more about pattern matching in \type
{find}.

\stopsummary

\startsummary[title={[lua] match gmatch}]

With \type {match} you can split of bits and pieces of a string. The parenthesis
indicate the captures.

\starttyping
local a, b, c, ... = string.match(str,pattern)
\stoptyping

The \type {gmatch} function is used to loop over a string, for instance the
following code prints the elements in a comma separated list, ignoring spaces
after commas.

\starttyping
for s in string.gmatch(str,"([^,%s])+") do
  print(s)
end
\stoptyping

A more detailed description of patterns can be found in the \LUA\ reference
manual, so we only mention the special directives. Characters are grouped in
classes:

\starttabulate[|lT|l|]
\HL
\NC \letterpercent a \NC letters                  \NC \NR
\NC \letterpercent l \NC lowercase letters        \NC \NR
\NC \letterpercent u \NC uppercase letters        \NC \NR
\NC \letterpercent d \NC digits                   \NC \NR
\NC \letterpercent w \NC letters and digits       \NC \NR
\NC \letterpercent c \NC control characters       \NC \NR
\NC \letterpercent p \NC punctuation              \NC \NR
\NC \letterpercent x \NC hexadecimal characters   \NC \NR
\NC \letterpercent s \NC space related characters \NC \NR
\HL
\stoptabulate

You can create sets too:

\starttabulate[|lT|l|]
\HL
\NC [\letterpercent l\letterpercent d]  \NC lowercase letters and digits \NC \NR
\NC [^\letterpercent d\letterpercent p] \NC all characters except digits and punctuation \NC \NR
\NC [p-z]                               \NC all characters in the range \type {p} upto \type {z} \NC \NR
\NC [pqr]                               \NC all characters \type {p}, \type {q} and \type {r} \NC \NR
\HL
\stoptabulate

There are some characters with special meanings:

\starttabulate[|lT|l|]
\HL
\NC \letterhat       \NC the beginning of a string \NC \NR
\NC \letterdollar    \NC end of a string \NC \NR
\NC .                \NC any character \NC \NR
\NC *                \NC zero or more of the preceding specifier, greedy \NC \NR
\NC -                \NC zero or more of the preceding specifier, least possible \NC \NR
\NC +                \NC one or more of the preceding specifier \NC \NR
\NC ?                \NC zero or one of the preceding specifier \NC \NR
\NC ( )              \NC encapsulate capture \NC \NR
\NC \letterpercent b \NC capture all between the following two characters \NC \NR
\HL
\stoptabulate

You can use whatever you like to be matched:

\starttabulate[|lT|l|]
\HL
\NC pqr                           \NC the sequence \type {pqr} \NC \NR
\NC my name is (\letterpercent w) \NC the word following \type {my name is} \NC \NR
\HL
\stoptabulate

If you want to specify such a token as it is, then you can precede it with a
percent sign, so to get a percent, you need two in a row.

\ShowLuaExampleThree {string} {match} {"before:after","^(.-):"}
\ShowLuaExampleThree {string} {match} {"before:after","^([^:])"}
\ShowLuaExampleThree {string} {match} {"before:after","bef(.*)ter"}
\ShowLuaExampleThree {string} {match} {"abcdef","[b-e]+"}
\ShowLuaExampleThree {string} {match} {"abcdef","[b-e]*"}
\ShowLuaExampleThree {string} {match} {"abcdef","b-e+"}
\ShowLuaExampleThree {string} {match} {"abcdef","b-e*"}

Such patterns should not be confused with regular expressions, although to some
extent they can do the same. If you really want to do complex matches, you should
look into \LPEG.

One trick is worth mentioning: matching a previous match. Here is an exmaple

\ShowLuaExampleZero {string} {match} {"AAoneBBtwoBBthreeAA","^(AA)(.*)(BB)(.*)%3(.*)%1$"}
\ShowLuaExampleZero {string} {gsub}  {"AAtestAA","^(AA)(.*)%1$","%2"}

It is easy to forget that such features exist, although the situations where you
have fenced that are the same are limited (single and double quotes as well as
vertical bars come to mind).

\stopsummary

\startsummary[title={[lua] lower upper}]

These two function spreak for themselves.

\ShowLuaExampleThree {string} {lower} {"LOW"}
\ShowLuaExampleThree {string} {upper} {"upper"}

\stopsummary

\startsummary[title={[lua] format}]

The \type {format} function takes a template as first argument and one or more
additional arguments depending on the format. The template is similar to the one
used in \CCODE\ but it has some extensions.

\starttyping
local s = format(format, str, ...)
\stoptyping

The following table gives an overview of the possible format directives. The
\type {s} is the most probably candidate and can handle numbers well as strings.
Watch how the minus sign influences the alignment. \footnote {There can be
differences between platforms although so far we haven't run into problems. Also,
\LUA\ 5.2 does a bit more checking on correct arguments and \LUA\ 5.3 is more
picky on integers.}

\starttabulate[|lB|lT|lT|lT|]
\HL
\NC integer     \NC \letterpercent i    \NC 12345  \NC \cldcontext{string.format("\letterpercent i",   12345 )} \NC \NR
\NC integer     \NC \letterpercent d    \NC 12345  \NC \cldcontext{string.format("\letterpercent d",   12345 )} \NC \NR
\NC unsigned    \NC \letterpercent u    \NC -12345 \NC \cldcontext{string.format("\letterpercent u",   12345 )} \NC \NR
\NC character   \NC \letterpercent c    \NC 123    \NC \cldcontext{string.format("\letterpercent c",   89    )} \NC \NR
\NC hexadecimal \NC \letterpercent x    \NC 123    \NC \cldcontext{string.format("\letterpercent x",   123   )} \NC \NR
\NC             \NC \letterpercent X    \NC 123    \NC \cldcontext{string.format("\letterpercent X",   123   )} \NC \NR
\NC octal       \NC \letterpercent o    \NC 12345  \NC \cldcontext{string.format("\letterpercent o",   12345 )} \NC \NR
\HL
\NC string      \NC \letterpercent s    \NC abc    \NC \cldcontext{string.format("\letterpercent s",   "abcd")} \NC \NR
\NC             \NC \letterpercent -8s  \NC 123    \NC \cldcontext{string.format("\letterpercent -8s", 123   )} \NC \NR
\NC             \NC \letterpercent 8s   \NC 123    \NC \cldcontext{string.format("\letterpercent 8s",  123   )} \NC \NR
\HL
\NC float       \NC \letterpercent 0.2f \NC 12.345 \NC \cldcontext{string.format("\letterpercent 0.2f",12.345)} \NC \NR
\NC exponential \NC \letterpercent 0.2e \NC 12.345 \NC \cldcontext{string.format("\letterpercent 0.2e",12.345)} \NC \NR
\NC             \NC \letterpercent 0.2E \NC 12.345 \NC \cldcontext{string.format("\letterpercent 0.2E",12.345)} \NC \NR
\NC autofloat   \NC \letterpercent 0.2g \NC 12.345 \NC \cldcontext{string.format("\letterpercent 0.2g",12.345)} \NC \NR
\NC             \NC \letterpercent 0.2G \NC 12.345 \NC \cldcontext{string.format("\letterpercent 0.2G",12.345)} \NC \NR
\HL
\stoptabulate

\startasciimode
\ShowLuaExampleThree {string} {format} {"U+\letterpercent 05X",2010}
\stopasciimode

\stopsummary

\startsummary[title={striplines}]

The \type {striplines} function can strip leading and trailing empty lines,
collapse or delete intermediate empty lines and strips leading and trailing
spaces. We will demonstrate this with string \type {str}:

\startluacode
local str = table.concat( {
"  ",
"    aap",
"  noot mies",
"  ",
"    ",
" wim    zus   jet",
"teun    vuur gijs",
"       lam    kees bok weide",
"    ",
"does hok duif schapen  ",
"  ",
}, "\n")

document.TempString = str

function document.ShowStrippedString(str)
    str = string.gsub(str," ","\\allowbreak<sp>\\allowbreak ")
    str = string.gsub(str,"([\010])","\\allowbreak<lf>\\allowbreak ")
    context.startalign { "flushleft,verytolerant" }
        context("{\\tttf %s}",str)
    context.stopalign()
end

function document.ShowStrippedBuffer(name,str)
    context.tobuffer(name,str)
    context.typebuffer( { name }, { numbering = "line" })
    context.resetbuffer { name }
end

function document.ShowStrippedCommand(option)
    context.type( { style = "ttbf" }, [[utilities.strings.striplines(str,"]] .. option .. [[")]])
end

context.blank { "big" }
document.ShowStrippedString(str)
document.ShowStrippedBuffer("dummy",str)
\stopluacode

The different options for stripping are demonstrated below, We use verbose
descriptions instead of vague boolean flags.

\startluacode
local str = document.TempString ; document.TempString = nil

for option in table.sortedhash(utilities.strings.striplinepatterns) do
    local s = utilities.strings.striplines(str,option)
    context.blank()
    document.ShowStrippedCommand(option)
    context.blank { "big,samepage" }
    document.ShowStrippedString(s)
    context.blank { "big,samepage" }
    document.ShowStrippedBuffer(option,str)
end
\stopluacode

You can of course mix usage with the normal \type {context} helper commands, for
instance put them in buffers. Buffers normally will prune leading and trailing
empty lines anyway.

\starttyping
context.tobuffer("dummy",utilities.strings.striplines(str))
context.typebuffer( { "dummy" }, { numbering = "line" })
\stoptyping

\stopsummary

\startsummary[title={formatters}]

The \type {format} function discussed before is the built|-|in. As an alternative
\CONTEXT\ provides an additional formatter that has some extensions. Interesting
is that that one is often more efficient, although there are cases where the
speed is comparable. As we run out of keys, some extra ones are a bit counter
intuitive, like \type {l} for booleans (logical).

\start \setuptype[color=]

\starttabulate[|lB|lT|lT|lT|]
\HL
\NC utf character       \NC \letterpercent c            \NC 322           \NC \cldcontext{"\letterpercent c",322} \NC \NR
\HL
\NC string              \NC \letterpercent s            \NC foo           \NC \cldcontext{"\letterpercent s","foo"} \NC \NR
\NC force tostring      \NC \letterpercent S            \NC nil           \NC \cldcontext{"\letterpercent S",nil} \NC \NR
\NC quoted string       \NC \letterpercent q            \NC foo           \NC \cldcontext{"\letterpercent q","foo"} \NC \NR
\NC force quoted string \NC \letterpercent Q            \NC nil           \NC \cldcontext{"\letterpercent Q",nil} \NC \NR
\NC                     \NC \letterpercent N            \NC 0123          \NC \cldcontext{"\letterpercent N","0123"} \NC \NR
\NC automatic quoted    \NC \letterpercent a            \NC true          \NC \cldcontext{"\letterpercent a",true} \NC \NR\NC \NR
\NC                     \NC \letterpercent A            \NC true          \NC \cldcontext{"\letterpercent A",true} \NC \NR\NC \NR
\NC left aligned utf    \NC \letterpercent 30<          \NC xx½xx         \NC \cldcontext{"\letterpercent 30<","xx½xx"} \NC \NR\NC \NR
\NC right aligned utf   \NC \letterpercent 30>          \NC xx½xx         \NC \cldcontext{"\letterpercent 30>","xx½xx"} \NC \NR\NC \NR
\HL
\NC integer             \NC \letterpercent i            \NC 1234          \NC \cldcontext{"\letterpercent i",1234} \NC \NR
\NC integer             \NC \letterpercent d            \NC 1234          \NC \cldcontext{"\letterpercent d",1234} \NC \NR
\NC signed number       \NC \letterpercent I            \NC 1234          \NC \cldcontext{"\letterpercent I",1234} \NC \NR
\NC rounded number      \NC \letterpercent r            \NC 1234.56       \NC \cldcontext{"\letterpercent r",1234.56} \NC \NR
\NC stripped number     \NC \letterpercent N            \NC 000123        \NC \cldcontext{"\letterpercent N","000123"} \NC \NR
\NC comma/period float  \NC \letterpercent m            \NC 12.34         \NC \cldcontext{"\letterpercent m",12.34} \NC \NR
\NC period/comma float  \NC \letterpercent M            \NC 12.34         \NC \cldcontext{"\letterpercent M",12.34} \NC \NR
\HL
\NC hexadecimal         \NC \letterpercent x            \NC 1234          \NC \cldcontext{"\letterpercent x",1234} \NC \NR
\NC                     \NC \letterpercent X            \NC 1234          \NC \cldcontext{"\letterpercent X",1234} \NC \NR
\NC octal               \NC \letterpercent o            \NC 1234          \NC \cldcontext{"\letterpercent o",1234} \NC \NR
\HL
\NC float               \NC \letterpercent 0.2f         \NC 12.345        \NC \cldcontext{"\letterpercent 0.2f",12.345} \NC \NR
\NC formatted float     \NC \letterpercent 2.3k         \NC 12.3456       \NC \cldcontext{"\letterpercent 2.3f",12.3456} \NC \NR
\NC checked float       \NC \letterpercent 0.2F         \NC 12.30         \NC \cldcontext{"\letterpercent 0.2F",12.3} \NC \NR
\NC exponential         \NC \letterpercent .2e          \NC 12.345e120    \NC \cldcontext{"\letterpercent 0.2j",12.345e120} \NC \NR
\NC                     \NC \letterpercent .2E          \NC 12.345e120    \NC \cldcontext{"\letterpercent 0.2J",12.345e120} \NC \NR
\NC sparse exp          \NC \letterpercent 0.2j         \NC 12.345e120    \NC \cldcontext{"\letterpercent 0.2j",12.345e120} \NC \NR
\NC                     \NC \letterpercent 0.2J         \NC 12.345e120    \NC \cldcontext{"\letterpercent 0.2J",12.345e120} \NC \NR
\NC autofloat           \NC \letterpercent g            \NC 12.345        \NC \cldcontext{"\letterpercent 0.2J",12.345} \NC \NR
\NC                     \NC \letterpercent G            \NC 12.345        \NC \cldcontext{"\letterpercent 0.2J",12.345} \NC \NR
\HL
\NC unicode value 0x    \NC \letterpercent h            \NC ł 1234        \NC \cldcontext{"\letterpercent v \letterpercent v", "ł",1234} \NC \NR
\NC                     \NC \letterpercent H            \NC ł 1234        \NC \cldcontext{"\letterpercent V \letterpercent V", "ł",1234} \NC \NR
\NC unicode value U+    \NC \letterpercent u            \NC ł 1234        \NC \cldcontext{"\letterpercent u \letterpercent u", "ł",1234} \NC \NR
\NC                     \NC \letterpercent U            \NC ł 1234        \NC \cldcontext{"\letterpercent U \letterpercent U", "ł",1234} \NC \NR
\HL
\NC points              \NC \letterpercent p            \NC 1234567       \NC \cldcontext{"\letterpercent p",1234567} \NC \NR
\NC points no unit      \NC \letterpercent P            \NC 1234567       \NC \cldcontext{"\letterpercent p",1234567} \NC \NR
\NC basepoints          \NC \letterpercent b            \NC 1234567       \NC \cldcontext{"\letterpercent b",1234567} \NC \NR
\NC basepoints no unit  \NC \letterpercent B            \NC 1234567       \NC \cldcontext{"\letterpercent b",1234567} \NC \NR
\HL
\NC table concat        \NC \letterpercent t            \NC \arg{1,2,3}   \NC \cldcontext{"\letterpercent t",{1,2,3}}  \NC \NR
\NC                     \NC \letterpercent *t           \NC \arg{1,2,3}   \NC \cldcontext{"\letterpercent *t",{1,2,3}} \NC \NR
\NC                     \NC \letterpercent \arg{ AND }t \NC \arg{a=1,b=3} \NC \cldcontext{"\letterpercent +{ AND }T",{a=1,b=2}} \NC \NR
\NC table serialize     \NC \letterpercent T            \NC \arg{1,2,3}   \NC \cldcontext{"\letterpercent *t",{1,2,3}} \NC \NR
\NC                     \NC \letterpercent T            \NC \arg{a=1,b=3} \NC \let|\relax\cldcontext{"\letterpercent T",{a=1,b=2}} \NC \NR
\NC                     \NC \letterpercent +T           \NC \arg{a=1,b=3} \NC \cldcontext{"\letterpercent [+T]",{a=1,b=2}} \NC \NR
\HL
\NC boolean (logic)     \NC \letterpercent l            \NC "a" == "b"    \NC \cldcontext{"\letterpercent l","a"=="b"} \NC \NR
\NC                     \NC \letterpercent L            \NC "a" == "b"    \NC \cldcontext{"\letterpercent L","a"=="b"} \NC \NR
\HL
\NC whitespace          \NC \letterpercent w            \NC 3             \NC \obeyspaces\vl\cldcontext{"\letterpercent w",3}\vl \NC \NR
\NC                     \NC \letterpercent 2w           \NC 3             \NC \obeyspaces\vl\cldcontext{"\letterpercent 2w",3}\vl \NC \NR
\NC (fixed)             \NC \letterpercent 4W           \NC               \NC \obeyspaces\vl\cldcontext{"\letterpercent 4W"}\vl \NC \NR
\HL
\NC skip                \NC \letterpercent 2z            \NC 1,2,3,4      \NC \obeyspaces\vl\cldcontext{"\letterpercent s\letterpercent 2z\letterpercent s",1,2,3,4}\vl \NC \NR
\NC reference argument  \NC \letterpercent 2Z            \NC 1,2,3,4      \NC \obeyspaces\vl\cldcontext{"\letterpercent s\letterpercent 2Z\letterpercent s",1,2,3,4}\vl \NC \NR
\HL
\stoptabulate

\stop

The generic formatters \type {a} and \type {A} convert the argument into a string
and deals with strings, number, booleans, tables and whatever. We mostly use
these in tracing. The lowercase variant uses single quotes, and the uppercase
variant uses double quotes.

A special one is the alignment formatter, which is a variant on the \type {s} one
that also takes an optional positive of negative number:

\startbuffer
\startluacode
context.start()
context.tttf()
context.verbatim("[[% 30<]]","xxaxx") context.par()
context.verbatim("[[% 30<]]","xx½xx") context.par()
context.verbatim("[[% 30>]]","xxaxx") context.par()
context.verbatim("[[% 30>]]","xx½xx") context.par()
context.verbatim("[[%-30<]]","xxaxx") context.par()
context.verbatim("[[%-30<]]","xx½xx") context.par()
context.verbatim("[[%-30>]]","xxaxx") context.par()
context.verbatim("[[%-30>]]","xx½xx") context.par()
context.stop()
\stopluacode
\stopbuffer

\typebuffer \getbuffer

There are two more formatters plugged in: \type {!xml!} and \type {!tex!}. These
are best demonstrated with an example:

\starttyping
local xf = formatter["xml escaped: %!xml!"]
local xr = formatter["tex escaped: %!tex!"]

print(xf("x > 1 && x < 10"))
print(xt("this will cost me $123.00 at least"))
\stoptyping

weird, this fails when cld-verbatim is there as part of the big thing:
catcodetable 4 suddenly lacks the comment being a other

The \type {context} command uses the formatter so one can say:

\startbuffer
\startluacode
context("first some xml: %!xml!, and now some %!tex!",
    "x > 1 && x < 10", "this will cost me $123.00 at least")
\stopluacode
\stopbuffer

\typebuffer

This renders as follows:

\blank \getbuffer \blank

You can extend the formatter but we advise you not to do that unless you're sure
what you're doing. You never know what \CONTEXT\ itself might add for its own
benefit.

However, you can define your own formatter and add to that without interference.
In fact, the main formatter is just defined that way. This is how it works:

\startbuffer[definition]
local MyFormatter = utilities.strings.formatters.new()

utilities.strings.formatters.add (
    MyFormatter,
    "upper",
    "globaldata.string.upper(%s)"
)
\stopbuffer

\typebuffer[definition]

Now you can use this one as:

\startbuffer[usage]
context.bold(MyFormatter["It's %s or %!upper!."]("this","that"))
\stopbuffer

\typebuffer[usage]

\blank \ctxluabuffer[definition,usage] \blank

Because we're running inside \CONTEXT, a better definition would be this:

\startbuffer
local MyFormatter = utilities.strings.formatters.new()

utilities.strings.formatters.add (
    MyFormatter,
    "uc",
    "myupper(%s)",
 -- "local myupper = globaldata.characters.upper"
    { myupper = globaldata.characters.upper }
)

utilities.strings.formatters.add (
    MyFormatter,
    "lc",
    "mylower(%s)",
 -- "local mylower = globaldata.characters.lower"
    { mylower = globaldata.characters.lower }
)

utilities.strings.formatters.add (
    MyFormatter,
    "sh",
    "myshaped(%s)",
 -- "local myshaped = globaldata.characters.shaped"
    { myshaped = globaldata.characters.shaped }
)

context(MyFormatter["Uppercased: %!uc!"]("ÀÁÂÃÄÅàáâãäå"))
context.par()
context(MyFormatter["Lowercased: %!lc!"]("ÀÁÂÃÄÅàáâãäå"))
context.par()
context(MyFormatter["Reduced: %!sh!"]("ÀÁÂÃÄÅàáâãäå"))
\stopbuffer

\typebuffer

The last arguments creates shortcuts. As expected we get:

\blank \ctxluabuffer \blank

Of course you can also apply the casing functions directly so in practice you
shouldn't use formatters without need. Among the advantages of using formatters
are:

\startitemize[packed]
\startitem They provide a level of abstraction. \stopitem
\startitem They can replace multiple calls to \type {\context}. \stopitem
\startitem Sometimes they make source code look better. \stopitem
\startitem Using them is often more efficient and faster. \stopitem
\stopitemize

The last argument might sound strange but considering the overhead involved in
the \type {context} (related) functions, doing more in one step has benefits.
Also, formatters are implemented quite efficiently, so their overhead can be
neglected.

In the examples you see that a formatter extension is itself a template.

\startbuffer
local FakeXML = utilities.strings.formatters.new()

utilities.strings.formatters.add(FakeXML,"b",[["<" ..%s..">" ]])
utilities.strings.formatters.add(FakeXML,"e",[["</"..%s..">" ]])
utilities.strings.formatters.add(FakeXML,"n",[["<" ..%s.."/>"]])

context(FakeXML["It looks like %!b!xml%!e! doesn't it?"]("it","it"))
\stopbuffer

\typebuffer

This gives: \ctxluabuffer. Of course we could go over the top here:

\startbuffer
local FakeXML = utilities.strings.formatters.new()

local stack = { }

function document.f_b(s)
    table.insert(stack,s)
    return "<" .. s .. ">"
end

function document.f_e()
    return "</" .. table.remove(stack) .. ">"
end

utilities.strings.formatters.add(FakeXML,"b",[[globaldata.document.f_b(%s)]])
utilities.strings.formatters.add(FakeXML,"e",[[globaldata.document.f_e()]])

context(FakeXML["It looks like %1!b!xml%0!e! doesn't it?"]("it"))
\stopbuffer

\typebuffer

This gives: \ctxluabuffer. Such a template look horrible, although it's not too
far from the regular format syntax: just compare \type {%1f} with \type {%1!e!}.
The zero trick permits us to inject information that we've put on the stack. As
this kind of duplicate usage might occur most, a better solution is available:

\startbuffer
local FakeXML = utilities.strings.formatters.new()

utilities.strings.formatters.add(FakeXML,"b",[["<"  .. %s .. ">"]])
utilities.strings.formatters.add(FakeXML,"e",[["</" .. %s .. ">"]])

context(FakeXML["It looks like %!b!xml%-1!e! doesn't it?"]("it"))
\stopbuffer

\typebuffer

We get: \ctxluabuffer. Anyhow, in most cases you will never feel the need for
such hackery and the regular formatter works fine. Adding this extension
mechanism was rather trivial and it doesn't influence the performance.

In \CONTEXT\ we have a few more extensions:

\starttyping
utilities.strings.formatters.add (
    strings.formatters, "unichr",
    [["U+" .. format("%%05X",%s) .. " (" .. utfchar(%s) .. ")"]]
)

utilities.strings.formatters.add (
    strings.formatters, "chruni",
    [[utfchar(%s) .. " (U+" .. format("%%05X",%s) .. ")"]]
)
\stoptyping

This one is used in messages:

\startbuffer
context("Missing character %!chruni! in font.",234) context.par()
context("Missing character %!unichr! in font.",234)
\stopbuffer

\typebuffer

This shows up as:

\blank \getbuffer \blank

If you look closely to the definition, you will notice that we use \type {%s}
twice. This is a feature of the definer function: if only one argument is
picked up (which is default) then the replacement format can use that two
times. Because we use a format in the constructor, we need to escape the
percent sign there.

\stopsummary

\startsummary[title={strip}]

This function removes any leading and trailing whitespace characters.

\starttyping
local s = string.strip(str)
\stoptyping

\ShowLuaExampleThree {string} {strip} {" lua + tex = luatex "}

\stopsummary

\startsummary[title={split splitlines checkedsplit}]

The line splitter is a special case of the generic splitter. The \type {split}
function can get a string as well an \type {lpeg} pattern. The \type
{checkedsplit} function removes empty substrings.

\starttyping
local t = string.split        (str, pattern)
local t = string.split        (str, lpeg)
local t = string.checkedsplit (str, lpeg)
local t = string.splitlines   (str)
\stoptyping

\start \let\ntex\relax % hack

\ShowLuaExampleTwo {string} {split}        {"a, b,c, d", ","}
\ShowLuaExampleTwo {string} {split}        {"p.q,r", lpeg.S(",.")}
\ShowLuaExampleTwo {string} {checkedsplit} {";one;;two", ";"}
\ShowLuaExampleTwo {string} {splitlines}   {"lua\ntex nic"}

\stop

\stopsummary

\startsummary[title={quoted unquoted}]

You will hardly need these functions. The \type {quoted} function can normally be
avoided using the \type {format} pattern \type {%q}. The \type {unquoted}
function removes single or double quotes but only when the string starts and ends
with the same quote.

\starttyping
local q = string.quoted  (str)
local u = string.unquoted(str)
\stoptyping

\ShowLuaExampleThree {string} {quoted}   {[[test]]}
\ShowLuaExampleThree {string} {quoted}   {[[test"test]]}
\ShowLuaExampleThree {string} {unquoted} {[["test]]}
\ShowLuaExampleThree {string} {unquoted} {[["t\"est"]]}
\ShowLuaExampleThree {string} {unquoted} {[["t\"est"x]]}
\ShowLuaExampleThree {string} {unquoted} {"\'test\'"}

\stopsummary

\startsummary[title={count}]

The function \type {count} returns the number of times that a given pattern
occurs. Beware: if you want to deal with \UTF\ strings, you need the variant that
sits in the \type {lpeg} namespace.

\starttyping
local n = count(str,pattern)
\stoptyping

\ShowLuaExampleThree {string} {count} {"test me", "e"}

\stopsummary

\startsummary[title={limit}]

This function can be handy when you need to print messages that can be rather
long. By default, three periods are appended when the string is chopped.

\starttyping
print(limit(str,max,sentinel)
\stoptyping

\ShowLuaExampleThree {string} {limit} {"too long", 6}
\ShowLuaExampleThree {string} {limit} {"too long", 6, " (etc)"}

\stopsummary

\startsummary[title={is_empty}]

A string considered empty by this function when its length is zero or when it
only contains spaces.

\starttyping
if is_empty(str) then ... end
\stoptyping

\ShowLuaExampleThree {string} {is_empty} {""}
\ShowLuaExampleThree {string} {is_empty} {" "}
\ShowLuaExampleThree {string} {is_empty} {" ? "}

\stopsummary

\startsummary[title={escapedpattern topattern}]

These two functions are rather specialized. They come in handy when you need to
escape a pattern, i.e.\ prefix characters with a special meaning by a \type {%}.

\starttyping
local e = escapedpattern(str, simple)
local p = topattern     (str, lowercase, strict)
\stoptyping

The simple variant does less escaping (only \type {-.?*} and is for instance used
in wildcard patterns when globbing directories. The \type {topattern} function
always does the simple escape. A strict pattern gets anchored to the beginning
and end. If you want to see what these functions do you can best look at their
implementation.

\stopsummary

\startsummary[title={autosingle autodouble}]

These are helpers that we use in formatters: they wrap their content in single or
double quotes. For practical reasons we keep them in the \type {string} but they
could as well be private because we don't use them in other cases.

\ShowLuaExampleZero {string} {autodouble} {'123"456'}
\ShowLuaExampleZero {string} {autosingle} {'123"456'}

\ShowLuaExampleZero {string} {autodouble} {{ 1, 2, 3 }}
\ShowLuaExampleZero {string} {autosingle} {{ 4, 5, 6 },"+"}

\stopsummary

\startsummary[title={tohex toHEX todec tobytes hextocharacters}]

Here are a few of the public converters. In an ecosystem like \CONTEXT\
hexadecimal strings are all over the place deep down, especially in the backend.

\ShowLuaExampleZero {string} {tohex}   {"A"}       % bytestohex
\ShowLuaExampleZero {string} {tohex}   {"A,B:C.D"} %
\ShowLuaExampleZero {string} {toHEX}   {"A,B:C.D"} % bytestoHEX
\ShowLuaExampleZero {string} {todec}   {"A"}       % bytestodec
\ShowLuaExampleZero {string} {todec}   {"A,B:C.D"} %
\ShowLuaExampleZero {string} {tobytes} {"6C7561"}  % hextobytes
\ShowLuaExampleZero {string} {tobytes} {"4c7561"}  % hextobytes

% string.hashes.X00[n] % [named] lower ascii (subset)
% string.hashes.X02[n]
% string.hashes.X04[n]
% string.hashes.spaces[n]

The \typ {tobytes} function is also available as \typ {hextocharacters}, due to
the history of the implementation.

\stopsummary

\startsummary[title={dectointeger hextointeger octtointeger chrtointeger}]

These convert from various (milti-)byte \quote {encodings} to an integer.

\ShowLuaExampleZero {string} {dectointeger} {"123"} % kind of tonumber
\ShowLuaExampleZero {string} {hextointeger} {"1B3"}
\ShowLuaExampleZero {string} {octtointeger} {"17"}
\ShowLuaExampleZero {string} {chrtointeger} {"1"}

\stopsummary

% -- inspect(string.totable("ABCD")) -- no utf
% -- inspect(string.toutf  ("\254\255\0A\0B\0C")) -- utf16 with bom
% -- inspect(string.toutf  ("\255\254A\0B\0C\0")) -- utf16 with bom
%

\startsummary[title={tracedchar}]

This is typically a function that we only provide because it is used in other
functions.

\ShowLuaExampleZero {string} {tracedchar} {"\0"}
\ShowLuaExampleZero {string} {tracedchar} {"\6"}

Only the range upto space (\type {0x20}) returns a string, otherwise you get
\type {nil}. This permits usage in \ASCII\ as well as \UTF\ tracing because we
might want to do something with the visible cases.

\stopsummary

% -- inspect(string.toutf8 ("A")) -- no utf
% -- inspect(string.toutf32("A")) -- no utf

% -- inspect(string.utfcharacter(84,69,88))

% -- local s = [[
% --     test
% --       test
% --         test
% -- ]]
% -- inspect(string.longtostring(s))

\startsummary[title={nospaces}]

If spaces bother you the next function can be of help:

\ShowLuaExampleZero {string} {nospaces} {" test test test "}

\stopsummary

% -- inspect(string.tobytes("6935")) -- hextobytes == hextocharacters (so we can alias)
%
% -- local c = os.clock() for i=1,100000 do string.hextocharacters("696969696969696969696969") end print(os.clock()-c)
% -- local c = os.clock() for i=1,100000 do string.tobytes        ("696969696969696969696969") end print(os.clock()-c)

\startsummary[title={[lua] rep}]

There are not that many helpers in the standard \type {string} library but \type
{rep} (\type {repeat} would clash with the keyword) is one of them:

\ShowLuaExampleZero {string} {rep} {"twice",2}

\stopsummary

\startsummary[title={[lua] reverse}]

This is a more recent addition which only makes sense if we talk bytes; why reverse
a string anyway?

\ShowLuaExampleZero {string} {reverse} {"hello dlrow"}

\stopsummary

\startsummary[title={[lua] pack unpack packsize}]

These are basically scanners driven by a syntax specification pattern. There is
not much to tell about them here as we don't use them in \CONTEXT. If you need
them you ave a good reason to buy the official \LUA\ manual.

\stopsummary

\startsummary[title={[lua] dump}]

This function dumps a function as bytecode that later can be loaded.

\stopsummary

\startsummary[title={toboolean}]

See the boolean section.

\stopsummary

\startsummary[title={bytes bytepairs}]

The byte iterators originally were meant for manipulating an eight bit (single byte encodings)
or sixteen bit (\UTF16) input stream in order to get \UTF8.

\startbuffer
\startluacode
for a in string.bytes("foof") do
    context("<%i> ",a)
end

for a, b in string.bytepairs("foof") do
    context("[%i,%i] ",a,b)
end

\stopluacode
\stopbuffer

\typebuffer

Both give: \inlinebuffer.

\stopsummary

\startsummary[title={characters characterpairs}]

These two iterators grab single byte character which limits their application:

\startbuffer
\startluacode
for a in string.characters("foof") do
    context("<%s> ",a)
end

for a, b in string.characterpairs("foof") do -- originally meant for utf16
    context("[%s,%s] ",a,b)
end
\stopluacode
\stopbuffer

\typebuffer

The two examples return: \inlinebuffer.

\stopsummary

\startsummary[title={utfcharacters utfvalues}]

These two iterators grab \UTF\ character:

\startbuffer
\startluacode
for a in string.utfcharacters("foof") do -- originally meant for utf16
    context("<%s> ",a)
end

for a in string.utfvalues("foof") do
    context("[%i] ",a)
end
\stopluacode
\stopbuffer

\typebuffer

They return: \inlinebuffer.

\stopsummary

\startsummary[title={packrowscolumns}]

This is mostly a helper for low level mechanisms like bitmap graphics. In that case
the entries in the passed table are rows.

\startbuffer
\startluacode
context.type (
    string.packrowscolumns {
        { 76, 85, 65 },
        { 84, 69, 88 },
    }
)
\stopluacode
\stopbuffer

\typebuffer

We get a string with bytes: \inlinebuffer. Here the horizontal resolution is 3
and the vertical resolution is 2 and of course in practice a 8-bit bitmap is not that
readable.

\stopsummary

\startsummary[title={len}]

Instead of the \type {#} operators you can use the \type {len} function, for
instance when a hash is interpreted in a special way.

\ShowLuaExampleZero {string} {len} {"In a TeX source a hash can be a bit of a problem!"}

\stopsummary

\startsummary[title={itself valid}]

In case you need a a function that returns its argument, you can use:

\ShowLuaExampleZero {string} {itself} {"indeed"}

The \type {valid} checks if we have a string and if it is not empty. The default value
is returned if these criteria are not met.

\ShowLuaExampleZero {string} {valid} {"whatever"}
\ShowLuaExampleZero {string} {valid} {"","default"}

\stopsummary

\startsummary[title=quote]

When you output a value that might need to be read in again, strings need to be
between quotes and non integer numbers need to be output as exact as possible.
The \type {format} function has \type {%q} for that. Here is that one as
function. Watch the hexadecimal double:

\ShowLuaExampleZero {string} {quote} {('That was "well" done!')}
\ShowLuaExampleZero {string} {quote} {(123)}
\ShowLuaExampleZero {string} {quote} {(123.456)}
\ShowLuaExampleZero {string} {quote} {(true)}

\stopsummary

\startsummary[title=optionalquoted]

This rather special function is used in command-line escaping. It is probably of
little use to the average user. So:

\startbuffer
\startluacode
local s = '"foo"bar \"and " whatever"'
context.formatted.type("%s : %s",s,string.optionalquoted(s))
\stopluacode
\stopbuffer

\typebuffer

escapes to:

\getbuffer

while:

\startbuffer
\startluacode
local s = 'foo"bar \"and " whatever'
context.formatted.type("%s : %s",s,string.optionalquoted(s))
\stopluacode
\stopbuffer

\typebuffer

becomes:

\getbuffer

\stopsummary

\startsummary[title={splitup}]

This splits a string in single byte characters. The pattern can be a string or
\LPEG\ specification.

\ShowLuaExampleZero {string} {splitup} {"a b c d"," "}
\ShowLuaExampleZero {string} {splitup} {"a,b,c,d",","}
\ShowLuaExampleZero {string} {splitup} {"a   b c  d",lpeg.patterns.whitespace^1}
\ShowLuaExampleZero {string} {splitup} {"a   b c  d",lpeg.patterns.whitespace^1}

\stopsummary

\startsummary[title={bytetable}]

This is yet another splitter, this time single byte characters become integers:

\ShowLuaExampleNine {string} {bytetable} {"just bytes"}

\stopsummary

\startsummary[title={linetable}]

When splitting a string into lines we look at \ASCII\ character 10 and 13:

\ShowLuaExampleNine {string} {linetable} {("a\013b\010c\013\010d")}

\startsummary[title={wordsplitter}]

This function returns a splitter function, so it's an indirect feature:

\starttyping
\startluacode
local split = string.wordsplitter(",")
local list  = split("words,of,wisdom")
\stopluacode
\stoptyping

\ShowLuaExampleNine {string} {wordsplitter} {(",")("words,of,wisdom")}
\ShowLuaExampleNine {string} {wordsplitter} {(",")(" words,   of,     wisdom ")}

\ShowLuaExampleNine {string} {wordsplitter} {(" ")("words of wisdom")}
\ShowLuaExampleNine {string} {wordsplitter} {(" ")(" words   of     wisdom ")}

\stopsummary

\startsummary[title={containsws}]

This is basically an option checker, where options are separated by spaces, like:

\ShowLuaExampleThree {string} {containsws} {"red green blue","green"}
\ShowLuaExampleThree {string} {containsws} {"high left bold","italic"}

\stopsummary

\startsummary[title={texnewlines replacenewlines}]

The \type {texnewlines} function normalizes newlines to those that \LUATEX\
expects (\ASCII\ 13 or \type {\r} in \CCODE\ or \LUA\ speak). It's not that
useful outside that context.

The \type {replacenewlines} normalizes newlines to the ones the operating system
expects. Normally in \CONTEXT\ you don't need to worry about these issues.

\stopsummary

\startsummary[title={unquoted escapedquotes unescapedquotes}]

Quotes are always a potential hazard. In general, all languages have special
characters which means that no matter what you prefer (some dislike \TEX's
backslashes but then like hashes and dots) you have to somehow escape them when
they have their normal meaning. Here is a helper. We show quite some examples so
that you can see the various catches.

\ShowLuaExampleNine {string} {unquoted} {("test")}
\ShowLuaExampleNine {string} {unquoted} {([["t\"est"]])}
\ShowLuaExampleNine {string} {unquoted} {([["t\"est"x]])}
\ShowLuaExampleNine {string} {unquoted} {("\'test\'")}
\ShowLuaExampleNine {string} {unquoted} {('"test"')}
\ShowLuaExampleNine {string} {unquoted} {('"test"')}
\ShowLuaExampleNine {string} {unquoted} {("'what'ever'")}
\ShowLuaExampleNine {string} {unquoted} {("'whatever'")}
\ShowLuaExampleNine {string} {unquoted} {('"what"ever"')}
\ShowLuaExampleNine {string} {unquoted} {('"whatever"')}

You can also use \type {string.unquote} which is just an alias. The other two
functions also deal with quotes; they complement each other.

\ShowLuaExampleNine {string} {escapedquotes}   {("test\"test")}
\ShowLuaExampleNine {string} {unescapedquotes} {("test\"test")}

\stopsummary

\startsummary[title={collapsespaces fullstrip}]

These two fit in the repertoire is space manipulators. Multiple spaces are seen
as one.

\ShowLuaExampleNine {string} {collapsespaces} {("a  b  c    d    ")}
\ShowLuaExampleNine {string} {fullstrip}      {(" test test ")}

\stopsummary

\startsummary[title={tformat}]

Because in a \TEX\ source a percent normally starts a comment, we provide a
variant where the \type {@} performs its function ion a pattern. Of course this
only makes sense when you run some \LUA\ from the \TEX\ end and pass an argument.

\ShowLuaExampleNine {string} {tformat} {("[%i]",123)}
\ShowLuaExampleNine {string} {tformat} {("[@i]",123)}

\stopsummary

% -- for now kind of private:
%
% \startsummary[title={f6}]
%     ...
% \stopsummary

% -- for now kind of private:
%
% local collect, flush = utilities.strings.newcollector()
%
% collect("first")
% collect("second %i",123)
% collect("third (%N,%N)",123.456,789)
%
% inspect(flush("\n"))

% -- for now kind of private:
%
% local hashes = utilities.strings.newrepeater("#")
%
% inspect(hashes[10])

% -- for now kind of private:
%
% inspect(utilities.strings.tabtospace("12\t34",2))
% inspect(utilities.strings.tabtospace("12\t34",4))
% inspect(utilities.strings.tabtospace("12\t34",6))

% -- private table
%
% inspect(striplinepatterns)

\startsummary[title={utflength utfvalue utfvaluetable utfcharactertable}]

We have an \UTF\ library but the string library also has some \UTF\ related
functions.

\ShowLuaExampleNine {string} {utflength}        {("TèX")}
\ShowLuaExampleNine {string} {utfvalue}         {("TèX")}
\ShowLuaExampleNine {string} {utfvaluetable}    {("TèX")}
\ShowLuaExampleNine {string} {utfcharactertable}{("TèX")}

\stopsummary

\startsummary[title={utfcharacter utftabletostring}]

The first one can handle a mix, the second just numbers:

\ShowLuaExampleNine {string} {utfcharacter}     {(232)}
\ShowLuaExampleNine {string} {utfcharacter}     {(84, 232, 88)}
\ShowLuaExampleNine {string} {utfcharacter}     {{ 84, 232, 88 }}
\ShowLuaExampleNine {string} {utftabletostring} {{ 84, 232, 88 }}

\stopsummary

\startsummary[title={utfpadd utfpadding}]

For \type {utfpadd} the default padding character is a space, but here we use a
dash to show the effect:

\ShowLuaExampleNine {string} {utfpadd} {("TèX",    20,"-")}
\ShowLuaExampleNine {string} {utfpadd} {("TèXTèX", 20,"-")}
\ShowLuaExampleNine {string} {utfpadd} {("TèX",   -20,"-")}
\ShowLuaExampleNine {string} {utfpadd} {("TèXTèX",-20,"-")}

\stopsummary

% -- used internally
%
% inspect("[TèX"    .. string.utfpadding("TèX",   20) .. "]")
% inspect("[TèXTèX" .. string.utfpadding("TèXTèX",20) .. "]")

\startsummary[title={toutf8 toutf16 toutf32}]

These three take a table and produce a byte string. We can't show the 16 and 32
bit variants so we just show a call:

\starttyping
\startluacode
local b = string.toutf8  { 84, 232, 88 }
local b = string.toutf16 { 84, 232, 88 }
local b = string.toutf32 { 84, 232, 88 }
\stopluacode
\stoptyping

The last two can also be called with a second argument:

\starttyping
\startluacode
local b = string.toutf16(t, true)
local b = string.toutf32(t, true)
\stopluacode
\stoptyping

in which case two or four zero characters will be added.

\stopsummary

\startsummary[title={tounicode16}]

A hexadecimal 16 bit \UNICODE\ string (as for instance needed in the backend code)
can be generated with:

\ShowLuaExampleNine {string} {tounicode16} {(232)}

\stopsummary

\stopsection

\startsection[title=\UTF]

We used to have the \type {slunicode} library available but as most of it is not
used and because it has a somewhat fuzzy state, we will no longer rely on it. In
fact we only used a few functions in the \type {utf} namespace so as \CONTEXT\
user you'd better stick to what is presented here. You don't have to worry how
they are implemented. Depending on the version of \LUATEX\ cq.\ \LUAMETATEX\ it
can be that a library, a native function, or \LPEG is used.

In \LUA\ 5.5 we have a \type {utf8} library with some helpers like \typ {char},
\typ {charpattern}, \typ {codes}, \typ {codepoint}, \typ {len} and \typ {offset}.
Here are a few examples:

\starttyping[option=LUA]
print(utf8.len("ãbcdêfgh"))
print(utf8.len("ãbcdêfgh",3))
print(utf8.len("ãbcdêfgh",3,4))
print(utf8.codepoint("ãbcdêfgh",3,5))
print(utf8.offset("ãbcdêfgh",3,4))
print(utf8.codepoint("ãbcdêfgh",
    utf8.offset("ãbcdêfgh",3,3),
    utf8.offset("ãbcdefgh",3,4)))
\stoptyping

We don't use these in \CONTEXT, as we had our own helpers around un the \type
{utf}, \typ {string}, \typ {table}, \typ {lpeg} and other namespaces already and
these perform quite well.

\startsummary[title={char byte}]

As \UTF\ is a multibyte encoding the term char in fact refers to a \LUA\
string of one upto four 8|-|bit characters.

\starttyping
local b = utf.byte("å")
local c = utf.char(0xE5)
\stoptyping

The number of places in \CONTEXT\ where do such conversion is not that large:
it happens mostly in tracing messages.

\starttyping
logs.report("panic","the character U+%05X is used",utf.byte("æ"))
\stoptyping

\ShowLuaExampleThree {utf} {byte} {"æ"}
\ShowLuaExampleThree {utf} {char} {0xE6}

\stopsummary

\startsummary[title={sub}]

If you need to take a slice of an \UTF\ encoded string the \type {sub} function
can come in handy. This function takes a string and a range defined by two
numbers. Negative numbers count from the end of the string.

\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",1,7}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",0,7}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",0,9}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",4}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",0}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",0,0}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",4,4}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",4,0}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",-3,0}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",0,-3}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",-5,-3}
\ShowLuaExampleThree {utf} {sub} {"123456àáâãäå",-3}

\stopsummary

\startsummary[title={len length chrlen}]

There are probably not that many people that can instantly see how many bytes the
string in the following example takes:

\starttyping
local l = utf.len("ÀÁÂÃÄÅàáâãäå")
\stoptyping

Programming languages use \ASCII\ mostly so there each characters takes one byte.
In \CJK\ scripts however, you end up with much longer sequences. If you ever did
some typesetting of such scripts you have noticed that the number of characters
on a page is less than in the case of a Latin script. As information is coded
in less characters, effectively the source of a Latin or \CJK\ document will not
differ that much.

\ShowLuaExampleThree {utf} {len} {"ÒÓÔÕÖòóôõö"}
\ShowLuaExampleThree {utf} {len} {"OOOOOooooo"}

The \type {length} function is just an equivalent. The \type {chrlen}
helper takes a \UNICODE\ slot:

\ShowLuaExampleThree {utf} {chrlen} {utf.byte("e")}
\ShowLuaExampleThree {utf} {chrlen} {utf.byte("ê")}

\stopsummary

\startsummary[title={count}]

These two exmaples demonstrate that this function returns the number of
occurrences:

\ShowLuaExampleThree {utf} {count} {"eêêêêe","e"}
\ShowLuaExampleThree {utf} {count} {"eêêêêe","ê"}

It actually checks for a pattern:

\ShowLuaExampleThree {utf} {count} {"eêêêêe","eê"}

\stopsummary

\startsummary[title={ustring xstring tocodes toentities}]

For tracing purposes we have these helpers:

\ShowLuaExampleThree {utf} {ustring}    {"ê"}
\ShowLuaExampleThree {utf} {xstring}    {"ê"}
\ShowLuaExampleThree {utf} {tocodes}    {"eêe"}
\ShowLuaExampleThree {utf} {toentities} {"eêe"}

\stopsummary

\startsummary[title={split splitlines totable}]

We have a few splitters. The \type {split} functions gives a table of characters,
optionally without spaces:

\ShowLuaExampleNine {utf} {split} {"e ê e"}
\ShowLuaExampleNine {utf} {split} {"e ê e",true} % ignores spaces

We can also split into lines:

\ShowLuaExampleNine {utf} {splitlines} {"eê\010e\010ê"}

This one also keeps the initial byte order marker:

\ShowLuaExampleNine {utf} {totable}    {"eêe"}        % keeps bom

\stopsummary

\startsummary[title={sub}]

Like \type {string.sub} we can slice of a substring:

\ShowLuaExampleThree {utf} {sub}    {"eêêe",2,3}

\stopsummary

\startsummary[title={toeight filetype}]

The first one returns a\ \UTF8\ string from a 8, 16 or 32 bit \UTF\ sequence. The
second one returns the type of \UTF:

\ShowLuaExampleThree {utf} {toeight}   {"eêêe"}
\ShowLuaExampleThree {utf} {filetype}  {"eêêe"}

\stopsummary

\startsummary[title={values characters}]

There are two iterators that deal with \UTF. In \LUATEX\ these are extensions to
the \type {string} library but for consistency we've move them to the \type {utf}
namespace.

The following function loops over the \UTF\ characters in a string and returns
the \UNICODE\ number in \type {u}:

\starttyping
for u in utf.values(str) do
    ... -- u is a number
end
\stoptyping

The next one returns a string \type {c} that has one or more characters as \UTF\
characters can have upto 4 bytes.

\starttyping
for c in utf.characters(str) do
    ... -- c is a string
end
\stoptyping

\stopsummary

\startsummary[title={ustring xstring tocodes}]

These functions are mostly useful for logging where we want to see the \UNICODE\
number.

\ShowLuaExampleThree {utf} {ustring} {0xE6}
\ShowLuaExampleThree {utf} {ustring} {"ù"}
\ShowLuaExampleThree {utf} {xstring} {0xE6}
\ShowLuaExampleThree {utf} {xstring} {"à"}
\ShowLuaExampleThree {utf} {tocodes} {"ùúü"}
\ShowLuaExampleThree {utf} {tocodes} {"àáä",""}
\ShowLuaExampleThree {utf} {tocodes} {"òóö","+"}

\stopsummary

\startsummary[title={split splitlines totable}]

The \type {split} function splits a sequence of \UTF\ characters into a table
which one character per slot. The \type {splitlines} does the same but each slot
has a line instead. The \type {totable} function is similar to \type {split}, but
the later strips an optionally present \UTF\ bom.

\ShowLuaExampleThree {utf} {split} {"òóö"}

\stopsummary

\startsummary[title={count}]

This function counts the number of times that a given substring occurs in a
string. The patterns can be a string or an \LPEG\ pattern.

\ShowLuaExampleThree {utf} {count} {"òóöòóöòóö","ö"}
\ShowLuaExampleThree {utf} {count} {"äáàa",lpeg.P("á") + lpeg.P("à")}

\stopsummary

\startsummary[title={remapper replacer substituter}]

With \type {remapper} you can create a remapping function that remaps a given
string using a (hash) table.

\starttyping
local remap = utf.remapper { a = 'd', b = "c", c = "b", d = "a" }

print(remap("abcd 1234 abcd"))
\stoptyping

A remapper checks each character against the given mapping table. Its cousin
\type {replacer} is more efficient and skips non matches. The \type {substituter}
function only does a quick check first and avoids building a string with no
replacements. That one is much faster when you expect not that many replacements.

The \type {replacer} and \type {substituter} functions take table as argument
and an indexed as well as hashed one are acceptable. In fact you can even do
things like this:

\starttyping
local rep = utf.replacer { [lpeg.patterns.digit] = "!" }
\stoptyping

\stopsummary

\startsummary[title={is_valid}]

This function returns false if the argument is no valid \UTF\ string. As \LUATEX\
is pretty strict with respect to the input, this function is only useful when
dealing with external files.

\starttyping
function checkfile(filename)
  local data = io.loaddata(filename)
  if data and data ~= "" and not utf.is_valid(data) then
    logs.report("error","file %q contains invalid utf",filename)
  end
end
\stoptyping

\stopsummary

% not that relevant:
%
% -- utf.filetype
% -- string.toutf

\stopsection

\startsection[title=Numbers and bits]

In the \type {number} namespace we collect some helpers that deal with numbers as
well as bits. Starting with \LUA\ 5.2 a library \type {bit32} is but the language
itself doesn't provide for them via operators: the library uses functions to
manipulate numbers upto 2\high{32}. In the latest \LUATEX\ you can use the new
bit related operators.

% For advanced bit manipulations you should use the \type {bit32} library, otherwise
% it's best to stick to the functions described here.
%
% \startsummary[title={hasbit setbit clearbit}]
%
% As bitsets are numbers you will also use numbers to qualify them. So, if you want to
% set bits 1, 4 and 8, you can to that using the following specification:
%
% \starttyping
% local b = 1 + 4 + 8 -- 0x1 + 0x4 + 0x8
% local b = 13        -- or 0xC
% \stoptyping
%
% However, changing one bit by adding a number to an existing doesn't work out that well
% if that number already has that bit set. Instead we use:
%
% \starttyping
% local b = number.setbit(b,0x4)
% \stoptyping
%
% In a similar fashion you can turn of a bit:
%
% \starttyping
% local b = number.clearbit(b,0x4)
% \stoptyping
%
% Testing for a bit(set) is done as follows:
%
% \starttyping
% local okay = number.hasbit(b,0x4)
% \stoptyping
%
% \stopsummary
%
% \startsummary[title={bit}]
%
% Where the previously mentioned helpers work with numbers representing one or more
% bits, it is sometimes handy to work with positions. The \type {bit} function
% returns the associated number value.
%
% \ShowLuaExampleThree {number} {bit} {5}
%
% \stopsummary

\startsummary[title={tobitstring}]

There is no format option to go from number to bits in terms of zeros and ones so
we provide a helper: \type {tobitsting}.

\ShowLuaExampleThree {number} {tobitstring} {2013}
\ShowLuaExampleThree {number} {tobitstring} {2013,3}
\ShowLuaExampleThree {number} {tobitstring} {2013,1}

\stopsummary

% \startsummary[title={bits}]
%
% If you ever want to convert a bitset into a table containing the set bits you can
% use this function.
%
% \ShowLuaExampleTwo {number} {bits} {11}
%
% \stopsummary
%
% \startsummary[title={toset}]
%
% A string or number can be split into digits with \type {toset}. Beware, this
% function does not return a function but multiple numbers
%
% \starttyping
% local a, b, c, d = number.toset("1001")
% \stoptyping
%
% The returned values are either numbers or \type {nil} when an valid digit is
% seen.
%
% \ShowLuaExampleSeven {number} {toset} {100101}
% \ShowLuaExampleSeven {number} {toset} {"100101"}
% \ShowLuaExampleSeven {number} {toset} {"21546"}
%
% \stopsummary

\startsummary[title={valid}]

This function can be used to check or convert a number, for instance in user
interfaces.

\ShowLuaExampleThree {number} {valid} {12}
\ShowLuaExampleThree {number} {valid} {"34"}
\ShowLuaExampleThree {number} {valid} {"ab",56}

\stopsummary

\stopsection

\startsection[title=\LPEG\ patterns]

For \LUATEX\ and \CONTEXT\ \MKIV\ the \type {lpeg} library came at the right
moment as we can use it in lots of places. An in|-|depth discussion makes no
sense as it's easier to look into \type {l-lpeg.lua}, so we stick to an overview.
\footnote {If you search the web for \type {lua lpeg} you will end up at the
official documentation and tutorial.} Most functions return an \type {lpeg}
object that can be used in a match. In time critical situations it's more
efficient to use the match on a predefined pattern that to create the pattern new
each time. Patterns are cached so there is no penalty in predefining a pattern.
So, in the following example, the \type {splitter} that splits at the asterisk
will only be created once.

\starttyping
local splitter_1 = lpeg.splitat("*")
local splitter_2 = lpeg.splitat("*")

local n, m = lpeg.match(splitter_1,"2*4")
local n, m = lpeg.match(splitter_2,"2*4")
\stoptyping

\startsummary[title={[lua] match print P R S V C Cc Cs ...}]

The \type {match} function does the real work. Its first argument is a \type
{lpeg} object that is created using the functions with the short uppercase names.

\starttyping
local P, R, C, Ct = lpeg.P, lpeg.R, lpeg.C, lpeg.Ct

local pattern = Ct((P("[") * C(R("az")^0) * P(']') + P(1))^0)

local words = lpeg.match(pattern,"a [first] and [second] word")
\stoptyping

In this example the words between square brackets are collected in a table. There
are lots of examples of \type {lpeg} in the \CONTEXT\ code base.

\stopsummary

\startsummary[title={anywhere}]

\starttyping
local p = anywhere(pattern)
\stoptyping

\ShowLuaExampleTwo {lpeg} {match} {lpeg.Ct((lpeg.anywhere("->")/"!")^0), "oeps->what->more"}

\stopsummary

\startsummary[title={splitter splitat firstofsplit secondofsplit}]

The \type {splitter} function returns a pattern where each match gets an action
applied. The action can be a function, table or string.

\starttyping
local p = splitter(pattern, action)
\stoptyping

The \type {splitat} function returns a pattern that will return the split off
parts. Unless the second argument is \type {true} the splitter keeps splitting

\starttyping
local p = splitat(separator,single)
\stoptyping

When you need to split off a prefix (for instance in a label) you can use:

\starttyping
local p = firstofsplit(separator)
local p = secondofsplit(separator)
\stoptyping

The first function returns the original when there is no match but the second
function returns \type {nil} instead.

\ShowLuaExampleTwo {lpeg} {match} {lpeg.Ct(lpeg.splitat("->",false)), "oeps->what->more"}
\ShowLuaExampleTwo {lpeg} {match} {lpeg.Ct(lpeg.splitat("->",false)), "oeps"}
\ShowLuaExampleTwo {lpeg} {match} {lpeg.Ct(lpeg.splitat("->",true)), "oeps->what->more"}
\ShowLuaExampleTwo {lpeg} {match} {lpeg.Ct(lpeg.splitat("->",true)), "oeps"}

\ShowLuaExampleThree {lpeg} {match} {lpeg.firstofsplit(":"), "before:after"}
\ShowLuaExampleThree {lpeg} {match} {lpeg.firstofsplit(":"), "whatever"}
\ShowLuaExampleThree {lpeg} {match} {lpeg.secondofsplit(":"), "before:after"}
\ShowLuaExampleThree {lpeg} {match} {lpeg.secondofsplit(":"), "whatever"}

\stopsummary

\startsummary[title={split checkedsplit}]

The next two functions have counterparts in the \type {string} namespace. They
return a table with the split parts. The second function omits empty parts.

\starttyping
local t = split       (separator,str)
local t = checkedsplit(separator,str)
\stoptyping

\ShowLuaExampleTwo {lpeg} {split}        {",","a,b,c"}
\ShowLuaExampleTwo {lpeg} {split}        {",",",a,,b,c,"}
\ShowLuaExampleTwo {lpeg} {checkedsplit} {",",",a,,b,c,"}

\stopsummary

\startsummary[title={stripper keeper replacer}]

These three functions return patterns that manipulate a string. The \type
{replacer} gets a mapping table passed.

\starttyping
local p = stripper(str or pattern)
local p = keeper  (str or pattern)
local p = replacer(mapping)
\stoptyping

\ShowLuaExampleThree {lpeg} {match} {lpeg.stripper(lpeg.R("az")), "[-a-b-c-d-]"}
\ShowLuaExampleThree {lpeg} {match} {lpeg.stripper("ab"), "[-a-b-c-d-]"}
\ShowLuaExampleThree {lpeg} {match} {lpeg.keeper(lpeg.R("az")), "[-a-b-c-d-]"}
\ShowLuaExampleThree {lpeg} {match} {lpeg.keeper("ab"), "[-a-b-c-d-]"}
\ShowLuaExampleThree {lpeg} {match} {lpeg.replacer{{"a","p"},{"b","q"}}, "[-a-b-c-d-]"}

\stopsummary

\startsummary[title={balancer}]

One of the nice things about \type {lpeg} is that it can handle all kind of
balanced input. So, a function is provided that returns a balancer pattern:

\starttyping
local p = balancer(left,right)
\stoptyping

\ShowLuaExampleTwo {lpeg} {match} {lpeg.Ct((lpeg.C(lpeg.balancer("{","}"))+1)^0),"{a} {b{c}}"}
\ShowLuaExampleTwo {lpeg} {match} {lpeg.Ct((lpeg.C(lpeg.balancer("((","]"))+1)^0),"((a] ((b((c]]"}

\stopsummary

\startsummary[title={counter}]

The \type {counter} function returns a function that returns the length of a
given string. The \type {count} function differs from its counterpart living in
the \type {string} namespace in that it deals with \UTF\ and accepts strings as
well as patterns.

\starttyping
local fnc = counter(lpeg.P("á") + lpeg.P("à"))
local len = fnc("äáàa")
\stoptyping

\stopsummary

\startsummary[title={UP US UR}]

In order to make working with \UTF-8 input somewhat more convenient a few helpers
are provided.

\starttyping
local p = lpeg.UP(utfstring)
local p = lpeg.US(utfstring)
local p = lpeg.UR(utfpair)
local p = lpeg.UR(first,last)
\stoptyping

\ShowLuaExampleThree {utf} {count} {"äáàa",lpeg.UP("áà")}
\ShowLuaExampleThree {utf} {count} {"äáàa",lpeg.US("àá")}
\ShowLuaExampleThree {utf} {count} {"äáàa",lpeg.UR("aá")}
\ShowLuaExampleThree {utf} {count} {"äáàa",lpeg.UR("àá")}
\ShowLuaExampleThree {utf} {count} {"äáàa",lpeg.UR(0x0000,0xFFFF)}

\stopsummary

\startsummary[title={patterns}]

The following patterns are available in the \type {patterns} table in the \type
{lpeg} namespace:

\startluacode
context.startalignment { "flushleft" }
local done = false
for k, v in table.sortedpairs(lpeg.patterns) do
    if done then
        context.space()
    else
        done = true
    end
    context.type(k)
end
context.stopalignment()
\stopluacode

There will probably be more of them in the future.

\stopsummary

\stopsection

\startsection[title=IO]

The \type {io} library is extended with a couple of functions as well and
variables but first we mention a few predefined functions.

\startsummary[title={[lua] open popen...}]

The IO library deals with in- and output from the console and
files.

\starttyping
local f  = io.open(filename)
\stoptyping

When the call succeeds \type {f} is a file object. You close this file
with:

\starttyping
f:close()
\stoptyping

Reading from a file is done with \type {f:read(...)} and writing to a file with
\type {f:write(...)}. In order to write to a file, when opening a second argument
has to be given, often \type {wb} for writing (binary) data. Although there are
more efficient ways, you can use the \type {f:lines()} iterator to process a file
line by line.

You can open a process with \type {io.popen} but dealing with this one depends a
bit on the operating system.

\stopsummary

\startsummary[title={fileseparator pathseparator}]

The value of the following two strings depends on the operating system that is
used.

\starttyping
io.fileseparator
io.pathseparator
\stoptyping

\ShowLuaExampleFive {io} {fileseparator}
\ShowLuaExampleFive {io} {pathseparator}

\stopsummary

\startsummary[title={loaddata savedata}]

These two functions save you some programming. The first function loads a whole
file in a string. By default the file is loaded in binary mode, but when the
second argument is \type {true}, some interpretation takes place (for instance
line endings). In practice the second argument can best be left alone.

\starttyping
io.loaddata(filename,textmode)
\stoptyping

Saving the data is done with:

\starttyping
io.savedata(filename,str)
io.savedata(filename,tab,joiner)
\stoptyping

When a table is given, you can optionally specify a string that
ends up between the elements that make the table.

\stopsummary

\startsummary[title={exists size noflines}]

These three function don't need much comment.

\starttyping
io.exists(filename)
io.size(filename)
io.noflines(fileobject)
io.noflines(filename)
\stoptyping

\stopsummary

\startsummary[title={characters bytes readnumber readstring}]

When I wrote the icc profile loader, I needed a few helpers for reading strings
of a certain length and numbers of a given width. Both accept five values of
\type {n}: \type {-4}, \type {-2}, \type {1}, \type {2} and \type {4} where the
negative values swap the characters or bytes.

\starttyping
io.characters(f,n) --
io.bytes(f,n)
\stoptyping

The function \type {readnumber} accepts five sizes: \type {1}, \type {2}, \type
{4}, \type {8}, \type {12}. The string function handles any size and strings zero
bytes from the string.

\starttyping
io.readnumber(f,size)
io.readstring(f,size)
\stoptyping

Optionally you can give the position where the reading has to start:

\starttyping
io.readnumber(f,position,size)
io.readstring(f,position,size)
\stoptyping

\stopsummary

\startsummary[title={ask}]

In practice you will probably make your own variant of the following function,
but at least a template is there:

\starttyping
io.ask(question,default,options)
\stoptyping

For example:

\starttyping
local answer = io.ask("choice", "two", { "one", "two" })
\stoptyping

\stopsummary

\stopsection

\startsection[title=File]

The file library is one of the larger core libraries that comes with
\CONTEXT.

\startsummary[title={pathpart basename suffix suffixes nameonly}]

We start with a few filename manipulators.

\starttyping
local path   = file.pathpart(name,default) -- or file.dirname
local base   = file.basename(name)
local suffix = file.suffix(name,default)   -- or file.extname or file.suffixonly
local name   = file.nameonly(name)
\stoptyping

\ShowLuaExampleThree {file} {pathpart} {"/data/temp/whatever.cld"}
\ShowLuaExampleThree {file} {pathpart} {"c:/data/temp/whatever.cld"}
\ShowLuaExampleThree {file} {basename} {"/data/temp/whatever.cld"}
\ShowLuaExampleThree {file} {suffix}   {"c:/data/temp/whatever.cld"}
\ShowLuaExampleThree {file} {nameonly} {"/data/temp/whatever.cld"}

\ShowLuaExampleZero  {file} {suffixes} {"c:/data/temp/whatever.a.b"}

\stopsummary

\startsummary[title={addsuffix replacesuffix removesuffix}]

These functions are used quite often:

\starttyping
local filename = file.addsuffix(filename, suffix, criterium)
local filename = file.replacesuffix(filename, suffix)
local filename = file.removesuffix(filename)
\stoptyping

The first one adds a suffix unless one is present. When \type {criterium} is
\type {true} no checking is done and the suffix is always appended. The second
function replaces the current suffix or add one when there is none.

\ShowLuaExampleThree {file} {addsuffix} {"whatever","cld"}
\ShowLuaExampleThree {file} {addsuffix} {"whatever.tex","cld"}
\ShowLuaExampleThree {file} {addsuffix} {"whatever.tex","cld",true}

\ShowLuaExampleThree {file} {replacesuffix} {"whatever","cld"}
\ShowLuaExampleThree {file} {replacesuffix} {"whatever.tex","cld"}

\ShowLuaExampleThree {file} {removesuffix} {"whatever.tex"}

\stopsummary

\startsummary[title={size}]

As the name suggests, this function returns the file size (here from a file that
we create runtime elsewhere):

\ShowLuaExampleThree {file} {size} {"temp.log"}

\stopsummary

\startsummary[title={iswritable isreadable}]

These two test the nature of a file:

\starttyping
file.iswritable(name) -- file.is_writable lfs.writablefile
file.isreadable(name) -- file.is_readable lfs.readablefile
\stoptyping

\stopsummary

\startsummary[title={splitname nametotable join collapsepath}]

Instead of splitting off individual components you can get them all in one go:

\starttyping
local drive, path, base, suffix = file.splitname(name)
\stoptyping

The \type {drive} variable is empty on operating systems other than \MSWINDOWS.
Such components are joined with the function:

\starttyping
file.join(...)
\stoptyping

The given snippets are joined using the \type {/} as this is rather platform
independent. Some checking takes place in order to make sure that nu funny paths
result from this. There is also \type {collapsepath} that does some cleanup on a
path with relative components, like \type {..}.

%ShowLuaExampleSix   {file} {splitname}    {"a:/b/c/d.e"}
\ShowLuaExampleZero  {file} {splitname}    {"a:/b/c/d.e"}
\ShowLuaExampleNine  {file} {nametotable}  {(resolvers.findfile("context.mkxl"))}
\ShowLuaExampleThree {file} {join}         {"a","b","c.d"}
\ShowLuaExampleThree {file} {collapsepath} {"a/b/../c.d"}
\ShowLuaExampleThree {file} {collapsepath} {"a/b/../c.d",true}

\stopsummary

\startsummary[title={splitpath joinpath splitbase}]

By default splitting a execution path specification is done using the operating
system dependent separator, but you can force one as well:

\starttyping
file.splitpath(str,separator)
\stoptyping

The reverse operation is done with:

\starttyping
file.joinpath(tab,separator)
\stoptyping

Beware: in the following examples the separator is system dependent so
the outcome depends on the platform you run on.

\ShowLuaExampleTwo   {file} {splitpath} {"a:b:c"}
\ShowLuaExampleTwo   {file} {splitpath} {"a;b;c"}
\ShowLuaExampleThree {file} {joinpath}  {{"a","b","c"}}

As bonus we have:

\ShowLuaExampleThree {file} {splitbase}  {"temp.log"}
\ShowLuaExampleThree {file} {splitbase}  {"foo/temp.log"}

\stopsummary

\startsummary[title={collapsepath expandname reslash}]

Sometimes you need the full path, without embedded \type {.} or \type {..}
placeholders:

\ShowLuaExampleThree {file} {collapsepath} {"a/b/c/d/../e.tex"}

The next variant assembles a name from the current path and the given filename:

\ShowLuaExampleThree {file} {expandname} {"e.tex"}

You can normalize slashes to forward slashes, which are also understood
in \MSWINDOWS\ when used in function calls:

\ShowLuaExampleThree {file} {reslash} {"a\\b/c.tex"}

\stopsummary

\startsummary[title={robustname}]

In workflows filenames with special characters can be a pain so the following
function replaces characters other than letters, digits, periods, slashes and
hyphens by hyphens.

\starttyping
file.robustname(str,strict)
\stoptyping

\ShowLuaExampleThree {file} {robustname} {"We don't like this!"}
\ShowLuaExampleThree {file} {robustname} {"We don't like this!",true}

\stopsummary

\startsummary[title={savedata readdata}]

A fast way to save data to a file and read if back is to use the following,
where the load function is equivalent to \type {io.loaddata}.

\ShowLuaExampleThree {file} {savedata} {"temp.log","demo line"}
\ShowLuaExampleThree {file} {readdata} {"temp.log"}

The boolean result permits checking:

\starttyping
if file.savedata("temp.log","demo line") then
    local data = file.readdata("temp.log"))
else
    print("something went wrong")
end
\stoptyping

\stopsummary

\startsummary[title={copy}]

There is not much to comment on this one:

\starttyping
file.copy(oldname,newname)
\stoptyping

\stopsummary

\startsummary[title={is_qualified_path is_rootbased_path}]

A qualified path has at least one directory component while a rootbased path is
anchored to the root of a filesystem or drive.

\starttyping
file.is_qualified_path(filename)
file.is_rootbased_path(filename)
\stoptyping

\ShowLuaExampleThree {file} {is_qualified_path} {"a"}
\ShowLuaExampleThree {file} {is_qualified_path} {"a/b"}
\ShowLuaExampleThree {file} {is_rootbased_path} {"a/b"}
\ShowLuaExampleThree {file} {is_rootbased_path} {"/a/b"}

\stopsummary

% prevents backtracking, subscure, private and it might move elswhere:
%
% file.withinbase("temp.log")
% file.withinbase("foo/temp.log")
% file.withinbase("../foo/temp.log")
% file.withinbase("./foo/temp.log")

\startsummary[title={checksum savechecksum loadchecksum}]

The next two functions manage a \type {MD5} checksum for a given file. Although
there is always a change that two different files give the same checksum, in
practice that seldom happens.

\ShowLuaExampleThree {file} {savechecksum} {"temp.log"}
\ShowLuaExampleThree {file} {loadchecksum} {"temp.log"}

The \type {checksum} function returns the checksum:

\ShowLuaExampleThree {file} {checksum} {"temp.log"}
%ShowLuaExampleThree {file} {checksum} {resolvers.findfile("context.mkxl")}

\stopsummary

\startsummary[title={syncmtimes needsupdating}]

Instead of comparing checksums you can also check timestamps, for instance when
one file is derived from another.

\starttyping
if file.needsupdating(oldname,newname,threshold) then -- default threshold 1
    -- create file "newname" from "oldname"
    file.syncmtimes(oldname,newname)
end
\stoptyping

\stopsummary

\stopsection

\startsection[title=Dir]

The \type {dir} library uses functions of the \type {lfs} library that is linked
into \LUATEX.

\startsummary[title={current}]

This returns the current directory:

\starttyping
dir.current()
\stoptyping

\stopsummary

\startsummary[title={glob globpattern globfiles globdirs}]

% not yet documented: dir.collectpattern(path,patt,recurse,result) -- collects tree

The \type {glob} function collects files with names that match a given pattern.
The pattern can have wildcards: \type {*} (oen of more characters), \type {?}
(one character) or \type {**} (one or more directories). You can pass the
function a string or a table with strings. Optionally a second argument can be
passed, a table that the results are appended to.

\starttyping
local files = dir.glob(pattern,target)
local files = dir.glob({pattern,...},target)
\stoptyping

The target is optional and often you end up with simple calls like:

\starttyping
local files = dir.glob("*.tex")
\stoptyping

There is a more extensive version where you start at a path, and applies an
action to each file that matches the pattern. You can either or not force
recursion.

\starttyping
dir.globpattern(path,patt,recurse,action)
\stoptyping

The \type {globfiles} function collects matches in a table that is returned at
the end. You can pass an existing table as last argument. The first argument is
the starting path, the second arguments controls analyzing directories and the
third argument has to be a function that gets a name passed and is supposed to
return \type {true} or \type {false}. This function determines what gets
collected.

\starttyping
dir.globfiles(path,recurse,func,files)
\stoptyping

The \type {globdirs} function collects directories below the given directory and
also takes up to four arguments.

\stopsummary

\startsummary[title={push pop}]

You can (temporary) jump to another directory, assuming that this is permitted.
After you've done the work there you can return to where you came from.

\starttyping
if dir.push("foo") then
    -- whatever you need to do
    dir.pop()
end
\stoptyping

\startsummary[title={ls}]

The list files function takes one argument, when no argument is given an empty
string is assumed. The return value is a string where every match is on its own
line.

\starttyping
local s = dir.ls()
local s = dir.ls("*.tex")
\stoptyping

\stopsummary

\startsummary[title={makedirs}]

With \type {makedirs} (or its alias \type {mkdirs}) you can create the given
directory. If more than one name is given they are concatinated.

\starttyping
dir.makedirs(name,...)
\stoptyping

\stopsummary

\startsummary[title={expandname}]

This function tries to resolve the given path, including relative paths.

\starttyping
local fullname = dir.expandname(name)
\stoptyping

\ShowLuaExampleThree {dir} {expandname} {"."}

\stopsummary

\startsummary[title={found}]

This function takes a list of potential path names and returns the first one that
exists like:

\starttyping
local found = dir.found("here","there")
\stoptyping

\stopsummary

\startsummary[title={expandlink}]

This function is a bit special and depends on the operating system. Users
normally don't need to call this one.

\starttyping
local realname = dir.expandlink(usedname)
\stoptyping

\stopsummary

\startsummary[title={collectpattern}]

We don't use this one often, it returns a table of paths where we can find files
that match the given pattern.

\starttyping
local locations = dir.collectpattern("t:/sources","*.lua",true) -- result
\stoptyping

\stopsummary

\stopsection

\startsection[title=lfs]

In \LUATEX\ we have the so called \quotation {Lua File System} library but in
\LUAMETATEX\ that has been replaced. However, we still have that namespace,
mainly because there can be references to its functions all over the place. You
only need these when the \type {file} and \type {dir} collections lack similar
helpers. \footnote {Right from the start functions that deal with files and other
operating system specifics have been written in a way so that \MSWINDOWS\ 16 bit
\UNICODE\ deals with them properly. Users can use \UTF8\ as everywhere in the
engine and translation back and forth happen automatically. This is different
from \LUATEX, although that might of course change at some point.} This library
is discussed in the \LUAMETATEX\ manual.

% attributes modification
% chdir mkdir mkdirs mkdirs rmdir
% currentdir
% walkdir dir
% isdir isfile isfound
% isreadabledir isreadablefile iswriteabledir iswriteablefile
% setexecutable
% link readlink symlink symlinktarget
% touch

\stopsection

\startsection[title=URL]

\startsummary[title={split hashed construct}]

This is a specialized library. You can split an \type {url} into its components.
An \URL\ is constructed like this:

\starttyping
foo://example.com:2010/alpha/beta?gamma=delta#epsilon
\stoptyping

\starttabulate[|T|T|]
\NC scheme    \NC foo://           \NC \NR
\NC authority \NC example.com:2010 \NC \NR
\NC path      \NC /alpha/beta      \NC \NR
\NC query     \NC gamma=delta      \NC \NR
\NC fragment  \NC epsilon          \NC \NR
\stoptabulate

A string is split into a hash table with these keys using the following function:

\starttyping
url.hashed(str)
\stoptyping

or in strings with:

\starttyping
url.split(str)
\stoptyping

The hash variant is more tolerant than the split. In the hash
there is also a key \type {original} that holds the original \URL\
and and the boolean \type {noscheme} indicates if there is a
scheme at all.

The reverse operation is done with:

\starttyping
url.construct(hash)
\stoptyping

\startasciimode
\ShowLuaExampleTwo {url} {hashed} {"foo://example.com:2010/alpha/beta?gamma=delta#epsilon"}
\ShowLuaExampleTwo {url} {hashed} {"alpha/beta"}
\ShowLuaExampleTwo {url} {split}  {"foo://example.com:2010/alpha/beta?gamma=delta#epsilon"}
\ShowLuaExampleTwo {url} {split}  {"alpha/beta"}
\stopasciimode

\stopsummary

\startsummary[title={hasscheme addscheme filename query}]

There are a couple of helpers and their names speaks for themselves:

\starttyping
url.hasscheme(str)
url.addscheme(str,scheme)
url.filename(filename)
url.query(str)
\stoptyping

\ShowLuaExampleThree {url} {hasscheme} {"http://www.pragma-ade.com/cow.png"}
\ShowLuaExampleThree {url} {hasscheme} {"www.pragma-ade.com/cow.png"}
\ShowLuaExampleThree {url} {addscheme} {"www.pragma-ade.com/cow.png","http://"}
\ShowLuaExampleThree {url} {addscheme} {"www.pragma-ade.com/cow.png"}
\ShowLuaExampleThree {url} {filename}  {"http://www.pragma-ade.com/cow.png"}
\ShowLuaExampleTwo   {url} {query}     {"a=b&c=d"}

\stopsection

\startsection[title=OS]

\startsummary[title={[lua luatex] env setenv getenv}]

In \CONTEXT\ normally you will use the resolver functions to deal with the
environment and files. However, a more low level interface is still available.
You can query and set environment variables with two functions. In addition there
is the \type {env} table as interface to the environment. This threesome replaces
the built in functions.

\starttyping
os.setenv(key,value)
os.getenv(key)
os.env[key]
\stoptyping

\stopsummary

\startsummary[title={[lua] execute}]

There are several functions for running programs. One comes directly from \LUA,
the otheres come with \LUATEX. All of them are are overloaded in \CONTEXT\ in
order to get more control.

\starttyping
os.execute(...)
\stoptyping

\stopsummary

\stopsummary

% These are gone in luametatex so they're also gone here.
%
% \startsummary[title={[luatex] spawn exec}]
%
% Two other runners are:
%
% \starttyping
% os.spawn(...)
% os.exec (...)
% \stoptyping
%
% The \type {exec} variant will transfer control from the current process to the
% new one and not return to the current job. There is a more detailed explanation
% in the \LUATEX\ manual.
%
% \stopsummary

\startsummary[title={resultof launch pipeto}]

The following function runs the command and returns the result as string.
Multiple lines are combined.

\starttyping
os.resultof(command)
\stoptyping

The next one launches a file assuming that the operating system knows
what application to use.

\starttyping
os.launch(filename)
\stoptyping

You can open a pipe with:

\starttyping
local handle = os.pipeto(command)
\stoptyping

\stopsummary

\startsummary[title={type name platform libsuffix binsuffix binsuffixes}]

There are a couple of strings that reflect the current machinery: \type {type}
returns either \type {windows} or \type {unix}. The variable \type {name} is more
detailed: \type {windows}, \type {msdos}, \type {linux}, \type {macosx}, etc. If
you also want the architecture you can consult \type {platform}.

\starttyping
local t = os.type
local n = os.name
local p = os.platform
\stoptyping

These three variables as well as the next two are used internally and normally
they are not needed in your applications as most functions that matter are aware
of what platform specific things they have to deal with.

\starttyping
local s = os.libsuffix
local b = os.binsuffix
\stoptyping

These are string, not functions.

\ShowLuaExampleFive {os} {type}
\ShowLuaExampleFive {os} {name}
\ShowLuaExampleFive {os} {platform}
\ShowLuaExampleFive {os} {libsuffix}
\ShowLuaExampleFive {os} {binsuffix}

The \type {binsuffixes} array contains a list of suffixes specific for the
operating system in use.

\stopsummary

\startsummary[title={[lua] time [lua] difftime}]

The built in time function returns a number. The accuracy is
implementation dependent and not that large.

\ShowLuaExampleThree {os} {time} {}

The \type {difftime} function returns the difference (a double) between two given
times (integers). It is kind of inaccurate so we don't use this in \CONTEXT.

\stopsummary

\startsummary[title={[lua] sleep}]

You can freeze the program by passing a number, but real small values can be
unreliable as this also depends on the operating system and workload.

\stopsummary

\startsummary[title={[lua] date}]

This command returns a formatted date. You can pass a template; consult the
official \LUA\ manual for the possibilities.

\ShowLuaExampleThree {os} {date} {}

\stopsummary

\startsummary[title={[lua] remove [lua] rename [lua] tmpname}]

The names tell what these commands do. Of course you have to be careful in
using them:

\starttyping
if os.remove(filename) then
    -- success
end
if os.rename(oldname,newname) then
    -- success
end
\stoptyping

The \type {os.tmpname} function returns a temporary filename but we don't
rely on that in \CONTEXT. Use it at your own risk.

\stopsummary

\startsummary[title={[lua] clock}]

This function returns the time elapsed since we started in milliseconds. The
accuracy is okay for most case but next we will discuss a more precise variant.

\ShowLuaExampleThree {os} {clock} {}

\stopsummary


\startsummary[title={[luatex] times gettimeofday}]

Although \LUA\ has a built in type {os.time} function, we normally will use the
one provided by \LUATEX\ as it is more precise:

\starttyping
os.gettimeofday()
\stoptyping

% % This one is gone in luametatex:
%
% There is also a more extensive variant:
%
% \starttyping
% os.times()
% \stoptyping
%
% This one is platform dependent and returns a table with \type {utime} (use time),
% \type {stime} (system time), \type {cutime} (children user time), and \type
% {cstime} (children system time).

\ShowLuaExampleThree {os} {gettimeofday} {}
%ShowLuaExampleTwo   {os} {times}        {}

\stopsummary

\startsummary[title={runtime}]

More interesting is:

\starttyping
os.runtime()
\stoptyping

which returns the time spent in the application so far.

\ShowLuaExampleThree {os} {runtime} {}

\stopsummary

\startsummary[title={timezone}]

Sometimes you need to add the timezone to a verbose time and the following
function does that for you.

\starttyping
os.timezone(delta)
\stoptyping

\ShowLuaExampleThree {os} {timezone} {}
\ShowLuaExampleThree {os} {timezone} {1}
\ShowLuaExampleThree {os} {timezone} {-1}

\stopsummary

\startsummary[title={startuptime localtime fulltime now}]

Here are a few more time functions that were introduced in \LUAMETATEX:

\ShowLuaExampleThree {os} {startuptime} {}
\ShowLuaExampleThree {os} {localtime}   {}
\ShowLuaExampleThree {os} {fulltime}    {}
\ShowLuaExampleThree {os} {now}         {}

\stopsummary

\startsummary[title={today}]

Here is a function that returns a table with various date and time related values
as of today:

\ShowLuaExampleNine {os} {today} {()}

\stopsummary

\startsummary[title={nofdays isleapyear weekday}]

Here we need to pass a year:

\ShowLuaExampleThree {os} {nofdays}    {2021}
\ShowLuaExampleThree {os} {isleapyear} {2020}

Here we pass a day, month and year (in that order):

\ShowLuaExampleThree {os} {weekday} {18,11,2025}

\stopsummary

\startsummary[title={validdata}]

This is just a checker:

\starttyping
local year, month, day = os.validdate(year,month,day)
\stoptyping

\stopsummary

\startsummary[title={converttime}]

This is our default time converter which returns a optional second default value
when zero or an invalid time number is passed.

\ShowLuaExampleThree {os} {converttime} {os.time()}

\stopsummary

\startsummary[title={uuid}]

A version 4 UUID can be generated with:

\starttyping
os.uuid()
\stoptyping

The generator is good enough for our purpose.

\ShowLuaExampleThree {os} {uuid} {}

\stopsummary

\startsummary[title={self*}]

\ShowLuaExampleNine {os} {selfbin}  {}
\ShowLuaExampleNine {os} {selfcore} {}
\ShowLuaExampleNine {os} {selfdir}  {}
\ShowLuaExampleNine {os} {selflink} {}
\ShowLuaExampleNine {os} {selfname} {}
\ShowLuaExampleNine {os} {selfpath} {}

The \type {os.selfarg} table holds all the command line arguments as passed to
the runner or engine. The zero entry has the program name. The \type {os.bits}
variable has the value 32 (unlikely) or 64 (likely).

\stopsummary

% undocumented as not to be used:
%
% os.getcodepage() % rather useless as we assume utf
% os.setlocale()   % a dummy

\startsummary[title={enableansi}]

This is a rather specific command: it checks if a console supports \ANSI\ escape
sequences and enables that when possible. Here is an example of usage. It is rather
\CONTEXT\ specific:

\starttyping
if os.enableansi() then
    logs.setformatters("ansi")
else
    logs.setformatters("ansilog")
end

logs.report("system","OEPS") -- now system comes out in green

logs.setformatters()

logs.report("system","OEPS") -- we're back in black and white mode
\stoptyping

In \CONTEXT\ you can pass \type {--ansi} to the \type {mtxrun} and \type
{context} runners, so normally users never have to deal with this themselves.

\stopsummary

\startsummary[title={where}]

This function returns the location where a program resides. You can also \type
{which} which is the command on \UNIX.

\ShowLuaExampleNine {os} {where} {("luametatex")}

\stopsummary

\startsummary[title={uname}]

Here we get some information about the running system, for as far as known:

\ShowLuaExampleNine {os} {uname} {()}

\stopsummary

\startsummary[title={getunamefields getnamevalues gettypevalues}]

As with many \LUAMETATEX\ features that have a \LUA\ interface, we can get some
information about the to-be-expected keys and/or values:

\ShowLuaExampleNine {os} {getunamefields} {()}
\ShowLuaExampleNine {os} {getnamevalues}  {()}
\ShowLuaExampleNine {os} {gettypevalues}  {()}

\stopsummary

\startsummary[title={[lua] exit setexitcode}]

You can exit the running program. Setting the exit code is mostly relevant from
within a \TEX\ run. A set exit code wins over a passed one, so in the next case
the \type {123} is used instead of \type {456}:

\starttyping
os.setexitcode(123)
os.exit()
os.exit(456)
\stoptyping

The set exit code is not used when the program exits otherwise. You can actually
change that with \typ {lua.setexitcode} which has a companion \typ
{lua.getexitcode}.

\stopsummary

% This one will not be mentioned:
%
% print(os.newline)

% This one is meant for signal (context clock):
%
% os.serialwrite(...)

\stopsection

\stopchapter

\stopcomponent
