% language=us runpath=texruns:manuals/lowlevel

\environment lowlevel-style

\startdocument
  [title=registers,
   color=darkmagenta]

\startsectionlevel[title=Preamble]

Registers are sets of variables that are accessed by index and a such resemble
registers in a processing unit. You can store a quantity in a register, retrieve
it, and also manipulate it.

There is hardly any need to use them in \CONTEXT\ so we keep it simple.

\stopsectionlevel

\startsectionlevel[title={\TEX\ primitives}]

There are several categories:

\startitemize
\startitem
    Integers (int): in order to be portable (at the time it surfaced) there are only
    integers and no floats. The only place where \TEX\ uses floats internally is
    when glue gets effective which happens in the backend.
\stopitem
\startitem
    Dimensions (dimen): internally these are just integers but when they are entered they
    are sliced into two parts so that we have a fractional part. The internal
    representation is called a scaled point.
\stopitem
\startitem
    Glue (skip): these are dimensions with a few additional properties: stretch and
    shrink. Being a compound entity they are stored differently and thereby a bit
    less efficient than numbers and dimensions.
\stopitem
\startitem
    Math glue (muskip): this is the same as glue but with a unit that adapts to
    the current math style properties. It's best to think about them as being
    relative measures.
\stopitem
\startitem
    Token lists (toks): these contain a list of tokens coming from the input
    or coming from a place where they already have been converted.
\stopitem
\stopitemize

The original \TEX\ engine had 256 entries per set. The first ten of each set are
normally reserved for scratch purposes: the even ones for local use, and the odd
ones for global usage. On top of that macro packages can reserve some for its own
use. It was quite easy to reach the maximum but there were tricks around that.
This limitation is no longer present in the variants in use today.

Let's set a few dimension registers:

\startbuffer[1]
\dimen 0 = 10 pt
\dimen2=10pt
\dimen4 10pt
\scratchdimen 10pt
\stopbuffer

\typebuffer[1][option=TEX]

We can serialize them with:

\startbuffer[2]
\the    \dimen0
\number \dimen2
\meaning\dimen4
\meaning\scratchdimen
\stopbuffer

\typebuffer[2][option=TEX]

The results of these operations are:

\startlines\tt
\getbuffer[1,2]
\stoplines

The last two is not really useful but it is what you see when tracing options are
set. Here \type {\scratchdimen} is a shortcut for a register. This is {\em not} a
macro but a defined register. The low level \type {\dimendef} is used for this
but in a macro package you should not use that one but the higher level \type
{\newdimen} macro that uses it.

\startbuffer[1]
\newdimen\MyDimenA
\def     \MyDimenB{\dimen999}
\dimendef\MyDimenC998
\stopbuffer

\typebuffer[1][option=TEX]

\startbuffer[2]
\meaning\MyDimenA
\meaning\MyDimenB
\meaning\MyDimenC
\stopbuffer

\typebuffer[2][option=TEX]

Watch the difference:

\pushoverloadmode
\startlines\tt
\getbuffer[1,2]
\stoplines
\popoverloadmode

The first definition uses a yet free register so you won't get a clash. The
second one is just a shortcut using a macro and the third one too but again
direct shortcut. Try to imagine how the second line gets interpreted:

\starttyping[option=TEX]
\MyDimenA10pt \MyDimenA10.5pt
\MyDimenB10pt \MyDimenB10.5pt
\MyDimenC10pt \MyDimenC10.5pt
\stoptyping

Also try to imagine what messing around with \type {\MyDimenC} will do when we
also have defined a few hundred extra dimensions with \type {\newdimen}.

In the case of dimensions the \type {\number} primitive will make the register
serialize as scaled points without unit \type {sp}.

Next we see some of the other registers being assigned:

\starttyping[option=TEX]
\count  0 = 100
\skip   0 = 10pt plus 3pt minus 2pt
\skip   0 = 10pt plus 1fill
\muskip 0 = 10mu plus 3mu minus 2mu
\muskip 0 = 10mu minus 1 fil
\toks   0 = {hundred}
\stoptyping

When a number is expected, you can use for instance this:

\starttyping[option=TEX]
\scratchcounter\scratchcounterone
\stoptyping

Here we use a few predefined scratch registers. You can also do this:

\starttyping[option=TEX]
\scratchcounter\numexpr\scratchcounterone+\scratchcountertwo\relax
\stoptyping

There are some quantities that also qualify as number:

\starttyping[option=TEX]
\chardef\MyChar=123 % refers to character 123 (if present)
\scratchcounter\MyChar
\stoptyping

In the past using \type {\chardef} was a way to get around the limited number of
registers, but it still had (in traditional \TEX) a limitation: you could not go
beyond 255. The \type {\mathchardef} could fo higher as it also encodes a family
number and class. This limitation has been lifted in \LUATEX.

A character itself can also be interpreted as number, in which case it has to be
prefixed with a reverse quote: \type {`}, so:

\startbuffer
\scratchcounter\numexpr`0+5\relax
\char\scratchcounter
\stopbuffer

\typebuffer[option=TEX]

produces \quotation {\inlinebuffer} because the \type {`0} expands into the
(\ASCII\ and \UTF8) slot {\tt \number`0} which represents the character zero. In
this case the next makes more sense:

\starttyping[option=TEX]
\char\numexpr`0+5\relax
\stoptyping

If you want to know more about all these quantities, \quotation {\TEX\ By Topic}
provides a good summary of what \TEX\ has to offer, and there is no need to repeat
it here.

\stopsectionlevel

\startsectionlevel[title={\ETEX\ primitives}]

Apart from the ability to use expressions, the contribution to registers that
\ETEX\ brought was that suddenly we could use upto 65K of them, which is more
than enough. The extra registers were not as efficient as the first 256 because
they were stored in the hash table, but that was not really a problem. In \OMEGA\
and later \LUATEX\ regular arrays were used, at the cost of more memory which in
the meantime has become cheap. As \CONTEXT\ moved to \ETEX\ rather early its
users never had to worry about it.

\stopsectionlevel

\startsectionlevel[title={\LUATEX\ primitives}]

The \LUATEX\ engine introduced attributes. These are numeric properties that are
bound to the nodes that are the result of typesetting operations. They are
basically like integer registers but when set their values get bound and when
unset they are kind of invisible.

\startitemize
\startitem
    Attribute (attribute): a numeric property that when set becomes part of the
    current attribute list that gets assigned to nodes.
\stopitem
\stopitemize

Attributes can be used to communicate properties to \LUA\ callbacks. There are
several functions available for setting them and querying them.

\starttyping[option=TEX]
\attribute999 = 123
\stoptyping

Using attributes this way is dangerous (of course I can only speak for \CONTEXT)
because an attribute value might trigger some action in a callback that gives
unwanted side effects. For convenience \CONTEXT\ provides:

\startbuffer
\newattribute\MyAttribute
\stopbuffer

\typebuffer[option=TEX] \getbuffer

Which currently defines \type {\MyAttribute} as {\tt \meaning\MyAttribute} and is
meant to be used as: \footnote {The low level \type {\attributedef} command is
rather useless in the perspective of \CONTEXT.}

\starttyping[option=TEX]
\attribute\MyAttribute = 123
\stoptyping

Just be aware that defining attributes can have an impact on performance. As you
cannot access them at the \TEX\ end you seldom need them. If you do you can
better use the proper more high level definers (not discussed here).

\stopsectionlevel

\startsectionlevel[title={\LUAMETATEX\ primitives}]

The fact that scanning stops at a non-number or \type {\relax} can be sort of
unpredictable which is why in \LUAMETATEX\ we also support the following variant:

\starttyping[option=TEX]
\scratchdimen\dimexpr  10pt + 3pt \relax
\scratchdimen\dimexpr {10pt + 3pt}
\stoptyping

At the cost of one more token braces can be used as boundaries instead of the
single \type {\relax} boundary.

An important property of registers is that they can be accessed by a number. This
has big consequences for the implementation: they are part of the big memory
store and consume dedicated ranges. If we had only named access \TEX's memory
layout could be a bit leaner. In principle we could make the number of registers
smaller because any limit on the amount at some point can be an obstacle. It is
for that reason that we also have name-only variants:

\starttyping[option=TEX]
\dimensiondef \MyDimenA   12pt
\integerdef   \MyIntegerA 12
\gluespecdef  \MyGlueA    12pt + 3pt minus 4pt
\mugluespecdef\MyMuA      12mu + 3mu minus 4mu
\stoptyping

These are as efficient but not accessible by number but they behave like
registers which means that you (can) use \type {\the}, \type {\advance}, \type
{\multiply} and \type {\divide} with them. \footnote {There are also the slightly
more efficient \type {\advanceby}, \type {\multiplyby} and \type {\divideby} that
don't check for the \type {by} keyword.} In case you wonder why there is no
alternative for \type {\toksdef}, there actually are multiple: they are called
macros.

{\em todo: expressions}

\stopsectionlevel

\startsectionlevel[title=Units]

The \LUAMETATEX\ engine supports the following units. The first batch is constant
with hard coded fine tuned values. The second set is related to the current font.
The last group is kind of special, the \type {es} is a replacement for the \type
{in} and has a little sister in \type {ts}. The \type {dk} is dedicated to the
master and makes a nice offset for so called \TEX\ pages that we use for demos.

\starttabulate[|Tl|Tr|l|]
\FL
\NC pt \NC \thewithoutunit\dimexpr1pt \NC point \NC \NR
\NC bp \NC \thewithoutunit\dimexpr1bp \NC big point (aka postscript point) \NC \NR
\NC in \NC \thewithoutunit\dimexpr1in \NC inch \NC \NR
\NC cm \NC \thewithoutunit\dimexpr1cm \NC centimeter \NC \NR
\NC mm \NC \thewithoutunit\dimexpr1mm \NC milimeter \NC \NR
\NC dd \NC \thewithoutunit\dimexpr1dd \NC didot \NC \NR
\NC cc \NC \thewithoutunit\dimexpr1cc \NC cicero \NC \NR
\NC pc \NC \thewithoutunit\dimexpr1pc \NC pica \NC \NR
\NC sp \NC \thewithoutunit\dimexpr1sp \NC scaled points \NC \NR
\NC px \NC \thewithoutunit\dimexpr1sp \NC pixel \NC \NR
\ML
\NC ex \NC \thewithoutunit\dimexpr1ex \NC ex height \NC \NR
\NC em \NC \thewithoutunit\dimexpr1em \NC em width \NC \NR
\NC mu \NC $\tt\thewithoutunit\onemuskip$ \NC math unit \NC \NR
\ML
\NC ts \NC \thewithoutunit\dimexpr1ts \NC tove \NC \NR
\NC es \NC \thewithoutunit\dimexpr1es \NC edith \NC \NR
\NC eu \NC \thewithoutunit\dimexpr1eu \NC european unit \NC \NR
\NC dk \NC \thewithoutunit\dimexpr1dk \NC knuth \NC \NR
\LL
\stoptabulate

The \type {fi[lll]} unit is not really a unit but a multiplier for infinite
stretch and shrink; original \TEX\ doesn't have the simple \type {fi}.

In addition to these we can have many more. In principle a user can define
additional ones but there's always a danger of clashing. For users we reserve the
units starting with an \type{u}. Here is how you define your own, we show three
variants:

\startbuffer
\newdimension \FooA   \FooA 1.23pt
\newdimen     \FooB   \FooB 12.3pt
\protected\def\FooC   {\the\dimexpr\FooA +\FooB\relax}

\pushoverloadmode % just in case
    \newuserunit\FooA ua
    \newuserunit\FooB ub
    \newuserunit\FooC uc
\popoverloadmode
\stopbuffer

\typebuffer[option=TEX] \getbuffer

And this is how they show up:

\startbuffer
\the\dimexpr 2 ua \relax\quad
\the\dimexpr 2 ub \relax\quad
\the\dimexpr 2 uc \relax
\stopbuffer

\getbuffer

with

\typebuffer[option=TEX]

The following additional units are predefined (reserved). The values are in
points and some depend on the current layout and document font.

\starttabulate[|T|Tr|l|]
\NC pi \NC \thewithoutunit\dimexpr 1pi \NC {\tt\pi} for Mikael        \NC \NR
\NC ft \NC \thewithoutunit\dimexpr 1ft \NC \type {foot} for Alan      \NC \NR
\NC fs \NC \thewithoutunit\dimexpr 1fs \NC (global body) font size    \NC \NR
\NC tw \NC \thewithoutunit\dimexpr 1tw \NC (layout) text width        \NC \NR
\NC th \NC \thewithoutunit\dimexpr 1th \NC (layout) text height       \NC \NR
\NC hs \NC \thewithoutunit\dimexpr 1hs \NC (current) hsize            \NC \NR
\NC vs \NC \thewithoutunit\dimexpr 1vs \NC (current) vsize            \NC \NR
\NC cd \NC \thewithoutunit\dimexpr 1cd \NC (when set) column distance \NC \NR % todo
\NC cw \NC \thewithoutunit\dimexpr 1cw \NC (when set) column width    \NC \NR % todo
\NC cx \NC \thewithoutunit\dimexpr 1cx \NC combination cell width     \NC \NR
\NC uu \NC \thewithoutunit\dimexpr 1uu \NC user unit (\METAFUN)       \NC \NR
\NC fw \NC \thewithoutunit\dimexpr 1fw \NC framed width               \NC \NR
\NC fh \NC \thewithoutunit\dimexpr 1fh \NC framed height              \NC \NR
\NC fo \NC \thewithoutunit\dimexpr 1fo \NC framed offset              \NC \NR
\NC lw \NC \thewithoutunit\dimexpr 1lw \NC line width                 \NC \NR
\NC sh \NC \thewithoutunit\dimexpr 1sh \NC strut height               \NC \NR
\NC sd \NC \thewithoutunit\dimexpr 1sd \NC strut depth                \NC \NR
\NC st \NC \thewithoutunit\dimexpr 1st \NC strut total                \NC \NR
\NC ch \NC \thewithoutunit\dimexpr 1ch \NC width of zero (css)        \NC \NR
\NC fa \NC \thewithoutunit\dimexpr 1fa \NC font ascender              \NC \NR
\NC fd \NC \thewithoutunit\dimexpr 1fd \NC font descender             \NC \NR
\NC fc \NC \thewithoutunit\dimexpr 1fc \NC font cap height            \NC \NR
\stoptabulate

Here is an example of usage:

\startbuffer
\startcombination[nx=4,ny=1]
    {\ruledhbox to 1cx{\strut one}}   {1}
    {\ruledhbox to 1cx{\strut two}}   {2}
    {\ruledhbox to 1cx{\strut three}} {3}
    {\ruledhbox to 1cx{\strut four}}  {4}
\stopcombination
\stopbuffer

\usemodule[system-units]

\startplacefigure[title=A map of available units,reference=fig:unitsmap]
    \framed[offset=1ex]{\showunitsmap}
\stopplacefigure

\typebuffer[option=TEX]

\startlinecorrection
\getbuffer
\stoplinecorrection

The \type {uu} can be set by users using the \type {\uunit} dimension variable.
The default valu sis 1cm. Its current value is also known at the \METAPOST\ end,
as demonstrated in \in {figure} [fig:userunits].

\startbuffer
\startcombination[nx=2,ny=1]
    \startcontent
        \uunit=1cm
        \framed[offset=1uu]
            \bgroup
                \startMPcode
                    fill fullcircle scaled 3uu withcolor "darkred"   ;
                    fill fullcircle scaled 2cm withcolor "darkgreen" ;
                \stopMPcode
            \egroup
    \stopcontent
    \startcaption
        \type {\uunit = 1cm}
    \stopcaption
    \startcontent
        \uunit=1cx
        \framed[offset=.1uu]
            \bgroup
                \startMPcode
                    fill fullcircle scaled .5uu withcolor "darkblue"   ;
                    fill fullcircle scaled  2cm withcolor "darkyellow" ;
                \stopMPcode
            \egroup
    \stopcontent
    \startcaption
        \type {\uunit = 1cx}
    \stopcaption
\stopcombination
\stopbuffer

% \framed[offset=1uu]
%     \bgroup
%         \startMPcode
%             save uu ; numeric uu ; uu := 1cm ;
%             fill fullcircle scaled 3uu withcolor "darkcyan"    ;
%             fill fullcircle scaled 2cm withcolor "darkmagenta" ;
%         \stopMPcode
%     \egroup

\typebuffer[option=TEX]

There is one catch here. If you use your own \type {uu} as numeric, you might
need this:

\starttyping[option=MP]
save uu ; numeric uu ; uu := 1cm ;
\stoptyping

That is: make sure the meaning is restored afterwards and explicitly declare the
variable. But this is good practice anyway when you generate multiple graphics
using the same \METAPOST\ instance.

\startplacefigure[reference=fig:userunits,title={Shared user units in \TEX\ and \METAFUN.}]
    \getbuffer
\stopplacefigure

There a few units not mentioned yet and those concern math, where we need to adapt to
the current style.

\protected\def\ShowMathUnits#1%
  {\im{#1x\mathord{+
     {\darkgray \vrule width 1pt depth 0pt height 1ex \relax}%
     {\darkred  \vrule width 1pt depth 0pt height 1ma \relax}%
     {\darkgreen\vrule width 1pt depth 0pt height 1mx \relax}%
     {\darkblue \vrule width 1pt depth 0pt height 1mq \relax}%
    x}}}

\startlinecorrection
\startcombination[nx=3,ny=1]
    {\scale[scale=6000]{\ShowMathUnits\textstyle}}         {text style}
    {\scale[scale=6000]{\ShowMathUnits\scriptstyle}}       {script style}
    {\scale[scale=6000]{\ShowMathUnits\scriptscriptstyle}} {script script style}
\stopcombination
\stoplinecorrection

The bars show \type {1ex}, \type {1ma} (axis), \type {1mx} (ex|-|height) and
\type {1mq} (em|-|width or quad). The last three adapt themselves to the style.
Often the \type {mx} makes more sense than \type {ex}.

\stopsectionlevel

\startsectionlevel[title=How many]

The original \TEX\ engine provides 256 registers for the various categories. The
\ETEX\ extension bumped that to 32768 while \LUATEX\ offers 65536. But how many
do we actually need? Just as we limit the number of languages and fonts to a
reasonable amount, we can as well do that for the registers. In \LUAMETATEX\ we
currently have this (the engine can have the 64K maxima but can also be compiled
with much less.

\starttabulate [|lT|lT|]
\NC toks      \NC \cldcontext{tex.magicconstants.max_toks_register_index}      \NC \NR
\NC box       \NC \cldcontext{tex.magicconstants.max_box_register_index}       \NC \NR
\NC integer   \NC \cldcontext{tex.magicconstants.max_integer_register_index}   \NC \NR
\NC float     \NC \cldcontext{tex.magicconstants.max_float_register_index}     \NC \NR
\NC dimension \NC \cldcontext{tex.magicconstants.max_dimension_register_index} \NC \NR
\NC attribute \NC \cldcontext{tex.magicconstants.max_attribute_register_index} \NC \NR
\NC glue      \NC \cldcontext{tex.magicconstants.max_glue_register_index}      \NC \NR
\NC muglue    \NC \cldcontext{tex.magicconstants.max_muglue_register_index}    \NC \NR
\stoptabulate

In \CONTEXT\ the maxima available to users are actually half that because we also
support local register allocation. Maxima of 8K or even 4K work fine but for
boxes one might like a bit more.

%D \begingroup
%D   \newlocaldimen\mydimena % \mydimena1\onepoint
%D   \newlocaldimen\mydimenb % \mydimenb2\onepoint
%D   \begingroup
%D     \newlocaldimen\mydimena % \mydimena3\onepoint
%D     \newlocaldimen\mydimenb % \mydimenb4\onepoint
%D     \newlocaldimen\mydimenc % \mydimenc5\onepoint
%D   \endgroup
%D \endgroup

%D \begingroup
%D   \setlocaldimen\mydimena 1\onepoint
%D   \setlocaldimen\mydimenb 2\onepoint
%D   \begingrou
%D     \setlocaldimen\mydimena 3\onepoint
%D     \setlocaldimen\mydimenb 4\onepoint
%D     \setlocaldimen\mydimenc 5\onepoint
%D   \endgroup
%D \endgroup
%D \stoptyping

\stopsectionlevel

\page

\stopdocument



