% language=us runpath=texruns:manuals/luametatex

\environment luametatex-style

\startdocument[title=Engines]

\startsubject[title=Contents]
    \setups[sectioncontents]
\stopsubject

\startsection[title={Introduction}]

There are good reasons why we initialized the \LUATEX\ and later \LUAMETATEX\
projects. Here I will go into some of them. It is just short wrap up of how it
started, how other engines influenced the process and how we see usage. There are
plenty of documents out there that go into more detail. The main objective of
this section is to put documentation into perspective.

\stopsection

\startsection[title={How it started}]

When we started with \CONTEXT, hardware was rather limited compared to what we
have today. A personal computer had some 640kB memory, possibly bumped to 1MB
with help from a memory extender. This put some restrictions to how macro
packages could be defined, also because that memory had to be shared with the
baseline operating system. However, over time, memory and runtime became less of
an issue and the \TEX\ engine could be configured to use whatever was available.
Extending the program other than increasing the available memory became more
feasible.

As with any program, there is always something to wish for which is why the
\ETEX\ variant came into view. Before those extensions could be used, \PDFTEX\
showed up. That variant simplified the \quote {\TEX\ plus separate backend
driver} model to a one-step process. Eventually \ETEX\ was merged into \PDFTEX,
and that became the de facto standard engine. There was never a follow up on
\ETEX, and more drastic deviations like \OMEGA\ were never ready for production.
At some point \XETEX\ came around but that was mostly a font specific extension.
We were kind of stuck with a wish-list that never would be fulfilled but we
occasionally pondered a follow up. We drafted an extended \ETEX\ proposal, played
with some features related to \PDF, improved a few things but that was it.

Having some experiences with \LUA\ as extension language in \SCITE, I wondered
what something like that would bring to \TEX\ and after discussing this with
Hartmut he made variant of \PDFTEX\ that has some basic interfaces: we could
access properties of registers and print something to \TEX\ as if it came from
file. As is common with some variant, a new name was coined and \LUATEX\ came
into existence. We're talking 2005.

Because Idris wanted to typeset high quality scholar manuscripts mixing Arabic
and Latin we discussed how to do that in \CONTEXT\ and his experiences with
\OMEGA\ were such that alternatives had to be considered: the Oriental \TEX\
project was started and \LUATEX\ was the starting point. Taco merged some parts
of \ALEPH\ (a somewhat stable variant of \OMEGA) into the code base and stepwise
some primitives were added. It was overall a rather large and serious project
that took a lot of our time. It was not commercially driven, mostly for \CONTEXT\
users and therefore also a lot of fun to do. As often with such projects, early
adapters keep things going.

It took a while before \LUATEX\ was stable in the sense that nothing more was
added. Because the engine was developed alongside what is called \CONTEXT\ \MKIV,
we could easily adapt both to each other. Even better: users could use both in
production. However, in order for other macro packages to use \LUATEX\ (per
request) it had to be frozen, and that happened around 2015, some 10 years after
we started. However, we were not done yet and in order not to violate this stability
principle the follow up was called \LUAMETATEX. Because it was a more drastic
extension project, and also a somewhat drastic separation of the code base from
the complex \LUATEX\ one, the related \CONTEXT\ code was also separated, this time
tagged \MKXL, or \LMTX\ when we talk about the combination. The project started
around 2019 and soon again entered a state of combined development and use in
production and most users switched to this variant.

There are more complete wrap ups of these developments and we systematically
reported on them in various documents that are available in the distribution
and|/|or published in user group journals.

\stopsection

\startsection[title={The engines}]

Of course all starts with original \TEX. We want to be compatible so we keep that
functionality. However, for practical reasons \LUAMETATEX\ omits two core
components. Font loading is not present in the frontend and there is no backend.
Both are supposed to be provided via \LUA\ plugins. This makes sense because in
the meantime font technologies have changed and keep changing and backend also
are a moving target. In \CONTEXT\ we already did all that in \LUA, so there was
no need to keep that font and \PDF\ generation code around in the engine. There
are a few more deviations, like dropping some system specific features (terminal
related) and in former times practical features like outer and long macros that
no longer made sense and complicated integrating new features unnecessarily.

As mentioned in the introduction, \PDFTEX\ is the basis for \LUATEX\ and \LUATEX\ is
where we started with \LUAMETATEX. If we compare \PDFTEX\ with traditional \TEX\
the main additions are:

\startitemize[packed]
\startitem
    There is an integrated \PDF\ backend that also supports for instance
    hyperlinks and various annotations.
\stopitem
\startitem
    Expansion of glyphs (aka hz) has been added to the engine and integrated in
    the par builder. The same is true for character protrusion (in the margin).
\stopitem
\startitem
    There is, to some extend, support for inter-character kerning.
\stopitem
\startitem
    There are some handy helpers, for instance for calculating hashes,
    randomization, etc.
\stopitem
\startitem
    There is an extension to injection between lines (adjust).
\stopitem
\startitem
    We have few more conditionals (like testing for a csname and absolute values).
\stopitem
\startitem
    A few helpers like \type {\quitvmode} (that we liked to have in \CONTEXT)
    were added.
\stopitem
\stopitemize

Because \PDFTEX\ was actively developed and maintained over many years,
extensions showed up stepwise, also depending on usage and needs. That is also
why the \ETEX\ extensions were included:

\startitemize[packed]
\startitem
    More that 256 registers, including marks.
\stopitem
\startitem
    Access to discarded material in the vertical splitting code.
\stopitem
\startitem
    Protection against expansion of macros (the \type {\protected} prefix).
\stopitem
\startitem
    A simple right to left typesetting mechanism.
\stopitem
\startitem
    Access to some states, a limited set of last nodes, etc.
\stopitem
\startitem
    There are some additional tracing features.
\stopitem
\startitem
    One can reprocess tokens and produce detokenized lists.
\stopitem
\stopitemize

In \LUATEX\ we also looked at what \OMEGA\ could bring:

\startitemize[packed]
\startitem
    More that 256 registers.
\stopitem
\startitem
    Multi-directional typesetting.
\stopitem
\startitem
    Local boxes (in lines).
\stopitem
\startitem
    Input processing.
\stopitem
\stopitemize

If we combine these lists, we see font expansion and protrusion coming back in
\LUAMETATEX. However, already in \LUATEX\ expansion and protrusion were dealt
with a bit differently and even more so in \LUAMETATEX, while protection in
\LUAMETATEX\ is implemented differently. We also kept injection of vertical
material but in \LUAMETATEX\ that done quite differently. Most of \ETEX\ is there
but not right to left typesetting and the register approach. Of course we kept
the additional conditionals but implemented them a bit different.

In \LUATEX\ we took the \OMEGA\ enlarged register approach and directional
typesetting although that has been stripped down and redone to right to left
only. Local boxes are there but redone in \LUAMETATEX. There was no need for
input processing because we have \LUA. In the end there is little that we kept
from the other engines which also means that one cannot take the manuals that
come with these engines and simply assume that it is there.

We should of course mention \METAPOST. That graphical subsystem was integrated in
\LUATEX\ and on the one hand stripped down (less backend) and on the other
extended (remove bottlenecks and add some functionality) in \LUAMETATEX. With
respect to \LUA\ we moved to more recent versions and dropped support for just in
time compilation. This also means that in \CONTEXT\ \MKXL\ we have more \METAFUN\
and therefore talk \LUAMETAFUN.

There is of course a lot in \LUATEX\ that can be found also in \LUAMETATEX\ but
the later one goes way beyond its predecessor. It actually provides what we always
wanted (as \CONTEXT\ developers) but never showed up. And this brings us to a next
topic.

\stopsection

\startsection[title=Reflections]

The previous section, somewhat derived from the \LUATEX\ manual, might suggest
that \LUATEX\ and therefore \LUAMETATEX\ provides most of what \PDFTEX, \ETEX,
\XETEX\ and \OMEGA\ provide but here I must disappoint the reader. So in addition
to or variation on the above here are some reflections.

We were quite involved in the early days of \PDFTEX\ development, so some
features of that program we kept in \LUATEX, like expansion, protrusion, basic
\PDF\ features like annotations, destinations, outlines and literals,
transformations, image inclusion, plus a few handy extensions like \type
{\vadjust pre}, \type {\insertht} and \type {\quitvmode} that were introduced for
\CONTEXT, as well as positioning that when brought into \PDFTEX\ made that we no
longer needed the indirect method using specials that we used (first with a post
processing script filtering specials and providing positional information, later
that became a the \type {dvipos} program). Experimental features (at that time
introduced for \CONTEXT) like snapping lines could make sense but were easier to
handle in \LUA\ so even those were dropped. We never used the features that were
introduced for other macro packages, like color stacks and (un)escaping because
we already did that otherwise. We also didn't want to burden the evolving
\LUATEX\ engine with the other kerning features because they lacked control
anyway. Even the mentioned adjust and insert extensions were redone and much more
was added in those departments.

Because we started in 2005 (with a first release in 2006) the \PDFTEX\ of that
time is of course not the same as of today. I'm not sure what the last version is
that \CONTEXT\ \MKII\ is targeting at because the version numbering changed a bit;
at some point versions like \type {14e} became \type {140.XX}, so it might be
around\type {140.17} or so. It might even be that recent versions break \MKII\
without us noticing. For \LUATEX\ we basically only took what we needed for
\CONTEXT\ at that time and assumed that \LUA\ could fill in the gaps. Because
\CONTEXT\ didn't really use much of the \LUATEX\ backend in the end what we kept
from \PDFTEX\ in \LUAMETATEX\ was a follow up on expansion and protrusion, for
which that engine set the standard. If it wasn't for \PDFTEX\ the \TEX\ community
would not be where it was now.

For as much as they make sense \ETEX\ extensions are mostly there but that
project was basically stopped after the first major release. In fact because
\PDFTEX\ has these extensions, we never implicitly had to include \ETEX. When we
started with \LUATEX\ we actually kept in mind the ideas we had at that time
because before we started with \LUATEX\ we already had plans for extensions
(flagged \type {eetex}) but those never came to fruit, just as we had some ideas
about extending \DVI\ which were superseded by the arrival of \PDF. The token
prepend and append primitives actually were examples of that. The \ETEX\ project
demonstrated that extending \TEX\ was an option. In that sense, \LUATEX\ and
\LUAMETATEX\ were very much unavoidable from the perspective of those involved in
\CONTEXT.

Then there is \XETEX. It is supported by \MKII\ but only the font mechanism was
extended to handle \OPENTYPE\ fonts. For a while \LUATEX\ had some compatible
math character definition primitives but none of its features. We can assume that
its internals are quite different when it comes to fonts from \LUATEX\ because
\LUATEX\ basically provides support for traditional fonts and delegates
everything \OPENTYPE\ to \LUA. In \LUATEX\ we used font loader code from
\FONTFORGE\ and some backend code from \DVIPDFMX, although \CONTEXT\ eventually
did all loading in \LUA. But anyway this program demonstrated that a \UNICODE\
engine supporting publicly available (fancy) fonts was possible by adapting \TEX\
so \XETEX\ introduced \TEX ies to the at that point still evolving world of
\OPENTYPE. And if someone is a happy \XETEX\ user, there is no need to switch to
\LUATEX.

In \OMEGA\ there were input translation mechanisms that some \CONTEXT\ users used
but that I never looked into myself. These are of course not present in \LUATEX\
because we can use \LUA\ for input processing. In the end a bit of the
directionality is all that we kept, most noticeably the initial \type {parnode}
and \type {dirnode} but we made them first class nodes instead of whatsits and in
\LUAMETATEX\ the first one serves different purposes. It got us started with
directions. Comparing the paragraph nodes of \LUATEX\ and \LUAMETATEX\ makes
little sense: they are too different.

Over the last decades other engines showed up, most noticeably those for e.g.\
Japanese but I never looked into these. I'm not sure if \LUATEX\ can do the same
with help from \LUA, but \LUAMETATEX\ has some more features so maybe it can.
These engines serve a specific (language and script audience) but if Japanese
\CONTEXT\ user need something more than we provide they can ask. One reason why
we have for instance an orientation feature in boxes is that we can support
vertical typesetting with rotated glyphs and boxes.

In the end the extensions in \LUAMETATEX\ come from our own demands combined with
trying to be complete but of course I might have missed something. It also means
that flaws in the design are just mine (or ours in the case of \LUATEX). Of
course quite some are common sense additions, often based what we need and what
makes macro programming easier, but if users depend on some feature present in
the other engines that cannot be handled by \LUA, they might ask for something
similar but then we need details and examples and not some reference to a manual
or macros that we are unaware of, have never seen, never used or haven't catched
up with.

There is very little overlap and common ground to cover between macro packages so
if someone thinks that \LUAMETATEX\ is a \CONTEXT\ thing, they might be right. Of
course \LUAMETATEX\ has plenty of what \LUATEX\ has and the core of \LUATEX\ is
pretty much original \TEX. However, in \LUAMETATEX\ nearly all mechanism have
been extended, optimized, and in the process made a bit more \CCODE-ish. The
original documentation describes what happens, the principles behind \TEX, so to
say. More details will be added but can also found in numerous documents in the
\CONTEXT\ distribution, articles and presentations. In the end we owe most to Don
Knuth, who gave is the (very original) original that we can build upon.

An important decision we made when we started with \LUATEX\ is that we don't lock
ourselves in specific solutions: callbacks are the way to go. Of course in
\LUAMETATEX\ we have some more helpers, and in the extended \METAPOST\ library we
do implement some features in \CCODE\ after having prototyped them in \LUA, but
there runtime performance was important. But in general \LUA\ solutions work out
fine. Good examples are the font loading and processing code and the \METAPOST\
backend. We can easily adapt, have complete control, can meet specific demands,
and as with more plugins we just needed them right from the start. And that is an
important consideration: we want to be able to experiment without constraints.
There are always the other engines: \PDFTEX\ for eight bit speed and \XETEX\ for
hard coded solutions and if they suit you, why not use them. We don't have to
cover every niche out there.

\stopsection

\startsection[title=Usage]

Why is it that there has been little fundamental development around \TEX\
engines? One of the reasons is that macro packages have to be stable. New
features can be added but if they are only available in one engine (and there are
a few more around now, like \XETEX\ and the \CJK\ specific ones) a macro package
has to provide ways around them when they are not available. Risking some
criticism I dare to say that in order to use \LUATEX\ to its full potential,
macro package has to be set up such that this is possible and \CONTEXT\ does just
that. When we talk backends it's relatively easy, and when we talk fonts it's
doable. But if you are not willing to adapt the core of your code dramatically
(and conceptually) all you get from \LUATEX\ is a built-in scripting language and
some occasional messing around with node lists. In \CONTEXT\ we could transition
rather well because the user interfaces permitted to do so without users
noticing. Of course there were changes, for instance because encodings matter
less, which is also true for e.g.\ \XETEX, and font technologies changed. But for
macro packages other than \CONTEXT\ just the availability of \LUA\ might be
enough reasons to use that engine. That also means that documentation of the more
intricate features is less important: one can just learn by example and \CONTEXT\
is that example.

With \LUAMETATEX\ we go further because here one really has to make some
fundamental choices. Again this could be done within the existing user
interfaces, but here we are not only talking of fundamental improvements, like
rendering math or breaking paragraphs into lines, but also of more flexible
handling of alignments, inserts, adjusts, marks, par and page building, etc.\
Basically all mechanism got extended and opened up. In order to profit from this
you have to be able to throw away existing solution and use these extensions to
come up with better ones. If one can put sentiments aside, this also takes quite
some time. This also means that where \LUAMETATEX\ is fine for the current
\CONTEXT, it might be less so for other macro packages. For instance, a font
frontend and \PDF\ backend in \LUA\ comes at a performance penalty. Some can be
gained back elsewhere, and actually the engine itself is more efficient too, but
there are no guarantees that it also works out for others.

A very important aspect (at least for me) is that I want the macro code to look
nice and in that respect stick to the \TEX\ syntax as much as possible. That
means that we have more programming related primitives, enhanced macro argument
parsing, more (flexible) conditionals, additional registers, extra expansion
related features and so on. Instead of some intermediate layer (like the helpers
in \CONTEXT) we can stick closer to the language itself. Of course this is not
something that most users will notice. What users might notice, is that on the
average \CONTEXT\ with \LUAMETATEX\ performs better than with \LUATEX\ or even
\LUAMETATEX. Even with more performance critical components delegated to \LUA\
(like the backend \PDF\ generation) we gain and often can compete performance
wise well with the faster eight bit \PDFTEX\ engine.

The fact that one has to make (and/or cannot make) drastic choices has a consequence
for documentation. Most of what is new and interesting is discussed in articles
and low level manuals. However, it is often discussed in the perspective of
\CONTEXT. Although we do discuss and show generic solutions it makes little sense
to go into details there, simply because in the end only \CONTEXT\ might use them
as intended. It's just a waste of time to implement variants that are more
generic because they will never be used elsewhere, especially in situations where
the solutions are considered \quote {standard} and will not change. In \CONTEXT\
we always followed the principle that if we can do better, we will do better, and
interfaces are such that this can be done.

Of course that brings up the question \quotation {How do you know that these are
the best solutions} and the answer is that we don't. However, we're not talking
of quick and dirty solutions. For instance it took years to enhance math support:
experiments, discussion, reconsideration, documenting, writing articles, looking
at usage, fonts, etc. A wider discussion would not have brought better solutions,
if at all. If that were the case, there would already have been successors. The
same is true for most extensions: there was little need for them outside the
\CONTEXT\ community. So in the end that's what those interested should look at:
how is \LUAMETATEX\ used in \CONTEXT. It is the combined development together
with acceptance by users that makes this possible.

\stopsection

\startsection[title=Dependencies]

When we started using \LUATEX\ in \CONTEXT\ we needed to add support for locating
and reading files as well as for \OPENTYPE\ processing. We also had to provide a
backend driver especially when we diverted from the \PDFTEX\ approach. In \MKII\
locating files is delegated to \KPSE\ but we already had code in the runners that
made it more efficient because nested lookup calls (as we have for starting up
and running \METAPOST) have quite some overhead. In a full blown \TEXLIVE\
installation we could measure multiple seconds overhead. So, we carried these
optimizations into the \MKIV\ code base and replaced \KPSE\ completely. This why
we could remove it from \LUAMETATEX, which also removed a complex dependency.
When a \KPSE\ library is present you can, at your own risk, use the optional
library which (as with any optional) provides a minimal set of runtime bindings.

When it comes to fonts, in \LUATEX\ we also had to test plain \TEX, so we made
the font loader and processing code generic, which also is the reason why Kai
Eigner is involved in maintaining it. We are aware of usage in \LATEX\ but as far
as we know it is a patched version (likely with limited functionality) and there
using a library is the recommended way anyway. The font loader and processor in
\MKXL\ is extended and tuned for \LUAMETATEX\ and because we have no reason to
support plain (it can if needed be emulated in \CONTEXT) we don't need to make
that variant generic. We could implement the base mode (native) font engine in
\LUA\ but the built-in one is fine and we see no reason to cripple \TEX\ too
much, but you need to load the font via \LUA.

The same is true for the (\PDF) backend. We already did most in \LUA\ and
prototyped in \MKIV\ before we moved on to \LUAMETATEX. A disadvantage of not
having a hard coded backend is that it increases runtime. The first versions of
\LUAMETATEX\ took between 10 and 20 percent more runtime and partly that was due
to the backend. Because the engine became faster and a few more primitive
features were added we could gain back performance and currently \MKXL\ is
outperforming \MKIV, for various reasons. A backend is only partially doing pure
textual output. There are images to be dealt with, fonts to be embedded (which
can also involve virtual fonts), specific features involved (like color or
hyperlinks), etc. Most of that is actually macro package specific. One could
argue that a generic backend makes sense, but there is little use in that when it
is only used by \CONTEXT. I could code a \LUATEX\ compatible backend but I would
not be able to test it, unless I made a setup for it, which is a waste of time.

So, these three components, although we could provide them, are currently very
\CONTEXT\ specific so if you want to use \LUAMETATEX\ elsewhere you need to come
up with something like that. Given history, changes are slim that any agreement
can be reached on a generic approach. \footnote {We also have to be realistic.
Apart maybe from fonts and hyphenation patterns there is little that macro
packages have in common. The only package that from the perspective of \CONTEXT\
qualifies as \quote {generic} is TikZ as that was designed with multiple macro
packages in mind. We therefore try to keep supporting it in the \CONTEXT\
ecosystem.} This is no real problem because in order to use \LUAMETATEX\ to its
full potential you probably need to rewrite a lot and therefore likely end up in
the \CONTEXT\ mindset. If someone needs specific features that \LUAMETATEX\ seems
to offer, just give \CONTEXT\ a try. \footnote {This is why we have no
\LUAMETATEX\ specific mailing list, we use the \CONTEXT\ mailing list(s) for
that. The official source distribution is the \CONTEXT\ distribution where it is
included in order to enable users to compile binaries if needed. This also fits
into the open source \quote {source included} paradigm. If your \CONTEXT\
installation doesn't ship with sources it is likely not an official one!}

\stopsection

\stopdocument

