% ----------------------------------------------------------------------

\chapter{The matrix}
\label{matrix}
\index{matrix}
\index{linear algebra}
\index{algebra!linear}
\index{matrix rudiments}
\index{rudiments}

\index{applied mathematics!foundations of}
\index{mathematics!applied, foundations of}
Chapters~\ref{alggeo} through~\ref{inttx} have solidly laid the basic
foundations of applied mathematics.  This chapter begins to build
on those foundations, demanding some heavier mathematical lifting.

Taken by themselves, most of the foundational methods of the earlier
chapters have handled only one or at most a few numbers (or functions)
at a time.  However, in practical applications the need to handle large
arrays of numbers at once arises often.  Some nonobvious effects then
emerge, as for example the eigenvalue of chapter~\ref{eigen}.

\index{matrix!motivation for}
Regarding the eigenvalue: the eigenvalue was always there, but
prior to this point in the book it was usually trivial---the eigenvalue
of~$5$ is just~$5$, for instance---so we didn't bother much to talk about it.
It is when numbers are laid out in orderly grids like
\[
  C = \mf{rrr}{
    6 & 4 & 0 \\
    3 & 0 & 1 \\
    3 & 1 & 0
  }
\]
that nontrivial eigenvalues arise (though you cannot tell just by
looking, the eigenvalues of~$C$ happen to be~$-1$
and $[7\pm\sqrt{\mbox{0x49}}]/2$).  But, just what is an
\emph{eigenvalue?}  Answer: an eigenvalue is the value by which an
object like~$C$ scales an eigenvector without altering the eigenvector's
direction.  Of course, we have not yet said what an \emph{eigenvector}
is, either, or how~$C$ might scale something, but it is to answer
precisely such questions that this chapter and the three which follow it
are written.

So, we are getting ahead of ourselves.
Let's back up.

\index{multiplier}
\index{coefficient}
An object like~$C$ is called a \emph{matrix.}  It serves as a
generalized coefficient or multiplier.  Where we have used single
numbers as coefficients or multipliers heretofore, one can with
sufficient care often use matrices instead.  The matrix interests us for
this reason among others.

\index{vector!generalized}
\index{vector!matrix}
\index{matrix vector}
\index{vector!$n$-dimensional}
\index{$n$-dimensional vector}
\index{complex number!being a scalar not a vector}
\index{element}
\index{dimension}
The technical name for the ``single number'' is the \emph{scalar.}  Such
a number, as for instance~$5$ or $-4+i3$, is called a scalar because its
action alone during multiplication is simply to scale the thing it
multiplies.  Besides acting alone, however, scalars can also act in
concert---in orderly formations---thus constituting any of three basic
kinds of arithmetical object:
\bi
  \item \index{scalar} the \emph{scalar} itself, a single number
    like $\alpha=5$ or $\beta=-4+i3$;
  \item \index{vector} the \emph{vector,} a column of~$m$ scalars like
    \[
      \ve u = \mf{c}{
        5 \\ -4+i3
      },
    \]
    which can be written in-line with the notation
    $\ve u=[5\ \ \mbox{$-4+i3$}]^T$ (here there are two scalar
    elements,~$5$ and $-4+i3$, so in this example $m=2$);
  \item \index{matrix}\index{row}\index{column}\index{vector!row of}
    the \emph{matrix,} an $m\times n$ grid of scalars, or equivalently a
    row of~$n$ vectors, like
    \[
      A = \mf{rrr}{
        0 & 6 &  2 \\
        1 & 1 & -1
      },
    \]
    which can be written in-line with the notation $A=[0\;6\;2;
    1\;1\;\mbox{$-1$}]$ or the notation $A=[0\;1; 6\;1; 2\;\mbox{$-1$}]^T$
    (here there are two rows and three columns of scalar elements, so in
    this example $m=2$ and $n=3$).
\ei
% An x which solves A*x==b is x=[1 1 -2]', where b==[2 4]'.
Several general points are immediately to be observed about these
various objects.  First, despite the geometrical Argand interpretation
of the complex number, a complex number is not a two-element vector but
a scalar; therefore any or all of a vector's or matrix's scalar elements
can be complex.  Second, an $m$-element vector does not differ for most
purposes from an $m\times 1$ matrix; generally the two can be regarded
as the same thing.
Third, the three-element (that is, three-dimensional) geometrical vector
of \S~\ref{trig:230} is just an $m$-element vector with $m=3$.
Fourth,~$m$ and~$n$ can be any nonnegative integers, even one, even
zero, even infinity.%
\footnote{
  Fifth, though the progression \emph{scalar, vector, matrix} suggests
  next a ``matrix stack'' or stack of~$p$ matrices, such objects
  in fact are seldom used.  As we shall see in \S~\ref{matrix:120}, the
  chief advantage of the standard matrix is that it neatly represents
  the linear transformation of one vector into another.  ``Matrix
  stacks'' bring no such advantage.  This book does not treat them.
}

Where one needs visually to distinguish a symbol like~$A$
representing a matrix, one can write it~$[A]$, in square brackets.%
\footnote{
  Alternate notations seen in print include~$\dyad A$ and~$\ve A$.
}
Normally however a simple~$A$ suffices.

The matrix is a notoriously hard topic to motivate.  The idea of the
matrix is deceptively simple.  The mechanics of matrix arithmetic are
deceptively intricate.  The most basic body of matrix theory, without
which little or no useful matrix work can be done, is deceptively
extensive.  The matrix neatly encapsulates a substantial knot of
arithmetical tedium and clutter, but to understand the matrix one must
first understand the tedium and clutter the matrix encapsulates.  As far
as the author is aware, no one has ever devised a way to introduce the
matrix which does not seem shallow, tiresome, irksome, even interminable
at first encounter; yet the matrix is too important to ignore.  Applied
mathematics brings nothing else quite like it.%
\footnote{
  In most of its chapters, the book seeks a balance between terseness
  the determined beginner cannot penetrate and prolixity the seasoned
  veteran will not abide.  The matrix upsets this balance.

  Part of the trouble with the matrix is that its arithmetic is just that, an
  arithmetic, no more likely to be mastered by mere theoretical study
  than was the classical arithmetic of childhood.  To master matrix
  arithmetic, one must drill it; yet the book you hold is fundamentally
  one of theory not drill.

  The reader who has previously drilled matrix arithmetic will meet here
  the essential applied theory of the matrix.  That reader will find
  this chapter and the next three tedious enough.  The reader who has not
  previously drilled matrix arithmetic, however, is likely to find these
  chapters positively hostile.  Only the doggedly determined beginner
  will learn the matrix here alone; others will find it more amenable to
  drill matrix arithmetic first in the early chapters of an introductory
  linear algebra textbook, dull though such chapters be (see~\cite{Lay}
  or better yet the fine, surprisingly less dull~\cite{Hefferon} for
  instance, though the early chapters of almost any such book give the
  needed arithmetical drill.) Returning here thereafter, the beginner
  can expect to find \emph{these} chapters still tedious but no longer
  impenetrable.  The reward is worth the effort.  That is the approach
  the author recommends.

  To the mathematical rebel, the young warrior with face painted and
  sword agleam, still determined to learn the matrix here alone, the
  author salutes his honorable defiance.  Would the rebel consider
  alternate counsel?  If so, then the rebel might compose a dozen
  matrices of various sizes and shapes, broad, square and tall,
  decomposing each carefully by pencil per the Gauss-Jordan method of
  \S~\ref{gjrank:341}, checking results (again by pencil; using a
  machine defeats the point of the exercise, and using a sword, well, it
  won't work) by multiplying factors to restore the original matrices.
  Several hours of such drill should build the young warrior the
  practical arithmetical foundation to master---with commensurate
  effort---the theory these chapters bring.  The way of the warrior is
  hard, but conquest is not impossible.

  To the matrix veteran, the author presents these four chapters
  with grim enthusiasm.  Substantial, logical, necessary the chapters
  may be, but exciting they are not.  At least, the earlier parts are
  not very exciting (later parts are better).  As a reasonable
  compromise, the veteran seeking more interesting reading might skip
  directly to chapters~\ref{mtxinv} and~\ref{eigen}, referring back to
  chapters~\ref{matrix} and \ref{gjrank} as need arises.
}

Chapters~\ref{matrix} through~\ref{eigen} treat the matrix and its
algebra.  This chapter, chapter~\ref{matrix}, introduces the rudiments of
the matrix itself.%
\footnote{%
  \cite{Beattie}%
  \cite{Franklin}%
  \cite{Hefferon}%
  \cite{Lay}
}

% ----------------------------------------------------------------------

\section{Provenance and basic use}
\label{matrix:120}
\index{matrix!basic operations of}
\index{matrix!motivation for}
\index{matrix!provenance of}
\index{matrix!basic use of}

It is in the study of linear transformations that the concept of the
matrix first arises.  We begin there.

\subsection{The linear transformation}
\label{matrix:120.10}
\index{linear transformation}
\index{transformation!linear}

\index{index}
Section~\ref{integ:240.05} has introduced the idea of linearity.  The
\emph{linear transformation}%
\footnote{
  Professional mathematicians conventionally are careful to begin by
  drawing a clear distinction between the ideas of the linear
  transformation, the basis set and the simultaneous system of linear
  equations---proving from suitable axioms that the three amount more or
  less to the same thing, rather than implicitly assuming the fact.  The
  professional approach \cite[chapters~1 and~2]{Beattie}\cite[chapters~1, 2
  and~5]{Lay} has much to recommend it, but it is not the approach we
  will follow here.
}
is the operation of an $m\times n$ matrix~$A$, as in
\bq{matrix:000:10}
  A\ve x = \ve b,
\eq
to transform an $n$-element vector~$\ve x$ into an $m$-element vector~$\ve
b$, while respecting the rules of linearity
\bq{matrix:000:20}
  \renewcommand\arraystretch{1.5}
  \br{rclcl}
    A(\ve x_1 + \ve x_2) &=& A\ve x_1 + A\ve x_2 &=& \ve b_1 + \ve b_2, \\
    A(\alpha \ve x) &=& \alpha A\ve x &=& \alpha\ve b, \\
    A(0) &=& 0. &&
  \er
\eq
For example,
\[
  A = \mf{rrr}{
    0 & 6 &  2 \\
    1 & 1 & -1
  }
\]
is the $2\times 3$ matrix which transforms a three-element vector~$\ve
x$ into a two-element vector~$\ve b$ such that
\[
  A\ve x = \mf{c}{
    0x_1 + 6x_2 + 2x_3 \\
    1x_1 + 1x_2 - 1x_3
  } = \ve b,
\]
where
\[
  \ve x = \mf{c}{ x_1 \\ x_2 \\ x_3 },
  \ \ %
  \ve b = \mf{c}{ b_1 \\ b_2 }.
\]
In general, the operation of a matrix~$A$ is that%
\footnote{
  As observed in appendix~\ref{greek}, there are unfortunately not
  enough distinct Roman and Greek letters available to serve the needs
  of higher mathematics.  In matrix work, the Roman letters~$ijk$
  conventionally serve as indices, but the same letter~$i$ also serves
  as the imaginary unit, which is not an index and has nothing to do
  with indices.  Fortunately, the meaning is usually clear from the
  context:~$i$ in~$\sum_i$ or~$a_{ij}$ is an index;~$i$ in $-4+i3$
  or~$e^{i\phi}$ is the imaginary unit.  Should a case arise in which
  the meaning is not clear, one can use~$\ell jk$ or some other
  convenient letters for the indices.
}%
$\mbox{}^,$%
\footnote{
  Whether to let the index~$j$ run from~$0$ to $n-1$ or from~$1$ to~$n$
  is an awkward question of applied mathematical style.  In computers,
  the index normally runs from~$0$ to $n-1$, and in many ways this
  really is the more sensible way to do it.  In mathematical
  theory, however, a~$0$ index normally implies something special or
  basic about the object it identifies.  The book you are reading tends
  to let the index run from~$1$ to~$n$, following mathematical
  convention in the matter for this reason.

  Conceived more generally, an $m\times n$ matrix can be considered an
  $\infty\times\infty$ matrix with zeros in the unused cells.  Here,
  both indices~$i$ and~$j$ run from~$-\infty$ to~$+\infty$ anyway, so
  the computer's indexing convention poses no dilemma in this case.
  See \S~\ref{matrix:180}.
}
\bq{matrix:000:30}
  b_i = \sum_{j=1}^{n} a_{ij} x_j,
\eq
where~$x_j$ is the $j$th element of~$\ve x$, $b_i$ is the $i$th element
of~$\ve b$, and
\[
  a_{ij} \equiv [A]_{ij}
\]
is the element at the $i$th row and $j$th column of~$A$, counting from
top left (in the example for instance, $a_{12}=6$).

\index{equation!solving a set of simultaneously}
\index{equation!simultaneous linear system of}
\index{simultaneous system of linear equations}
Besides representing linear transformations as such, matrices can also
represent simultaneous systems of linear equations.  For example, the
system
\[
  \begin{split}
    0x_1 + 6x_2 +  2x_3 &= 2, \\
    1x_1 + 1x_2 -  1x_3 &= 4,
  \end{split}
\]
is compactly represented as
\[
  A\ve x = \ve b,
\]
with~$A$ as given above and $\ve b = [2 \; 4 ]^T$.  Seen from this point
of view, a simultaneous system of linear equations is itself neither
more nor less than a linear transformation.

\subsection{Matrix multiplication (and addition)}
\label{matrix:120.20}
\index{matrix!multiplication of}
\index{multiplication!of matrices}
\index{matrix!arithmetic of}
\index{arithmetic!of matrices}

\index{matrix!addition of}
\index{addition!of matrices}
\index{associativity!of matrix multiplication}
\index{commutivity!noncommutivity of matrix multiplication}
\index{noncommutivity!of matrix multiplication}
\index{matrix!associativity of the multiplication of}
\index{matrix!noncommutivity of the multiplication of}
\index{matrix!multiplication of by a scalar}
\index{counterexample}
Nothing prevents one from lining several vectors~$\ve x_k$ up in a row,
industrial mass production-style, transforming them at once into the
corresponding vectors~$\ve b_k$ by the same matrix~$A$.  In this case,
\bq{matrix:000:40}
  \br{rcrccccl}
    X &\equiv& [& \ve x_1 & \ve x_2 &\cdots& \ve x_p &], \\
    B &\equiv& [& \ve b_1 & \ve b_2 &\cdots& \ve b_p &], \\
    AX &=& \multicolumn{6}{l}{B,} \\
    b_{ik} &=& \multicolumn{6}{l}{
      \ds\sum_{j=1}^{n} a_{ij}x_{jk}.
    }
  \er
\eq
Equation~(\ref{matrix:000:40}) implies a definition for matrix
multiplication.  Such matrix multiplication is associative since
\bqa
  \left[(A)(XY)\right]_{ik} &=& \sum_{j=1}^{n} a_{ij}[XY]_{jk}
  \xn\\&=& \sum_{j=1}^{n} a_{ij}\left[
      \sum_{\ell=1}^{p} x_{j\ell} y_{\ell k}
    \right]
  \xn\\&=& \sum_{\ell=1}^{p} \sum_{j=1}^{n} a_{ij} x_{j\ell} y_{\ell k}
  \xn\\&=& \left[(AX)(Y)\right]_{ik}.
  \label{matrix:000:42}
\eqa
Matrix multiplication is not generally commutative, however;
\bq{matrix:000:41}
  AX \neq XA,
\eq
as one can show by a suitable counterexample like
$A=[0\;1; 0\;0],$ $X=[1\;0; 0\;0]$.
To multiply a matrix by a scalar, one multiplies each of the matrix's
elements individually by the scalar:
\bq{matrix:000:45}
  [\alpha A]_{ij} = \alpha a_{ij}.
\eq
Evidently multiplication by a scalar \emph{is} commutative: $\alpha A\ve
x = A\alpha\ve x$.

Matrix addition works in the way one would expect, element by element;
and as one can see from~(\ref{matrix:000:40}), under multiplication,
matrix addition is indeed distributive:
\bq{matrix:000:48}
  \begin{split}
    [X+Y]_{ij} &= x_{ij} + y_{ij}; \\
    (A)(X+Y) &= AX+AY; \\
    (A+C)(X) &= AX+CX.
  \end{split}
\eq

\subsection{Row and column operators}
\label{matrix:120.27}
\index{row operator}
\index{column operator}

The matrix equation $A\ve x = \ve b$ represents the linear transformation
of~$\ve x$ into~$\ve b$, as we have seen.  Viewed from another
perspective, however, the same matrix equation represents something
else; it represents a weighted sum of the columns of~$A$, with the
elements of~$\ve x$ as the weights.  In this view, one
writes~(\ref{matrix:000:30}) as
\bq{matrix:230:05}
  \ve b = \sum_{j=1}^{n} [A]_{*j} x_j,
\eq
where~$[A]_{*j}$ is the $j$th column of~$A$.
Here~$\ve x$ is not only a vector; it is also an operator.
It operates on $A$'s columns.  By virtue of multiplying~$A$ from the
right, the vector~$\ve x$ is a \emph{column operator} acting on~$A$.

If several vectors~$\ve x_k$ line up in a row to form a matrix~$X$,
such that $AX = B$, then the matrix~$X$ is likewise a column operator:
\bq{matrix:230:10}
  [B]_{*k} = \sum_{j=1}^{n} [A]_{*j} x_{jk}.
\eq
The $k$th column of~$X$ weights the several columns of~$A$ to yield the
$k$th column of~$B$.

If a matrix multiplying from the right is a column operator, is a matrix
multiplying from the left a \emph{row operator?}  Indeed it is.  Another
way to write that $AX=B$, besides~(\ref{matrix:230:10}), is
\bq{matrix:230:20}
  [B]_{i*} = \sum_{j=1}^{n} a_{ij} [X]_{j*}.
\eq
The $i$th row of~$A$ weights the several rows of~$X$ to yield the
$i$th row of~$B$.  The matrix~$A$ is a row operator.
%
(Observe the notation.  The~${*}$ here means ``any'' or ``all.''
Hence~$[X]_{j{*}}$ means ``$j$th row, all columns of~$X$''---that is,
the $j$th row of~$X$.  Similarly,~$[A]_{{*}j}$ means ``all rows, $j$th
column of~$A$''---that is, the $j$th column of~$A$.)

\emph{Column operators attack from the right; row operators, from the
left.}  This rule is worth memorizing; the concept is important.  In
$AX=B$, the matrix~$X$ operates on $A$'s columns; the matrix~$A$
operates on $X$'s rows.

Since matrix multiplication produces the
same result whether one views it as a linear
transformation~(\ref{matrix:000:40}), a column
operation~(\ref{matrix:230:10}) or a row
operation~(\ref{matrix:230:20}), one might wonder what purpose lies in
defining matrix multiplication three separate ways.  However, it is not
so much for the sake of the mathematics that we define it three ways as
it is for the sake of the mathematician.  We do it for ourselves.
Mathematically, the latter two do indeed expand to
yield~(\ref{matrix:000:40}), but as written the three represent three
different perspectives on the matrix.  A tedious, nonintuitive matrix
theorem from one perspective can appear suddenly obvious from
another (see for example eqn.~\ref{matrix:330:25}).  Results hard to
visualize one way are easy to visualize another.  It is worth developing
the mental agility to view and handle matrices all three ways for this
reason.

\subsection{The transpose and the adjoint}
\label{matrix:120.30}
\index{transpose}
\index{transpose!conjugate}
\index{conjugate transpose}
\index{adjoint}

\index{Hermite, Charles (1822--1901)}
\index{conjugate}
One function peculiar to matrix algebra is the \emph{transpose}
\bq{matrix:000:50}
  \begin{split}
    C &= A^{T}, \\
    c_{ij} &= a_{ji},
  \end{split}
\eq
which mirrors an $m\times n$ matrix into an $n\times m$ matrix.
For example,
\[
  A^{T} = \mf{rr}{
      0 &  1 \\
      6 &  1 \\
      2 & -1
  }.
\]
Similar and even more useful is the \emph{conjugate transpose} or
\emph{adjoint}%
\footnote{
  Alternate notations sometimes seen in print for the adjoint
  include~$A^\dagger$ (a notation which in this book means something
  unrelated) and~$A^H$ (a notation which recalls the name of the
  mathematician Charles Hermite).  However, the book you are reading
  writes the adjoint only as~$A^{*}$, a notation which better captures
  the sense of the thing in the author's view.
}
%\footnote{
%  More couth than the noun ``transpose'' would seem to
%  ``transposition,'' but the former appears to be entrenched in the
%  literature.
%}
\bq{matrix:000:52}
  \begin{split}
    C &= A^{*}, \\
    c_{ij} &= a_{ji}^{*},
  \end{split}
\eq
which again mirrors an $m\times n$ matrix into an $n\times m$ matrix,
but conjugates each element as it goes.

The transpose is convenient notationally to write vectors and matrices
in-line and to express certain matrix-arithmetical mechanics; but
algebraically the transpose is artificial.  It is the adjoint rather
which mirrors a matrix properly.  (If the transpose and adjoint
functions applied to words as to matrices, then the transpose of
``derivations'' would be ``snoitavired,'' whereas the adjoint would be
``\reflectbox{derivations}.''  See the difference?)  On real-valued
matrices like the~$A$ in the example, of course, the transpose and the
adjoint amount to the same thing.

If one needed to conjugate the elements of a matrix without transposing
the matrix itself, one could contrive notation like~$A^{*T}$.
Such a need seldom arises, however.

Observe that
\bq{matrix:120:30}
  \begin{split}
    (A_2A_1)^{T} &= A_1^{T}A_2^{T}, \\
    (A_2A_1)^{*} &= A_1^{*}A_2^{*},
  \end{split}
\eq
and more generally that%
\footnote{
  Recall from \S~\ref{alggeo:227} that $\prod_k A_k=\cdots A_3 A_2 A_1$,
  whereas $\coprod_k A_k=A_1 A_2 A_3 \cdots$.
}
\bq{matrix:120:33}
  \begin{split}
    \left(\prod_k A_k\right)^{T} &= \coprod_k A_k^{T}, \\
    \left(\prod_k A_k\right)^{*} &= \coprod_k A_k^{*}.
  \end{split}
\eq

% ----------------------------------------------------------------------

\section{The Kronecker delta}
\label{matrix:150}
\index{Kronecker delta}
\index{Kronecker, Leopold (1823--1891)}
\index{delta, Kronecker}
\index{$\delta$}

\index{sifting property}
\index{Kronecker delta!sifting property of}
\index{delta, Kronecker!sifting property of}
Section~\ref{integ:670} has introduced the Dirac delta.  The discrete
analog of the Dirac delta is the \emph{Kronecker delta}%
\footnote{\cite[``Kronecker delta,'' 15:59, 31 May 2006]{wikip}}
\bq{matrix:150:kron1}
  \delta_{i} \equiv \begin{cases}
    1 &\mbox{if $i=0$,} \\
    0 &\mbox{otherwise;}
  \end{cases}
\eq
or
\bq{matrix:150:kron2}
  \delta_{ij} \equiv \begin{cases}
    1 &\mbox{if $i=j$,} \\
    0 &\mbox{otherwise.}
  \end{cases}
\eq
The Kronecker delta enjoys the Dirac-like properties that
\bq{matrix:150:sift0}
  \sum_{i=-\infty}^{\infty} \delta_i
  = \sum_{i=-\infty}^{\infty} \delta_{ij}
  = \sum_{j=-\infty}^{\infty} \delta_{ij} = 1
\eq
and that
\bq{matrix:150:sift}
  \sum_{j=-\infty}^{\infty} \delta_{ij} a_{jk} = a_{ik},
\eq
the latter of which is the Kronecker sifting property.
The Kronecker equations~(\ref{matrix:150:sift0})
and~(\ref{matrix:150:sift}) parallel the Dirac
equations~(\ref{integ:670:sift0}) and~(\ref{integ:670:sift}).

Chapters~\ref{matrix} and~\ref{eigen} will find frequent use for the
Kronecker delta.  Later, \S~\ref{vector:240.30} will revisit the
Kronecker delta in another light.

% ----------------------------------------------------------------------

\section{Dimensionality and matrix forms}
\label{matrix:180}
\index{dimensionality}
\index{infinite dimensionality}
\index{matrix!form of}

\index{matrix!multiplication of}
\index{multiplication!of matrices}
\index{padding a matrix with zeros}
\index{zero!padding a matrix with}
\index{matrix!padding of with zeros}
An $m \times n$ matrix like
\[
  X = \mf{rrr}{
   -4 & 0 \\
    1 & 2 \\
    2 &-1
  }
\]
can be viewed as the $\infty \times \infty$ matrix
{
  \settowidth\tla{\fn$0$}
  \[
    X = \mf{crrrrrrrc}{
      \ddots&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \\
      \cdots&0&0& 0 & 0 &0&0&0&\cdots \\
      \cdots&0&0& 0 & 0 &0&0&0&\cdots \\
      \cdots&0&0& \makebox[\tla][r]{$-\!4$} & 0 &0&0&0&\cdots \\
      \cdots&0&0& 1 & 2 &0&0&0&\cdots \\
      \cdots&0&0& 2 & \makebox[\tla][r]{$-\!1$} &0&0&0&\cdots \\
      \cdots&0&0& 0 & 0 &0&0&0&\cdots \\
      \cdots&0&0& 0 & 0 &0&0&0&\cdots \\
      &
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \multicolumn{1}{c}{\vdots}&
      \ddots
    },
  \]
}%
with zeros in the unused cells.  As before, $x_{11}=-4$ and $x_{32}=-1$,
but now~$x_{ij}$ exists for all integral~$i$ and~$j$; for instance,
$x_{(-1)(-1)}=0$.  For such a matrix, indeed for all matrices, the
matrix multiplication rule~(\ref{matrix:000:40}) generalizes to
\bqa
  B &=& AX, \xn\\
  b_{ik} &=& \sum_{j=-\infty}^{\infty} a_{ij}x_{jk}.
  \label{matrix:180:05}
\eqa

\index{matrix!square}
\index{square matrix}
\index{matrix operator}
For square matrices whose purpose is to manipulate other matrices or
vectors in place, merely padding with zeros often does not suit.
Consider for example the square matrix
\[
  A_3 = \mf{ccc}{
    1 & 0 & 0 \\
    5 & 1 & 0 \\
    0 & 0 & 1
  }.
\]
This~$A_3$ is indeed a matrix, but when it acts~$A_3X$ as a row operator
on some $3 \times p$ matrix~$X$, its effect is to add to $X$'s second
row,~$5$ times the first.  Further consider
\[
  A_4 = \mf{cccc}{
    1 & 0 & 0 & 0 \\
    5 & 1 & 0 & 0 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1
  },
\]
which does the same to a $4 \times p$ matrix~$X$.  We can also
define $A_5, A_6, A_7, \ldots$, if we want; but, really, all these
express the same operation: ``to add to the second row,~$5$ times the
first.''

\index{main diagonal}
\index{diagonal!main}
\index{matrix!main diagonal of}
The $\infty \times \infty$ matrix
\[
  A = \mf{ccccccccc}{
    \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
    \cdots & 1 & 0 & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 1 & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 1 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 5 & 1 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 1 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 & 1 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 & 0 & 1 & \cdots \\
    &\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
  }
\]
expresses the operation generally.  As before, $a_{11} = 1$ and $a_{21}
= 5$, but now also $a_{(-1)(-1)} = 1$ and $a_{09} = 0$, among others.
By running ones infinitely both ways out the \emph{main diagonal,} we
guarantee by~(\ref{matrix:180:05}) that when~$A$ acts~$AX$
on a matrix~$X$ of any dimensionality whatsoever,~$A$ adds to
the second row of~$X$, $5$ times the first---and affects no other row.
(But what if~$X$ is a $1 \times p$ matrix, and \emph{has} no second row?
Then the operation~$AX$ creates a new second row,~$5$ times the first---or
rather so fills in $X$'s previously null second row.)

\index{bit}
\index{extended operator}
In the infinite-dimensional view, the matrices~$A$ and~$X$ differ
essentially.%
\footnote{
  This particular section happens to use the symbols~$A$ and~$X$ to
  represent certain specific matrix forms because such usage flows
  naturally from the usage $A\ve x = \ve b$ of \S~\ref{matrix:120}.
  Such usage admittedly proves awkward in other contexts.  Traditionally
  in matrix work and elsewhere in the book, the letter~$A$ does not
  necessarily represent an extended operator as it does here, but rather
  an arbitrary matrix of no particular form.
}
This section explains, developing some nonstandard formalisms the
derivations of later sections and chapters can use.%
\footnote{
  The idea of infinite dimensionality is sure to discomfit some readers,
  who have studied matrices before and are used to thinking of a matrix
  as having some definite size.  There is nothing wrong with thinking of
  a matrix as having some definite size, only that that view does not
  suit the present book's development.  And really, the idea of an
  $\infty \times 1$ vector or an $\infty \times \infty$ matrix should
  not seem so strange.  After all, consider the vector~$\ve u$
  such that
  \[
    u_\ell = \sin \ell\ep,
  \]
  where $0 < \ep \ll 1$ and~$\ell$ is an integer, which holds all
  values of the function $\sin\theta$ of a real argument~$\theta$.  Of
  course one does not actually write down or store all the elements of
  an infinite-dimensional vector or matrix, any more than one actually
  writes down or stores all the bits (or digits) of~$2\pi$.  Writing
  them down or storing them is not the point.  The point is that
  infinite dimensionality is all right; that the idea thereof does not
  threaten to overturn the reader's pre\"existing matrix knowledge;
  that, though the construct seem unfamiliar, no fundamental conceptual
  barrier rises against it.

  Different ways of looking at the same mathematics can be extremely
  useful to the applied mathematician.  The applied mathematical reader
  who has never heretofore considered infinite dimensionality in vectors
  and matrices would be well served to take the opportunity to do so
  here.  As we shall discover in chapter~\ref{gjrank}, dimensionality is a
  poor measure of a matrix's size in any case.  What really counts is
  not a matrix's $m \times n$ dimensionality but rather its \emph{rank.}
}

\subsection{The null and dimension-limited matrices}
\label{matrix:180.25}
\index{null matrix}
\index{matrix!null}
\index{dimension-limited matrix}
\index{matrix!dimension-limited}

\index{zero matrix}
\index{zero!matrix}
\index{$0$ (zero)!matrix}
The \emph{null matrix} is just what its name implies:
\[
  0 = \mf{ccccccc}{
    \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
    \cdots & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 & \cdots \\
    & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  };
\]
or more compactly,
\[
  [0]_{ij} = 0.
\]
Special symbols like~$\dyad 0$, $\ve 0$ or~$O$ are possible for the null
matrix, but usually a simple~$0$ suffices.  There are no surprises here;
the null matrix brings all the expected properties of a zero, like
\bqb
  0 + A &=& A, \\
  \mbox{} [0][X] &=& 0.
\eqb

\index{zero vector}
\index{zero!vector}
\index{$0$ (zero)!vector}
The same symbol~$0$ used for the null scalar (zero) and the null
matrix is used for the null vector, too.  Whether the scalar~$0$, the
vector~$0$ and the matrix~$0$ actually represent different things is a
matter of semantics, but the three are interchangeable for most
practical purposes in any case.  Basically, a zero is a zero is a zero;
there's not much else to it.%
\footnote{
  Well, of course, there's a lot else to it, when it comes to
  dividing by zero as in chapter~\ref{drvtv}, or to summing an infinity of
  zeros as in chapter~\ref{integ}, but those aren't what we were speaking of
  here.
}

\index{active region}
Now a formality: the ordinary $m \times n$ matrix~$X$ can be viewed,
infinite-dimensionally, as a variation on the null matrix, inasmuch
as~$X$ differs from the null matrix only in the~$mn$ elements~$x_{ij}$,
$1 \le i \le m$, $1 \le j \le n$.  Though the theoretical dimensionality
of~$X$ be $\infty \times \infty$, one need record only the~$mn$
elements, plus the values of~$m$ and~$n$, to retain complete information
about such a matrix.  So the semantics are these: when we call a
matrix~$X$ an \emph{$m \times n$ matrix,} or more precisely a
\emph{dimension-limited matrix} with an $m \times n$ \emph{active
region,} we will mean formally that~$X$ is an $\infty \times \infty$
matrix whose elements are all zero outside the $m \times n$ rectangle:
\bq{matrix:180:27}
  \mbox{$x_{ij} = 0$ except where $1 \le i \le m$ and $1 \le j \le n$.}
\eq
By these semantics, every $3 \times 2$ matrix (for example) is also a
formally a $4 \times 4$ matrix; but a $4 \times 4$ matrix is not in
general a $3 \times 2$ matrix.

\subsection[The identity, scalar and extended matrices]
  {The identity and scalar matrices and the extended operator}
\label{matrix:180.35}
\index{extended operator}
\index{identity matrix}
\index{matrix!identity}

\index{general identity matrix}
\index{matrix!general identity}
\index{scalar matrix}
\index{matrix!scalar}
\index{one}
\index{$1$ (one)}
The \emph{general identity matrix}---or simply, the \emph{identity
matrix}---is
\[
  I = \mf{ccccccc}{
    \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
    \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 1 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 1 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
    & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  },
\]
or more compactly,
\bq{matrix:180:31}
  [I]_{ij} = \delta_{ij},
\eq
where~$\delta_{ij}$ is the Kronecker delta of \S~\ref{matrix:150}.
The identity matrix~$I$ is a matrix~$1$, as it were,%
\footnote{
  In fact you can write it as~$1$ if you like.  That is essentially what
  it is.  The~$I$ can be regarded as standing for ``identity'' or as the
  Roman numeral~I\@.
}
bringing the essential property one expects of a~$1$:
\bq{matrix:180:32}
  IX = X = XI.
\eq
The \emph{scalar matrix} is
\[
  \lambda I = \mf{ccccccc}{
    \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
    \cdots & \lambda & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & \lambda & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & \lambda & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & \lambda & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & \lambda & \cdots \\
    & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  },
\]
or more compactly,
\bq{matrix:180:33}
  [\lambda I]_{ij} = \lambda \delta_{ij},
\eq
If the identity matrix~$I$ is a matrix~$1$, then the scalar
matrix~$\lambda I$ is a matrix~$\lambda$, such that
\bq{matrix:180:34}
  [\lambda I] X = \lambda X = X [\lambda I].
\eq
The identity matrix is (to state the obvious) just the scalar matrix
with $\lambda=1$.

\index{sparsity}
\index{matrix!sparse}
The \emph{extended operator}~$A$ is a variation on the scalar
matrix~$\lambda I$, $\lambda \neq 0$, inasmuch as~$A$ differs
from~$\lambda I$ only in~$p$ specific elements, with~$p$ a finite
number.  Symbolically,
\bqa
  a_{ij} &=&
  \begin{cases}
    (\lambda)(\delta_{ij} + \alpha_k) & \mbox{if $(i,j) = (i_k,j_k)$, $1 \le k \le p$,} \\
    \lambda\delta_{ij} & \mbox{otherwise;}
  \end{cases} \label{matrix:180:29} \\
  \lambda &\neq& 0. \xn
\eqa
The several~$\alpha_k$ control how the extended operator~$A$ differs
from~$\lambda I$.  One need record only the several~$\alpha_k$ along
with their respective addresses $(i_k,j_k)$, plus the scale~$\lambda$,
to retain complete information about such a matrix.  For example, for an
extended operator fitting the pattern
\[
  A = \mf{ccccccc}{
    \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
    \cdots & \lambda & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & \lambda \alpha_1 & \lambda & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & \lambda & \lambda \alpha_2 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & \lambda(1+\alpha_3) & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 & \lambda & \cdots \\
    & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  },
\]
one need record only the values of~$\alpha_1$, $\alpha_2$
and~$\alpha_3$, the respective addresses
% bad break
$(2,1)$, $(3,4)$ and~$(4,4)$, and the value of the scale~$\lambda$; this
information alone implies the entire $\infty \times \infty$ matrix~$A$.

When we call a matrix~$A$ an \emph{extended $n \times n$ operator,} or
an extended operator with an $n \times n$ \emph{active region,} we will mean
formally that~$A$ is an $\infty \times \infty$ matrix and is further an
extended operator for which
\bq{matrix:180:28}
  \mbox{$1 \le i_k \le n$ and $1 \le j_k \le n$ for all $1 \le k \le p$.}
\eq
That is, an extended $n \times n$ operator is one whose several~$\alpha_k$
all lie within the $n \times n$ square.  The~$A$ in the example is
an extended $4 \times 4$ operator (and also a $5 \times 5$, a $6 \times 6$,
etc., but not a $3 \times 3$).

(Often in practice for smaller operators---especially in the typical
case that $\lambda = 1$---one finds it easier just to record all the $n
\times n$ elements of the active region.  This is fine.  Large matrix
operators however tend to be \emph{sparse,} meaning that they depart
from~$\lambda I$ in only a very few of their many elements.  It would
waste a lot of computer memory explicitly to store all those zeros, so
one normally stores just the few elements, instead.)

Implicit in the definition of the extended operator is that the identity
matrix~$I$ and the scalar matrix~$\lambda I$, $\lambda \neq 0$, are extended
operators with $0 \times 0$ active regions (and also $1 \times 1$, $2
\times 2$, etc.).  If $\lambda = 0$, however, the scalar
matrix~$\lambda I$ is just the null matrix, which is no extended
operator but rather by definition a $0 \times 0$ dimension-limited matrix.

\subsection{The active region}
\label{matrix:180.38}
\index{active region}

Though maybe obvious, it bears stating explicitly that a product of
% bad break
di\-men\-sion-lim\-ited
and/or extended-operational matrices with $n \times n$
active regions itself has an $n \times n$ active region.%
\footnote{
  The section's earlier subsections formally define the term
  \emph{active region} with respect to each of the two matrix forms.
}
(Remember that a matrix with an $m' \times n'$
active region also by definition has an $n \times n$ active region if
$m' \le n$ and $n' \le n$.)  If any of the factors has dimension-limited
form then so does the product; otherwise the product is an extended
operator.%
\footnote{
  If symbolic proof of the subsection's claims is wanted, here it is in outline:
  \settowidth\tla{$\sum_k a_{ik}(\lambda_b\delta{kj})$}
  \settowidth\tld{$\makebox[\tla][l]{$\sum_k a_{ik}(\lambda_b\delta_{kj})$} = \lambda_b a_{ij}$}
  \settowidth\tlc{$j$}
  \settowidth\tlb{$\lambda_a\lambda_b\delta_{ij}$}
  \bqb
    a_{ij} &=&
      \makebox[\tlb][l]{$\lambda_a\delta_{ij}$} \ \ \mbox{unless $1 \le (i,j) \le n$}, \\
    b_{ij} &=&
      \makebox[\tlb][l]{$\lambda_b\delta_{ij}$} \ \ \mbox{unless $1 \le (i,j) \le n$}; \\
    \mbox{} [AB]_{ij} &=& \ds\sum_k a_{ik}b_{kj}
    \\&=& \begin{cases}
      \makebox[\tld][l]{$\makebox[\tla][l]{$\sum_k (\lambda_a\delta_{ik})b_{kj}$} = \lambda_a b_{ij}$}
        \ \ \mbox{unless $1 \le \makebox[\tlc][c]{$i$} \le n$} \\
      % Deliberately left the comma off the end of the last line.  It looks funny.
      \makebox[\tld][l]{$\makebox[\tla][l]{$\sum_k a_{ik}(\lambda_b\delta_{kj})$} = \lambda_b a_{ij}$}
        \ \ \mbox{unless $1 \le \makebox[\tlc][c]{$j$} \le n$} \\
    \end{cases}
    \\&=& \makebox[\tlb][l]{$\lambda_a\lambda_b \delta_{ij}$} \ \ \mbox{unless $1 \le (i,j) \le n$}.
  \eqb
  It's probably easier just to sketch the matrices and look at them, though.
}

\subsection{Other matrix forms}
\label{matrix:180.45}

Besides the dimension-limited form of \S~\ref{matrix:180.25}
and the extended-operational
form of \S~\ref{matrix:180.35}, other infinite-dimensional matrix
forms are certainly possible.  One could for example advantageously
define a ``null sparse'' form, recording only nonzero elements and their
addresses in an otherwise null matrix; or a ``tridiagonal extended''
form, bearing repeated entries not only along the main diagonal but also
along the diagonals just above and just below.  Section~\ref{matrix:340}
introduces one worthwhile matrix which fits neither the
dimension-limited nor the extended-operational form.  Still, the
dimension-limited and extended-operational forms are normally the most
useful, and they are the ones we will principally be handling in this
book.

One reason to have defined specific infinite-dimensional matrix forms is
to show how straightforwardly one can fully represent a practical matrix
of an infinity of elements by a modest, finite quantity of information.
Further reasons to have defined such forms will soon occur.

\subsection{The rank-$r$ identity matrix}
\label{matrix:180.22}
\index{identity matrix}
\index{matrix!identity}
\index{identity matrix!$r$-dimensional}
\index{identity matrix!rank-$r$}

The \emph{rank-$r$ identity matrix}~$I_r$ is the dimension-limited matrix
for which
\bq{matrix:180:IM}
  [I_r]_{ij} = \begin{cases}
    \delta_{ij} &\mbox{if $1 \le i \le r$ and/or $1 \le j \le r$,} \\
    0 &\mbox{otherwise,}
  \end{cases}
\eq
where either the ``and'' or the ``or'' can be regarded (it makes no
difference).  The effect of~$I_r$ is that
\bq{matrix:180:22}
  \begin{split}
    I_m X &= X = XI_n, \\
    I_m\ve x &= \ve x,
  \end{split}
\eq
where~$X$ is an $m\times n$ matrix and~$\ve x$, an $m\times 1$ vector.
Examples of~$I_r$ include
\[
  I_3 = \mf{rrr}{
    1&0&0 \\
    0&1&0 \\
    0&0&1
  }.
\]
(Remember that in the infinite-dimensional view,~$I_3$, though a $3
\times 3$ matrix, is formally an $\infty\times\infty$ matrix with
zeros in the unused cells.  It has only the three ones and fits the $3
\times 3$ dimension-limited form of \S~\ref{matrix:180.25}.  The areas
of~$I_3$ not shown are all zero, even along the main diagonal.)

The rank~$r$ can be any nonnegative integer, even zero (though the
rank-zero identity matrix~$I_0$ is in fact the null matrix, normally
just written~$0$).  If alternate indexing limits are needed (for
instance for a computer-indexed identity matrix whose indices run
from~$0$ to $r-1$), the notation~$I_a^b$, where
\bq{matrix:180:IMC}
  [I_a^b]_{ij} \equiv \begin{cases}
    \delta_{ij} &\mbox{if $a \le i \le b$ and/or $a \le j \le b$,} \\
    0 &\mbox{otherwise,}
  \end{cases}
\eq
can be used; the rank in this case is $r=b-a+1$, which is just the count
of ones along the matrix's main diagonal.

The name ``rank-$r$'' implies that~$I_r$ has a ``rank'' of~$r$, and
indeed it does.  For the moment, however, we will discern the attribute
of rank only in the rank-$r$ identity matrix itself.
Section~\ref{gjrank:340} defines \emph{rank} for matrices more generally.

\subsection{The truncation operator}
\label{matrix:180.23}
\index{operator!truncation}
\index{truncation operator}
\index{matrix!truncating}

The rank-$r$ identity matrix~$I_r$ is also the \emph{truncation
operator.}  Attacking from the left, as in~$I_rA$, it retains the first
through $r$th rows of~$A$ but cancels other rows.  Attacking from the
right, as in~$AI_r$, it retains the first through $r$th columns.  Such
truncation is useful symbolically to reduce an extended operator to
dimension-limited form.

\index{commutation!of the identity matrix}
\index{identity matrix!commutation of}
Whether a matrix~$C$ has dimension-limited or extended-operational form
(though not necessarily if it has some other form), if it has an $m
\times n$ active region%
\footnote{
  Refer to the definitions of \emph{active region} in
  \S\S~\ref{matrix:180.25} and~\ref{matrix:180.35}.  That a matrix has
  an $m \times n$ active region does not necessarily mean that it is
  all zero outside the $m \times n$ rectangle.  (After all, if it were
  always all zero outside, then there would be little point in applying
  a truncation operator.  There would be nothing there to truncate.)
}
and
\[
  \begin{split}
    m &\le r, \\
    n &\le r,
  \end{split}
\]
then
\bq{matrix:180:35}
  I_rC = I_rCI_r = CI_r.
\eq
For such a matrix,~(\ref{matrix:180:35}) says at least two things:
\bi
  \item
    It is superfluous to truncate both rows and columns;
    it suffices to truncate one or the other.
  \item
    The rank-$r$ identity matrix~$I_r$ commutes freely past~$C$.
\ei
Evidently big identity matrices commute freely where small ones cannot
(and the general identity matrix $I=I_{-\infty}^\infty$ commutes freely past
everything).

\subsection
  [The elementary vector and lone-element matrix]
  {The elementary vector and the lone-element matrix}
\label{matrix:180.30}
\index{lone-element matrix}
\index{matrix!lone-element}
\index{elementary vector}
\index{vector!elementary}

The \emph{lone-element matrix}~$E_{mn}$ is the matrix with
a one in the $mn$th cell and zeros elsewhere:
\bq{matrix:180:E}
  [E_{mn}]_{ij} \equiv \delta_{im}\delta_{jn} = \begin{cases}
    1 &\mbox{if $i=m$ and $j=n$,} \\
    0 &\mbox{otherwise.}
  \end{cases}
\eq
By this definition, $C = \sum_{i,j} c_{ij} E_{ij}$ for any matrix~$C$.
The vector analog of the lone-element matrix is the \emph{elementary
vector}~$\ve e_m$, which has a one as the $m$th element:
\bq{matrix:180:e}
  [\ve e_{m}]_{i} \equiv \delta_{im} = \begin{cases}
    1 &\mbox{if $i=m$,} \\
    0 &\mbox{otherwise.}
  \end{cases}
\eq
By this definition, $[I]_{*j} = \ve e_j$ and
$[I]_{i*} = \ve e_i^T$.

\subsection{Off-diagonal entries}
\label{matrix:180.60}
\index{off-diagonal entries}

It is interesting to observe and useful to note that if
\[
  \left[ C_1 \right]_{i*} =
  \left[ C_2 \right]_{i*} =
  \ve e_i^T,
\]
then also
\bq{matrix:180:61}
  \left[ C_1 C_2 \right]_{i*} =
  \ve e_i^T;
\eq
and likewise that if
\[
  \left[ C_1 \right]_{*j} =
  \left[ C_2 \right]_{*j} =
  \ve e_j,
\]
then also
\bq{matrix:180:62}
  \left[ C_1 C_2 \right]_{*j} =
  \ve e_j.
\eq
The product of matrices has off-diagonal entries in a row or column only
if at least one of the factors itself has off-diagonal entries in that
row or column.  Or, less readably but more precisely, \emph{the $i$th
row or $j$th column of the product of matrices can depart from~$\ve
e_i^T$ or~$\ve e_j$, respectively, only if the corresponding row or
column of at least one of the factors so departs.} The reason is that
in~(\ref{matrix:180:61}), $C_1$ acts as a row operator
on~$C_2$; that if $C_1$'s $i$th row is~$\ve e_i^T$, then its action is
merely to duplicate $C_2$'s $i$th row, which itself is just~$\ve
e_i^T$.  Parallel logic naturally applies to~(\ref{matrix:180:62}).

% ----------------------------------------------------------------------

\section{The elementary operator}
\label{matrix:320}
\index{elementary operator}
\index{operator!elementary}

% The tactical decision is made here to count the self-interchange as a
% valid interchange operator.  The opposite tactic is equally possible,
% and each tactic has minor, subtle advantages over the other.  A future
% revision of the book might reverse the choice.

\index{elementary operator!inverse of}
Section~\ref{matrix:120.27} has introduced the general row or column operator.
Denoted~$T$, the \emph{elementary operator} is a
simple extended row or column operator from sequences of
which more complicated extended operators can be built.  The elementary
operator~$T$ comes in three kinds.%
\footnote{
  In \S~\ref{matrix:180}, the symbol~$A$ specifically represented an
  extended operator, but here and generally the symbol represents any
  matrix.
}
\bi
  \item \index{elementary operator!interchange}
    \index{interchange operator!elementary}
    The first is the \emph{interchange elementary}
    \bq{matrix:320:Tdefxchg}
      T_{[i\lra j]} = I - (E_{ii}+E_{jj}) + (E_{ij}+E_{ji}),
    \eq
    which by operating $T_{[i\lra j]}A$ or $AT_{[i\lra j]}$ respectively
    interchanges $A$'s $i$th row or column with its $j$th.%
    \footnote{%
      \label{matrix:320:fn10}%
      As a matter of definition, some authors~\cite{Lay} forbid
      $T_{[i\lra i]}$ as an elementary operator, where $j=i$, since
      after all $T_{[i\lra i]}=I$; which is to say that the
      operator doesn't actually do anything.  There exist legitimate
      tactical reasons to forbid (as in \S~\ref{matrix:322}),
      but normally this book permits.
      %
      % --> I don't think that the rest of this footnote is needed;
      % --> delete the rest?  --THB--
      %It is good to define a concept aesthetically.  One should usually
      %do so when one can; and indeed in this case one might reasonably
      %promote either definition on aesthetic grounds.  However, an
      %applied mathematician ought not to let a mere definition entangle
      %him.  What matters is the underlying concept.  Where the
      %definition does not serve the concept well, the applied
      %mathematician considers whether it were not worth the effort to
      %adapt the definition accordingly.%
    }
  \item \index{elementary operator!scaling}
    \index{scaling operator!elementary}
    The second is the \emph{scaling elementary}
    \bq{matrix:320:Tdefsc}
      T_{\alpha[i]} = I + (\alpha-1) E_{ii}, \ \ \alpha \neq 0,
    \eq
    which by operating $T_{\alpha[i]}A$ or $AT_{\alpha[i]}$ scales
    (multiplies) $A$'s $i$th row or column, respectively, by the
    factor~$\alpha$.
  \item \index{elementary operator!addition}
    \index{addition operator!elementary}
    The third and last is the \emph{addition elementary}
    \bq{matrix:320:Tdefadd}
      T_{\alpha[ij]} = I + \alpha E_{ij}, \ \ i\neq j,
    \eq
    which by operating $T_{\alpha[ij]}A$ adds to the $i$th row of~$A$, $\alpha$
    times the $j$th row; or
    which by operating $AT_{\alpha[ij]}$ adds to the $j$th column of~$A$, $\alpha$
    times the $i$th column.
\ei
Examples of the elementary operators include
\bqb
  T_{[1\lra 2]} &=&
    \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 0 & 1 & 0 & 0 & 0 & \cdots \\
      \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 1 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
    }, \\
  T_{5[4]} &=&
    \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 1 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 5 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
    }, \\
  T_{5[21]} &=&
    \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 5 & 1 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 1 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
    }.
\eqb
Note that none of these, and in fact no elementary operator of any kind,
differs from~$I$ in more than four elements.

\subsection{Properties}
\label{matrix:320.10}

\index{invertibility!of the elementary operator}
\index{elementary operator!invertibility of}
Significantly, elementary operators as defined above are always
invertible (which is to say, reversible in effect), with
\bq{matrix:320:30}
  \begin{split}
    T_{[i\lra j]}^{-1} &= T_{[j\lra i]} = T_{[i\lra j]}, \\
    T_{\alpha[i]}^{-1} &= T_{(1/\alpha)[i]}, \\
    T_{\alpha[ij]}^{-1} &= T_{-\alpha[ij]},
  \end{split}
\eq
being themselves elementary operators such that
\bq{matrix:320:33}
  T^{-1}T = I = TT^{-1}
\eq
in each case.%
\footnote{
  The addition elementary $T_{\alpha[ii]}$ and the scaling elementary
  $T_{0[i]}$ are forbidden precisely because they are not generally invertible.
}
This means that any sequence of elementaries
$\prod_k T_k$ can safely be undone by the reverse sequence
$\coprod_k T_k^{-1}$:
\bq{matrix:320:34}
  \coprod_k T_k^{-1} \prod_k T_k = I = \prod_k T_k \coprod_k T_k^{-1}.
\eq

The rank-$r$ identity matrix~$I_r$ is no elementary operator,%
\footnote{
  If the statement seems to contradict statements of some other books,
  it is only a matter of definition.  This book finds it convenient to
  define the elementary operator in infinite-dimensional,
  extended-operational form.  The other books are not wrong; their
  underlying definitions just differ slightly.
}
nor is the lone-element matrix~$E_{mn}$; but the general
identity matrix~$I$ is indeed an elementary operator.
The last can be considered a distinct, fourth kind of elementary
operator if desired; but it is probably easier just to regard it as an
elementary of any of the first three kinds, since
$I=T_{[i\lra i]}=T_{1[i]}=T_{0[ij]}$.

From~(\ref{matrix:180:35}), we have that
\bq{matrix:320:39}
  I_rT = I_rTI_r = TI_r \ \ \mbox{if}\ 1\le i\le r\ \mbox{and}\ 1\le j\le r
\eq
for any elementary operator~$T$ which operates within the given bounds.
Equation~(\ref{matrix:320:39}) lets an identity matrix with sufficiently
high rank pass through a sequence of elementaries as needed.

In general, the transpose of an elementary row operator is the
corresponding elementary column operator.  Curiously, the interchange
elementary is its own transpose and adjoint:
\bq{matrix:320:50}
  T_{[i\lra j]}^{*} = T_{[i\lra j]} = T_{[i\lra j]}^{T}.
\eq

\subsection{Commutation and sorting}
\label{matrix:320.20}
\index{commutation!of elementary operators}
\index{elementary operator!commutation of}
\index{elementary operator!sorting of}

Elementary operators often occur in long chains like
\[
  A =
  T_{-4[32]}
  T_{[2\lra 3]}
  T_{(1/5)[3]}
  T_{(1/2)[31]}
  T_{5[21]}
  T_{[1\lra 3]},
\]
with several elementaries of all kinds intermixed.  Some applications
demand that the elementaries be sorted and grouped by kind, as
\[
  A =
 \left(
  T_{[2\lra 3]}
  T_{[1\lra 3]}
 \right)
 \left(
  T_{-4[21]}
  T_{(1/\mbox{\scriptsize 0xA})[13]}
  T_{5[23]}
 \right)
 \left(
  T_{(1/5)[1]}
 \right)
\]
or as
\[
  A =
 \left(
  T_{-4[32]}
  T_{(1/\mbox{\scriptsize 0xA})[21]}
  T_{5[31]}
 \right)
 \left(
  T_{(1/5)[2]}
 \right)
 \left(
  T_{[2\lra 3]}
  T_{[1\lra 3]}
 \right),
\]
among other possible orderings.  Though you probably cannot
tell just by looking, the three products above are different orderings
of the same elementary chain; they yield the same~$A$ and thus represent
exactly the same matrix operation.  Interesting is that
the act of reordering the elementaries has altered some of them
into other elementaries of the same kind, but has changed the kind of
none of them.

\index{commutation!hierarchy of}
\index{elementary operator!two, of different kinds}
One sorts a chain of elementary operators by repeatedly exchanging
adjacent pairs.  This of course supposes that one can exchange
adjacent pairs, which seems impossible since matrix multiplication is not
commutative: $A_1A_2 \neq A_2A_1$.  However, at the moment we are dealing in
elementary operators only; and for most pairs~$T_1$ and~$T_2$ of
elementary operators, though indeed $T_1T_2 \neq T_2T_1$, it so happens
that there exists either a~$T_1'$ such that $T_1T_2 = T_2T_1'$ or
a~$T_2'$ such that $T_1T_2 = T_2'T_1$, where~$T_1'$ and~$T_2'$ are
elementaries of the same kinds respectively as~$T_1$ and~$T_2$.  The
attempt sometimes fails when both~$T_1$ and~$T_2$ are addition
elementaries, but all other pairs commute in this way.  Significantly,
\emph{elementaries of different kinds always commute.} And, though
commutation can alter one (never both) of the two elementaries, it
changes the kind of neither.

Many qualitatively distinct pairs of elementaries exist; we will list
these exhaustively in a moment.  First, however, we should
like to observe a natural hierarchy among the three kinds of elementary:
(i)~interchange; (ii)~scaling; (iii)~addition.
\bi
  \item
    The interchange elementary is the strongest.  Itself subject to
    alteration only by another interchange elementary, it can alter any
    elementary by commuting past.  When an interchange elementary
    commutes past another elementary of any kind, what it alters are the
    other elementary's indices~$i$ and/or~$j$ (or~$m$ and/or~$n$, or
    whatever symbols happen to represent the indices in question).  When
    two interchange elementaries commute past one another, only one of
    the two is altered.  (Which one?  Either.  The mathematician
    chooses.) Refer to Table~\ref{matrix:Txchg2}.
  \item
    Next in strength is the scaling elementary.  Only an interchange
    elementary can alter it, and it in turn can alter only an addition
    elementary.  Scaling elementaries do not alter one another during
    commutation.  When a scaling elementary commutes past an addition
    elementary, what it alters is the latter's scale~$\alpha$
    (or~$\beta$, or whatever symbol happens to represent the scale in
    question).
    Refer to Table~\ref{matrix:Txchg3}.
  \item
    The addition elementary, last and weakest, is subject to alteration by
    either of the other two, itself having no power to alter any
    elementary during commutation.  A pair of addition elementaries are
    the only pair that can altogether fail to commute---they fail when
    the row index of one equals the column index of the other---but when
    they do commute, neither alters the other.
    Refer to Table~\ref{matrix:Txchg1}.
\ei
{
  \nc\tcaption[2]{
    \caption[Elementary operators: {#1}.]
    {Inverting, commuting, combining and expanding elementary operators: {#1}.
    In the table, $i\neq j\neq m\neq n$; no two indices are the same.  {#2}}
  }
  \index{elementary operator!inverse of}
  \index{elementary operator!combination of}
  \index{elementary operator!expansion of}
  Tables~\ref{matrix:Txchg2}, \ref{matrix:Txchg3}
  and~\ref{matrix:Txchg1} list all possible
  pairs of elementary operators, as the reader can check.
  The only pairs that fail to commute are the last three of
  Table~\ref{matrix:Txchg1}.
  \begin{table}
    \tcaption{interchange}{
      Notice that the effect an interchange elementary $T_{[m\lra n]}$
      has in passing any other elementary, even another interchange
      elementary, is simply to replace~$m$ by~$n$ and~$n$ by~$m$ among
      the indices of the other elementary.
    }
    \label{matrix:Txchg2}
    \index{elementary operator!interchange}
    \[
      \renewcommand\arraystretch{1.5}
      \br{rcl}
        T_{[m\lra n]} &=& T_{[n\lra m]} \\
        T_{[m\lra m]} &=& I \\
        I T_{[m\lra n]} &=& T_{[m\lra n]} I \\
        T_{[m\lra n]}T_{[m\lra n]} &=&
          T_{[m\lra n]}T_{[n\lra m]}
          = T_{[n\lra m]}T_{[m\lra n]} = I \\
        T_{[m\lra n]}T_{[i\lra n]}
          &=& T_{[i\lra n]}T_{[m\lra i]}
           =  T_{[i\lra m]}T_{[m\lra n]} \\
          &=& \left(T_{[i\lra n]}T_{[m\lra n]}\right)^2 \\
        T_{[m\lra n]}T_{[i\lra j]} &=& T_{[i\lra j]}T_{[m\lra n]} \\
        T_{[m\lra n]}T_{\alpha[m]} &=& T_{\alpha[n]}T_{[m\lra n]} \\
        T_{[m\lra n]}T_{\alpha[i]} &=& T_{\alpha[i]}T_{[m\lra n]} \\
        T_{[m\lra n]}T_{\alpha[ij]} &=& T_{\alpha[ij]}T_{[m\lra n]} \\
        T_{[m\lra n]}T_{\alpha[in]} &=& T_{\alpha[im]}T_{[m\lra n]} \\
        T_{[m\lra n]}T_{\alpha[mj]} &=& T_{\alpha[nj]}T_{[m\lra n]} \\
        T_{[m\lra n]}T_{\alpha[mn]} &=& T_{\alpha[nm]}T_{[m\lra n]}
      \er
    \]
  \end{table}
  \begin{table}
    \tcaption{scaling}{}
    \label{matrix:Txchg3}
    \index{elementary operator!scaling}
    \[
      \renewcommand\arraystretch{1.5}
      \br{rcl}
        T_{1[m]} &=& I \\
        I T_{\beta[m]} &=& T_{\beta[m]} I \\
        T_{(1/\beta)[m]}T_{\beta[m]} &=& I \\
        T_{\beta[m]}T_{\alpha[m]} &=& T_{\alpha[m]}T_{\beta[m]} = T_{\alpha\beta[m]} \\
        T_{\beta[m]}T_{\alpha[i]} &=& T_{\alpha[i]}T_{\beta[m]} \\
        T_{\beta[m]}T_{\alpha[ij]} &=& T_{\alpha[ij]}T_{\beta[m]} \\
        T_{\beta[m]}T_{\alpha\beta[im]} &=& T_{\alpha[im]}T_{\beta[m]} \\
        T_{\beta[m]}T_{\alpha[mj]} &=& T_{\alpha\beta[mj]}T_{\beta[m]}
      \er
    \]
  \end{table}
  \begin{table}
    \tcaption{addition}{The last three lines give pairs of
      addition elementaries that do not commute.}
    \label{matrix:Txchg1}
    \index{elementary operator!addition}
    \[
      \renewcommand\arraystretch{1.5}
      \br{rcl}
        T_{0[ij]} &=& I \\
        I T_{\alpha[ij]} &=& T_{\alpha[ij]} I \\
        T_{-\alpha[ij]} T_{\alpha[ij]} &=& I \\
        T_{\beta[ij]}T_{\alpha[ij]} &=&
          T_{\alpha[ij]}T_{\beta[ij]}
          = T_{(\alpha+\beta)[ij]} \\
        T_{\beta[mj]}T_{\alpha[ij]} &=& T_{\alpha[ij]}T_{\beta[mj]} \\
        T_{\beta[in]}T_{\alpha[ij]} &=& T_{\alpha[ij]}T_{\beta[in]} \\
        T_{\beta[mn]}T_{\alpha[ij]} &=& T_{\alpha[ij]}T_{\beta[mn]} \\
        T_{\beta[mi]}T_{\alpha[ij]} &=&
          T_{\alpha[ij]}T_{\alpha\beta[mj]}T_{\beta[mi]} \\
        T_{\beta[jn]}T_{\alpha[ij]} &=&
          T_{\alpha[ij]}T_{-\alpha\beta[in]}T_{\beta[jn]} \\
        T_{\beta[ji]}T_{\alpha[ij]} &\neq& T_{\alpha[ij]}T_{\beta[ji]}
      \er
    \]
  \end{table}
}

% ----------------------------------------------------------------------

\section{Inversion and similarity (introduction)}
\label{matrix:321}
\index{inversion}
\index{matrix!inversion of}
\index{similarity transformation}

\index{commutation!of matrices}
\index{matrix!commutation of}
If Tables~\ref{matrix:Txchg2}, \ref{matrix:Txchg3}
and~\ref{matrix:Txchg1} exhaustively describe the commutation of one
elementary past another elementary, then what can one write of the
commutation of an elementary past the general matrix~$A$?
With some matrix algebra,
\[
  \begin{split}
    TA &= (TA)(I) = (TA)(T^{-1}T), \\
    AT &= (I)(AT) = (TT^{-1})(AT),
  \end{split}
\]
one can write that
\bq{matrix:321:59}
  \begin{split}
    TA &= [TAT^{-1}]T, \\
    AT &= T[T^{-1}AT],
  \end{split}
\eq
where~$T^{-1}$ is given by~(\ref{matrix:320:30}).  An elementary
commuting rightward changes
% bad break
$A$ to~$TAT^{-1}$; commuting leftward, to~$T^{-1}AT$.

First encountered in \S~\ref{matrix:320}, the notation~$T^{-1}$
means the \emph{inverse} of the elementary operator~$T$, such that
\[
  T^{-1}T = I = TT^{-1}.
\]
Matrix inversion is not for elementary operators only, though.  Many
matrices~$C$ that are more general also have inverses such that
\bq{matrix:321:10}
  C^{-1}C = I = CC^{-1}.
\eq
(Do all matrices have such inverses?  No.  For example, the null matrix
has no such inverse.)  The broad question of how to invert a general
matrix~$C$, we leave for chapters~\ref{gjrank} and~\ref{mtxinv} to address.
For the moment however we should like to observe three simple rules
involving matrix inversion.

First, nothing in the logic leading to~(\ref{matrix:321:59}) actually
requires the matrix~$T$ there to be an elementary operator.  Any
matrix~$C$ for which~$C^{-1}$ is known can fill the role.  Hence,
\bq{matrix:321:60}
  \begin{split}
    CA &= [CAC^{-1}]C, \\
    AC &= C[C^{-1}AC].
  \end{split}
\eq
The transformation~$CAC^{-1}$ or~$C^{-1}AC$ is called a \emph{similarity
transformation.}  Sections~\ref{gjrank:337}
and~\ref{eigen:505}
speak further of this.

\index{inverse!of a matrix transpose or adjoint}
\index{transpose!of a matrix inverse}
\index{conjugate transpose!of a matrix inverse}
\index{adjoint!of a matrix inverse}
Second,
\settowidth\tla{$C^{-T}$}
\bq{matrix:321:70}
  \begin{split}
    \big(C^{T}\big)^{-1} &= \makebox[\tla][l]{$C^{-T}$} = \left(C^{-1}\right)^{T}, \\
    \big(C^{*}\big)^{-1} &= \makebox[\tla][l]{$C^{-*}$} = \left(C^{-1}\right)^{*},
  \end{split}
\eq
where~$C^{-*}$ is condensed notation for conjugate transposition and
inversion in either order and~$C^{-T}$ is of like style.
Equation~(\ref{matrix:321:70}) is a consequence
of~(\ref{matrix:120:30}), since for conjugate transposition
\[
  \left(C^{-1}\right)^{*} C^{*}
  = \left[CC^{-1}\right]^{*}
  = \left[I\right]^{*} = I = \left[I\right]^{*}
  = \left[C^{-1}C\right]^{*}
  = C^{*} \left(C^{-1}\right)^{*}
\]
and similarly for nonconjugate transposition.

\index{inverse!of a matrix product}
Third,
\bq{matrix:321:80}
  \left( \prod_k C_k \right)^{-1} = \coprod_k C_k^{-1}.
\eq
This rule emerges upon repeated application of~(\ref{matrix:321:10}),
which yields that
\[
  \coprod_k C_k^{-1} \prod_k C_k = I = \prod_k C_k \coprod_k C_k^{-1}.
\]

\index{matrix!rank-$r$ inverse of}
\index{rank-$r$ inverse}
A more limited form of the inverse exists than the infinite-dimensional
form of~(\ref{matrix:321:10}).  This is the rank-$r$ inverse, a
matrix $C^{-1(r)}$ such that
\bq{matrix:321:20}
  C^{-1(r)}C = I_r = CC^{-1(r)}.
\eq
The full notation $C^{-1(r)}$ is not standard and usually is not needed,
since the context usually implies the rank.  When so, one can abbreviate
the notation to~$C^{-1}$.  In either notation,~(\ref{matrix:321:70})
and~(\ref{matrix:321:80}) apply equally for the rank-$r$ inverse as for
the infinite-dimensional inverse.  Because of~(\ref{matrix:180:35}),
eqn.~(\ref{matrix:321:60}) too applies for the rank-$r$ inverse if $A$'s
active region is limited to $r \times r$.  (Section~\ref{mtxinv:230} uses
the rank-$r$ inverse to solve an exactly determined linear system.  This
is a famous way to use the inverse, with which many or most readers will
already be familiar; but before using it so in chapter~\ref{mtxinv}, we shall
first learn how to compute it reliably in chapter~\ref{gjrank}.)

Table~\ref{matrix:321:tbl} summarizes.
\begin{table}
  \caption[Matrix inversion properties.]
    {Matrix inversion properties.
      (The similarity properties work equally for $C^{-1(r)}$ as for~$C^{-1}$ if~$A$
      honors an $r \times r$ active region.  The full notation $C^{-1(r)}$
      for the rank-$r$ inverse incidentally is not standard, usually is
      not needed, and normally is not used.)}
  \label{matrix:321:tbl}
  \index{matrix!inversion properties of}
  \[
    \renewcommand\arraystretch{1.3}
    \br{rcccl}
      C^{-1}C &=& I &=& CC^{-1} \\
      C^{-1(r)}C &=& I_r &=& CC^{-1(r)} \\
      \big(C^{T}\big)^{-1} &=& C^{-T} &=& \left(C^{-1}\right)^{T} \\
      \big(C^{*}\big)^{-1} &=& C^{-*} &=& \left(C^{-1}\right)^{*}
    \er
  \]
  \bqb
    CA &=& [CAC^{-1}]C \\
    AC &=& C[C^{-1}AC] \\
    \left( \prod_k C_k \right)^{-1} &=& \coprod_k C_k^{-1}
  \eqb
\end{table}

% ----------------------------------------------------------------------

\section{Parity}
\label{matrix:322}
\index{parity}
\index{oddness}
\index{evenness}

Consider the sequence of integers or other objects $1,2,3,\ldots,n$.  By
successively interchanging pairs of the objects (any pairs, not just
adjacent pairs), one can achieve any desired permutation
(\S~\ref{drvtv:220.20}).  For example, beginning with $1,2,3,4,5$, one
can achieve the permutation $3,5,1,4,2$ by interchanging first the~$1$
and~$3$, then the~$2$ and~$5$.

\index{proper sequence}
\index{improper sequence}
\index{sequence!proper or improper}
Now contemplate all possible pairs:
\settowidth\tla{$(0,n)$}
\bqb
  \br{lllcl}
    (1,2) & (1,3) & (1,4) & \cdots & (1,n); \\
          & (2,3) & (2,4) & \cdots & (2,n); \\
          &       & (3,4) & \cdots & (3,n); \\
          &       &       & \ddots & \makebox[\tla][c]{\vdots} \\
          &       &       &        & (n-1,n).
  \er
\eqb
In a given permutation (like $3,5,1,4,2$),
some pairs will appear in proper sequence with respect to one another,
while others will appear in improper sequence.  (In $3,5,1,4,2$, the pair
$[1,2]$ appears in proper sequence in that the larger~$2$ stands to the right
of the smaller~$1$; but the pair $[1,3]$ appears in improper sequence in
that the larger~$3$ stands to the \emph{left} of the smaller~$1$.)
If~$p$ is the number of pairs which appear in improper sequence (in the
example, $p=6$), and if~$p$ is even, then we say that the permutation has
\emph{even} or \emph{positive parity;} if odd, then \emph{odd} or
\emph{negative parity.}%
\footnote{
  For readers who learned arithmetic in another language than English,
  the \emph{even} integers are $\ldots, -4, -2, 0, 2, 4, 6, \ldots$;
  the \emph{odd} integers are $\ldots, -3, -1, 1, 3, 5, 7, \ldots$\,.
}

Now consider: every interchange of adjacent elements must either
increment or decrement~$p$ by one, reversing parity.
Why?  Well, think about it.  If two elements are adjacent and
their order is correct, then interchanging falsifies the order, but
only of that pair (no other element interposes, so the interchange
affects the ordering of no other pair).  Complementarily, if the order
is incorrect, then interchanging rectifies the order.  Either way, an
adjacent interchange alters~$p$ by exactly~$\pm 1$, thus reversing
parity.

What about nonadjacent elements?  Does interchanging a pair of these
reverse parity, too?  To answer the question, let~$u$ and~$v$ represent
the two elements interchanged, with $a_1,a_2,\ldots,a_m$ the elements
lying between.  Before the interchange:
\[
  \ldots,u,a_1,a_2,\ldots,a_{m-1},a_m,v,\ldots
\]
After the interchange:
\[
  \ldots,v,a_1,a_2,\ldots,a_{m-1},a_m,u,\ldots
\]
The interchange reverses with respect to one another just
the pairs
\[
  \br{lllll}
    (u,a_1) & (u,a_2) & \cdots & (u,a_{m-1}) & (u,a_m) \\
    (a_1,v) & (a_2,v) & \cdots & (a_{m-1},v) & (a_m,v) \\
    (u,v)
  \er
\]
The number of pairs reversed is odd.  Since each reversal alters~$p$
by~$\pm 1$, the net change in~$p$ apparently also is odd, reversing
parity.  It seems that regardless of how distant the pair,
\emph{interchanging any pair of elements reverses the permutation's
parity.}

The sole exception arises when an element is interchanged with
itself.  This does not change parity, but it does not change anything
else, either, so in parity calculations we ignore it.%
\footnote{
  This is why some authors forbid self-interchanges, as explained in
  footnote~\ref{matrix:320:fn10}.
}
All other interchanges reverse parity.

We discuss parity in this, a chapter on matrices, because parity
concerns the elementary interchange operator of \S~\ref{matrix:320}.
The rows or columns of a matrix can be considered elements in a
sequence.  If so, then the interchange operator $T_{[i\lra j]}$, $i\neq
j$, acts precisely in the manner described, interchanging rows or
columns and thus reversing parity.  It follows that if $i_k\neq j_k$
and~$q$ is odd, then $\prod_{k=1}^q T_{[i_k\lra j_k]} \neq I$.
However, it is possible that $\prod_{k=1}^q T_{[i_k\lra j_k]} =
I$ if~$q$ is even.  In any event, even~$q$ implies even~$p$,
which means even (positive) parity; odd~$q$ implies odd~$p$, which means
odd (negative) parity.

We shall have more to say about parity in \S\S~\ref{matrix:325.10}
and~\ref{eigen:310}.

% ----------------------------------------------------------------------

\section{The quasielementary operator}
\label{matrix:325}
\index{quasielementary operator}
\index{operator!quasielementary}

Multiplying sequences of the elementary operators of
\S~\ref{matrix:320}, one can form much more complicated operators, which
per~(\ref{matrix:320:34}) are always invertible.  Such complicated
operators are not trivial to analyze, however, so one finds it
convenient to define an intermediate class of operators, called in this
book the \emph{quasielementary operators,} more complicated than
elementary operators but less so than arbitrary matrices.

A quasielementary operator is composed of elementaries only of a single
kind.  There are thus three kinds of quasielementary---interchange,
scaling and addition---to match the three kinds of elementary.  With
respect to interchange and scaling, any sequences of elementaries of the
respective kinds are allowed.  With respect to addition, there are some
extra rules, explained in \S~\ref{matrix:325.30}.

The three subsections which follow respectively introduce the three
kinds of quasielementary operator.

% bad break
\subsection[The general interchange operator]
  {The interchange quasielementary or general inter-\linebreak change operator}
\label{matrix:325.10}
\index{quasielementary operator!interchange}
\index{interchange quasielementary}
\index{general interchange operator}
\index{operator!general interchange}
\index{interchange operator!general}

\index{permutation matrix}
\index{permutor}
Any product~$P$ of zero or more interchange elementaries,
\bq{325:05}
  P = \prod_k T_{[i_k\lra j_k]},
\eq
constitutes an \emph{interchange quasielementary,} \emph{permutation
matrix}, \emph{permutor} or \emph{general interchange operator.}%
\footnote{
  The letter~$P$ here recalls the verb ``to permute.''
}
An example is
\[
  P = T_{[2\lra 5]}T_{[1\lra 3]} =
    \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 1 & 0 & \cdots \\
      \cdots & 0 & 1 & 0 & 0 & 0 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  }.
\]
This operator resembles~$I$ in that it has a single one in each
row and in each column, but the ones here do not necessarily run along
the main diagonal.  The effect of the operator is to shuffle the rows or
columns of the matrix it operates on, without altering any of the rows
or columns it shuffles.

By~(\ref{matrix:320:34}), (\ref{matrix:320:30}), (\ref{matrix:320:50})
and~(\ref{matrix:120:33}), the inverse of the general interchange
operator is
\bqa
  P^{-1} &=& \left( \prod_k T_{[i_k\lra j_k]} \right)^{-1}
  = \coprod_k T^{-1}_{[i_k\lra j_k]}
  \xn\\&=& \coprod_k T_{[i_k\lra j_k]}
  \xn\\&=& \coprod_k T_{[i_k\lra j_k]}^{*}
    = \left( \prod_k T_{[i_k\lra j_k]} \right)^{*}
  \xn\\&=& P^{*} = P^{T}
  \label{matrix:325:10}
\eqa
(where $P^{*}=P^T$ because~$P$ has only real elements).  The inverse,
transpose and adjoint of the general interchange operator are thus the
same:
\bq{matrix:325:11}
  P^{T}P = P^{*}P = I = P P^{*} = P P^{T}.
\eq

A significant attribute of the general interchange operator~$P$
is its parity: positive or even parity if the number of interchange
elementaries $T_{[i_k\lra j_k]}$ which compose it is even; negative
or odd parity if the number is odd.  This works precisely as described
in \S~\ref{matrix:322}.  For the purpose of parity determination, only
interchange elementaries $T_{[i_k\lra j_k]}$ for which $i_k \neq j_k$ are
counted; any $T_{[i\lra i]}=I$ noninterchanges are ignored.  Thus
the example's~$P$ above has even parity (two interchanges), as
does~$I$ itself (zero interchanges), but $T_{[i\lra j]}$ alone
(one interchange) has odd parity if $i\neq j$.  As we shall see in
\S~\ref{eigen:310}, the positive (even) and negative (odd) parities
sometimes lend actual positive and negative senses to the matrices they
describe.  The parity of the general interchange operator~$P$ concerns
us for this reason.

Parity, incidentally, is a property of the matrix~$P$ itself, not just
of the operation~$P$ represents.  No interchange quasielementary~$P$ has
positive parity as a row operator but negative as a column
operator.  The reason is that, regardless of whether one ultimately
means to use~$P$ as a row or column operator, the matrix is nonetheless
composable as a definite sequence of interchange elementaries.  It is
the number of interchanges, not the use, which determines $P$'s parity.

\subsection[The general scaling operator]
  {The scaling quasielementary or general scaling operator}
\label{matrix:325.20}
\index{quasielementary operator!scaling}
\index{scaling quasielementary}
\index{general scaling operator}
\index{operator!general scaling}
\index{scaling operator!general}

\index{diagonal matrix}
Like the interchange quasielementary~$P$ of \S~\ref{matrix:325.10}, the
\emph{scaling quasielementary,} \emph{diagonal matrix} or \emph{general
scaling operator}~$D$ consists of a product of zero or more elementary
operators, in this case elementary scaling operators:%
\footnote{
  The letter~$D$ here recalls the adjective ``diagonal.''
}
\bq{matrix:325:15}
  D = \prod_{i=-\infty}^\infty T_{\alpha_i[i]} = \coprod_{i=-\infty}^\infty T_{\alpha_i[i]}
  = \sum_{i=-\infty}^\infty \alpha_{i} E_{ii}
  = \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots &{*}& 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 &{*}& 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 &{*}& 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 &{*}& 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 &{*}& \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  }
\eq
(of course it might be that $\alpha_i = 1$, and thus that $T_{\alpha_i[i]}
= I$, for some, most or even all~$i$; however, $\alpha_i = 0$ is
forbidden by the definition of the scaling elementary).  An example is
\settowidth\tla{\fn$0$}
\[
  D = T_{-5[4]}T_{4[2]}T_{7[1]} =
    \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 7 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 4 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & \makebox[\tla][r]{$-\!5$} & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  }.
\]
This operator resembles~$I$ in that all its entries run down the
main diagonal; but these entries, though never zeros, are not necessarily
ones, either.  They are nonzero scaling factors.  The effect of the
operator is to scale the rows or columns of the matrix it operates on.

The general scaling operator is a particularly simple matrix.  Its
inverse is evidently
\bq{matrix:325:20}
  D^{-1} = \coprod_{i=-\infty}^\infty T_{(1/\alpha_i)[i]}
    = \prod_{i=-\infty}^\infty T_{(1/\alpha_i)[i]}
    = \sum_{i=-\infty}^\infty \frac{E_{ii}}{\alpha_i},
\eq
where each element down the main diagonal is individually inverted.

\index{diagonal matrix}
\index{diag notation, the}
A superset of the general scaling operator is the \emph{diagonal
matrix,} defined less restrictively that $[A]_{ij} = 0$ for $i \neq j$,
where zeros along the main diagonal are allowed.  The conventional
notation
\bqa
  \left[ \mopx{diag}\{\ve x\} \right]_{ij}
  &\equiv& \delta_{ij}x_i = \delta_{ij}x_j,
  \label{matrix:325:diag}\\
  \mopx{diag}\{\ve x\} &=& \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & x_1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & x_2 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & x_3 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & x_4 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 & x_5 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  },
  \xn
\eqa
converts a vector~$\ve x$ into a diagonal matrix.  The diagonal matrix
in general is not invertible and is no quasielementary operator, but is
sometimes useful nevertheless.

\subsection{Addition quasielementaries}
\label{matrix:325.30}
\index{quasielementary operator!addition}
\index{addition quasielementary}
\index{multitarget addition operator}
\index{operator!multitarget addition}
\index{addition operator!multitarget}

Any product of interchange elementaries (\S~\ref{matrix:325.10}), any
product of scaling elementaries (\S~\ref{matrix:325.20}), qualifies as a
quasielementary operator.  Not so, any product of addition
elementaries.  To qualify as a quasielementary, a product of elementary
addition operators must meet some additional restrictions.

\index{row addition quasielementary}
\index{addition quasielementary!row}
\index{quasielementary operator!row addition}
Four types of addition quasielementary are defined:%
\footnote{
  In this subsection the explanations are briefer than in the last two,
  but the pattern is similar.  The reader can fill in the details.
}
\bi
  \item
    \index{downward multitarget addition operator}
    \index{multitarget addition operator!downward}
    \index{operator!downward multitarget addition}
    \index{addition operator!downward multitarget}
    the \emph{downward multitarget row addition operator,}%
    \footnote{
      The letter~$L$ here recalls the adjective ``lower.''
    }
    \bqa
      L_{[j]}
        &=& \prod_{i=j+1}^\infty T_{\alpha_{ij}[ij]}
        = \coprod_{i=j+1}^\infty T_{\alpha_{ij}[ij]}
      \label{matrix:325:41}
      \\&=& I + \sum_{i=j+1}^\infty \alpha_{ij} E_{ij} \xn
      \\&=& \mf{ccccccc}{
            \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
            \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
            \cdots & 0 & 1 & 0 & 0 & 0 & \cdots \\
            \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
            \cdots & 0 & 0 &{*}& 1 & 0 & \cdots \\
            \cdots & 0 & 0 &{*}& 0 & 1 & \cdots \\
            & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
        }, \xn
    \eqa
    whose inverse is
    \bqa
      L_{[j]}^{-1}
        &=& \coprod_{i=j+1}^\infty T_{-\alpha_{ij}[ij]}
        = \prod_{i=j+1}^\infty T_{-\alpha_{ij}[ij]}
      \label{matrix:325:42}
      \\&=& I - \sum_{i=j+1}^\infty \alpha_{ij} E_{ij}
        = 2I - L_{[j]}; \xn
    \eqa
  \item
    \index{upward multitarget addition operator}
    \index{multitarget addition operator!upward}
    \index{operator!upward multitarget addition}
    \index{addition operator!upward multitarget}
    the \emph{upward multitarget row addition operator,}%
    \footnote{
      The letter~$U$ here recalls the adjective ``upper.''
    }
    \bqa
      U_{[j]}
        &=& \coprod_{i=-\infty}^{j-1} T_{\alpha_{ij}[ij]}
        = \prod_{i=-\infty}^{j-1} T_{\alpha_{ij}[ij]}
      \label{matrix:325:43}
      \\&=& I + \sum_{i=-\infty}^{j-1} \alpha_{ij} E_{ij} \xn
      \\&=& \mf{ccccccc}{
            \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
            \cdots & 1 & 0 &{*}& 0 & 0 & \cdots \\
            \cdots & 0 & 1 &{*}& 0 & 0 & \cdots \\
            \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
            \cdots & 0 & 0 & 0 & 1 & 0 & \cdots \\
            \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
            & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
        }, \xn
    \eqa
    whose inverse is
    \bqa
      U_{[j]}^{-1}
        &=& \prod_{i=-\infty}^{j-1} T_{-\alpha_{ij}[ij]}
        = \coprod_{i=-\infty}^{j-1} T_{-\alpha_{ij}[ij]}
      \label{matrix:325:44}
      \\&=& I - \sum_{i=-\infty}^{j-1} \alpha_{ij} E_{ij}
        = 2I - U_{[j]}; \xn
    \eqa
  \item
    \index{rightward multitarget addition operator}
    \index{multitarget addition operator!rightward}
    \index{operator!rightward multitarget addition}
    \index{addition operator!rightward multitarget}
    the \emph{rightward multitarget column addition operator,} which is
    the transpose~$L_{[j]}^{T}$ of the downward operator; and
  \item
    \index{leftward multitarget addition operator}
    \index{multitarget addition operator!leftward}
    \index{operator!leftward multitarget addition}
    \index{addition operator!leftward multitarget}
    the \emph{leftward multitarget column addition operator,} which is
    the transpose~$U_{[j]}^{T}$ of the upward operator.
\ei

% ----------------------------------------------------------------------

\section{The unit triangular matrix}
\label{matrix:330}
\index{unit triangular matrix}
\index{matrix!unit triangular}

\index{unit lower triangular matrix}
\index{unit upper triangular matrix}
\index{lower triangular matrix}
\index{upper triangular matrix}
\index{matrix!unit lower triangular}
\index{matrix!unit upper triangular}
Yet more complicated than the quasielementary of \S~\ref{matrix:325} is
the \emph{unit triangular matrix,} with which we draw this necessary but
tedious chapter toward a long close:
\bqa
  L &=& I
    + \sum_{i=-\infty}^\infty \, \sum_{j=-\infty}^{i-1}
      \alpha_{ij}E_{ij}
  = I
    + \sum_{j=-\infty}^\infty \, \sum_{i=j+1}^\infty
      \alpha_{ij}E_{ij}
  \label{matrix:330:11}
  \\&=& \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots &{*}& 1 & 0 & 0 & 0 & \cdots \\
      \cdots &{*}&{*}& 1 & 0 & 0 & \cdots \\
      \cdots &{*}&{*}&{*}& 1 & 0 & \cdots \\
      \cdots &{*}&{*}&{*}&{*}& 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  }; \xn\\
  U &=& I
    + \sum_{i=-\infty}^\infty \, \sum_{j=i+1}^\infty
      \alpha_{ij}E_{ij}
  = I
    + \sum_{j=-\infty}^\infty \, \sum_{i=-\infty}^{j-1}
      \alpha_{ij}E_{ij}
  \label{matrix:330:12}
  \\&=& \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 &{*}&{*}&{*}&{*}& \cdots \\
      \cdots & 0 & 1 &{*}&{*}&{*}& \cdots \\
      \cdots & 0 & 0 & 1 &{*}&{*}& \cdots \\
      \cdots & 0 & 0 & 0 & 1 &{*}& \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  }. \xn
\eqa
The former is a \emph{unit lower triangular matrix;} the latter, a
\emph{unit upper triangular matrix.}
The unit triangular matrix is a generalized addition quasielementary, which
adds not only to multiple targets but also from multiple sources---but
in one direction only: downward or leftward for~$L$ or~$U^T$
(or~$U^{*}$); upward or rightward for~$U$ or~$L^T$ (or~$L^{*}$).

% The following belonged to a deleted passage.  The author kind of liked
% the expression, though, so it is retained here, commented out.
%
%\cdots  So, this being a book of derivations,
%though derivation be tedious, derive the formulas we must, derive the
%formulas we will, as follows.

\index{triangular matrix}
\index{general triangular matrix}
\index{strictly triangular matrix}
\index{Schur, Issai (1875--1941)}
The general \emph{triangular matrix}~$L_S$ or~$U_S$, which by definition
can have any values along its main diagonal, is sometimes of interest,
as in the Schur decomposition of \S~\ref{eigen:520}.%
\footnote{
  The subscript~$S$ here stands for Schur.  Other books typically
  use the symbols~$L$ and~$U$ for the general triangular matrix of
  Schur, but this book distinguishes by the subscript.
}
The \emph{strictly triangular matrix} $L-I$ or $U-I$ is likewise
sometimes of interest, as in Table~\ref{matrix:330:t18}.%
\footnote{\cite[``Schur decomposition,'' 00:32, 30~Aug.\ 2007]{wikip}}
However, such matrices cannot in general be expressed as products of
elementary operators and this section does not treat them.

This section presents and derives the basic properties of the unit triangular
matrix.

\subsection{Construction}
\label{matrix:330.20}
\index{unit triangular matrix!construction of}
\index{triangular matrix!construction of}
\index{matrix!triangular, construction of}

To make a unit triangular matrix is straightforward:
\bq{matrix:330:20}
  \begin{split}
    L &= \coprod_{j=-\infty}^\infty L_{[j]}; \\
    U &= \prod_{j=-\infty}^\infty U_{[j]}.
  \end{split}
\eq
So long as the multiplication is done in the order indicated,%
\footnote{
  Recall again from \S~\ref{alggeo:227} that $\prod_k A_k=\cdots A_3 A_2
  A_1$, whereas $\coprod_k A_k=A_1 A_2 A_3 \cdots$.  This means
  that $(\prod_k A_k)(C)$ applies first~$A_1$, then~$A_2$, $A_3$ and so on,
  as row operators to~$C$; whereas $(C)(\coprod_k A_k)$ applies
  first~$A_1$, then~$A_2$, $A_3$ and so on, as column operators to~$C$.
  The symbols~$\prod$ and~$\coprod$ as this book uses them thus can be
  thought of respectively as row and column sequencers.
}
then conveniently,
\bq{matrix:330:25}
  \begin{split}
    \Big[L\Big]_{ij} &= \Big[L_{[j]}\Big]_{ij}, \\
    \Big[U\Big]_{ij} &= \Big[U_{[j]}\Big]_{ij},
  \end{split}
\eq
which is to say that the entries of~$L$ and~$U$ are respectively
nothing more than the relevant entries of the several~$L_{[j]}$
and~$U_{[j]}$.  Equation~(\ref{matrix:330:25}) enables one to
use~(\ref{matrix:330:20}) immediately and directly, without calculation,
to build any unit triangular matrix desired.

The correctness of~(\ref{matrix:330:25}) is most easily seen if the
several~$L_{[j]}$ and~$U_{[j]}$ are regarded as column
operators acting sequentially on~$I$:
\[
  \begin{split}
    L &= (I)\left(\coprod_{j=-\infty}^\infty L_{[j]}\right); \\
    U &= (I)\left(\prod_{j=-\infty}^\infty U_{[j]}\right).
  \end{split}
\]
The reader can construct an inductive proof symbolically on this basis
without too much difficulty if desired, but just thinking about
how~$L_{[j]}$ adds columns leftward and~$U_{[j]}$, rightward, then
considering the order in which the several~$L_{[j]}$ and~$U_{[j]}$
act,~(\ref{matrix:330:25}) follows at once.

\subsection{The product of like unit triangular matrices}
\label{matrix:330.30}

The product of like unit triangular matrices,
\bq{matrix:330:30}
  \begin{split}
    L_1 L_2 &= L, \\
    U_1 U_2 &= U,
  \end{split}
\eq
is another unit triangular matrix of the same type.  The proof for unit lower and
unit upper triangular matrices is the same.  In the unit lower triangular
case, one starts from a form of the definition of a unit lower triangular
matrix:
\[
  [L_1]_{ij}\ \mbox{or}\ [L_2]_{ij} = \begin{cases}
    0 &\mbox{if $i < j$,} \\
    1 &\mbox{if $i = j$.}
  \end{cases}
\]
Then,
\[
  [ L_1 L_2 ]_{ij} = \sum_{m=-\infty}^\infty [L_1]_{im} [L_2]_{mj}.
\]
But as we have just observed,~$[L_1]_{im}$ is null when $i < m$,
and~$[L_2]_{mj}$ is null when $m < j$.  Therefore,
\[
  [ L_1 L_2 ]_{ij} = \begin{cases}
    0 &\mbox{if $i < j$,} \\
    \sum_{m=j}^i [L_1]_{im} [L_2]_{mj} &\mbox{if $i \ge j$.}
  \end{cases}
\]
Inasmuch as this is true, nothing prevents us from weakening
the statement to read
\[
  [ L_1 L_2 ]_{ij} = \begin{cases}
    0 &\mbox{if $i < j$,} \\
    \sum_{m=j}^i [L_1]_{im} [L_2]_{mj} &\mbox{if $i = j$.}
  \end{cases}
\]
But this is just
\[
  [ L_1 L_2 ]_{ij} = \begin{cases}
    0 &\mbox{if $i < j$,} \\
    [L_1]_{ij} [L_2]_{ij} = [L_1]_{ii} [L_2]_{ii} = (1)(1) = 1 &\mbox{if $i = j$,}
  \end{cases}
\]
which again is the very definition of a unit lower triangular matrix.
Hence
% bad break
(\ref{matrix:330:30}).

\subsection{Inversion}
\label{matrix:330.40}

Inasmuch as any unit triangular matrix can be constructed from addition
quasielementaries by~(\ref{matrix:330:20}), inasmuch
as~(\ref{matrix:330:25}) supplies the specific quasielementaries, and
inasmuch as~(\ref{matrix:325:42}) or~(\ref{matrix:325:44}) gives the
inverse of each such quasielementary, one can always invert a unit triangular
matrix easily by
\bq{matrix:330:40}
  \begin{split}
    L^{-1} &= \prod_{j=-\infty}^\infty L_{[j]}^{-1}, \\
    U^{-1} &= \coprod_{j=-\infty}^\infty U_{[j]}^{-1}.
  \end{split}
\eq
In view of~(\ref{matrix:330:30}), therefore, \emph{the inverse of a unit lower
triangular matrix is another unit lower triangular matrix; and the inverse of
a unit upper triangular matrix, another unit upper triangular matrix.}

It is plain to see but still interesting to note that---unlike the
inverse---the adjoint or transpose of a unit lower triangular matrix is a
unit upper triangular matrix; and that the adjoint or transpose of a unit upper
triangular matrix is a unit lower triangular matrix.  The adjoint reverses
the sense of the triangle.

\subsection{The parallel unit triangular matrix}
\label{matrix:330.50}
\index{parallel unit triangular matrix}
\index{triangular matrix!unit parallel}
\index{matrix!unit parallel triangular}

If a unit triangular matrix fits the special, restricted form
\settoheight\tlj{{\scriptsize $k$}}
\bqa
  \settowidth\tla{$,$}
  L_\|^{\{k\}} &=& I + \sum_{\rule{0pt}{\tlj}j=-\infty}^k \, \sum_{i=k+1}^\infty \alpha_{ij} E_{ij}
  \label{matrix:330:51} \\
  &=& \mf{ccccccc}{
      \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
      \cdots&1&0&0&0&0&\cdots\\
      \cdots&0&1&0&0&0&\cdots\\
      \cdots&0&0&1&0&0&\cdots\\
      \cdots&{*}&{*}&{*}&1&0&\cdots\\
      \cdots&{*}&{*}&{*}&0&1&\cdots\\
      &\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
  }\makebox[\tla][l]{} \xn
\eqa
or
\settoheight\tlj{{\scriptsize $k$}}
\bqa
  U_\|^{\{k\}} &=& I + \sum_{j=k}^{\infty} \, \sum_{\rule{0pt}{\tlj}i=-\infty}^{k-1} \alpha_{ij} E_{ij}
  \label{matrix:330:52} \\
  &=& \mf{ccccccc}{
      \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
      \cdots&1&0&{*}&{*}&{*}&\cdots\\
      \cdots&0&1&{*}&{*}&{*}&\cdots\\
      \cdots&0&0&1&0&0&\cdots\\
      \cdots&0&0&0&1&0&\cdots\\
      \cdots&0&0&0&0&1&\cdots\\
      &\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
  }\makebox[\tla][l]{,} \xn
\eqa
confining its nonzero elements to a rectangle within the triangle as
shown, then it is a \emph{parallel unit triangular matrix} and has some
special properties the general unit triangular matrix lacks.

\index{source}
\index{target}
\index{frontier}
The general unit lower triangular matrix~$L$ acting~$LA$ on a matrix~$A$ adds
the rows of~$A$ downward.  The parallel unit lower triangular
matrix $L_\|^{\{k\}}$ acting $L_\|^{\{k\}}A$ also adds rows downward,
but with the useful restriction that it makes no row of~$A$ both source
and target.  The addition is \emph{from} $A$'s rows through the $k$th,
\emph{to} $A$'s $(k+1)$th row onward.  A horizontal frontier separates
source from target, which thus march in~$A$ as separate squads.

Similar observations naturally apply with respect to the parallel unit upper
triangular matrix $U_\|^{\{k\}}$, which acting $U_\|^{\{k\}}A$ adds rows
upward, and also with respect to $L_\|^{\{k\}T}$ and $U_\|^{\{k\}T}$,
which acting $AL_\|^{\{k\}T}$ and $AU_\|^{\{k\}T}$ add columns
respectively rightward and leftward (remembering that $L_\|^{\{k\}T}$ is
no unit lower but a unit upper triangular matrix; that $U_\|^{\{k\}T}$ is the
lower).  Each separates source from target in the matrix~$A$ it operates
on.

The reason we care about the separation of source from target is that,
in matrix arithmetic generally, where source and target are not
separate but remain intermixed, the sequence matters in which rows or
columns are added.  That is, in general,
\[
  T_{\alpha_1[i_1j_1]}T_{\alpha_2[i_2j_2]}
  \neq I + \alpha_1E_{i_1j_1} + \alpha_2E_{i_2j_2}
  \neq T_{\alpha_2[i_2j_2]}T_{\alpha_1[i_1j_1]}.
\]
It makes a difference whether the one addition comes before, during or
after the other---but only because the target of the one addition might
be the source of the other.  The danger is that $i_1=j_2$ or $i_2=j_1$.
Remove this danger, and the sequence ceases to matter (refer to
Table~\ref{matrix:Txchg1}).

That is exactly what the parallel unit triangular matrix does: it separates
source from target and thus removes the danger.  It is for this reason
that the parallel unit triangular matrix brings the useful property that
\settoheight\tlj{{\scriptsize $k$}}
\bq{matrix:330:55}
  \begin{split}
    L_\|^{\{k\}} &= I + \sum_{\rule{0pt}{\tlj}j=-\infty}^k \, \sum_{i=k+1}^\infty \alpha_{ij} E_{ij} \\
      &= \coprod_{\rule{0pt}{\tlj}j=-\infty}^k \,   \prod_{i=k+1}^\infty T_{\alpha_{ij}[ij]}
       = \coprod_{\rule{0pt}{\tlj}j=-\infty}^k \, \coprod_{i=k+1}^\infty T_{\alpha_{ij}[ij]} \\
      &=   \prod_{\rule{0pt}{\tlj}j=-\infty}^k \,   \prod_{i=k+1}^\infty T_{\alpha_{ij}[ij]}
       =   \prod_{\rule{0pt}{\tlj}j=-\infty}^k \, \coprod_{i=k+1}^\infty T_{\alpha_{ij}[ij]} \\
      &=   \prod_{i=k+1}^\infty \, \coprod_{\rule{0pt}{\tlj}j=-\infty}^k T_{\alpha_{ij}[ij]}
       = \coprod_{i=k+1}^\infty \, \coprod_{\rule{0pt}{\tlj}j=-\infty}^k T_{\alpha_{ij}[ij]} \\
      &=   \prod_{i=k+1}^\infty \,   \prod_{\rule{0pt}{\tlj}j=-\infty}^k T_{\alpha_{ij}[ij]}
       = \coprod_{i=k+1}^\infty \,   \prod_{\rule{0pt}{\tlj}j=-\infty}^k T_{\alpha_{ij}[ij]}, \\
    U_\|^{\{k\}} &= I + \sum_{j=k}^\infty \, \sum_{\rule{0pt}{\tlj}i=-\infty}^{k-1} \alpha_{ij} E_{ij} \\
      &=   \prod_{j=k}^\infty \, \coprod_{\rule{0pt}{\tlj}i=-\infty}^{k-1} T_{\alpha_{ij}[ij]}
       = \cdots,
  \end{split}
\eq
which says that one can build a parallel unit triangular matrix equally well
in any sequence---in contrast to the case of the general unit triangular
matrix, whose construction per~(\ref{matrix:330:20}) one must sequence
carefully.  (Though eqn.~\ref{matrix:330:55} does not show them,
even more sequences are possible.  You can scramble the factors'
ordering any random way you like.  The multiplication is
fully commutative.)  Under such conditions, the inverse of
the parallel unit triangular matrix is particularly simple:%
\footnote{
  There is some odd parochiality at play in applied mathematics when one
  calls such collections of symbols as~(\ref{matrix:330:56})
  ``particularly simple.''  Nevertheless, in the present context the
  idea~(\ref{matrix:330:56}) represents is indeed simple: that one can
  multiply constituent elementaries in any order and still reach the
  same parallel unit triangular matrix; that the elementaries in this case do
  not interfere.
}
\settoheight\tlj{{\scriptsize $k$}}
\bq{matrix:330:56}
  \begin{split}
    L_\|^{\{k\}\,\mbox{\scriptsize$-1$}} &= I - \sum_{\rule{0pt}{\tlj}j=-\infty}^k \, \sum_{i=k+1}^\infty \alpha_{ij} E_{ij}
        = 2I - L_\|^{\{k\}} \\
      &= \prod_{\rule{0pt}{\tlj}j=-\infty}^k \, \coprod_{i=k+1}^\infty T_{-\alpha_{ij}[ij]}
       = \cdots, \\
    U_\|^{\{k\}\,\mbox{\scriptsize$-1$}} &= I - \sum_{j=k}^{\infty} \, \sum_{\rule{0pt}{\tlj}i=-\infty}^{k-1} \alpha_{ij} E_{ij}
        = 2I - U_\|^{\{k\}} \\
      &= \coprod_{j=k}^{\infty} \, \prod_{\rule{0pt}{\tlj}i=-\infty}^{k-1} T_{-\alpha_{ij}[ij]}
       = \cdots,
  \end{split}
\eq
where again the elementaries can be multiplied in any order.
Pictorially,
{
  \settowidth\tla{\fn${0}$}
  \nc\xx{\makebox[\tla][r]{$-{*}$}}
  \bqb
    L_\|^{\{k\}\,\mbox{\scriptsize$-1$}} &=& \mf{ccccccc}{
        \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
        \cdots&\,1\,&\,0\,&\,0\,&\,0\,&\,0\,&\cdots\\
        \cdots&0&1&0&0&0&\cdots\\
        \cdots&0&0&1&0&0&\cdots\\
        \cdots&\xx&\xx&\xx&1&0&\cdots\\
        \cdots&\xx&\xx&\xx&0&1&\cdots\\
        &\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
    }, \\
    U_\|^{\{k\}\,\mbox{\scriptsize$-1$}} &=& \mf{ccccccc}{
        \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
        \cdots&1&0&\xx&\xx&\xx&\cdots\\
        \cdots&0&1&\xx&\xx&\xx&\cdots\\
        \cdots&0&0&1&0&0&\cdots\\
        \cdots&0&0&0&1&0&\cdots\\
        \cdots&\,0\,&\,0\,&\,0\,&\,0\,&\,1\,&\cdots\\
        &\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
    }.
  \eqb
}%
The inverse of a parallel unit triangular matrix is just the matrix itself,
only with each element off the main diagonal negated.
Table~\ref{matrix:330:t18} records a few properties that come
immediately of the last observation and from the parallel unit triangular
matrix's basic layout.
% This book source has many idiosyncratic labels, but ":t18"?  Well, the
% table was originally equation ":18", by after several revisions it
% had grown too long and became Table ":t18".  Thus the label.  In
% general, the source's idiosyncratic labels---like this comment, never
% seen by the reader---are slightly annoying but not nearly annoying
% enough to merit a labor-intensive general revision.  Really good
% labels would have given a clue of what their equations are about, but
% *exactly* what an equation is about is sometimes not entirely clear to
% the author until he has typeset the equation and the surrounding
% narrative.  More importantly, revisions of the text often leave
% meaningful labels obsolete.  No.  Experience teaches that in a
% document of this size, meaningless numeric labels can be more
% practical for most numbered equations, and are acceptable for many
% tables, figures, etc., as well.  Experience also teaches that it is
% usually not a good idea to risk revising labels after the labels have
% been assigned, because every label revision risks breaking---or,
% worse, mislinking---some forgotten reference from somewhere else in
% the book.  In this, a document source differs from a program:
% programs benefit from tighter organization.
\begin{table}
  \caption[Properties of the parallel unit triangular matrix.]{
    Properties of the parallel unit triangular matrix.
    (In the table, the notation~$I_a^b$ represents the generalized
    dimension-limited identity matrix or truncator
    of eqn.~\ref{matrix:180:IMC}.  Note that the inverses
    $L_\|^{\{k\}\,\mbox{\scriptsize$-1$}} = L_\|^{\{k\}'}$ and
    $U_\|^{\{k\}\,\mbox{\scriptsize$-1$}} = U_\|^{\{k\}'}$ are parallel
    unit triangular matrices themselves, such that the table's properties
    hold for them, too.)
  }
  \label{matrix:330:t18}
  \index{parallel unit triangular matrix!properties of}
  \index{unit triangular matrix!parallel, properties of}
  \index{triangular matrix!parallel, properties of}
  \index{matrix!parallel unit triangular, properties of}
  \[
    \frac{
      L_\|^{\{k\}} + L_\|^{\{k\}\,\mbox{\scriptsize$-1$}}
    }{2}
    = I =
    \frac{
      U_\|^{\{k\}} + U_\|^{\{k\}\,\mbox{\scriptsize$-1$}}
    }{2}
  \]
  \[
    \renewcommand\arraystretch{1.3}
    \br{rcccl}
      \ds I_{k+1}^\infty L_\|^{\{k\}} I_{-\infty}^k
        &=& \ds L_\|^{\{k\}}-I
        &=& \ds I_{k+1}^\infty (L_\|^{\{k\}}-I) I_{-\infty}^k \\
      \ds I_{-\infty}^{k-1} U_\|^{\{k\}} I_k^\infty
        &=& \ds U_\|^{\{k\}}-I
        &=& \ds I_{-\infty}^{k-1} (U_\|^{\{k\}}-I) I_k^\infty
    \er
  \]
  \nc\xxa{If $L_\|^{\{k\}}$ honors an $n \times n$ active region, then}
  \nc\xxb{$(I_n-I_k) L_\|^{\{k\}} I_k = L_\|^{\{k\}}-I = (I_n-I_k) (L_\|^{\{k\}}-I) I_k$}
  \nc\xxe{and $(I-I_n)(L_\|^{\{k\}}-I) = 0 = (L_\|^{\{k\}}-I)(I-I_n)$.}
  \nc\xxc{If $U_\|^{\{k\}}$ honors an $n \times n$ active region, then}
  \nc\xxd{$I_{k-1} U_\|^{\{k\}} (I_n-I_{k-1}) = U_\|^{\{k\}}-I = I_{k-1} (U_\|^{\{k\}}-I) (I_n-I_{k-1})$}
  \nc\xxf{and $(I-I_n)(U_\|^{\{k\}}-I) = 0 = (U_\|^{\{k\}}-I)(I-I_n)$.}
  \settowidth\tla{\xxc}
  \settowidth\tlb{\xxd}
  \settowidth\tlc{\xxf}
  \[
    \renewcommand\arraystretch{1.3}
    \br{l}
      \makebox[\tla][l]{\xxa}
      \\\ \ \ \ \makebox[\tlb][l]{\xxb}
      \\\ \ \ \ \makebox[\tlc][l]{\xxe}
    \er
  \]
  \[
    \renewcommand\arraystretch{1.3}
    \br{l}
      \makebox[\tla][l]{\xxc}
      \\\ \ \ \ \makebox[\tlb][l]{\xxd}
      \\\ \ \ \ \makebox[\tlc][l]{\xxf}
    \er
  \]
\end{table}

\subsection{The partial unit triangular matrix}
\label{matrix:330.05}
\index{matrix!unit triangular, partial}
\index{unit triangular matrix!partial}
\index{triangular matrix!partial}
\index{partial unit triangular matrix}

Besides the notation~$L$ and~$U$ for the general unit lower and unit upper
triangular matrices and the notation~$L_\|^{\{k\}}$ and~$U_\|^{\{k\}}$
for the parallel unit lower and unit upper triangular matrices, we shall find it
useful to introduce the additional notation
\settoheight\tlj{{\scriptsize $k$}}
\bqa
  L^{[k]} &=& I
    + \sum_{j=k}^\infty \, \sum_{\rule{0pt}{\tlj}i=j+1}^\infty
      \alpha_{ij}E_{ij}
  \label{matrix:330:13}
  \\&=& \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 1 & 0 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 &{*}& 1 & 0 & \cdots \\
      \cdots & 0 & 0 &{*}&{*}& 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  }, \xn\\
  U^{[k]} &=& I
    + \sum_{j=-\infty}^k \, \sum_{i=-\infty}^{j-1}
      \alpha_{ij}E_{ij}
  \label{matrix:330:14}
  \\&=& \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 &{*}&{*}& 0 & 0 & \cdots \\
      \cdots & 0 & 1 &{*}& 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 1 & 0 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 1 & 0 & \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  } \xn
\eqa
for unit triangular matrices whose off-diagonal content is confined to a
narrow wedge and
\settoheight\tlj{{\scriptsize $k$}}
\bqa
  \xn \\
  L^{\{k\}} &=& I
    + \sum_{j=-\infty}^k \, \sum_{i=j+1}^\infty
      \alpha_{ij}E_{ij}
  \label{matrix:330:15}
  \\&=& \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 & 0 & 0 & 0 & 0 & \cdots \\
      \cdots &{*}& 1 & 0 & 0 & 0 & \cdots \\
      \cdots &{*}&{*}& 1 & 0 & 0 & \cdots \\
      \cdots &{*}&{*}&{*}& 1 & 0 & \cdots \\
      \cdots &{*}&{*}&{*}& 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  }, \xn\\
  U^{\{k\}} &=& I
    + \sum_{j=k}^\infty \, \sum_{\rule{0pt}{\tlj}i=-\infty}^{j-1}
      \alpha_{ij}E_{ij}
  \label{matrix:330:16}
  \\&=& \mf{ccccccc}{
      \ddots & \vdots & \vdots & \vdots & \vdots & \vdots & \\
      \cdots & 1 & 0 &{*}&{*}&{*}& \cdots \\
      \cdots & 0 & 1 &{*}&{*}&{*}& \cdots \\
      \cdots & 0 & 0 & 1 &{*}&{*}& \cdots \\
      \cdots & 0 & 0 & 0 & 1 &{*}& \cdots \\
      \cdots & 0 & 0 & 0 & 0 & 1 & \cdots \\
      & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
  } \xn
\eqa
for the supplementary forms.%
\footnote{
  The notation is arguably imperfect in that $L^{\{k\}}+L^{[k]}-I \neq
  L$ but rather that $L^{\{k\}}+L^{[k+1]}-I = L$.  The conventional
  notation $\sum_{k=a}^b f(k) + \sum_{k=b}^c f(k) \neq \sum_{k=a}^c
  f(k)$ suffers the same arguable imperfection.
}
Such notation is not standard in the literature, but
it serves a purpose in this book and is introduced here for this reason.
If names are needed for $L^{[k]}$, $U^{[k]}$, $L^{\{k\}}$ and
$U^{\{k\}}$, the former pair can be called \emph{minor partial
unit triangular matrices,} and the latter pair, \emph{major partial
unit triangular matrices.}  Whether minor or major, the partial unit triangular
matrix is a matrix which leftward or rightward of the $k$th column
resembles~$I$.  Of course partial unit triangular matrices which
resemble~$I$ above or below the $k$th \emph{row} are equally
possible, and can be denoted $L^{[k]T}$, $U^{[k]T}$, $L^{\{k\}T}$ and
$U^{\{k\}T}$.

Observe that the parallel unit triangular matrices~$L_\|^{\{k\}}$
and~$U_\|^{\{k\}}$ of \S~\ref{matrix:330.50} are in fact also major partial
unit triangular matrices, as the notation suggests.

% ----------------------------------------------------------------------

\section{The shift operator}
\label{matrix:340}
\index{shift operator}

Not all useful matrices fit the dimension-limited and
extended-operational forms of \S~\ref{matrix:180}.  An exception is the
\emph{shift operator}~$H_k$, defined that
\bq{matrix:340:10}
  [H_k]_{ij} = \delta_{i(j+k)}.
\eq
For example,
\[
  H_2 = \mf{ccccccccc}{
    &\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
    \cdots&0&0&0&0&0&0&0&\cdots\\
    \cdots&0&0&0&0&0&0&0&\cdots\\
    \cdots&1&0&0&0&0&0&0&\cdots\\
    \cdots&0&1&0&0&0&0&0&\cdots\\
    \cdots&0&0&1&0&0&0&0&\cdots\\
    \cdots&0&0&0&1&0&0&0&\cdots\\
    \cdots&0&0&0&0&1&0&0&\cdots\\
    &\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&
  }.
\]
Operating~$H_kA$, $H_k$ shifts $A$'s rows downward~$k$ steps.
Operating~$AH_k$, $H_k$ shifts $A$'s columns leftward~$k$ steps.
Inasmuch as the shift operator shifts all rows or columns of the matrix
it operates on, its active region is $\infty \times \infty$ in extent.
Obviously, the shift operator's inverse, transpose and adjoint are
the same:
\bq{matrix:340:60}
  \renewcommand\arraystretch{1.3}
  \br{c}
    H_k^T H_k = H_k^{*} H_k
      = I = H_k H_k^{*} = H_k H_k^T, \\
    H_k^{-1} = H_k^{T} = H_k^{*} = H_{-k}.
  \er
\eq
Further obvious but useful identities include that
\bq{matrix:340:61}
  \begin{split}
    (I_\ell-I_k)H_k &= H_kI_{\ell-k}, \\
    H_{-k}(I_\ell-I_k) &= I_{\ell-k}H_{-k}.
  \end{split}
\eq

% ----------------------------------------------------------------------

\section{The Jacobian derivative}
\label{matrix:350}
\index{Jacobian derivative}
\index{derivative!Jacobian}
\index{Jacobi, Carl Gustav Jacob (1804--1851)}

Chapter~\ref{drvtv} has introduced the derivative of a function with respect to
a scalar variable.  One can also take the derivative of a function with respect
to a vector variable, and the function itself can be vector-valued.
The derivative is
\bq{matrix:350:Jacobian}
  \left[\frac{d\ve f}{d\ve x}\right]_{ij} = \frac{\partial f_i}{\partial x_j}.
\eq
For instance, if~$\ve x$ has three elements and~$\ve f$ has two, then
\[
  \renewcommand\arraystretch{2.0}
  \frac{d\ve f}{d\ve x} = \mf{ccc}{
    \ds\frac{\partial f_1}{\partial x_1} & \ds\frac{\partial f_1}{\partial x_2} & \ds\frac{\partial f_1}{\partial x_3} \\
    \ds\frac{\partial f_2}{\partial x_1} & \ds\frac{\partial f_2}{\partial x_2} & \ds\frac{\partial f_2}{\partial x_3}
  }.
\]
This is called the \emph{Jacobian derivative,} the \emph{Jacobian
matrix,} or just the \emph{Jacobian.}%
\footnote{\cite[``Jacobian,'' 00:50, 15 Sept.\ 2007]{wikip}}
Each of its columns is the derivative with respect to one element
of~$\ve x$.

The Jacobian derivative of a vector with respect to itself is
\bq{matrix:350:Jacobian-self}
  \frac{d\ve x}{d\ve x} = I.
\eq
The derivative is not~$I_n$ as one might think, because, even if~$\ve x$
has only~$n$ elements, still, one could vary $x_{n+1}$ in principle, and
$\partial x_{n+1}/\partial x_{n+1} \neq 0$.

\index{derivative!product rule for}
\index{product rule, derivative}
The Jacobian derivative obeys the derivative product
rule~(\ref{drvtv:prod2}) in the form%
\footnote{
  Notice that the last term on~(\ref{matrix:350:Jacobian-prod})'s second
  line is transposed, not adjointed.
}
\settowidth\tla{\scriptsize${*}$}
\bq{matrix:350:Jacobian-prod}
  \begin{split}
    \frac{d}{d\ve x}\big(\ve g^{\makebox[\tla][l]{\scriptsize${T}$}} A \ve f\big)
    &= \Bigg[ \ve g^{\makebox[\tla][l]{\scriptsize${T}$}} A \left(\frac{d\ve f}{d\ve x}\right) \Bigg]
     + \Bigg[ \left(\frac{d\ve g}{d\ve x}\right)^{\makebox[\tla][l]{\scriptsize${T}$}} A \ve f \Bigg]^{T},
    \\
    \frac{d}{d\ve x}\big(\ve g^{\makebox[\tla][l]{\scriptsize${*}$}} A \ve f\big)
    &= \Bigg[ \ve g^{\makebox[\tla][l]{\scriptsize${*}$}} A \left(\frac{d\ve f}{d\ve x}\right) \Bigg]
     + \Bigg[ \left(\frac{d\ve g}{d\ve x}\right)^{\makebox[\tla][l]{\scriptsize${*}$}} A \ve f \Bigg]^{T},
  \end{split}
\eq
valid for any constant matrix~$A$---as is seen by applying the
definition~(\ref{drvtv:defz}) of the derivative, which here is
\[
  \frac{\partial \left(\ve g^{*} A \ve f\right)}{\partial x_j}
  = \lim_{\partial x_j\rightarrow 0}
  \frac{
     (\ve g + \partial \ve g/2)^{*}A(\ve f + \partial \ve f/2)
    -(\ve g - \partial \ve g/2)^{*}A(\ve f - \partial \ve f/2)
  }{\partial x_j},
\]
and simplifying.

% ----------------------------------------------------------------------

The shift operator of \S~\ref{matrix:340} and the Jacobian derivative of
this section complete the family of matrix rudiments we shall need to
begin to do increasingly interesting things with matrices in
chapters~\ref{mtxinv} and~\ref{eigen}.  Before doing interesting things,
however, we must treat two more foundational matrix matters.  The two
are the Gauss-Jordan decomposition and the matter of matrix rank, which
will be the subjects of chapter~\ref{gjrank}, next.

