% ----------------------------------------------------------------------

\chapter{Inversion and orthonormalization}
\label{mtxinv}
\index{inversion}
\index{orthonormalization}
\index{matrix!inversion of}

The undeniably tedious chapters~\ref{matrix} and~\ref{gjrank} have piled the
matrix theory deep while affording scant practical reward.  Building
upon the two tedious chapters, this chapter brings the first rewarding
matrix work.

\index{solution}
One might be forgiven for forgetting after so many pages of abstract
theory that the matrix afforded any reward or had any use at all.  Uses
however it has.  Sections~\ref{matrix:120.10} and~\ref{gjrank:340.24}
have already broached%
\footnote{
  The reader who has skipped chapter~\ref{gjrank} might at least review
  \S~\ref{gjrank:340.24}.
}
the matrix's most basic use, the primary subject of this chapter, to
represent a system of~$m$ linear scalar equations in~$n$ unknowns neatly
as
\[
  A \ve x = \ve b
\]
and to solve the whole system at once by inverting the matrix~$A$ that
characterizes it.

Now, before we go on, we want to confess that such a use alone, on the
surface of it---though interesting---might not have justified the whole
uncomfortable bulk of chapters~\ref{matrix} and~\ref{gjrank}.  We already
knew how to solve a simultaneous system of linear scalar equations in
principle without recourse to the formality of a matrix, after all, as
in the last step to derive~(\ref{trig:250:20}) as far back as
chapter~\ref{trig}.  Why should we have suffered two bulky chapters, if only
to prepare to do here something we already knew how to do?

The question is a fair one, but admits at least four answers.  First,
the matrix neatly solves a linear system not only for a particular
driving vector~$\ve b$ but for all possible driving vectors~$\ve b$ at
one stroke, as this chapter explains.  Second and yet more impressively,
the matrix allows \S~\ref{mtxinv:320} to introduce the
\emph{pseudoinverse} to approximate the solution to an unsolvable linear
system and, moreover, to do so both optimally and efficiently, whereas
such overdetermined systems arise commonly in applications.  Third, to
solve the linear system neatly is only the primary and most
straightforward use of the matrix, not its only use: the even more
interesting eigenvalue and its incidents await chapter~\ref{eigen}.  Fourth,
specific applications aside, one should never underestimate the blunt
practical benefit of reducing an arbitrarily large grid of scalars to a
single symbol~$A$, which one can then manipulate by known algebraic
rules.  Most students first learning the matrix have probably wondered at this
stage whether it were worth all the tedium; so, if the reader now
wonders, then he stands in good company.  The matrix finally begins to
show its worth here.

The chapter opens in \S~\ref{mtxinv:220} by inverting the square matrix
to solve the exactly determined, $n \times n$ linear system in
\S~\ref{mtxinv:230}.  It continues in \S~\ref{mtxinv:245} by computing
the rectangular matrix's kernel to solve the nonoverdetermined, $m
\times n$ linear system in \S~\ref{mtxinv:240}.  In \S~\ref{mtxinv:320},
it brings forth the aforementioned pseudoinverse, which rightly
approximates the solution to the unsolvable overdetermined linear
system.  After briefly revisiting the Newton-Raphson iteration in
\S~\ref{mtxinv:420}, it concludes by introducing the concept and
practice of vector orthonormalization in \S\S~\ref{mtxinv:445}
through~\ref{mtxinv:465}.

% ----------------------------------------------------------------------

\section{Inverting the square matrix}
\label{mtxinv:220}
\index{matrix!square}
\index{square matrix}
\index{matrix!inversion of}
\index{inversion}
\index{inverse!rank-$r$}
\index{rank-$r$ inverse}

Consider an $n \times n$ square matrix~$A$ of full rank $r=n$.  Suppose
that extended operators~$G_>$, $G_<$, $G_>^{-1}$ and~$G_<^{-1}$ can be
found, each with an $n \times n$ active region (\S~\ref{matrix:180.35}),
such that%
\footnote{
  The symbology and associated terminology might disorient a reader who
  had skipped chapters~\ref{matrix} and~\ref{gjrank}.  In this book, the
  symbol~$I$ theoretically represents an $\infty \times \infty$ identity
  matrix.  Outside the $m \times m$ or $n \times n$ square, the
  operators~$G_>$ and~$G_<$ each resemble the $\infty \times \infty$
  identity matrix~$I$, which means that the operators affect
  respectively only the first~$m$ rows or~$n$ columns of the thing they
  operate on.  (In the present section it happens that $m=n$ because the
  matrix~$A$ of interest is square, but this footnote uses both
  symbols because generally $m \neq n$.)

  The symbol~$I_r$ contrarily represents an identity matrix of only~$r$
  ones, though it too can be viewed as an $\infty \times \infty$ matrix
  with zeros in the unused regions.  If interpreted as an $\infty
  \times \infty$ matrix, the matrix~$A$ of the $m \times n$ system $A\ve
  x = \ve b$ has nonzero content only within the $m \times n$ rectangle.

  None of this is complicated, really.  Its purpose is merely to
  separate the essential features of a reversible operation like~$G_>$ or~$G_<$
  from the dimensionality of the vector or matrix on which the operation
  happens to operate.  The definitions do however necessarily, slightly
  diverge from definitions the reader may have been used to seeing in
  other books.  In this book, one can legally multiply any two matrices,
  because all matrices are theoretically $\infty \times \infty$, anyway
  (though whether it makes any sense in a given circumstance to multiply
  mismatched matrices is another question; sometimes it does make sense,
  as in eqns.~\ref{mtxinv:240:50} and~\ref{eigen:600:SVD},
  but more often it does not---which naturally is why the other books
  tend to forbid such multiplication).

  To the extent that the definitions confuse, the reader might briefly
  review the earlier chapters, especially \S~\ref{matrix:180}.
}
\bq{mtxinv:220:10}
  \begin{split}
    G_>^{-1}G_> &= I = G_>G_>^{-1}, \\
    G_<^{-1}G_< &= I = G_<G_<^{-1}, \\
    A &= G_>I_nG_<.
  \end{split}
\eq
Observing from~(\ref{matrix:180:35})
that
\[
  \renewcommand\arraystretch{1.3}
  \setlength\arraycolsep{0.50\arraycolsep} % normally would be 0.30
  \br{rcccl}
    I_nA &=& A &=& AI_n, \\
    I_nG_<^{-1}G_>^{-1} &=& G_<^{-1}I_nG_>^{-1} &=& G_<^{-1}G_>^{-1}I_n,
  \er
\]
we find by successive steps that
\bqb
  A &=& G_>I_nG_<, \\
  I_nA &=& G_>G_<I_n, \\
  G_<^{-1}G_>^{-1}I_nA &=& I_n, \\
  (G_<^{-1}I_nG_>^{-1})(A) &=& I_n;
\eqb
or alternately that
\bqb
  A &=& G_>I_nG_<, \\
  AI_n &=& I_nG_>G_<, \\
  AI_nG_<^{-1}G_>^{-1} &=& I_n, \\
  (A)(G_<^{-1}I_nG_>^{-1}) &=& I_n.
\eqb
Either way, we have that
\bq{mtxinv:220:20}
  \begin{split}
    A^{-1}A &= I_n = AA^{-1}, \\
    A^{-1} &\equiv G_<^{-1}I_nG_>^{-1}.
  \end{split}
\eq
Of course, for this to work,~$G_>$, $G_<$, $G_>^{-1}$ and~$G_<^{-1}$ must
exist, be known and honor $n \times n$ active regions, which might seem
a practical hurdle.  However,~(\ref{gjrank:341:GJ}),
(\ref{gjrank:341:GJinv}) and the body of \S~\ref{gjrank:341} have shown
exactly how to find just such a~$G_>$, $G_<$, $G_>^{-1}$ and~$G_<^{-1}$
for any square matrix~$A$ of full rank, without exception; so, there is
no trouble here.  The factors do exist, and indeed we know how to find
them.

Equation~(\ref{mtxinv:220:20}) features the important matrix~$A^{-1}$,
the \emph{rank-$n$ inverse} of~$A$.

We have not yet much studied the
rank-$n$ inverse, but have at least defined it in~(\ref{matrix:321:20}),
where we gave it the fuller, nonstandard notation $A^{-1(n)}$.  When
naming the rank-$n$ inverse in words one usually says simply, ``the
inverse,'' because the rank is implied by the size of the square active
region of the matrix inverted; but the rank-$n$ inverse
from~(\ref{matrix:321:20}) is not quite the infinite-dimensional inverse
from~(\ref{matrix:321:10}), which is what~$G_>^{-1}$ and~$G_<^{-1}$ are.
According to~(\ref{mtxinv:220:20}), the product of~$A^{-1}$
and~$A$---or, written more fully, the product of $A^{-1(n)}$
and~$A$---is, not~$I$, but~$I_n$.

\index{inverse!existence of}
\index{inverse!uniqueness of}
\index{inverse!mutual}
\index{noninvertibility}
\index{reciprocal pair}
Properties that emerge from~(\ref{mtxinv:220:20}) include the following.
\bi
  \item
    Like~$A$, the rank-$n$ inverse~$A^{-1}$ (more fully written as
    $A^{-1(n)}$) too is an $n \times n$ square matrix of full rank $r=n$.
  \item
    Since~$A$ is square and has full rank (\S~\ref{gjrank:340.25}), its
    rows and, separately, its columns are linearly independent, so it
    has only the one, unique inverse~$A^{-1}$.  No other rank-$n$
    inverse of~$A$ exists.
  \item
    On the other hand, inasmuch as~$A$ is square and has full rank, it
    does per~(\ref{mtxinv:220:20}) indeed have an inverse~$A^{-1}$.  The
    rank-$n$ inverse exists.
  \item
    If $B=A^{-1}$ then $B^{-1}=A$.  That is,~$A$ is itself the rank-$n$
    inverse of~$A^{-1}$.  The matrices~$A$ and~$A^{-1}$ thus form an
    exclusive, reciprocal pair.
    % The footnote just following is probably unneeded.
    %\footnote{%
    %  A professional mathematician might more rigorously prove such a
    %  proposition somewhat as follows.  Let~$A$ be an $n \times n$
    %  matrix of full rank.   Suppose two $n \times n$ matrices~$B_1$
    %  and~$B_2$ such that $B_1A = I_n$ and $B_2A = I_n$.  Subtracting
    %  the one equation from the other yields that $[B_2-B_1]A = 0$,
    %  which has each row of the $n \times n$ matrix $[B_2-B_1]$ to
    %  represent a null linear combination of the rows of~$A$.  However,
    %  \S~\ref{gjrank:340.25} has shown that a full-rank square matrix
    %  like~$A$ has independent rows, which according to the definition
    %  of \emph{independence} in \S~\ref{gjrank:335} implies that each
    %  row of $[B_2-B_1]$ is null.  That is, $[B_2-B_1]=0$ and thus
    %  $B_2=B_1$.
    %
    %  Judging such a proposition to be sufficiently obvious on the face
    %  of it, though, this book seldom bothers with such additional
    %  rigor.%
    %}
  \item
    If~$B$ is an $n \times n$ square matrix and either
    $BA = I_n$ or $AB = I_n$, then both equalities in fact hold; thus,
    $B = A^{-1}$.  One can have neither equality without the
    other.
  \item
    Only a square, $n \times n$ matrix of full rank $r=n$ has a rank-$n$
    inverse.  A matrix~$A'$ which is not square, or whose rank falls
    short of a full $r=n$, is not invertible in the rank-$n$ sense
    of~(\ref{mtxinv:220:20}).
\ei
That~$A^{-1}$ is an $n \times n$ square matrix of full rank and that~$A$
is itself the inverse of~$A^{-1}$ proceed from the
definition~(\ref{mtxinv:220:20}) of~$A^{-1}$ plus
\S~\ref{gjrank:340.20}'s finding that reversible operations
like~$G_>^{-1}$ and~$G_<^{-1}$ cannot change $I_n$'s rank.  That the
inverse exists is plain, inasmuch as the Gauss-Jordan decomposition
plus~(\ref{mtxinv:220:20}) reliably calculate it.  That the
inverse is unique begins from \S~\ref{gjrank:340.25}'s observation that
the columns (like the rows) of~$A$ are linearly independent because~$A$
is square and has full rank.  From this beginning and the fact that $I_n
= AA^{-1}$, it follows that $[A^{-1}]_{{*}1}$ represents%
\footnote{
  The notation $[A^{-1}]_{{*}j}$ means ``the $j$th column of~$A^{-1}$.''
  Refer to \S~\ref{matrix:120.27}.
}
the one and only possible combination of $A$'s columns which
achieves~$\ve e_1$, that $[A^{-1}]_{{*}2}$ represents the one and only
possible combination of $A$'s columns which achieves~$\ve e_2$, and so
on through~$\ve e_n$.  One could observe likewise respecting the
independent rows of~$A$.  Either way,~$A^{-1}$ is unique.
Moreover, no other $n \times n$ matrix $B \neq A^{-1}$ satisfies
\emph{either} requirement of~(\ref{mtxinv:220:20})---that $BA = I_n$ or
that $AB = I_n$---much less both.

\index{uniqueness}
It is not claimed that the matrix factors~$G_>$ and~$G_<$ themselves are
unique, incidentally.  On the contrary, many different pairs of matrix
factors~$G_>$ and~$G_<$ can yield $A = G_>I_nG_<$, no less than that
many different pairs of scalar factors~$\gamma_>$ and~$\gamma_<$ can
yield $\alpha = \gamma_>1\gamma_<$.  Though the Gauss-Jordan
decomposition is a convenient means to~$G_>$ and~$G_<$, it is hardly the
only means, and any proper~$G_>$ and~$G_<$ found by any means will
serve so long as they satisfy~(\ref{mtxinv:220:10}).  What are unique
are not the factors but the~$A$ and~$A^{-1}$ they produce.

\index{square matrix!degenerate}
\index{matrix!degenerate}
\index{matrix!singular}
\index{degenerate matrix}
\index{singular matrix}
\index{reciprocal}
What of the degenerate $n \times n$ square matrix~$A'$, of rank $r<n$?
Rank promotion is impossible as \S\S~\ref{gjrank:340.10}
and~\ref{gjrank:340.20} have shown, so in the sense
of~(\ref{mtxinv:220:20}) such a matrix has no inverse; for, if it had,
then~$A'^{-1}$ would by definition represent a row or column operation
which impossibly promoted~$A'$ to the full rank $r=n$ of~$I_n$.  Indeed,
in that it has no inverse such a degenerate matrix resembles
the scalar~$0$, which has no reciprocal.  Mathematical convention owns a
special name for a square matrix which is degenerate and thus has no
inverse; it calls it a \emph{singular} matrix.

\index{matrix!rectangular}
\index{rectangular matrix}
\index{column!null, appending a}
\index{row!null, appending a}
And what of a rectangular matrix?  Is it degenerate?  Well, no, not
exactly, not necessarily.  The definitions of the present particular
section are meant for square matrices; they do not neatly apply to
nonsquare ones.  Refer to \S\S~\ref{gjrank:340.20}
and~\ref{gjrank:340.25}.  However, appending the right number of null
rows or columns to a nonsquare matrix does turn it into a degenerate
square, in which case the preceding argument applies.  See also
\S\S~\ref{gjrank:340.24}, \ref{mtxinv:240} and~\ref{mtxinv:320}.

% ----------------------------------------------------------------------

\section{The exactly determined linear system}
\label{mtxinv:230}
\index{equation!solving a set of simultaneously}
\index{equation!simultaneous linear system of}
\index{simultaneous system of linear equations}
\index{linear system!exactly determined}
\index{exactly determined linear system}

\index{motivation}
Section~\ref{matrix:120.10} has shown how the single matrix equation
\bq{mtxinv:230:10}
  A \ve x = \ve b
\eq
concisely represents an entire simultaneous system of linear scalar
equations.  If the system has~$n$ scalar equations and~$n$ scalar
unknowns, then the matrix~$A$ has square, $n \times n$ dimensionality.
Furthermore, if the~$n$ scalar equations are independent of one another,
then the rows of~$A$ are similarly independent, which gives~$A$ full
rank and makes it invertible.  Under these
conditions, one can solve~(\ref{mtxinv:230:10}) and the corresponding
system of linear scalar equations by
left-multiplying~(\ref{mtxinv:230:10}) by the~$A^{-1}$
of~(\ref{mtxinv:220:20}) and~(\ref{mtxinv:220:10}) to reach the famous
formula
\bq{mtxinv:230:15}
  \ve x = A^{-1} \ve b.
\eq
Inverting the square matrix~$A$ of scalar
coefficients,~(\ref{mtxinv:230:15}) concisely solves a simultaneous
system of~$n$ linear scalar equations in~$n$ scalar unknowns.  It is the
classic motivational result of matrix theory.

\index{theory}
It has taken the book two long chapters to reach~(\ref{mtxinv:230:15}).
If one omits first to prepare the theoretical ground sufficiently to
support more advanced matrix work, then one can indeed
reach~(\ref{mtxinv:230:15}) with rather less effort than the book has
done.%
\footnote{%
  For motivational reasons, introductory, tutorial linear algebra
  textbooks like~\cite{Hefferon} and~\cite{Lay} rightly yet invariably
  invert the general square matrix of full rank much earlier,
  reaching~(\ref{mtxinv:230:15}) with less effort.  The deferred price
  the student pays for the simpler-seeming approach of the tutorials is
  twofold.  First, the student fails to develop the Gauss-Jordan
  decomposition properly, instead learning the less elegant but easier
  to grasp ``row echelon form'' of ``Gaussian elimination''
  \cite[chapter~1]{Hefferon}\cite[\S~1.2]{Lay}---which makes good
  matrix-arithmetic drill but leaves the student imperfectly prepared
  when the time comes to study kernels and eigensolutions or to read and
  write matrix-handling computer code.  Second, in the long run the
  tutorials save no effort, because the student still must at some point
  develop the theory underlying matrix rank and supporting each of the
  several co\"incident properties of \S~\ref{eigen:370}.  What the
  tutorials do is pedagogically necessary---it is how the writer first
  learned the matrix and probably how the reader first learned it,
  too---but it is appropriate to a tutorial, not to a study reference
  like this book.

  In this book, where derivations prevail, the proper place to invert
  the general square matrix of full rank is here.  Indeed, the inversion
  here goes smoothly, because chapters~\ref{matrix} and~\ref{gjrank} have
  laid under it a firm foundation upon which---and supplied it the right
  tools with which---to work.%
}
As the chapter's introduction has observed, however, we shall soon meet
additional interesting applications of the matrix which in any case
require the theoretical ground to have been prepared.
Equation~(\ref{mtxinv:230:15}) is only the first fruit of the effort.

\index{driving vector}
\index{vector!driving}
\index{equation!superfluous}
Where the inverse does not exist, where the square matrix~$A$ is
singular, the rows of the matrix are linearly dependent, meaning that
the corresponding system actually contains fewer than~$n$ useful
scalar equations.  Depending on the value of the driving vector~$\ve b$,
the superfluous
equations either merely reproduce or flatly contradict information the
other equations already supply.  Either way, no unique solution to a
linear system described by a singular square matrix is possible---though
a good approximate solution is given by the \emph{pseudoinverse} of
\S~\ref{mtxinv:320}.  In the language of \S~\ref{gjrank:340.24},
the singular square matrix characterizes a system that is both
underdetermined and overdetermined, and thus degenerate.

% ----------------------------------------------------------------------

\section{The kernel}
\label{mtxinv:245}
\index{kernel}

\index{rank!column}
If a matrix~$A$ has full column rank (\S~\ref{gjrank:340.25}), then the
columns of~$A$ are linearly independent and
\bq{mtxinv:245:05}
  A\ve x = 0
\eq
is impossible if $I_n\ve x \neq 0$.  If the matrix however lacks full
column rank then~(\ref{mtxinv:245:05}) is possible even if $I_n\ve x
\neq 0$.  In either case, any $n$-element~$\ve x$ (including $\ve x =
0$) that satisfies~(\ref{mtxinv:245:05}) belongs to the~\emph{kernel}
of~$A$.

\index{kernel matrix}
Let~$A$ be an $m \times n$ matrix of rank~$r$.  A second
matrix,%
\footnote{\label{mtxinv:245:08}%
  The conventional mathematical notation for the kernel of~$A$ is
  $\ker\{A\}$, $\mopx{null}\{A\}$ or something nearly resembling one of
  the two---the notation seems to vary from editor to editor---which technically
  represent the kernel space itself, as opposed to the notation~$A^K$
  which represents a matrix whose columns address the kernel space.
  This book de\"emphasizes the distinction and prefers the kernel matrix
  notation~$A^K$.

  If we were really precise, we might write not~$A^{K}$ but $A^{K(n)}$
  to match the $A^{-1(r)}$ of~(\ref{matrix:321:20}).  The abbreviated
  notation~$A^{K}$ is probably clear enough for most practical purposes,
  though, and surely more comprehensible to those who do not happen to
  have read this particular book.
}%
~$A^K$, minimally represents the kernel of~$A$ if and only if
\bi
  \item $A^K$ has $n \times (n-r)$ dimensionality (which
    gives~$A^K$ tall rectangular form unless $r=0$),
  \item $A^K$ has full rank $n-r$ (that is, the columns of~$A^K$ are
    linearly independent, which gives~$A^K$ full column rank), and
  \item $A^K$ satisfies the equation
    \bq{mtxinv:245:10}
      A A^K = 0.
    \eq
\ei
The $n-r$ independent columns of the kernel matrix~$A^K$ address the
complete space $\ve x = A^K \ve a$ of vectors in the kernel,
where the $(n-r)$-element vector~$\ve a$ can have any value.  In
symbols,
\[
  A\ve x = A (A^K \ve a) = (AA^K) \ve a = 0.
\]

\index{kernel space}
\index{space}
\index{vector space}
The definition does not pretend that the kernel matrix~$A^K$ is
unique.  Except when~$A$ has full column rank the kernel matrix is not
unique; there are infinitely many kernel matrices~$A^K$ to choose
from for a given matrix~$A$.  What is unique is not the kernel matrix
but rather the space its columns address, and it is the latter space
rather than~$A^K$ as such that is technically the kernel (if you
forget and call~$A^K$ ``a kernel,'' though, you'll be all right).

\index{Gauss-Jordan kernel formula}
\index{kernel!Gauss-Jordan formula for}
\index{shift operator}
The \emph{Gauss-Jordan kernel formula}%
\footnote{
  The name \emph{Gauss-Jordan kernel formula} is not standard as far as
  the writer is aware, but we would like a name
  for~(\ref{mtxinv:245:kernel}).  This name seems as fitting as any.
}
\bq{mtxinv:245:kernel}
  % I have checked this by computer for a 4x5 test case. --THB--
  A^K = S^{-1}K^{-1}H_rI_{n-r} = G_<^{-1}H_rI_{n-r}
\eq
gives a complete kernel~$A^K$ of~$A$, where~$S^{-1}$, $K^{-1}$
and~$G_<^{-1}$ are the factors their respective symbols indicate of the
Gauss-Jordan decomposition's complementary form~(\ref{gjrank:341:GJinv})
and~$H_r$ is the shift operator of \S~\ref{matrix:340}.
Section~\ref{mtxinv:245.10} derives the formula, next.

\subsection{The Gauss-Jordan kernel formula}
\label{mtxinv:245.10}
\index{Gauss-Jordan kernel formula}
\index{kernel!Gauss-Jordan formula for}
\index{Gauss, Carl Friedrich (1777--1855)}
\index{Jordan, Wilhelm (1842--1899)}

\index{proviso}
\index{generality}
To derive~(\ref{mtxinv:245:kernel}) is not easy.  It begins from the
statement of the linear system
\bq{mtxinv:245:11}
  \mbox{$A \ve x = \ve b$, where $\ve b = 0$ or $r = m$, or both;}
\eq
and where~$\ve b$ and~$\ve x$ are respectively~$m$- and $n$-element
vectors and~$A$ is an $m \times n$ matrix of rank~$r$.  This statement
is broader than~(\ref{mtxinv:245:05}) requires but it serves
\S~\ref{mtxinv:240}, too; so, for the moment, for generality's sake, we
leave~$\ve b$ unspecified but by the given proviso.  Gauss-Jordan
factoring~$A$, by successive steps,
\bqb
  G_>I_rKS\ve x &=& \ve b, \\
  I_rKS\ve x &=& G_>^{-1}\ve b, \\
  I_r(K-I)S\ve x + I_rS\ve x &=& G_>^{-1}\ve b.
\eqb
Applying an identity from Table~\ref{gjrank:341:t34} on
page~\pageref{gjrank:341:t34},
\[
  I_rK(I_n-I_r)S\ve x + I_rS\ve x = G_>^{-1}\ve b.
\]
Rearranging terms,
\bq{mtxinv:245:30}
  I_rS\ve x = G_>^{-1}\ve b - I_rK(I_n-I_r)S\ve x.
\eq

\index{free element}
\index{dependent element}
\index{element!free and dependent}
\index{driving vector}
\index{vector!driving}
Equation~(\ref{mtxinv:245:30}) is interesting.  It has~$S\ve x$ on both
sides, where~$S\ve x$ is the vector~$\ve x$ with elements reordered in
some particular way.  The equation has however on the left only~$I_rS\ve
x$, which is the first~$r$ elements of~$S\ve x$; and on the right only
$(I_n-I_r)S\ve x$, which is the remaining $n-r$ elements.%
\footnote{
  Notice how we now associate the factor $(I_n-I_r)$ rightward as a row
  truncator, though it had first entered acting leftward as a column
  truncator.  The flexibility to reassociate operators in such a way
  is one of many good reasons chapters~\ref{matrix} and~\ref{gjrank} have
  gone to such considerable trouble to develop the basic theory of the
  matrix.
}
No element of~$S\ve x$ appears on both sides.  Naturally this is no
accident; we have (probably after some trial and error not recorded
here) planned the steps leading to~(\ref{mtxinv:245:30}) to achieve
precisely this effect.  Equation~(\ref{mtxinv:245:30}) implies \emph{that
one can choose the last $n-r$ elements of~$S\ve x$ freely, but that the
choice then determines the first~$r$ elements.}

\index{vector space}
\index{column}
\index{matrix!column of}
\index{domain}
\index{range}
The implication is significant.  To express the implication more clearly
we can rewrite~(\ref{mtxinv:245:30}) in the improved form
\bq{mtxinv:245:32}
  \begin{split}
    \ve f &= G_>^{-1}\ve b - I_rKH_r\ve a, \\
    S\ve x &= \left[\br{c}\ve f\\\ve a\er\right]
      = \ve f + H_r\ve a, \\
    \ve f &\equiv I_rS\ve x, \\
    \ve a &\equiv H_{-r}(I_n-I_r)S\ve x,
  \end{split}
\eq
where~$\ve a$ represents the $n-r$ free elements of~$S\ve x$ and~$\ve f$
represents the~$r$ dependent elements.  This makes~$\ve f$ and thereby
also~$\ve x$ functions of the free parameter~$\ve a$ and the driving
vector~$\ve b$:
\bq{mtxinv:245:49}
  \begin{split}
    \ve f(\ve a, \ve b) &= G_>^{-1}\ve b - I_rKH_r\ve a, \\
    S\ve x(\ve a, \ve b) &= \left[\br{c}\ve f(\ve a, \ve b)\\\ve a\er\right]
      = \ve f(\ve a, \ve b) + H_r\ve a.
  \end{split}
\eq
If $\ve b=0$ as~(\ref{mtxinv:245:05}) requires, then
\[
  \begin{split}
    \ve f(\ve a, 0) &= - I_rKH_r\ve a, \\
    S\ve x(\ve a, 0) &= \left[\br{c}\ve f(\ve a, 0)\\\ve a\er\right]
      = \ve f(\ve a, 0) + H_r\ve a.
  \end{split}
\]
Substituting the first line into the second,
\bq{mtxinv:245:48}
  S\ve x(\ve a, 0) = (I - I_rK)H_r\ve a.
\eq
In the event that $\ve a = \ve e_j$, where $1 \le j \le n-r$,
\[
  S\ve x(\ve e_j, 0) = (I - I_rK)H_r\ve e_j.
\]
For all the~$\ve e_j$ at once,
\[
  S\ve x(I_{n-r}, 0) = (I - I_rK)H_rI_{n-r}.
\]
But if all the~$\ve e_j$ at once---that is, if all
the columns of~$I_{n-r}$---exactly
address the domain of~$\ve a$, then the columns of $\ve x(I_{n-r}, 0)$
likewise exactly address the range of $\ve x(\ve a, 0)$.
Equation~(\ref{mtxinv:245:10}) has already named this range~$A^K$,
by which%
\footnote{
  These are difficult steps.  How does one justify replacing~$\ve a$
  by~$\ve e_j$, then~$\ve e_j$ by~$I_{n-r}$, then~$\ve x$ by~$A^K$?  One
  justifies them in that the columns of~$I_{n-r}$ are the several~$\ve
  e_j$, of which any $(n-r)$-element vector~$\ve a$ can be constructed
  as the linear combination
  \[
    \ve a = I_{n-r}\ve a
    = [ \br{ccccc} \ve e_1 & \ve e_2 & \ve e_3 & \cdots & \ve e_{n-r} \er ] \ve a
    = \sum_{j=1}^{n-r} a_j \ve e_j
  \]
  weighted by the elements of~$\ve a$.  Seen from one perspective, this
  seems trivial; from another perspective, baffling; until one grasps
  what is really going on here.

  The idea is that if we can solve the problem for each elementary
  vector~$\ve e_j$---that is, in aggregate, if we can solve the problem
  for the identity matrix~$I_{n-r}$---then we shall implicitly have
  solved it for every~$\ve a$ because~$\ve a$ is a weighted combination
  of the~$\ve e_j$ and the whole problem is linear.  The solution
  \[
    \ve x = A^K\ve a
  \]
  for a given choice of~$\ve a$ becomes a weighted combination of the
  solutions for each~$\ve e_j$, with the elements of~$\ve a$ again as
  the weights.  And what are the solutions for each~$\ve e_j$?  Answer:
  the corresponding columns of~$A^K$, which by definition are the
  independent values of~$\ve x$ that cause $\ve b = 0$.
}
\bq{mtxinv:245:41}
  SA^K = (I-I_rK)H_rI_{n-r}.
\eq
Left-multiplying by
\bq{mtxinv:245:415}
  S^{-1}=S^{*}=S^{T}
\eq
produces the alternate kernel formula
\bq{mtxinv:245:42}
  A^K = S^{-1}(I-I_rK)H_rI_{n-r}.
\eq

\index{schematic}
\index{pencil}
\index{hypothesis}
\index{kernel!alternate formula for}
The alternate kernel formula~(\ref{mtxinv:245:42}) is correct but not
as simple as it could be.  By the identity~(\ref{matrix:340:61}),
eqn.~(\ref{mtxinv:245:41}) is
\bqa
  SA^K &=& (I-I_rK)(I_n-I_r)H_r \xn\\
    &=& [(I_n-I_r) - I_rK(I_n-I_r)]H_r \xn\\
    &=& [(I_n-I_r) - (K-I)]H_r,
    \label{mtxinv:245:33}
\eqa
where we have used Table~\ref{gjrank:341:t34} again in the last step.
%
How to proceed symbolically from~(\ref{mtxinv:245:33}) is not obvious,
but if one sketches the matrices of~(\ref{mtxinv:245:33}) schematically
with a pencil, and if one remembers that~$K^{-1}$ is just~$K$ with
elements off the main diagonal negated, then it appears that
\bq{mtxinv:245:34}
  SA^K = K^{-1}H_rI_{n-r}.
\eq
The appearance is not entirely convincing,%
\footnote{
  Well, no, actually, the appearance pretty much is entirely convincing,
  but let us finish the proof symbolically nonetheless.
}
but~(\ref{mtxinv:245:34}) though unproven still helps because it posits a
hypothesis toward which to target the analysis.

Two variations on the identities of Table~\ref{gjrank:341:t34} also
help.  First, from the identity that
\[
  \frac{K+K^{-1}}{2} = I,
\]
we have that
\bq{mtxinv:245:35}
  K-I = I-K^{-1}.
\eq
Second, right-multiplying by~$I_r$ the identity that
\[
  I_rK^{-1}(I_n-I_r) = K^{-1}-I
\]
and canceling terms, we have that
\bq{mtxinv:245:36}
  K^{-1}I_r = I_r
\eq
(which actually is pretty obvious if you think about it, since all
of $K$'s interesting content lies by construction right of its $r$th
column).  Now we have enough to go on with.
Substituting~(\ref{mtxinv:245:35}) and~(\ref{mtxinv:245:36})
into~(\ref{mtxinv:245:33}) yields that
\[
  SA^K = [(I_n-K^{-1}I_r) - (I-K^{-1})]H_r.
\]
Adding $0=K^{-1}I_nH_r-K^{-1}I_nH_r$ and rearranging terms,
\[
  SA^K =  K^{-1}(I_n-I_r)H_r + [K^{-1} - K^{-1}I_n - I + I_n]H_r.
\]
Factoring,
\[
  SA^K = K^{-1}(I_n-I_r)H_r + [(K^{-1}-I)(I-I_n)]H_r.
\]
According to Table~\ref{gjrank:341:t34}, the quantity in square brackets
is zero, so
\[
  SA^K = K^{-1}(I_n-I_r)H_r,
\]
which, considering that the identity~(\ref{matrix:340:61}) has that
$(I_n-I_r)H_r = H_rI_{n-r}$, proves~(\ref{mtxinv:245:34}).  The final
step is to left-multiply~(\ref{mtxinv:245:34}) by $S^{-1}=S^{*}=S^{T}$,
reaching~(\ref{mtxinv:245:kernel}) that was to be derived.

\index{vector space!address of}
\index{kernel space}
One would like to feel sure that the columns
of~(\ref{mtxinv:245:kernel})'s $A^K$ actually addressed the whole kernel
space of~$A$ rather than only part.  One would further like to feel sure
that~$A^K$ had no redundant columns; that is, that it had full rank.
Moreover, the definition of~$A^K$ in the section's introduction demands
both of these features.  In general such features would be hard to
establish, but here the factors conveniently are Gauss-Jordan factors.
Regarding the whole kernel space,~$A^K$ addresses it
because~$A^K$ comes from all~$\ve a$.  Regarding
redundancy,~$A^K$ lacks it because~$SA^K$ lacks it,
and~$SA^K$ lacks it because according to~(\ref{mtxinv:245:41}) the
last rows of~$SA^K$ are~$H_rI_{n-r}$.  So, in
fact,~(\ref{mtxinv:245:kernel}) has both features and does fit the
definition.

\subsection{Converting between kernel matrices}
\label{mtxinv:245.20}
\index{kernel matrix!converting between two of}

\index{vector!replacement of}
If~$C$ is a reversible $(n-r) \times (n-r)$ operator by which we
right-multiply~(\ref{mtxinv:245:10}),
then the matrix
\bq{mtxinv:245:17}
  A'^K = A^K C
\eq
like~$A^K$ evidently represents the kernel of~$A$:
\[
  AA'^K = A (A^K C) = (AA^K) C = 0.
\]
Indeed this makes sense: because the columns of~$A^K C$ address the
same space the columns of~$A^K$ address, the two matrices
necessarily represent the same underlying kernel.  Moreover,
\emph{some}~$C$ exists to convert~$A^K$ into every alternate kernel
matrix~$A'^K$ of~$A$.  We know this because \S~\ref{gjrank:338} lets one
replace the columns of~$A^K$ with those of~$A'^K$, reversibly, one
column at a time, without altering the space addressed.  (It might not
let one replace the columns in sequence, but if out of sequence then a
reversible permutation at the end corrects the order.  Refer to
\S\S~\ref{gjrank:340.05} and~\ref{gjrank:340.10} for the pattern by
which this is done.)

The orthonormalizing column operator~$R^{-1}$ of~(\ref{mtxinv:QR}) below
incidentally tends to make a good choice for~$C$.

\subsection{The degree of freedom}
\label{mtxinv:245.70}
\index{freedom, degree of}
\index{degree of freedom}

A slightly vague but extraordinarily useful concept has emerged in this
section, worth pausing briefly to appreciate.  The concept is the
concept of the \emph{degree of freedom.}

\index{artillerist}
\index{cannon}
\index{gunpowder}
\index{battlefield}
\index{azimuth}
\index{elevation}
\index{north}
\index{south}
\index{east}
\index{west}
\index{horse}
\index{battle}
\index{battlefield}
\index{Napoleon}
A degree of freedom is a parameter one remains free to
determine within some continuous domain.  For example, Napoleon's artillerist%
\footnote{
  The author, who has never fired an artillery piece (unless an arrow
  from a Boy Scout bow counts), invites any real artillerist among the
  readership to write in to improve the example.
}
might have enjoyed as many as six degrees of freedom in
firing a cannonball: two in where he chose to set up his cannon (one
degree in north-south position, one in east-west); two in aim (azimuth
and elevation); one in muzzle velocity (as governed by the quantity of
gunpowder used to propel the ball); and one in time.  A seventh
potential degree of freedom, the height from which the artillerist
fires, is of course restricted by the lay of the land: the artillerist
can fire from a high place only if the place he has chosen to fire from
happens to be up on a hill, for Napoleon had no flying cannon.  Yet even
among the six remaining degrees of freedom, the artillerist might find
some impractical to exercise.  The artillerist probably preloads the
cannon always with a standard charge of gunpowder because, when he finds
his target in the field, he cannot spare the time to unload the cannon
and alter the charge: this costs one degree of freedom.  Likewise, the
artillerist must limber up the cannon and hitch it to a horse to shift
it to better ground; for this too he cannot spare time in the heat of
battle: this costs two degrees.  And Napoleon might yell,
``Fire!'' canceling the time degree as well.  Two degrees of freedom
remain to the artillerist; but, since exactly two degrees are needed to
hit some particular target on the battlefield, the two are enough.

\index{carriage wheel}
Now consider what happens if the artillerist loses one of his last two
remaining degrees of freedom.  Maybe the cannon's carriage wheel is
broken and the artillerist can no longer turn the cannon; that is, he
can still choose firing elevation but no longer azimuth.  In such a
strait to hit some particular target on the battlefield, the artillerist
needs somehow to recover another degree of freedom, for he needs two but
has only one.  If he disregards Napoleon's order, ``Fire!'' (maybe not a
wise thing to do, but, anyway, \ldots)\ %
and waits for the target to traverse the cannon's fixed line of fire,
then he can still hope to hit even with the broken carriage wheel; for
could he choose neither azimuth nor the moment to fire, then he would
almost surely miss.

\index{nonlinearity}
Some apparent degrees of freedom are not real.  For example, muzzle
velocity gives the artillerist little control firing elevation does not
also give.  Other degrees of freedom are nonlinear in effect: a certain
firing elevation gives maximum range; nearer targets can be hit by
firing either higher or lower at the artillerist's discretion.  On the
other hand, too much gunpowder might break the cannon.

\index{engineer}
\index{aeronautical engineer}
\index{aileron}
\index{rudder}
\index{control surface, aeronautical}
\index{pilot}
\index{control}
\index{tuning}
\index{electrical engineer}
\index{circuit}
\index{electric circuit}
\index{potentiometer}
\index{technician}
All of this is hard to generalize in unambiguous mathematical terms, but
the count of the degrees of freedom in a system is of high conceptual
importance to the engineer nonetheless.  Basically, the count captures
the idea that to control~$n$ output variables of some system takes
at least~$n$ independent input variables.  The~$n$ may
possibly for various reasons still not suffice---it might be wise in
some cases to
allow $n+1$ or $n+2$---but in no event will fewer than~$n$ do.
Engineers of all kinds think in this way: an aeronautical engineer knows
in advance that an airplane needs at least~$n$ ailerons, rudders and
other control surfaces for the pilot adequately to control the airplane;
an electrical engineer knows in advance that a circuit needs at
least~$n$ potentiometers for the technician adequately to tune the
circuit; and so on.

\index{up}
\index{down}
\index{mountain road}
\index{road!mountain}
\index{point}
\index{line}
\index{plane}
\index{swimming pool}
\index{city street}
In geometry, a line brings a single degree of freedom.  A plane brings
two.  A point brings none.  If the line bends and turns like a mountain
road, it still brings a single degree of freedom.  And if the road
reaches an intersection?  Answer: still one degree.  A degree of freedom
has some continuous nature, not merely a discrete choice to turn left or
right.  On the other hand, a swimmer in a swimming pool enjoys three
degrees of freedom (up-down, north-south, east-west) even though
his domain in any of the three is limited to the small volume of the
pool.  The driver on the mountain road cannot claim a second degree of
freedom at the mountain intersection (he can indeed claim a choice, but
the choice being discrete lacks the proper character of a degree of
freedom), but he might plausibly claim a second degree of freedom upon
reaching the city, where the web or grid of streets is dense enough to
approximate access to any point on the city's surface.  Just how many streets it
takes to turn the driver's ``line'' experience into a ``plane''
experience is a matter for the mathematician's discretion.

Reviewing~(\ref{mtxinv:245:49}), we find $n-r$ degrees of freedom in the
general underdetermined linear system, represented by the $n-r$ free
elements of~$\ve a$.  If the underdetermined system is not also
overdetermined, if it is nondegenerate such that $r=m$, then it is
guaranteed to have a family of solutions~$\ve x$.  This family is the
topic of the next section.

% ----------------------------------------------------------------------

\section{The nonoverdetermined linear system}
\label{mtxinv:240}
\index{linear system!nonoverdetermined}
\index{nonoverdetermined linear system}

\index{matrix!broad}
\index{solution!family of}
\index{family of solutions}
The exactly determined linear system of \S~\ref{mtxinv:230} is common,
but also common is the more general, nonoverdetermined linear system
\bq{mtxinv:240:05}
  A \ve x = \ve b,
\eq
in which~$\ve b$ is a known, $m$-element vector;~$\ve x$ is an unknown,
$n$-element vector; and~$A$ is a square or broad, $m \times n$ matrix of
full row rank (\S~\ref{gjrank:340.25})
\bq{mtxinv:240:07}
  r = m \le n.
\eq
Except in the exactly determined edge case $r=m=n$ of
\S~\ref{mtxinv:230}, the nonoverdetermined linear system has no unique
solution but rather a family of solutions.  This section delineates the
family.

\subsection{Particular and homogeneous solutions}
\label{mtxinv:240.20}
\index{solution!particular and homogeneous}
\index{particular solution}
\index{homogeneous solution}

\index{driving vector}
\index{vector!driving}
\index{split form}
The nonoverdetermined linear system~(\ref{mtxinv:240:05}) by definition
admits more than one solution~$\ve x$ for a given driving vector~$\ve
b$.  Such a system is hard to solve all at once, though, so we prefer to
split the system as
\bq{mtxinv:240:20}
  \begin{split}
    A \ve x_1 &= \ve b, \\
    A (A^K \ve a) &= 0, \\
    \ve x &= \ve x_1 + A^K \ve a,
  \end{split}
\eq
which, when the second line is added to the first and the third is
substituted, makes the whole form~(\ref{mtxinv:240:05}).  Splitting the
system does not change it but does let us treat the system's first
and second lines in~(\ref{mtxinv:240:20}) separately.  In the split
form, the symbol~$\ve x_1$ represents any one $n$-element vector that
happens to satisfy the form's first line---many are possible; the
mathematician just picks one---and is called \emph{a particular
solution} of~(\ref{mtxinv:240:05}).  The $(n-r)$-element vector~$\ve a$
remains unspecified, whereupon~$A^K \ve a$ represents the complete
family of $n$-element vectors that satisfy the form's second line.  The
family of vectors expressible as~$A^K \ve a$ is called \emph{the
homogeneous solution} of~(\ref{mtxinv:240:05}).

\index{articles ``a'' and ``the''}
Notice the italicized articles \emph{a} and \emph{the.}

The Gauss-Jordan kernel formula~(\ref{mtxinv:245:kernel}) has given
us~$A^K$ and thereby the homogeneous solution, which renders the
analysis of~(\ref{mtxinv:240:05}) already half done.  To complete the
analysis, it remains in \S~\ref{mtxinv:240.30} to find a particular
solution.

\subsection{A particular solution}
\label{mtxinv:240.30}
\index{solution!particular}
\index{particular solution}
\index{linear system!nonoverdetermined, particular solution of}
\index{nonoverdetermined linear system!particular solution of}

Any particular solution will do.  Equation~(\ref{mtxinv:245:49}) has
that
\[
  \begin{split}
    \ve f(\ve a, \ve b) &= G_>^{-1}\ve b - I_rKH_r\ve a, \\
    (S) \left[ \ve x_1(\ve a, \ve b) + A^K \ve a \right]
      &= \left[\br{c}\ve f(\ve a, \ve b)\\\ve a\er\right]
      = \ve f(\ve a, \ve b) + H_r\ve a,
  \end{split}
\]
where we have substituted the last line of~(\ref{mtxinv:240:20}) for~$\ve
x$.  This holds for any~$\ve a$ and~$\ve b$.  We are not free to choose
the driving vector~$\ve b$, but since we need only one particular
solution,~$\ve a$ can be anything we want.  Why not
\[
  \ve a=0?
\]
Then
\[
  \begin{split}
    \ve f(0, \ve b) &= G_>^{-1}\ve b, \\
    S\ve x_1(0, \ve b)
      &= \left[\br{c}\ve f(0, \ve b)\\0\er\right]
      = \ve f(0, \ve b).
  \end{split}
\]
That is,
\bq{mtxinv:240:25}
  \ve x_1 = S^{-1}G_>^{-1}\ve b.
\eq

\subsection{The general solution}
\label{mtxinv:240.40}
\index{solution!general}
\index{general solution}
\index{linear system!nonoverdetermined, general solution of}
\index{nonoverdetermined linear system!general solution of}

Assembling~(\ref{mtxinv:245:kernel}), (\ref{mtxinv:240:20})
and~(\ref{mtxinv:240:25}) in light of~(\ref{gjrank:341:GJinv})
yields the general solution
\bq{mtxinv:240:50}
  \ve x = S^{-1}(G_>^{-1}\ve b + K^{-1}H_rI_{n-r} \ve a)
\eq
to the nonoverdetermined linear system~(\ref{mtxinv:240:05}).

\index{arithmetic!exact and inexact}
\index{exact arithmetic}
\index{inexact arithmetic}
In exact arithmetic~(\ref{mtxinv:240:50}) solves the nonoverdetermined
linear system in theory exactly.  Therefore~(\ref{mtxinv:240:50})
properly concludes the section.  Nevertheless, one should like to add a
significant practical observation regarding \emph{inexact arithmetic}
as follows.

\index{rounding error}
\index{error!due to rounding}
\index{pivot!small}
\index{matrix!large}
\index{register, computer's floating-point}
\index{computer register}
\index{floating-point register}
\index{library, programmer's}
\index{computer memory}
\index{memory, computer}
\index{mantissa}
Practical calculations are usually done in inexact arithmetic insofar as
they are done in the limited precision of a computer's floating-point
registers.  Exceptions are possible---exact-arithmetic libraries are
available for a programmer to call---but exact-arithmetic libraries are
slow and memory-intensive and, for this reason among others, are only
occasionally used in practice.  When they are not used,
compounded rounding error in a floating-point register's last
bit of mantissa\footnote{%
  % diagn: this footnote, substantially rewritten, wants review
  \index{floating-point exponent}%
  \index{exponent!floating-point}%
  \index{floating-point infinity}%
  \index{infinity!floating-point}%
  \index{floating-point zero}%
  \index{zero!floating-point}%
  \index{bit}%
  What is a \emph{mantissa?}
  Illustration: in the number $1.65\times 10^6$,
  the mantissa is 1.65.  However, computers do it in binary rather
  than in decimal, typically with fifty-two ($\text{0x34}$) stored
  bits of mantissa not counting
  the leading bit which, in binary, is not stored because
  it is always~1.  (There exist implementational details like
  floating-point ``denormals'' which might seem pedantically to
  contradict the always-1 rule, but that is a
  computer-science technicality which is hardly interesting even in
  computer science and is uninteresting in the context of the present
  discussion.  What might be interesting in the present context
  is this: a standard double-precision floating-point representation
  has---besides fifty-two bits of
  mantissa---also eleven, $\text{0xB}$,
  bits for an exponent and one bit for a sign.  The
  smallest positive number representable without denormalization
  is~$2^{-\text{0x3FE}}$; the
  largest is
  \mbox{$\text{0x1.FFFF\,FFFF\,FFFF\,F}\times 2^{\text{0x3FF}}$},
  just less than~$2^{\text{0x400}}$.  If the code for a
  full~$2^{\text{0x400}}$ is entered, then that is held to
  represent infinity.  Similarly, the code for~$2^{-\text{0x3FF}}$
  represents zero.  If you think that the code for~$2^{-\text{0x400}}$
  should instead represent zero, no such code can actually be entered,
  for the exponent's representation is offset by one; and there are many
  other details beyond the book's scope.)}
eventually disrupts~(\ref{mtxinv:240:50}) for matrices
larger than some moderately large size.  Avoiding unduly small pivots
early in the Gauss-Jordan extends~(\ref{mtxinv:240:50})'s reach to larger
matrices, and for yet larger matrices a bewildering variety of more
sophisticated techniques exists to mitigate the problem, which can be
vexing because the problem arises even when the matrix~$A$ is exactly
known.

Equation~(\ref{mtxinv:240:50}) is thus useful and correct, but one
should at least be aware that it can in practice lose floating-point
accuracy when the matrix it attacks grows too large.  (It can also lose
accuracy when the matrix's rows are almost dependent, but that is more
the fault of the matrix than of the formula.  See \S~\ref{eigen:470},
which addresses a related problem.)

% ----------------------------------------------------------------------

\section{The residual}
\label{mtxinv:315}
\index{residual}

Equations~(\ref{mtxinv:220:20}) and~(\ref{mtxinv:230:15}) solve the
exactly determined linear system $A\ve x = \ve b$.
Equation~(\ref{mtxinv:240:50}) broadens the solution to include the
nonoverdetermined linear system.  None of those equations however can
handle the overdetermined linear system, because for general~$\ve b$ the
overdetermined linear system
\bq{mtxinv:315:05}
  A \ve x \approx \ve b
\eq
has no exact solution.  (See \S~\ref{gjrank:340.24} for the definitions
of \emph{underdetermined, overdetermined,} etc.)

\index{linear system!overdetermined}
\index{overdetermined linear system}
\index{datum}
One is tempted to declare the overdetermined system uninteresting
because it has no solution and to leave the matter there, but this would
be a serious mistake.  In fact the overdetermined system is especially
interesting, and the more so because it arises so frequently in
applications.  One seldom trusts a minimal set of data for important
measurements, yet extra data imply an overdetermined system.  We need to
develop the mathematics to handle the overdetermined system properly.

\index{squared residual norm}
\index{residual!squared norm of}
The quantity%
\footnote{\label{mtxinv:315:95}%
  Alas, the alphabet has only so many letters (see
  appendix~\ref{greek}).  The~$\ve r$ here is unrelated to matrix
  rank~$r$.
}$\mbox{}^,$%
\footnote{
  This is as~\cite{vdVorst} defines it.  Some
  authors~\cite{Peterson/Mittra:1986} however prefer to define $\ve
  r(\ve x) \equiv A\ve x - \ve b$, instead.
}
\bq{mtxinv:residual}
  \ve r(\ve x) \equiv \ve b - A\ve x
\eq
measures how nearly some candidate solution~$\ve x$ solves the
system~(\ref{mtxinv:315:05}).  We call this quantity the
\emph{residual,} and the smaller, the better.  More precisely, the
smaller the nonnegative real scalar
\bq{mtxinv:residual-norm}
  [\ve r(\ve x)]^{*}[\ve r(\ve x)] =
  \sum_i \left|r_i(\ve x)\right|^2
\eq
is, called the \emph{squared residual norm,} the more favorably we
regard the candidate solution~$\ve x$.

% ----------------------------------------------------------------------

\section[The pseudoinverse and least squares]
  {The Moore-Penrose pseudoinverse and the least-squares problem}
\label{mtxinv:320}
\index{pseudoinverse}
\index{Moore-Penrose pseudoinverse}
\index{Moore, E.~H.\ (1862--1932)}
\index{Penrose, Roger (1931--)}
\index{least squares}
\index{squares, least}

\index{datum}
\index{line!fitting a}
\index{freeway}
\index{building construction}
\index{contractor}
\index{Saturday}
\index{worker}
\index{labor union}
A typical problem is to fit a straight line to some data.  For example,
suppose that we are building-construction contractors with a unionized
work force, whose labor union can supply additional, fully trained labor
on demand.  Suppose further that we are contracted to build a long
freeway and have been adding workers to the job in recent weeks to speed
construction.  On Saturday morning at the end of the second week, we
gather and plot the production data on the left of
Fig.~\ref{mtxinv:320:fig1}.  If~$u_i$ and~$b_i$ respectively represent
the number of workers and the length of freeway completed during
week~$i$, then we can fit a straight line $b = \sigma u + \gamma$ to the
measured production data such that
\[
  \mf{cc}{u_1&1\\u_2&1} \mf{c}{\sigma\\\gamma} = \mf{c}{b_1\\b_2},
\]
inverting the matrix per \S\S~\ref{mtxinv:220} and~\ref{mtxinv:230} to
solve for $\ve x \equiv[\sigma\;\gamma]^T$, in the hope that the
resulting line will predict future production accurately.
\begin{figure}
  \caption{Fitting a line to measured data.}
  \label{mtxinv:320:fig1}
  \bc
    \nc\fxa{-5.8} \nc\fxb{5.8}
    \nc\fya{-2.5} \nc\fyb{2.5}
    \nc\xxb{3.5}
    \nc\xxc{3.0}
    \nc\xxd{0.30}
    \nc\xxf{-0.85}
    \begin{pspicture}(\fxa,\fya)(\fxb,\fyb)
      %\psframe[linewidth=0.5pt,dimen=outer](\fxa,\fya)(\fxb,\fyb)
      {
        \small
        \psset{dimen=middle}
        \rput(-4.2,\xxf){
          {
            \psset{linewidth=0.5pt}
            \psline{<->}(0,\xxc)(0,0)(\xxb,0)
            \footnotesize
            \rput[tl](0,-\xxd)
              {Number of workers}
            \rput[bl]{90}(-\xxd,0)
              {\parbox{3.0cm}{Length newly\\completed\\during the week}}
          }
          {
            \psset{linewidth=2.0pt}
            \psdot(1.2,1.5)
            \psdot(1.5,1.6)
          }
          {
            \psset{linewidth=1.0pt}
            % slope          0.33333
            % y intercept at 1.10000
            \psline(0.7,1.3333)(3.2,2.1667)
          }
        }
        \rput(1.8,\xxf){
          {
            \psset{linewidth=0.5pt}
            \psline{<->}(0,\xxc)(0,0)(\xxb,0)
            \footnotesize
            \rput[tl](0,-\xxd)
              {Number of workers}
            \rput[bl]{90}(-\xxd,0)
              {\parbox{3.0cm}{Length newly\\completed\\during the week}}
          }
          {
            \psset{linewidth=2.0pt}
            % slope          0.52212
            % y intercept at 0.86372
            \psdot(1.2,1.5)
            \psdot(1.5,1.6)
            \psdot(1.6,1.8)
            \psdot(1.8,1.7)
            \psdot(2.1,2.0)
          }
          {
            \psset{linewidth=1.0pt}
            % slope          0.52212
            % y intercept at 0.86372
            \psline(0.7,1.2292)(3.2,2.5345)
          }
        }
      }
    \end{pspicture}
  \ec
\end{figure}

\index{linear system!overdetermined}
\index{overdetermined linear system}
The foregoing is all mathematically irreproachable.  By the fifth Saturday
however we shall have gathered more production data, plotted on the
figure's right, to which we should like to fit a better line to predict
production more accurately.
The added data present a problem.
Statistically, the added data are welcome, but geometrically
we need only two points to specify a line; what are we to do with the
other three?  The five points together overdetermine the linear system
\[
  \mf{cc}{u_1&1\\u_2&1\\u_3&1\\u_4&1\\u_5&1} \mf{c}{\sigma\\\gamma} = \mf{c}{b_1\\b_2\\b_3\\b_4\\b_5}.
\]
There is no way to draw a single straight line $b = \sigma u + \gamma$
exactly through all five, for in placing the line we enjoy only two
degrees of freedom.%
\footnote{
  Section~\ref{mtxinv:245.70} characterized a line as enjoying only one
  degree of freedom.  Why now two?  The answer is that
  \S~\ref{mtxinv:245.70} discussed travel along a line rather than
  placement of a line as here.  Though both involve lines, they differ
  as driving an automobile differs from washing one.  Do not let this
  confuse you.
}

The proper approach is to draw among the data points a single straight
line that misses the points as narrowly as possible.  More precisely,
the proper approach
chooses parameters~$\sigma$ and~$\gamma$ to minimize the squared
residual norm $[\ve r(\ve x)]^{*}[\ve r(\ve x)]$ of \S~\ref{mtxinv:315},
given that
\[
  A = \mf{cc}{u_1&1\\u_2&1\\u_3&1\\u_4&1\\u_5&1\\\multicolumn{2}{c}{\vdots}},
  \ \ \ve x = \mf{c}{\sigma\\\gamma},
  \ \ \ve b = \mf{c}{b_1\\b_2\\b_3\\b_4\\b_5\\\vdots}.
\]
Such parameters constitute a \emph{least-squares} solution.

The matrix~$A$ in the example has two columns, data marching on the
left, all ones on the right.  This is a typical structure for~$A$, but
in general any matrix~$A$ with any number of columns of any content
might arise (because there were more than two relevant variables or
because some data merited heavier weight than others, among many further
reasons).  Whatever matrix~$A$ might arise from whatever source, this
section attacks the difficult but important problem of approximating
optimally a solution to the general, possibly unsolvable linear
system~(\ref{mtxinv:315:05}), $A\ve x \approx \ve b$.

\subsection{Least squares in the real domain}
\label{mtxinv:320.20}
\index{solution!of least-squares}
\index{least-squares solution}

\index{Jacobian derivative}
\index{derivative!Jacobian}
\index{matrix!real}
The least-squares problem is simplest when the matrix~$A$ enjoys full
column rank and no complex numbers are involved.  In this case, we seek to
minimize the squared residual norm
\bqb
  [\ve r(\ve x)]^{T}[\ve r(\ve x)]
    &=& (\ve b - A\ve x)^{T}(\ve b - A\ve x)
  \\&=& \ve x^{T}A^{T}A\ve x + \ve b^{T}\ve b
    - \left(
      \ve x^{T} A^{T}\ve b + \ve b^{T}A\ve x
    \right)
  \\&=& \ve x^{T}A^{T}A\ve x + \ve b^{T}\ve b
    - 2\ve x^{T}A^{T}\ve b
  \\&=& \ve x^{T}A^{T}\left(A\ve x - 2\ve b\right) + \ve b^{T}\ve b,
\eqb
in which the transpose is used interchangeably for the adjoint because
all the numbers involved happen to be real.  The norm is minimized where
\[
  \frac{d}{d\ve x}\left(\ve r^{T}\ve r\right) = 0
\]
(in which $d/d\ve x$ is the Jacobian operator of \S~\ref{matrix:350}).
A requirement that
\[
  \frac{d}{d\ve x}\left[\ve x^{T}A^{T}\left(A\ve x - 2\ve b\right) + \ve b^{T}\ve b \right] = 0
\]
comes of combining the last two equations.  Differentiating by the
Jacobian product rule~(\ref{matrix:350:Jacobian-prod}) yields the
equation
\[
  \ve x^{T}A^{T}A
  + \left[A^{T}\left(A\ve x - 2\ve b\right)\right]^{T}
  = 0;
\]
or, after transposing the equation, rearranging terms and dividing by 2,
the simplified equation
\[
  A^{T}A\ve x = A^{T}\ve b.
\]
Assuming (as warranted by \S~\ref{mtxinv:320.30}, next) that the $n
\times n$ square matrix~$A^{T}A$ is invertible, the simplified equation
implies the approximate but optimal least-squares solution
\bq{mtxinv:320:leastsq}
  \ve x = \left(A^{T}A\right)^{-1}A^{T}\ve b
\eq
to the unsolvable linear system~(\ref{mtxinv:315:05}) in the restricted
but quite typical case that~$A$ and~$\ve b$ are real and~$A$ has full
column rank.

Equation~(\ref{mtxinv:320:leastsq}) plots the line on
Fig.~\ref{mtxinv:320:fig1}'s right.  As the reader can see, the line
does not pass through all the points, for no line can; but it does pass
pretty convincingly nearly among them.  In fact it passes optimally
nearly among them.  No line can pass more nearly, in the
squared-residual norm sense of~(\ref{mtxinv:residual-norm}).%
\footnote{%
  \index{metric}%
  Here is a nice example of the use of the mathematical adjective
  \emph{optimal} in its adverbial form.  ``Optimal'' means ``best.''
  Many problems in applied mathematics involve discovering the best of
  something.  What constitutes the best however can be a matter of
  judgment, even of dispute.  We will leave to the philosopher and the
  theologian the important question of what constitutes objective good,
  for applied mathematics is a poor guide to such mysteries.  The role
  of applied mathematics is to construct suitable models to calculate
  quantities needed to achieve some definite good; its role is not,
  usually, to identify the good as good in the first place.

  \index{optimality}%
  \index{cost function}%
  One generally establishes mathematical optimality by some suitable,
  nonnegative, real \emph{cost function} or \emph{metric,} and the less,
  the better.  Strictly speaking, the mathematics cannot tell us which
  metric to use, but where no other consideration prevails the applied
  mathematician tends to choose the metric that best simplifies the
  mathematics at hand---and, really, that is about as good a way to
  choose a metric as any.  The metric~(\ref{mtxinv:residual-norm}) is so
  chosen.

  ``But,'' comes the objection, ``what if some more complicated
  metric is better?''

  Well, if the other metric really, objectively is better, then one
  should probably use it.  In general however the mathematical question
  is: what does one mean by ``better?''  Better by which metric?  Each
  metric is better according to itself.  This is where the
  mathematician's experience, taste and judgment come in.

  In the present section's example, too much labor on the freeway job
  might actually slow construction rather than speed it.  One could
  therefore seek to fit not a line but some downward-turning curve to
  the data.  Mathematics offers many downward-turning curves.  A circle,
  maybe?  Not likely.  An experienced mathematician would probably
  reject the circle on the aesthetic yet practical ground that the
  parabola $b = \alpha u^2 + \sigma u + \gamma$ lends itself to easier
  analysis.  Yet even fitting a mere straight line offers choices.  One
  might fit the line to the points $(b_i,u_i)$ or $(\ln u_i,\ln b_i)$
  rather than to the points $(u_i,b_i)$.  The three resulting lines
  differ subtly.  They predict production differently.  The adjective
  ``optimal'' alone evidently does not always tell us all we need to
  know.

  Section~\ref{noth:420} offers a choice between averages that resembles
  in spirit this footnote's choice between metrics.
}

\subsection{The invertibility of~$A^{*}A$}
\label{mtxinv:320.30}
\index{invertibility}

Section~\ref{mtxinv:320.20} has assumed correctly but unwarrantedly that
the product~$A^{T}A$ were invertible for real~$A$ of full column rank.
For real~$A$, it happens that $A^{T}=A^{*}$, so it only broadens the
same assumption to suppose that the product~$A^{*}A$ were invertible for
complex~$A$ of full column rank.%
\footnote{
  Notice that if~$A$ is tall, then~$A^{*}A$ is a compact, $n \times n$
  square, whereas~$AA^{*}$ is a big, $m \times m$ square.  It is the
  compact square that concerns this section.  The big square is not very
  interesting and in any case is not invertible.
}
This subsection warrants the latter assumption, thereby incidentally
also warranting the former.

\index{column rank}
\index{rank!column}
Let~$A$ be a complex, $m \times n$ matrix of full column rank $r=n \le
m$.  Suppose falsely that~$A^{*}A$ were not invertible but singular.
Since the product~$A^{*}A$ is a square, $n \times n$ matrix,
this is to suppose (\S~\ref{mtxinv:220}) that the product's rank $r'<n$
were less than full, implying (\S~\ref{gjrank:340.25}) that its columns
(as its rows) depended on one another.  This would mean that there
existed a nonzero, $n$-element vector~$\ve u$ for which
\[
  A^{*}A\ve u = 0, \ \ I_n\ve u \neq 0.
\]
Left-multiplying by~$\ve u^{*}$ would give that
\[
  \ve u^{*}A^{*}A\ve u = 0, \ \ I_n\ve u \neq 0,
\]
or in other words that
\[
  (A\ve u)^{*}(A\ve u) = \sum_{i=1}^{n} \left| [A\ve u]_i \right|^2 = 0, \ \ I_n\ve u \neq 0.
\]
But this could only be so if
\[
  A\ve u = 0, \ \ I_n\ve u \neq 0,
\]
impossible when the columns of~$A$ are independent.  The contradiction
proves false the assumption which gave rise to it.  The false
assumption: that~$A^{*}A$ were singular.

Thus, \emph{the $n \times n$ product~$A^{*}A$ is invertible for any tall
or square, $m \times n$ matrix~$A$ of full column rank $r=n \le m$.}

\subsection{Positive definiteness}
\label{mtxinv:320.34}
\index{positive definiteness}

% As long as we are on the topic, now seems as good a time as any to
% introduce the positive definite matrix.

An $n \times n$ matrix~$C$ is \emph{positive definite} if and only if
\bq{mtxinv:320:34}
  \mbox{
    $\Im(\ve u^{*}C\ve u) = 0$ and
    $\Re(\ve u^{*}C\ve u) > 0$
    for all $I_n\ve u \neq 0$.
  }
\eq
As in \S~\ref{mtxinv:320.30}, here also when a matrix~$A$ has full
column rank $r = n \le m$ the product $\ve u^{*}A^{*}A\ve u = (A\ve
u)^{*}(A\ve u)$ is real and positive for all nonzero, $n$-element
vectors~$\ve u$.  Thus per~(\ref{mtxinv:320:34}) \emph{the
product~$A^{*}A$ is positive definite for any matrix~$A$ of full column
rank.}

\index{nonnegative definiteness}
An $n \times n$ matrix~$C$ is \emph{nonnegative definite} if and only if
\bq{mtxinv:320:35}
  \mbox{
    $\Im(\ve u^{*}C\ve u) = 0$ and
    $\Re(\ve u^{*}C\ve u) \ge 0$
    for all~$\ve u$.
  }
\eq
By reasoning like the last paragraph's, \emph{the product~$A^{*}A$ is
nonnegative definite for any matrix~$A$ whatsoever.}

Such definitions might seem opaque, but their sense is that a positive
definite operator never reverses the thing it operates on, that the
product~$A\ve u$ points more in the direction of~$\ve u$ than of~$-\ve
u$.  Section~\ref{mtxinv:445} explains further.  A positive definite
operator resembles a positive scalar in this sense.

\subsection{The Moore-Penrose pseudoinverse}
\label{mtxinv:320.50}

\index{factorization!full-rank}
\index{full-rank factorization}
\index{conjecture}
\index{residual!minimizing the}
Not every $m \times n$ matrix~$A$ enjoys full rank.  According
to~(\ref{gjrank:340:26}), however, every $m \times n$ matrix~$A$ of
rank~$r$ can be factored into a product%
\footnote{
  This subsection uses the symbols~$B$ and~$\ve b$ for unrelated
  purposes, which is unfortunate but conventional.  See
  footnote~\ref{mtxinv:315:95}.
}
\[
  A = BC
\]
of an $m \times r$ tall or square matrix~$B$ and an $r \times n$ broad
or square matrix~$C$, both of which factors themselves enjoy full
rank~$r$.  (If~$A$ happens to have full row or column rank, then one can
just choose $B=I_m$ or $C=I_n$; but even if~$A$ lacks full rank, the
Gauss-Jordan decomposition of eqn.~\ref{gjrank:341:GJ} finds at least
the full-rank factorization $B=G_>I_r$, $C=I_rG_<$.)  This being so, a
conjecture seems warranted.  Suppose that, inspired
by~(\ref{mtxinv:320:leastsq}), we manipulated~(\ref{mtxinv:315:05}) by
the successive steps
\bqb
  A \ve x &\approx& \ve b, \\
  BC \ve x &\approx& \ve b, \\
  (B^{*}B)^{-1}B^{*}BC \ve x &\approx& (B^{*}B)^{-1}B^{*}\ve b, \\
  C \ve x &\approx& (B^{*}B)^{-1}B^{*}\ve b.
\eqb
Then suppose that we changed
\[
  C^{*} \ve u \la \ve x,
\]
thus restricting~$\ve x$ to the space addressed by the independent
columns of~$C^{*}$.  Continuing,
\bqb
  CC^{*} \ve u &\approx& (B^{*}B)^{-1}B^{*}\ve b, \\
  \ve u &\approx& (CC^{*})^{-1}(B^{*}B)^{-1}B^{*}\ve b.
\eqb
Changing the variable back and (because we are conjecturing and can do
as we like), altering the~``$\approx$'' sign to~``$=$,''
\bq{mtxinv:320:50}
  \ve x = C^{*}(CC^{*})^{-1}(B^{*}B)^{-1}B^{*}\ve b.
\eq
Equation~(\ref{mtxinv:320:50}) has a pleasingly symmetrical form, and we
know from \S~\ref{mtxinv:320.30} at least that the two matrices it
tries to invert are invertible.  So here is our conjecture:
\bi
  \item no~$\ve x$ enjoys a smaller squared residual norm~$\ve r^{*}\ve
    r$ than the~$\ve x$ of~(\ref{mtxinv:320:50}) does; and
  \item among all~$\ve x$ that enjoy the same, minimal squared residual
    norm, the~$\ve x$ of~(\ref{mtxinv:320:50}) is strictly least in
    magnitude.
\ei
The conjecture is bold, but if you think about it in the right way it
is not unwarranted under the circumstance.  After
all,~(\ref{mtxinv:320:50}) does resemble~(\ref{mtxinv:320:leastsq}), the
latter of which admittedly requires real~$A$ of full column rank but
does minimize the residual when its requirements are met; and, even if
there were more than one~$\ve x$ which minimized the residual, one of
them might be smaller than the others: why not the~$\ve x$
of~(\ref{mtxinv:320:50})?  One can but investigate.

\index{deviation}
\index{reverse logic}
\index{logic!reverse}
\index{inequality}
The first point of the conjecture is symbolized
\[
  \ve r^{*}(\ve x) \ve r(\ve x) \le \ve r^{*}(\ve x + \Delta\ve x) \ve r(\ve x + \Delta\ve x),
\]
where~$\Delta\ve x$ represents the deviation, whether small, moderate or
large, of some alternate~$\ve x$ from the~$\ve x$
of~(\ref{mtxinv:320:50}).  According to~(\ref{mtxinv:residual}), this is
\[
  [\ve b-A\ve x]^{*}[\ve b-A\ve x]
  \le
  [\ve b-(A)(\ve x+\Delta\ve x)]^{*}[\ve b-(A)(\ve x+\Delta\ve x)].
\]
Reorganizing,
\[
  [\ve b-A\ve x]^{*}[\ve b-A\ve x]
  \le
  [(\ve b-A\ve x)-A\,\Delta\ve x]^{*}[(\ve b-A\ve x)-A\,\Delta\ve x].
\]
Distributing factors and canceling like terms,
\[
  0 \le
    - \Delta\ve x^{*}\,A^{*}(\ve b - A\ve x)
    - (\ve b - A\ve x)^{*}A\,\Delta\ve x
    + \Delta\ve x^{*}\,A^{*}A\,\Delta\ve x.
\]
But according to~(\ref{mtxinv:320:50}) and the full-rank factorization
$A=BC$,
\bqb
  A^{*}(\ve b-A\ve x) &=& A^{*}\ve b - A^{*}A\ve x
  \\&=& [C^{*}B^{*}][\ve b] - [C^{*}B^{*}][BC][C^{*}(CC^{*})^{-1}(B^{*}B)^{-1}B^{*}\ve b]
  \\&=& C^{*}B^{*}\ve b - C^{*}(B^{*}B)(CC^{*})(CC^{*})^{-1}(B^{*}B)^{-1}B^{*}\ve b
  \\&=& C^{*}B^{*}\ve b - C^{*}B^{*}\ve b = 0,
\eqb
which reveals two of the inequality's remaining three terms to be zero,
leaving an assertion that
\[
  0 \le \Delta\ve x^{*}\,A^{*}A\,\Delta\ve x.
\]
Each step in the present paragraph is reversible,%
\footnote{
  The paragraph might inscrutably but logically instead have ordered the
  steps in reverse as in \S\S~\ref{noth:420.20} and~\ref{inttx:250}.
  See chapter~\ref{noth}'s footnote~\ref{noth:420:85}.
}
so the assertion in the last form is logically equivalent to the
conjecture's first point, with which the paragraph began.  Moreover, the
assertion in the last form is correct because the product of any matrix
and its adjoint according to \S~\ref{mtxinv:320.34} is a nonnegative
definite operator, thus establishing the conjecture's first point.

The conjecture's first point, now established, has it that no $\ve x +
\Delta\ve x$ enjoys a smaller squared residual norm than the~$\ve x$
of~(\ref{mtxinv:320:50}) does.  It does not claim that no $\ve x +
\Delta\ve x$ enjoys the same, minimal squared residual norm.  The latter
case is symbolized
\[
  \ve r^{*}(\ve x) \ve r(\ve x) = \ve r^{*}(\ve x + \Delta\ve x) \ve r(\ve x + \Delta\ve x),
\]
or equivalently by the last paragraph's logic,
\[
  0 = \Delta\ve x^{*}\,A^{*}A\,\Delta\ve x;
\]
or in other words,
\[
  A\,\Delta\ve x = 0.
\]
But $A=BC$, so this is to claim that
\[
  B(C\,\Delta\ve x) = 0,
\]
which since~$B$ has full column rank is possible only if
\[
  C\,\Delta\ve x = 0.
\]
Considering the product $\Delta\ve x^{*}\,\ve x$ in light
of~(\ref{mtxinv:320:50}) and the last equation, we observe that
\bqb
  \Delta\ve x^{*}\,\ve x
  &=& \Delta\ve x^{*}\,[C^{*}(CC^{*})^{-1}(B^{*}B)^{-1}B^{*}\ve b]
  \\&=& [C\,\Delta\ve x]^{*}[(CC^{*})^{-1}(B^{*}B)^{-1}B^{*}\ve b],
\eqb
which is to observe that
\[
  \Delta\ve x^{*}\,\ve x = 0
\]
for any~$\Delta\ve x$ for which $\ve x + \Delta\ve x$ achieves minimal
squared residual norm.

Returning attention to the conjecture, its second point is symbolized
\[
  \ve x^{*}\ve x
  <
  (\ve x+\Delta\ve x)^{*}(\ve x+\Delta\ve x)
\]
for any
\[
  \Delta\ve x \neq 0
\]
for which $\ve x + \Delta\ve x$ achieves minimal squared residual norm
(note that it's ``$<$'' this time, not ``$\le$'' as in the conjecture's
first point).  Distributing factors and canceling like terms,
\[
  0 <
  \ve x^{*}\,\Delta\ve x
  + \Delta\ve x^{*}\,\ve x
  + \Delta\ve x^{*}\,\Delta\ve x.
\]
But the last paragraph has found that $\Delta\ve x^{*}\,\ve x = 0$ for
precisely such~$\Delta\ve x$ as we are considering here, so the last
inequality reduces to read
\[
  0 < \Delta\ve x^{*}\,\Delta\ve x,
\]
which naturally for $\Delta\ve x \neq 0$ is true.
Since each step in the paragraph is reversible, reverse logic
establishes the conjecture's second point.

With both its points established, the conjecture is true.

If $A=BC$ is a full-rank factorization, then the matrix%
\footnote{
  Some books print~$A^\dagger$ as~$A^{+}$.
}
\bq{mtxinv:psinv}
  A^\dagger \equiv C^{*}(CC^{*})^{-1}(B^{*}B)^{-1}B^{*}
\eq
of~(\ref{mtxinv:320:50}) is called the \emph{Moore-Penrose
pseudoinverse} of~$A$, more briefly the \emph{pseudoinverse}
of~$A$.  Whether underdetermined, exactly determined, overdetermined or
even degenerate, every matrix has a Moore-Penrose pseudoinverse.
Yielding the optimal approximation
\bq{mtxinv:320:70}
  \ve x = A^\dagger \ve b,
\eq
the Moore-Penrose solves the linear system~(\ref{mtxinv:315:05}) as well
as the system can be solved---exactly if possible, with minimal squared
residual norm if impossible.  If~$A$ is square and invertible, then the
Moore-Penrose $A^\dagger = A^{-1}$ is just the inverse, and then of
course~(\ref{mtxinv:320:70}) solves the system uniquely and exactly.
Nothing can solve the system uniquely if~$A$ has broad shape
but the Moore-Penrose still solves the system exactly in that case as
long as~$A$ has full row rank, moreover minimizing the solution's
squared magnitude~$\ve x^{*}\ve x$ (which the solution of
eqn.~\ref{mtxinv:240:25} fails to do).  If~$A$ lacks full row rank, then
the Moore-Penrose solves the system as nearly as the system can be
solved (as in Fig.~\ref{mtxinv:320:fig1}) and as a side-benefit also
minimizes~$\ve x^{*}\ve x$.  The Moore-Penrose is thus a general-purpose
solver and approximator for linear systems.  It is a significant
discovery.%
\footnote{
  \cite[\S~3.3]{Beattie}%
  \cite[``Moore-Penrose generalized inverse'']{planetm}
}

% ----------------------------------------------------------------------

\section{The multivariate Newton-Raphson iteration}
\label{mtxinv:420}
\index{Newton-Raphson iteration!multivariate}
\index{multivariate Newton-Raphson iteration}
\index{Raphson, Joseph (1648--1715)}
\index{Newton, Sir Isaac (1642--1727)}
\index{root!finding of numerically}

When we first met the Newton-Raphson iteration in \S~\ref{drvtv:270} we
lacked the matrix notation and algebra to express and handle
vector-valued functions adeptly.  Now that we have the notation and
algebra we can write down the multivariate Newton-Raphson iteration
almost at once.

\index{Jacobian derivative}
\index{derivative!Jacobian}
The iteration approximates the nonlinear vector function $\ve f(\ve x)$
by its tangent
\[
  \tilde{\ve f}_k(\ve x) = \ve f(\ve x_k) + \left[\frac{d}{d\ve x}\ve
  f(\ve x)\right]_{\ve x=\ve x_k} (\ve x-\ve x_k),
\]
where $d\ve f/d\ve x$ is the Jacobian derivative of \S~\ref{matrix:350}.
It then approximates the root $\ve x_{k+1}$ as the point at which
$\tilde{\ve f}_k(\ve x_{k+1}) = 0$:
\[
  \tilde{\ve f}_k(\ve x_{k+1}) = 0
  = \ve f(\ve x_k) + \left[\frac{d}{d\ve x}\ve f(\ve x)\right]_{\ve x=\ve
  x_k} (\ve x_{k+1}-\ve x_k).
\]
Solving for $\ve x_{k+1}$ (approximately if necessary), we have that
\bq{mtxinv:NR}
  \ve x_{k+1} = \left\{ \ve x -
  \left[\frac{d}{d\ve x}\ve f(\ve x)\right]^\dagger
  \ve f(\ve x) \right\}_{\ve x=\ve x_k},
\eq
where~$[\cdot]^\dagger$ is the Moore-Penrose pseudoinverse of
\S~\ref{mtxinv:320}---which is just the ordinary inverse~$[\cdot]^{-1}$
of \S~\ref{mtxinv:220} if~$\ve f$ and~$\ve x$ happen each to have the
same number of elements.  Refer to
\S~\ref{drvtv:270} and Fig.~\ref{drvtv:270:fig1}.%
\footnote{
  \cite{Scales-lecture}
}

Despite the Moore-Penrose notation of~(\ref{mtxinv:NR}), the
Newton-Raphson iteration is not normally meant to be applied at a
value of~$\ve x$ for which the Jacobian is degenerate.  The iteration
intends rather in light of~(\ref{mtxinv:psinv}) that
\bq{mtxinv:420:70}
  \left[\frac{d}{d\ve x}\ve f(\ve x)\right]^\dagger
  = \begin{cases}
      \left[d\ve f/d\ve x\right]^{*}
      \left(
        \left[d\ve f/d\ve x\right]
        \left[d\ve f/d\ve x\right]^{*}
      \right)^{-1}
      & \mbox{
        if $r=m \le n$,
      }
    \\
      \left[d\ve f/d\ve x\right]^{-1}
      & \mbox{
        if $r=m=n$,
      }
    \\
      \left(
        \left[d\ve f/d\ve x\right]^{*}
        \left[d\ve f/d\ve x\right]
      \right)^{-1}
      \left[d\ve f/d\ve x\right]^{*}
      & \mbox{
        if $r=n \le m$,
      }
  \end{cases}
\eq
where $B=I_m$ in the first case and $C=I_n$ in the last.  It does not
intend to use the full~(\ref{mtxinv:psinv}).  If both $r<m$ and
$r<n$---which is to say, if the Jacobian is
degenerate---then~(\ref{mtxinv:420:70}) fails, as though the curve of
Fig.~\ref{drvtv:270:fig1} ran horizontally at the test point---when one
quits, restarting the iteration from another point.

% ----------------------------------------------------------------------

\section{The dot product}
\label{mtxinv:445}
\index{dot product}
\index{inner product}
\index{product!dot or inner}
\index{vector!dot or inner product of two}

\index{elementary vector}
\index{vector!elementary}
\index{column vector}
\index{row vector}
\index{vector!column}
\index{vector!row}
\index{vector!orientation of}
The \emph{dot product} of two vectors, also called the \emph{inner
product,}%
\footnote{
  The term \emph{inner product} is often used to indicate a broader
  class of products than the one defined here, especially in some of
  the older literature.  Where used, the notation
  usually resembles $\langle \ve a, \ve b \rangle$ or $(\ve b,\ve a)$,
  both of which mean $\ve a^{*} \cdot \ve b$ (or, more broadly, some
  similar product), except that
  which of~$\ve a$ and~$\ve b$ is conjugated depends on the author.
  Most recently, at least in the author's country, the usage $\langle
  \ve a, \ve b \rangle \equiv \ve a^{*} \cdot \ve b$ seems
  to be emerging as standard where the dot is not used, as in
  \cite[\S~3.1]{Beattie}\cite[chapter~4]{Franklin} (but slightly
  contrary to \cite[\S~2.1]{HFDavis}, for example).  At any rate, this
  book prefers the dot.
}
is the product of the two vectors to the extent to which they run in the
same direction.  It is written as
\[
  \ve a \cdot \ve b.
\]
In general,
\[
  \ve a \cdot \ve b =
    (a_1\ve e_1 + a_2\ve e_2 + \cdots + a_n\ve e_n)
    \cdot
    (b_1\ve e_1 + b_2\ve e_2 + \cdots + b_n\ve e_n).
\]
But if the dot product is to mean anything, it must be that
\bq{mtxinv:445:05}
  \ve e_i \cdot \ve e_j = \delta_{ij},
\eq
where the Kronecker delta~$\delta_{ij}$ is as defined in
\S~\ref{matrix:150}.  Therefore,
\[
  \ve a \cdot \ve b =
    a_1b_1 + a_2b_2 + \cdots + a_nb_n;
\]
or, more concisely,
\bq{mtxinv:445:dotre}
  \ve a \cdot \ve b \equiv \ve a^T\ve b
    = \sum_{j=-\infty}^\infty a_jb_j.
\eq
The dot notation does not worry whether its arguments are column or row
vectors, incidentally:
\[
  \ve a \cdot \ve b
  = \ve a \cdot \ve b^{T}
  = \ve a^{T} \cdot \ve b
  = \ve a^{T} \cdot \ve b^{T}
  = \ve a^T\ve b.
\]
That is, if either vector is wrongly oriented, the notation implicitly
reorients it before using it.  (The more orderly notation~$\ve a^T\ve
b$ by contrast assumes that both are proper column vectors.)

Where vectors may have complex elements, usually one is not interested
in $\ve a \cdot \ve b$ so much as in
\bq{mtxinv:445:dot}
  \ve a^{*} \cdot \ve b \equiv \ve a^{*}\ve b
    = \sum_{j=-\infty}^\infty a_j^{*}b_j.
\eq
The reason is that
\[
  \Re(\ve a^{*} \cdot \ve b)
  = \Re(\ve a)\cdot\Re(\ve b) + \Im(\ve a)\cdot\Im(\ve b),
\]
with the product of the imaginary parts added not subtracted, thus
honoring the right Argand sense of ``the product of the two vectors to
the extent to which they run in the same direction.''

\index{magnitude!of a vector}
\index{vector!magnitude of}
\index{vector!unit}
\index{unit vector}
By the Pythagorean theorem, the dot product
\bq{mtxinv:445:dotmag}
  \left|\ve a\right|^2 = \ve a^{*}\cdot\ve a
\eq
gives the square of a vector's magnitude, always real, never negative.
The unit vector in $\ve a$'s direction then is
\bq{mtxinv:445:20}
  \vu a \equiv \frac{\ve a}{\left|\ve a\right|}
  = \frac{\ve a}{\sqrt{\ve a^{*}\cdot\ve a}},
\eq
from which
\bq{mtxinv:445:25}
  \left|\vu a\right|^2 = \vu a^{*}\cdot\vu a = 1.
\eq

\index{orthogonal vector}
\index{vector!orthogonal}
\index{vector!angle between two}
When two vectors do not run in the same direction at all, such that
\bq{mtxinv:445:30}
  \ve a^{*} \cdot \ve b = 0,
\eq
the two vectors are said to lie \emph{orthogonal} to one another.
Geometrically this puts them at right angles.  For other angles~$\theta$
between two vectors,
\bq{mtxinv:445:35}
  \vu a^{*} \cdot \vu b = \cos \theta,
\eq
which formally defines the angle~$\theta$ even when~$\ve a$ and~$\ve b$
have more than three elements each.

% ----------------------------------------------------------------------

\section{Complex vector inequalities}
\label{mtxinv:447}
\index{inequalities!complex vector}
\index{inequalities!vector}

The triangle inequalities~(\ref{alggeo:323:20})
and~(\ref{trig:278:triangle2}) lead one to hypothesize generally that
\bq{mtxinv:447:10}
  \left|\ve a\right|-\left|\ve b\right| \le \left|\ve a
  + \ve b\right| \le \left|\ve a\right| + \left|\ve b\right|
\eq
for any complex, $n$-dimensional vectors~$\ve a$ and~$\ve b$.
Section~\ref{mtxinv:447.30} will prove
% bad break
(\ref{mtxinv:447:10}); but first, \S~\ref{mtxinv:447.20} develops
a related inequality by Schwarz.

\subsection{The Schwarz inequality}
\label{mtxinv:447.20}
\index{inequality!Schwarz}
\index{Schwarz inequality}
\index{Cauchy-Schwarz inequality}
\index{Schwarz, Hermann (1843--1921)}

The \emph{Schwarz inequality,} alternately the \emph{Cauchy-Schwarz
inequality,}\footnote{%
  Pronounced as ``Schwartz,'' almost as ``{S}{c}{h}{w}{o}{r}{t}{z}.''
  You can sound out the German~\emph{w} like an English~\emph{v} if you
  wish.  The other name being French is pronounced as
  ``Co-{s}{h}{e}{e},'' preferably with little stress but---to the extent
  necessary while speaking English---with stress laid on the first
  syllable.
  %
  %And why isn't it Schwartz, like Leibniz is Leibnitz?  Various
  %reasons, too boring to explain here.%
}
has that
\bq{mtxinv:schwarz}
  |\ve a^{*} \cdot \ve b| \le |\ve a| |\ve b|.
\eq
Roughly in words: the dot product does not exceed the product of
lengths.

If the three-dimensional geometrical vectors with their dot products of
chapters~\ref{trig} and~\ref{vector} are already familiar to you
then~(\ref{mtxinv:schwarz}) might seem too obvious to bother proving.
The present chapter however brings an arbitrary number~$n$ of
dimensions.  Furthermore, elements in any or every dimension can be complex.
Therefore, the geometry is not so easy to visualize in the general case.
One would prefer an algebraic proof.\footnote{See~\cite{Wu/Wu}
and~\cite[\S~2.1]{HFDavis} for various other proofs, one of which partly
resembles the proof given here.  See also \cite[``Cauchy-Schwarz
inequality,'' 17:56, 22~May 2017]{wikip}.}

\index{symmetry}
\index{asymmetry}
\index{index!swapping of}
\index{swapping of indices}
The proof is by contradiction.  We suppose falsely that
\[
  |\ve a^{*} \cdot \ve b| > |\ve a| |\ve b|.
\]
Squaring and using~(\ref{alggeo:225:24}) and~(\ref{mtxinv:445:dotmag}),
\[
  (\ve a^{*} \cdot \ve b)(\ve b^{*} \cdot \ve a)
    > (\ve a^{*} \cdot \ve a)(\ve b^{*} \cdot \ve b),
\]
or in other words,
\[
  \sum_{i,j} a_i^{*}b_i^{\mbox{}}b_j^{*}a_j^{\mbox{}}
    > \sum_{i,j} a_i^{*}a_i^{\mbox{}}b_j^{*}b_j^{\mbox{}},
\]
wherein each side of the inequality is real-valued by construction (that
is, each side is real-valued because we had started with a real
inequality and---despite that elements on either side may be
complex---no step since the start has made either side of the inequality
complex as a whole).  One would like to segregate conjugated elements
for separate handling; it is not easy to see how to segregate them all
at once but to reorder factors as
\[
  \sum_{i,j} \left[(a_ib_j)^{*}(b_ia_j)\right]
    > \sum_{i,j} \left[(a_ib_j)^{*}(a_ib_j)\right]
\]
at least makes a step in the right direction.  The last inequality is
unhelpfully asymmetric, though, so we swap indices $i\lra j$ to write
the same inequality as that
\[
  \sum_{i,j} \left[(b_ia_j)^{*}(a_ib_j)\right]
    > \sum_{i,j} \left[(b_ia_j)^{*}(b_ia_j)\right].
\]
The swapped inequality is asymmetric too but one can add it to the
earlier, unswapped inequality to achieve the symmetric form
\[
  \sum_{i,j} \left[(a_ib_j)^{*}(b_ia_j)
    + (b_ia_j)^{*}(a_ib_j)\right]
    > \sum_{i,j} \left[(a_ib_j)^{*}(a_ib_j)
    +(b_ia_j)^{*}(b_ia_j)\right].
\]
Does this help?  Indeed it does.  Transferring all terms to the
inequality's right side,
\[
  0 > \sum_{i,j} \left[
    (a_ib_j)^{*}(a_ib_j) +(b_ia_j)^{*}(b_ia_j)
    - (a_ib_j)^{*}(b_ia_j) - (b_ia_j)^{*}(a_ib_j)
    \right],
\]
Factoring,
\[
  0 > \sum_{i,j} \left[
    (a_ib_j-b_ia_j)^{*}
    (a_ib_j-b_ia_j)
  \right] = \sum_{i,j} \left| a_ib_j-b_ia_j \right|^2,
\]
which inequality is impossible because $0 \le \mbox{$|\cdot|$}^2$ regardless of
what the~\mbox{$|\cdot|$} might be.  The contradiction proves false the
assumption that gave rise to it, thus establishing the Schwarz
inequality of~(\ref{mtxinv:schwarz}).

\subsection{Triangle inequalities}
\label{mtxinv:447.30}
\index{triangle inequalities!complex vector}
\index{triangle inequalities!vector}

The proof of the sum hypothesis~(\ref{mtxinv:447:10}) that $\left|\ve a
+ \ve b\right| \le \left|\ve a\right| + \left|\ve b\right|$ is again by
contradiction.  We suppose falsely that
\[
  \left|\ve a + \ve b\right| > \left|\ve a\right| + \left|\ve b\right|.
\]
Squaring and using~(\ref{mtxinv:445:dotmag}),
\[
  (\ve a + \ve b)^{*} \cdot (\ve a + \ve b)
    > \ve a^{*}\cdot\ve a + 2\left|\ve a\right|\left|\ve b\right| + \ve b^{*}\cdot\ve b.
\]
Distributing factors and canceling like terms,
\[
  \ve a^{*}\cdot\ve b + \ve b^{*}\cdot\ve a
    > 2\left|\ve a\right|\left|\ve b\right|,
\]
where both sides of the inequality remain real for the same reason as in
the last subsection.  On the left, the imaginary parts offset
because $(\ve b^{*}\cdot\ve a)^{*} = \ve a^{*}\cdot\ve b$,
leaving
\[
  2\Re\left({\ve a^{*}\cdot\ve b}\right) > 2\left|\ve a\right|\left|\ve b\right|.
\]
However, the real part of the Schwarz inequality~(\ref{mtxinv:schwarz}) has that
\[
  \Re\left({\ve a^{*}\cdot\ve b}\right)
  \le \left|{\ve a^{*}\cdot\ve b}\right|
  \le \left|\ve a\right|\left|\ve b\right|,
\]
which, when doubled, contradicts the last finding.
The contradiction proves false the
assumption that gave rise to it, thus establishing the sum hypothesis
of~(\ref{mtxinv:447:10}).

The difference hypothesis that $\left|\ve a\right|-\left|\ve b\right|
\le \left|\ve a + \ve b\right|$ is established by defining a vector $\ve
c$ such that
\[
  \ve a + \ve b + \ve c = 0,
\]
whereupon according to the sum hypothesis
\[
  \begin{split}
    \left|\ve a + \ve c\right| &\le \left|\ve a\right| + \left|\ve c\right|, \\
    \left|\ve b + \ve c\right| &\le \left|\ve b\right| + \left|\ve c\right|.
  \end{split}
\]
That is,
\[
  \begin{split}
    \left|-\ve b\right| &\le \left|\ve a\right| + \left|-\ve a-\ve b\right|, \\
    \left|-\ve a\right| &\le \left|\ve b\right| + \left|-\ve a-\ve b\right|,
  \end{split}
\]
which is the difference hypothesis in disguise.  This completes the proof
of the triangle inequalities~(\ref{mtxinv:447:10})

\index{Minkowski, Hermann (1864--1909)}
\index{Minkowski inequality}
The triangle sum inequality is alternately called the \emph{Minkowski
inequality.}\footnote{\cite[\S~2.1]{HFDavis}}

As in \S~\ref{trig:278}, here too we can extend the sum inequality to
the even more general form
\bq{mtxinv:447:20}
  \left|\sum_k \ve a_k \right| \le \sum_k \left| \ve a_k \right|.
\eq

% ----------------------------------------------------------------------

\section{The orthogonal complement}
\label{mtxinv:450}
\index{orthogonal complement}
\index{matrix!orthogonally complementary}

\index{perpendicular matrix}
\index{matrix!perpendicular}
The $m \times (m-r)$ kernel (\S~\ref{mtxinv:245})%
\footnote{\label{mtxinv:450:08}%
  The symbol~$A^\perp$
  \cite{Hefferon}\cite{Beattie}\cite{Lay}
  can be pronounced ``A perp,'' short for ``A perpendicular,'' since
  by~(\ref{mtxinv:450:20}) $A^\perp$ is in some sense perpendicular
  to~$A$.

  If we were really precise, we might write not~$A^{\perp}$ but
  $A^{\perp(m)}$.  Refer to footnote~\ref{mtxinv:245:08}.
}
\bq{mtxinv:450:perp}
  A^\perp \equiv A^{*K}
\eq
is an interesting matrix.  By definition of the kernel, the columns
of~$A^{*K}$ are the independent vectors~$\ve u_j$ for which $A^{*}\ve
u_j = 0$, which---inasmuch as the \emph{rows} of~$A^{*}$ are the adjoints of
the columns of~$A$---is possible only when each~$\ve u_j$ lies
orthogonal to every column of~$A$.  This says that the columns of
$A^\perp \equiv A^{*K}$ address the complete space of vectors that lie
orthogonal to $A$'s columns, such that
\bq{mtxinv:450:20}
  A^{\perp{*}} A = 0 = A^{*} A^\perp.
\eq
The matrix~$A^\perp$ is called the \emph{orthogonal complement}%
\footnote{\cite[\S~3.VI.3]{Hefferon}}
or \emph{perpendicular matrix} to~$A$.

\index{rank!row}
\index{full row rank}
Among other uses, the orthogonal complement~$A^\perp$ supplies the
columns
% bad break
$A$
lacks to reach full row rank.  Properties include that
\bq{mtxinv:450:30}
  \begin{split}
    A^{*K} &= A^\perp, \\
    A^{*\perp} &= A^K.
  \end{split}
\eq

% ----------------------------------------------------------------------

\section{Gram-Schmidt orthonormalization}
\label{mtxinv:460}
\index{orthonormalization}
\index{vector!orthonormalization of}
\index{Gram-Schmidt process}
\index{Gram, J\o rgen Pedersen (1850--1916)}
\index{Schmidt, Erhard (1876--1959)}

\index{normalization}
\index{vector!normalization of}
If a vector $\ve x=A^K\ve a$ belongs to a kernel space~$A^K$
(\S~\ref{mtxinv:245}), then so equally does any~$\alpha \ve x$.  If the
vectors $\ve x_1=A^K\ve a_1$ and $\ve x_2=A^K\ve a_2$ both
belong, then so does $\alpha_1 \ve x_1 + \alpha_2 \ve x_2$.  If I claim
$A^K=[3\;4\;5;\mbox{$-1$}\;1\;0]^T$ to represent a kernel, then you
are not mistaken arbitrarily to rescale each column of my~$A^K$ by
a separate nonzero factor, instead for instance representing the same
kernel as $A^K=[6\;8\;\mbox{0xA};\frac 1 7\;\mbox{$-\frac 1 7$}\;0]^T$.
Kernel vectors have no inherent scale.  Style generally asks one
to remove the false appearance of scale by
using~(\ref{mtxinv:445:20}) \emph{to normalize} the columns of a kernel
matrix to unit magnitude before reporting them.  The same goes for the
eigenvectors of chapter~\ref{eigen} to come.

\index{orthogonalization}
\index{vector!orthogonalization of}
Where a kernel matrix~$A^K$ has two or more columns (or a repeated
eigenvalue has two or more eigenvectors), style generally asks one
not only to normalize but also \emph{to
orthogonalize} the columns before reporting them.  One orthogonalizes a
vector~$\ve b$ with respect to a vector~$\ve a$ by subtracting from~$\ve
b$ a multiple of~$\ve a$ such that
\[
  \begin{split}
    \ve a^{*} \cdot \ve b_\perp &= 0, \\
    \ve b_\perp &\equiv \ve b - \beta \ve a,
  \end{split}
\]
where the symbol~$\ve b_\perp$ represents the orthogonalized vector.
Substituting the second of these equations into the first and solving
for~$\beta$ yields that
\[
  \beta = \frac{\ve a^{*} \cdot \ve b}{\ve a^{*} \cdot \ve a}.
\]
Hence,
\bq{mtxinv:460:10}
  \begin{split}
    \ve a^{*} \cdot \ve b_\perp &= 0, \\
    \ve b_\perp &\equiv \ve b -
      \frac{\ve a^{*} \cdot \ve b}{\ve a^{*} \cdot \ve a} \ve a.
  \end{split}
\eq
But according to~(\ref{mtxinv:445:20}), $\ve a = \vu a\sqrt{\ve a^{*}
\cdot \ve a}$; and according to~(\ref{mtxinv:445:25}),
$\vu a^{*}\cdot\vu a = 1$; so,
\bq{mtxinv:460:34}
  \ve b_\perp = \ve b -
     \vu a (\vu a^{*} \cdot \ve b);
\eq
or, in matrix notation,
\[
  \ve b_\perp = \ve b -
     \vu a ( \vu a^{*} )( \ve b ).
\]
This is arguably better written,
\bq{mtxinv:460:35}
  \ve b_\perp = \left[I - (\vu a)(\vu a^{*})\right]\ve b
\eq
(observe that it's $[\vu a][\vu a^{*}]$, a matrix, rather
than the scalar $[\vu a^{*}][\vu a]$).

One \emph{orthonormalizes} a set of vectors by orthogonalizing them with
respect to one another and then normalizing each of them to unit magnitude.
The procedure to orthonormalize several vectors
\[
  \left\{ \ve x_1,\ve x_2,\ve x_3,\ldots,\ve x_n\right\}
\]
therefore is as follows.  First, normalize~$\ve x_1$
by~(\ref{mtxinv:445:20}); call the result~$\vu x_{1\perp}$.  Second,
orthogonalize~$\ve x_2$ with respect to~$\vu x_{1\perp}$
by~(\ref{mtxinv:460:34}) or~(\ref{mtxinv:460:35}), then
normalize it; call the result~$\vu x_{2\perp}$.  Third,
orthogonalize~$\ve x_3$ with respect to~$\vu x_{1\perp}$ then to~$\vu
x_{2\perp}$, then normalize it; call the result~$\vu x_{3\perp}$.
Proceed in this manner through the several~$\ve x_j$.  Symbolically,
\bq{mtxinv:460:50}
  \begin{split}
    \vu x_{j\perp} &= \frac{
      \ve x_{j\perp}
    }{
      \sqrt{
        \ve x_{j\perp}^{*}
        \ve x_{j\perp}
      }
    }, \\
    \ve x_{j\perp} &\equiv
      \left[
        \prod_{i=1}^{j-1} \left(
          I -
          \vu x_{i\perp}
          \vu x_{i\perp}^{*}
        \right)
      \right]
      \ve x_j.
  \end{split}
\eq
By the vector replacement principle of \S~\ref{gjrank:338} in light
of~(\ref{mtxinv:460:10}), the resulting orthonormal set of vectors
\[
  \left\{ \vu x_{1\perp},\vu x_{2\perp},
  \vu x_{3\perp},\ldots,\vu x_{n\perp}\right\}
\]
addresses the same space as did the original set.

Orthonormalization naturally works equally for any linearly independent
set of vectors, not only for kernel vectors or eigenvectors.  By the
technique, one can conveniently replace a set of independent vectors by
an equivalent, neater, orthonormal set which addresses precisely the
same space.

\subsection{Efficient implementation}
\label{mtxinv:460.20}
\index{efficient implementation}
\index{implementation!efficient}

\index{computer memory}
\index{memory, computer}
\index{algorithm!implementation of from an equation}
To turn an equation like the latter line of~(\ref{mtxinv:460:50}) into
an efficient numerical algorithm sometimes demands some extra thought,
in perspective of whatever it happens to be that one is trying to
accomplish.  If all one wants is some vectors orthonormalized, then the
equation as written is neat but is overkill because the product~$\vu
x_{i\perp}\vu x_{i\perp}^{*}$ is a matrix, whereas the product~$\vu
x_{i\perp}^{*}\ve x_j$ implied by~(\ref{mtxinv:460:34}) is just a
scalar.  Fortunately, one need not apply the latter line
of~(\ref{mtxinv:460:50}) exactly as written.  One can instead introduce
intermediate vectors~$\ve x_{ji}$, representing the~$\prod$
multiplication in the admittedly messier form
\bq{mtxinv:460:55}
  \begin{split}
    \ve x_{j1} &\equiv \ve x_j, \\
    \ve x_{j(i+1)} &\equiv \ve x_{ji}
      -\left( \vu x_{i\perp}^{*}
      \cdot \ve x_{ji} \right) \vu x_{i\perp}, \\
    \ve x_{j\perp} &= \ve x_{jj}.
  \end{split}
\eq
Besides obviating the matrix $I - \vu x_{i\perp}\vu x_{i\perp}^{*}$ and
the associated matrix multiplication, the messier
form~(\ref{mtxinv:460:55}) has the significant additional practical
virtue that it lets one forget each intermediate vector~$\ve x_{ji}$
immediately after using it.  (A well-written orthonormalizing computer
program reserves memory for one intermediate vector only, which memory
it repeatedly overwrites---and, actually, probably does not even reserve
that much, working rather in the memory space it has already reserved
for~$\vu x_{j\perp}$.)%
\footnote{
  \cite[``Gram-Schmidt process,'' 04:48, 11~Aug.\ 2007]{wikip}
}

Other equations one algorithmizes can likewise benefit from thoughtful
rendering.

\subsection{The Gram-Schmidt decomposition}
\label{mtxinv:460.30}
\index{$QR$ decomposition}
\index{Gram-Schmidt decomposition}
\index{orthonormalizing decomposition}
\index{decomposition!$QR$}
\index{decomposition!Gram-Schmidt}
\index{decomposition!orthonormalizing}
\index{factorization!$QR$}
\index{factorization!Gram-Schmidt}
\index{factorization!orthonormalizing}

\index{interchange!refusing an}
The orthonormalization technique this section has developed is
named the \emph{Gram-Schmidt process.}  One can turn it into the
\emph{Gram-Schmidt decomposition}
\bq{mtxinv:QR}
  \begin{split}
    A &= QR = QUDS, \\
    R &\equiv UDS,
  \end{split}
\eq
also called the \emph{orthonormalizing} or \emph{QR decomposition,}
by an algorithm that somewhat resembles the Gauss-Jordan algorithm of
\S~\ref{gjrank:341.10}; except that~(\ref{gjrank:341:05}) here becomes
\bq{mtxinv:460:60}
  A = \tilde Q \tilde U \tilde D \tilde S
\eq
and initially $\tilde Q \la A$.  By elementary column operations based
on~(\ref{mtxinv:460:50}) and~(\ref{mtxinv:460:55}), the algorithm
gradually transforms~$\tilde Q$ into a dimension-limited, $m \times r$
matrix~$Q$ of orthonormal columns, distributing the inverse elementaries
to~$\tilde U$, $\tilde D$ and~$\tilde S$ according to
Table~\ref{gjrank:337:table}---where the latter three working matrices
ultimately become the extended-operational factors~$U$, $D$ and~$S$
of~(\ref{mtxinv:QR}).

\index{nesting}
\index{loop}
\index{majority}
Borrowing the language of computer science we observe that the
indices~$i$ and~$j$ of~(\ref{mtxinv:460:50}) and~(\ref{mtxinv:460:55})
imply a two-level nested loop, one level looping over~$j$ and the other
over~$i$.  The equations suggest \emph{$j$-major nesting,} with the loop
over~$j$ at the outer level and the loop over~$i$ at the inner, such
that the several $(i,j)$ index pairs occur in the sequence (reading left
to right then top to bottom)
\[
  \br{cccc}
    (1,2) &&& \\
    (1,3) & (2,3) && \\
    (1,4) & (2,4) & (3,4) & \\
    \cdots & \cdots & \cdots & \ddots
  \er
\]
In reality, however, (\ref{mtxinv:460:55})'s middle line requires only
that no~$\vu x_{i\perp}$ be used before it is fully calculated;
otherwise that line does not care which $(i,j)$ pair follows
which.  The $i$-major nesting
\[
  \br{cccc}
    (1,2) & (1,3) & (1,4) & \cdots \\
          & (2,3) & (2,4) & \cdots \\
          &       & (3,4) & \cdots \\
    &&& \ddots
  \er
\]
bringing the very same index pairs in a different sequence, is just as
valid.  We choose $i$-major nesting on the subtle ground that it affords
better information to the choice of column index~$p$ during the
algorithm's step~\ref{mtxinv:461:s20}.

The algorithm, in detail:
\begin{enumerate}
  \item \label{mtxinv:461:s10}
    Begin by initializing
    \[
      \br{c}
        \tilde U \la I, \
        \tilde D \la I, \
        \tilde S \la I, \\
        \setlength\arraycolsep{0.30\arraycolsep}
        \br{rcl}
          \tilde Q &\la& A, \\
          i &\la& 1.
        \er
      \er
    \]
  \item \label{mtxinv:461:s15}
    (Besides arriving at this point from step~\ref{mtxinv:461:s10}
    above, the algorithm also re\"enters here from
    step~\ref{mtxinv:461:s80} below.)
    Observe that~$\tilde U$ enjoys the major partial unit triangular
    form $L^{\{i-1\}T}$ (\S~\ref{matrix:330.05}), that~$\tilde D$ is a
    general scaling operator (\S~\ref{matrix:325.20}) with $\tilde
    d_{jj} = 1$ for all $j \ge i$, that~$\tilde S$ is permutor
    (\S~\ref{matrix:325.10}), and that the first through $(i-1)$th
    columns of~$\tilde Q$ consist of mutually orthonormal unit vectors.
  \item \label{mtxinv:461:s20}
    \index{column}
    \index{matrix!column of}
    \index{null column}
    \index{matrix!null column of}
    Choose a column $p \ge i$ of~$\tilde Q$ containing at least one
    nonzero element.  (The simplest choice is perhaps $p=i$ as long as
    the $i$th column does not happen to be null, but one might instead
    prefer to choose the column of greatest magnitude, or to choose
    randomly, among other heuristics.)  If~$\tilde Q$ is null in and
    rightward of its $i$th column such that no column $p \ge i$ remains
    available to choose, then skip directly to
    step~\ref{mtxinv:461:s90}.
  \item \label{mtxinv:461:s30}
    \index{interchange}
    Observing that~(\ref{mtxinv:460:60})
    can be expanded to read
    \bqb
      A &=&
        \Big( \tilde Q T_{[i \lra p]} \Big)
        \Big( T_{[i \lra p]} \tilde U T_{[i \lra p]} \Big)
        \Big( T_{[i \lra p]} \tilde D T_{[i \lra p]} \Big)
        \Big( T_{[i \lra p]} \tilde S \Big)
      \\&=&
        \Big( \tilde Q T_{[i \lra p]} \Big)
        \Big( T_{[i \lra p]} \tilde U T_{[i \lra p]} \Big)
        \tilde D
        \Big( T_{[i \lra p]} \tilde S \Big),
    \eqb
    where the latter line has applied a rule from
    Table~\ref{gjrank:337:table}, interchange the chosen $p$th column to
    the $i$th position by
    \[
      \begin{split}
        \tilde Q &\la \tilde Q T_{[i \lra p]}, \\
        \tilde U &\la T_{[i \lra p]} \tilde U T_{[i \lra p]}, \\
        \tilde S &\la T_{[i \lra p]} \tilde S.
      \end{split}
    \]
  \item \label{mtxinv:461:s35}
    \index{normalization}
    Observing that~(\ref{mtxinv:460:60}) can be expanded to read
    \[
      A =
        \Big( \tilde Q T_{(1/\alpha)[i]} \Big)
        \Big( T_{\alpha[i]} \tilde U T_{(1/\alpha)[i]} \Big)
        \Big( T_{\alpha[i]} \tilde D \Big)
        \tilde S,
    \]
    normalize the $i$th column of~$\tilde Q$ by
    \[
      \begin{split}
        \tilde Q &\la \tilde Q T_{(1/\alpha)[i]}, \\
        \tilde U &\la T_{\alpha[i]} \tilde U T_{(1/\alpha)[i]}, \\
        \tilde D &\la T_{\alpha[i]} \tilde D,
     \end{split}
    \]
    where
    \[
      \alpha = \sqrt{
        \left[ \tilde Q \right]_{*i}^{*} \cdot \left[ \tilde Q \right]_{*i}
      }.
    \]
  \item \label{mtxinv:461:s40}
    Initialize
    \[
      j \la i+1.
    \]
  \item \label{mtxinv:461:s50}
    \index{orthogonalization}
    (Besides arriving at this point from step~\ref{mtxinv:461:s40} above, the
    algorithm also re\"enters here from step~\ref{mtxinv:461:s60} below.)
    If $j>n$ then skip directly to step~\ref{mtxinv:461:s80}.
    Otherwise, observing that~(\ref{mtxinv:460:60}) can be expanded to read
    \[
      A =
        \Big( \tilde Q T_{-\beta[ij]} \Big)
        \Big( T_{\beta[ij]} \tilde U \Big)
        \tilde D
        \tilde S,
    \]
    orthogonalize the $j$th column of~$\tilde Q$
    per~(\ref{mtxinv:460:55}) with respect to the $i$th column by
    \[
      \begin{split}
        \tilde Q &\la \tilde Q T_{-\beta[ij]}, \\
        \tilde U &\la T_{\beta[ij]} \tilde U,
      \end{split}
    \]
    where
    \[
      \beta = \left[ \tilde Q \right]_{*i}^{*} \cdot \left[ \tilde Q \right]_{*j}.
    \]
  \item \label{mtxinv:461:s60}
    Increment
    \[
      j \la j + 1
    \]
    and return to step~\ref{mtxinv:461:s50}.
  \item \label{mtxinv:461:s80}
    Increment
    \[
      i \la i + 1
    \]
    and return to step~\ref{mtxinv:461:s15}.
  \item \label{mtxinv:461:s90}
    Let
    \[
      \br{c}
        Q \equiv  \tilde Q, \
        U \equiv  \tilde U, \
        D \equiv  \tilde D, \
        S \equiv  \tilde S, \\
        r = i-1.
      \er
    \]
    End.
\end{enumerate}

% bad break: the index entries that follow have bad breaks
\index{Gram-Schmidt decomposition!differences of against the
  Gauss-\newline Jordan}
\index{Gauss-Jordan decomposition!differences of against the
  Gram-\newline Schmidt}
\index{decomposition!differences between the
  Gram-\newline Schmidt and Gauss-Jordan}
\noindent
Though the Gram-Schmidt algorithm broadly resembles the Gauss-Jordan,
at least two significant differences stand out: (i)~the Gram-Schmidt is
one-sided because it operates only on the columns of~$\tilde Q$, never
on the rows; (ii)~since $Q$ is itself dimension-limited, the
Gram-Schmidt decomposition
% bad break
(\ref{mtxinv:QR}) needs and has no explicit factor~$I_r$.

\index{Gram-Schmidt decomposition!factor~$S$ of}
As in \S~\ref{gjrank:340.60}, here also one sometimes prefers that
$S=I$.  The algorithm optionally supports this preference if the $m
\times n$ matrix~$A$ has full column rank $r=n$, when null columns
cannot arise, if one always chooses $p=i$ during the algorithm's
step~\ref{mtxinv:461:s20}.  Such optional discipline maintains $S=I$
when desired.

\index{factorization!full-rank}
\index{full-rank factorization}
\index{pseudoinverse}
\index{Moore-Penrose pseudoinverse}
Whether $S=I$ or not, the matrix $Q=QI_r$ has only~$r$ columns, so one
can write~(\ref{mtxinv:QR}) as
\[
  A = (QI_r)(R).
\]
Reassociating factors, this is
\bq{mtxinv:460:70}
  A = (Q)(I_rR),
\eq
which per~(\ref{gjrank:340:26}) is a proper full-rank factorization
with which one can compute the pseudoinverse~$A^\dagger$ of~$A$ (see
eqn.~\ref{mtxinv:psinv}, above; but see also eqn.~\ref{mtxinv:QRinv},
below).

\index{Gram-Schmidt decomposition!factor~$Q$ of}
If the Gram-Schmidt decomposition~(\ref{mtxinv:QR}) looks useful, it is
even more useful than it looks.  The most interesting of its several
factors is the $m \times r$ orthonormalized matrix~$Q$, whose
orthonormal columns address the same space the columns of~$A$ themselves
address.  If~$Q$ reaches the maximum possible rank $r=m$, achieving
square, $m \times m$ shape, then it becomes a \emph{unitary
matrix}---the subject of \S~\ref{mtxinv:465}.

Before treating the unitary matrix, however, let us pause to develop the
orthogonal complement by Gram-Schmidt in \S~\ref{mtxinv:460.40}, next.

\subsection{The orthogonal complement by Gram-Schmidt}
\label{mtxinv:460.40}
\index{Gram-Schmidt orthogonal complement}
\index{orthogonal complement!by Gram-Schmidt}

Having decomposed an $m \times n$ matrix as
\bq{mtxinv:460:74}
  A=QR=(QI_r)R,
\eq
observing that the~$r$ independent columns of the $m \times r$
matrix~$Q=QI_r$ address the same space the columns of~$A$ address,
Gram-Schmidt computes an orthogonal complement~(\ref{mtxinv:450:perp})
by constructing the $m \times (r+m)$ matrix
\bq{mtxinv:460:76}
  A' \equiv QI_r + I_mH_{-r} = \left[ \br{cc}QI_r&I_m\er \right].
\eq
This constructed matrix~$A'$ is then itself decomposed,
\bq{mtxinv:460:77}
  A' = Q'R',
\eq
again by Gram-Schmidt---with the differences that, this time, one
chooses $p=1,2,3,\ldots,r$ during the first~$r$ instances of the
algorithm's step~\ref{mtxinv:461:s20} and that one skips the
unnecessary step~\ref{mtxinv:461:s50} for all $j \le r$,
on the ground that the earlier Gram-Schmidt application
of~(\ref{mtxinv:460:74}) has already orthonormalized first~$r$ columns
of~$A'$, which columns, after all, are just~$Q=QI_r$.  The resulting $m
\times m$, full-rank square matrix
\bq{mtxinv:460:perp0}
  Q' = QI_r + A^\perp H_{-r} = \left[ \br{cc}QI_r&A^\perp\er \right]
\eq
consists of
\bi
  \item
    $r$ columns on the left that address the same space the columns
    of~$A$ address and
  \item
    $m-r$ columns on the right that give a complete orthogonal
    complement (\S~\ref{mtxinv:450})~$A^\perp$ of~$A$.
\ei
Each column has unit magnitude and conveniently lies
orthogonal---indeed, orthonormal---to every
other column, left and right.

\index{Gram-Schmidt orthogonal-complement\\formula} % bad break
\index{orthogonal complement!Gram-Schmidt formula for}
Equation~(\ref{mtxinv:460:perp0}) is probably the more useful form,
but the
% bad break
\emph{Gram-\linebreak Schmidt
orthogonal-complement formula} as such is that
\bq{mtxinv:460:perp}
  A^{*K} = A^\perp = Q'H_rI_{m-r}.
\eq
%extracts the rightward columns that express the kernel, not of~$A$, but
%of~$A^{*}$.

%In either the form~(\ref{}) or the
%form~(\ref{}), the Gram-Schmidt kernel formula does
%everything the Gauss-Jordan kernel formula~(\ref{mtxinv:245:kernel})
%does and in at least one sense does it better; for, if one wants a
%Gauss-Jordan kernel orthonormalized, then one must orthonormalize it as
%an extra step, whereas the Gram-Schmidt kernel comes already
%orthonormalized.

\index{Gram-Schmidt kernel formula}
\index{kernel!Gram-Schmidt formula for}
The writer has not encountered a Gram-Schmidt kernel
in the style of~(\ref{mtxinv:245:kernel}) to accompany
the Gram-Schmidt orthogonal complement of
\linebreak % bad break
(\ref{mtxinv:460:perp})---though
if necessary one could maybe combine~(\ref{mtxinv:460:perp})
with~(\ref{mtxinv:450:perp}) for the purpose.  Instead, normally, as far
as the writer knows, the Gauss-Jordan~(\ref{mtxinv:245:kernel}) is used.
Meanwhile however, the matrix~$Q'$ of this subsection is interesting.
Being square and orthonormal, the $m \times m$ matrix~$Q'$ is a unitary
matrix.  Unitary matrices will be the subject of \S~\ref{mtxinv:465},
next.

% ----------------------------------------------------------------------

\section{The unitary matrix}
\label{mtxinv:465}
\index{unitary matrix}
\index{matrix!unitary}

When the orthonormalized matrix~$Q$ of the Gram-Schmidt
decomposition
% bad break
(\ref{mtxinv:QR})
is square, having the maximum possible rank $r=m$, it brings one
property so interesting that the property merits a section of its own.
The property is that
\bq{mtxinv:unitary}
  Q^{*}Q = I_m = QQ^{*}.
\eq
The reason that $Q^{*}Q = I_m$ is that $Q$'s columns are orthonormal,
and that the very definition of orthonormality demands that the dot
product $[Q]_{*i}^{*} \cdot [Q]_{*j}$ of orthonormal columns be zero
unless $i=j$, when the dot product of a unit vector with itself is
unity.  That $I_m=QQ^{*}$ is unexpected, however, until one realizes%
\footnote{\cite[\S~4.4]{Franklin}}
that the equation $Q^{*}Q = I_m$ characterizes~$Q^{*}$ to be the
rank-$m$ inverse of~$Q$, and that \S~\ref{mtxinv:220} lets any rank-$m$
inverse (orthonormal or otherwise) attack just as well from the right as
from the left.  Thus,
\bq{mtxinv:465:15}
  Q^{-1} = Q^{*},
\eq
a very useful property.  A matrix~$Q$ that
satisfies~(\ref{mtxinv:unitary}), whether derived from the Gram-Schmidt
or from elsewhere, is called a \emph{unitary matrix.}  (Note that the
permutor of \S~\ref{matrix:325.10} enjoys the property of
eqn.~\ref{mtxinv:465:15} precisely because it is unitary.)

\index{orthonormal rows and columns}
\index{row!orthonormal}
\index{column!orthonormal}
One immediate consequence of~(\ref{mtxinv:unitary}) is that \emph{a
square matrix with either orthonormal columns or orthonormal rows is
unitary and has both.}

\emph{The product of two or more unitary matrices is itself unitary} if
the matrices are of the same dimensionality.  To prove it, consider
the product
\bq{mtxinv:465:20}
  Q = Q_aQ_b
\eq
of $m \times m$ unitary matrices~$Q_a$ and~$Q_b$.  Let the
symbols~$\ve q_j$, $\ve q_{aj}$ and~$\ve q_{bj}$ respectively
represent the $j$th columns of~$Q$, $Q_a$ and~$Q_b$ and let the
symbol~$q_{bij}$ represent the $i$th element of~$\ve q_{bj}$.  By the
columnwise interpretation (\S~\ref{matrix:120.27}) of matrix
multiplication,
\[
  \ve q_j = \sum_i q_{bij} \ve q_{ai}.
\]
The adjoint dot product of any two of $Q$'s columns then is
\[
  \ve q_{j'}^{*} \cdot \ve q_j = \sum_{i,i'}
    q_{bi'j'}^{*}q_{bij}\ve q_{ai'}^{*}\cdot \ve q_{ai}.
\]
But $\ve q_{ai'}^{*}\cdot \ve q_{ai} = \delta_{i'i}$ because~$Q_a$ is
unitary,%
\footnote{
  This is true only for $1 \le i \le m$, but you knew that already.
}
so
\[
  \ve q_{j'}^{*} \cdot \ve q_j = \sum_i q_{bij'}^{*}q_{bij}
  = \ve q_{bj'}^{*} \cdot \ve q_{bj} = \delta_{j'j},
\]
which says neither more nor less than that the columns of~$Q$ are
orthonormal, which is to say that~$Q$ is unitary, as was to be
demonstrated.

\index{length!preservation of}
\index{magnitude!preservation of}
\emph{Unitary operations preserve length.}  That is, operating on an
$m$-element vector by an $m \times m$ unitary matrix does not alter the
vector's magnitude.  To prove it, consider the system
\[
  Q\ve x = \ve b.
\]
Multiplying the system by its own adjoint yields that
\[
  \ve x^{*}Q^{*}Q\ve x = \ve b^{*}\ve b.
\]
But according to~(\ref{mtxinv:unitary}), $Q^{*}Q=I_m$; so,
\[
  \ve x^{*}\ve x = \ve b^{*}\ve b,
\]
as was to be demonstrated.

\index{$QR$ decomposition!inverting a matrix by}
\index{Gram-Schmidt decomposition!inverting a matrix by}
\index{orthonormalizing decomposition!inverting a matrix by}
\index{decomposition!Gram-Schmidt, inverting a matrix by}
Equation~(\ref{mtxinv:465:15}) lets one use the Gram-Schmidt
decomposition~(\ref{mtxinv:QR}) to invert a square matrix as
\bq{mtxinv:QRinv}
  A^{-1} = R^{-1}Q^{*} = S^{*}D^{-1}U^{-1}Q^{*}.
\eq

\index{extended operator}
Unitary extended operators are certainly possible, for if~$Q$ is an $m
\times m$ dimension-limited matrix, then the extended
operator
\[
  Q_\infty = Q + (I-I_m),
\]
which is just~$Q$ with ones running out the main diagonal from its
active region, itself meets the unitary
criterion~(\ref{mtxinv:unitary}) for $m=\infty$.

Unitary matrices are so easy to handle that they can sometimes justify
significant effort to convert a model to work in terms of them if
possible.  We shall meet the unitary matrix again in
\S\S~\ref{eigen:520} and~\ref{eigen:600}.

% ----------------------------------------------------------------------

The chapter as a whole has demonstrated at least in theory (and usually
in practice) techniques to solve any linear system characterized by a
matrix of finite dimensionality, whatever the matrix's rank or shape.
It has explained how to orthonormalize a set of vectors and has derived
from the explanation the useful Gram-Schmidt decomposition.  As the
chapter's introduction had promised, the matrix has shown its worth
here; for without the matrix's notation, arithmetic and algebra most of
the chapter's findings would have lain beyond practical reach.  And even
so, the single most interesting agent of matrix arithmetic remains yet
to be treated.  This last is the eigenvalue, and it is the subject of
chapter~\ref{eigen}, next.

