% ----------------------------------------------------------------------

\chapter{The eigenvalue}
\label{eigen}
\index{eigenvalue}
\index{determinant}

The \emph{eigenvalue} is a scalar by which a square matrix scales a
vector without otherwise changing it, such that
\[
  A \ve v = \lambda \ve v.
\]
This chapter analyzes the eigenvalue and the associated
\emph{eigenvector} it scales.

Before treating the eigenvalue proper, the chapter gathers from
across chapters~\ref{matrix} through~\ref{eigen} several properties all
invertible square matrices share, assembling them in \S~\ref{eigen:370}
for reference.  One of these regards the \emph{determinant,} which opens
the chapter.

% ----------------------------------------------------------------------

\section{The determinant}
\label{eigen:310}
\index{determinant}

Through chapters~\ref{matrix}, \ref{gjrank} and~\ref{mtxinv} the theory of
the matrix has developed slowly but pretty straightforwardly.  Here comes the
first unexpected turn.

\index{permutor}
\index{parity}
It begins with an arbitrary-seeming definition.  The \emph{determinant}
of an $n \times n$ square matrix~$A$ is the sum of~$n!$ terms, each term
the product of~$n$ elements, no two elements from the same row or
column, terms of positive parity adding to and terms of negative parity
subtracting from the sum---a term's parity (\S~\ref{matrix:322}) being
the parity of the permutor (\S~\ref{matrix:325.10}) marking the
positions of the term's elements.

\index{inversion!symbolic}
Unless you already know about determinants, the definition alone might
seem hard to parse, so try this.  The inverse of the general
$2\times 2$ square matrix
\[
  A_2 = \left[\br{rr}
    a_{11} & a_{12} \\
    a_{21} & a_{22}
  \er\right],
\]
by the Gauss-Jordan method or any other convenient technique, is found
to be
\[
  A_2^{-1} = \frac{
  \left[\br{rr}
    a_{22} &-a_{12} \\
   -a_{21} & a_{11}
  \er\right]
  }{a_{11}a_{22} - a_{12}a_{21}}.
\]
The quantity%
\footnote{
  The determinant $\det A$ used to be written~$\left|A\right|$, an
  appropriately terse notation for which the author confesses some
  nostalgia.  The older notation~$\left|A\right|$ however unluckily
  suggests ``the magnitude of~$A$,'' which though not quite the wrong
  idea is not quite the right idea, either.  The
  magnitude~$\left|z\right|$ of a scalar or~$\left|\ve u\right|$ of a
  vector is a real-valued, nonnegative, nonanalytic function of the
  elements of the quantity in question, whereas the determinant
  $\det A$ is a complex-valued, analytic function.
  The book follows convention by denoting the determinant as $\det A$
  for this reason among others.
}
\[
  \det A_2 = a_{11}a_{22} - a_{12}a_{21}
\]
in the denominator is defined to be the \emph{determinant} of~$A_2$.
Each of the determinant's terms includes one element from each column of
the matrix and one from each row, with parity giving the term its~$\pm$
sign.  The determinant of the general $3 \times 3$ square matrix by the
same rule is
\bqb
  \det A_3 &=& ( a_{11}a_{22}a_{33} + a_{12}a_{23}a_{31} + a_{13}a_{21}a_{32} ) \\
  &&\ \ \mbox{} - ( a_{13}a_{22}a_{31} + a_{12}a_{21}a_{33} + a_{11}a_{23}a_{32} );
\eqb
and indeed if we tediously invert such a matrix symbolically, we do find
that quantity in the denominator there.

\index{determinant!definition of}
\index{marking quasielementary}
\index{quasielementary operator!marking}
\index{marking!permutor}
\index{permutor!marking}
The parity rule merits a more careful description.  The parity of a term
like $a_{12}a_{23}a_{31}$ is positive because the parity of the
permutor, or interchange quasielementary (\S~\ref{matrix:325.10}),
\[
  P = \mf{ccc}{
    0 & 1 & 0 \\
    0 & 0 & 1 \\
    1 & 0 & 0
  }
\]
marking the positions of the term's elements is positive.  The parity of
a term like $a_{13}a_{22}a_{31}$ is negative for the same reason.  The
determinant comprehends all possible such terms,~$n!$ in number, half of
positive parity and half of negative.  (How do we know that exactly half
are of positive and half, negative?  Answer: by pairing the terms.  For
every term like $a_{12}a_{23}a_{31}$ whose marking permutor
is~$P$, there is a corresponding $a_{13}a_{22}a_{31}$ whose marking
permutor is $T_{[1 \lra 2]}P$, necessarily of opposite parity.
The sole exception to the rule is the $1 \times 1$ square matrix, which
has no second term to pair.)

\index{rank-$n$ determinant}
\index{determinant!rank-$n$}
Normally the context implies a determinant's rank~$n$, but the
nonstandard notation
\[
  \mopx{det}^{(n)} A
\]
is available especially to call the rank out, stating explicitly that
the determinant has exactly~$n!$ terms.  (See also
\S\S~\ref{matrix:180.22} and~\ref{matrix:321} and
eqn.~\ref{matrix:321:20}.%
\footnote{
  And see further chapter~\ref{mtxinv}'s footnotes~\ref{mtxinv:245:08}
  and~\ref{mtxinv:450:08}.
}%
)

It is admitted%
\footnote{\cite[\S~1.2]{Franklin}}
that we have not, as yet,
actually shown the determinant to be a generally useful quantity; we have
merely motivated and defined it.  The true history of the determinant is
unknown to this writer, but one might suppose that the determinant had originally
emerged not from abstract considerations but for the mundane reason that
the quantity it represents occurs frequently in practice (as in
the~$A_2^{-1}$ of the example above).  Nothing however logically prevents one
from simply defining some quantity which, at first, one merely
suspects will later prove useful.  So we do here.%
\footnote{\cite[chapter~1]{Franklin}}

\subsection{Basic properties}
\label{eigen:310.25}
\index{determinant!properties of}

The determinant $\det A$ enjoys several useful basic properties.
\bi
  \item
    \index{interchange}
    If
    \[
      c_{i*} = \begin{cases}
        a_{i''*} & \mbox{when $i = i'$,} \\
        a_{i'*} & \mbox{when $i = i''$,} \\
        a_{i*} & \mbox{otherwise,}
      \end{cases}
    \]
    or if
    \[
      c_{*j} = \begin{cases}
        a_{*j''} & \mbox{when $j = j'$,} \\
        a_{*j'} & \mbox{when $j = j''$,} \\
        a_{*j} & \mbox{otherwise,}
      \end{cases}
    \]
    where $i'' \neq i'$ and $j'' \neq j'$, then
    \bq{eigen:310:16}
      \det C = -\det A.
    \eq
    Interchanging rows or columns negates the determinant.
  \item
    \index{scaling}
    If
    \[
      c_{i*} = \begin{cases}
        \alpha a_{i*} & \mbox{when $i = i'$,} \\
        a_{i*} & \mbox{otherwise,}
      \end{cases}
    \]
    or if
    \[
      c_{*j} = \begin{cases}
        \alpha a_{*j} & \mbox{when $j = j'$,} \\
        a_{*j} & \mbox{otherwise,}
      \end{cases}
    \]
    then
    \bq{eigen:310:11}
      \det C = \alpha\det A.
    \eq
    Scaling a single row or column of a matrix scales the matrix's
    determinant by the same factor.  (Equation~\ref{eigen:310:11}
    tracks the linear scaling property of \S~\ref{integ:240.05} and
    of eqn.~\ref{matrix:000:20}.)
  \item
    \index{superposition}
    If
    \[
      c_{i*} = \begin{cases}
        a_{i*} + b_{i*} & \mbox{when $i = i'$,} \\
        a_{i*} = b_{i*} & \mbox{otherwise,}
      \end{cases}
    \]
    or if
    \[
      c_{*j} = \begin{cases}
        a_{*j} + b_{*j} & \mbox{when $j = j'$,} \\
        a_{*j} = b_{*j} & \mbox{otherwise,}
      \end{cases}
    \]
    then
    \bq{eigen:310:12}
      \det C = \det A + \det B.
    \eq
    If one row or column of a matrix~$C$ is the sum of the corresponding
    rows or columns of two other matrices~$A$ and~$B$, while the three
    matrices remain otherwise identical, then the determinant of the one
    matrix is the sum of the determinants of the other two.
    (Equation~\ref{eigen:310:12} tracks the linear superposition
    property of \S~\ref{integ:240.05} and of eqn.~\ref{matrix:000:20}.)
  \item
    \index{row!null}
    \index{column!null}
    If
    \[
      c_{i'*} = 0,
    \]
    or if
    \[
      c_{*j'} = 0,
    \]
    then
    \bq{eigen:310:18}
      \det C = 0.
    \eq
    A matrix with a null row or column also has a null determinant.
  \item
    \index{row!scaled and repeated}
    \index{column!scaled and repeated}
    If
    \[
      c_{i''*} = \gamma c_{i'*},
    \]
    or if
    \[
      c_{*j''} = \gamma c_{*j'},
    \]
    where $i'' \neq i'$ and $j'' \neq j'$, then
    \bq{eigen:310:17}
      \det C = 0.
    \eq
    The determinant is zero if one row or column of the matrix is a
    multiple of another.
  \item
    \index{adjoint}
    \index{transpose}
    The determinant of the adjoint is just the determinant's conjugate, and
    the determinant of the transpose is just the determinant itself:
    \bq{eigen:310:19}
      \begin{split}
        \det C^{*} &= \left(\det C\right)^{*}; \\
        \det C^{T} &= \det C.
      \end{split}
    \eq
\ei
These basic properties are all fairly easy to see if the definition of
the determinant is clearly understood.
%
Equations~(\ref{eigen:310:11}), (\ref{eigen:310:12})
and~(\ref{eigen:310:18}) come because each of the~$n!$ terms in the
determinant's expansion has exactly one element from row~$i'$
or column~$j'$.
%
Equation~(\ref{eigen:310:16}) comes because a row or column interchange
reverses parity.
%
Equation~(\ref{eigen:310:19}) comes because according to
\S~\ref{matrix:325.10}, the permutors~$P$
and~$P^{*}$ always have the same parity, and because the adjoint
operation individually conjugates each element of~$C$.
%
Finally,~(\ref{eigen:310:17}) comes because, in this case, every term in
the determinant's expansion finds an equal term of opposite parity to
offset it.  Or, more formally,~(\ref{eigen:310:17}) comes because the
following procedure does not alter the matrix:
(i) scale row~$i''$ or column~$j''$ by $1/\gamma$;
(ii) scale row~$i'$ or column~$j'$ by~$\gamma$;
(iii) interchange rows $i' \lra i''$ or columns $j' \lra j''$.  Not
altering the matrix, the procedure does not alter the determinant
either; and indeed according to~(\ref{eigen:310:11}), step~(ii)'s effect
on the determinant cancels that of step~(i).  However, according
to~(\ref{eigen:310:16}), step~(iii) negates the determinant.  Hence the
net effect of the procedure is to negate the determinant---to negate the
very determinant the procedure is not permitted to alter.  The apparent
contradiction can be reconciled only if the determinant is zero to
begin with.

From the foregoing properties the following further property can be
deduced.
\bi
  \item
    \index{addition!of rows or columns}
    \index{row!addition of}
    \index{column!addition of}
    If
    \[
      c_{i*} = \begin{cases}
        a_{i*} + \alpha a_{i'*} & \mbox{when $i = i''$,} \\
        a_{i*}& \mbox{otherwise,}
      \end{cases}
    \]
    or if
    \[
      c_{*j} = \begin{cases}
        a_{*j} + \alpha a_{*j'} & \mbox{when $j = j''$,} \\
        a_{*j} & \mbox{otherwise,}
      \end{cases}
    \]
    where $i'' \neq i'$ and $j'' \neq j'$, then
    \bq{eigen:310:21}
      \det C = \det A.
    \eq
    Adding to a row or column of a matrix a multiple of another row or
    column does not change the matrix's determinant.
\ei
To derive~(\ref{eigen:310:21}) for rows (the column proof is similar),
one defines a matrix~$B$ such that
\[
  b_{i*} \equiv \begin{cases}
    \alpha a_{i'*} & \mbox{when $i = i''$,} \\
    a_{i*} & \mbox{otherwise.}
  \end{cases}
\]
From this definition, $b_{i''*} = \alpha a_{i'*}$ whereas
$b_{i'*} = a_{i'*}$, so
\[
  b_{i''*} = \alpha b_{i'*},
\]
which by~(\ref{eigen:310:17}) guarantees that
\[
  \det B=0.
\]
On the other hand, the three matrices~$A$, $B$ and~$C$ differ only in
the ($i''$)th row, where $[C]_{i''*} = [A]_{i''*} + [B]_{i''*}$;
so, according to~(\ref{eigen:310:12}),
\[
  \det C = \det A + \det B.
\]
Equation~(\ref{eigen:310:21}) results from combining the last two
equations.

\subsection{The determinant and the elementary operator}
\label{eigen:310.35}
\index{elementary operator!and the determinant}
\index{determinant!and the elementary operator}

Section~\ref{eigen:310.25} has it that interchanging, scaling or adding
rows or columns of a matrix respectively negates, scales or does not
alter the matrix's determinant.  But the three operations named are
precisely the operations of the three elementaries of
\S~\ref{matrix:320}.  Therefore,
\bq{eigen:310:25}
  \renewcommand\arraystretch{1.3}
  \br{c}
    \br{rcrcl}
      \det T_{[i\lra j]}A  &=& -\det A &=& \det AT_{[i\lra j]} , \\
      \det T_{\alpha[i]}A  &=& \alpha\det A &=& \det AT_{\alpha[j]} , \\
      \det T_{\alpha[ij]}A  &=& \det A &=& \det AT_{\alpha[ij]} ,
    \er \\
    1 \le (i,j) \le n,\ i \neq j,
  \er
\eq
for any $n \times n$ square matrix~$A$.  Obviously also,
\bq{eigen:310:27}
  \renewcommand\arraystretch{1.3}
  \br{rcccl}
    \det IA  &=& \det A &=& \det AI , \\
    \det I_nA  &=& \det A &=& \det AI_n , \\
    \det I  &=& 1 &=& \det I_n .
  \er
\eq

If~$A$ is taken to represent an arbitrary product of identity matrices
($I_n$ and/or~$I$) and elementary operators, then a significant
consequence of~(\ref{eigen:310:25}) and~(\ref{eigen:310:27}), applied
recursively, is that the determinant of a product is the product of the
determinants, at least where identity matrices and elementary operators
are concerned.  In symbols,%
\footnote{\label{eigen:310:fn1}%
  Notation like ``$\in$'', first met in \S~\ref{alggeo:227},
  can be too fancy for applied mathematics, but
  it does help here.  The notation $M_k \in \{ \ldots \}$
  restricts~$M_k$ to be any of the things between the braces.  As it
  happens though, in this case,~(\ref{eigen:310:30}) below is going to
  erase the restriction.
}
\bqa
  \det\left( \prod_k M_k \right) &=& \prod_k \det  M_k ,
  \label{eigen:310:26}\\
  M_k &\in& \left\{
    I_n, I, T_{[i \lra j]}, T_{\alpha[i]}, T_{\alpha[ij]}
  \right\}, \xn\\
  1 \ \le \ (i,j) &\le& n. \xn
\eqa
This matters because, as the Gauss-Jordan decomposition of
\S~\ref{gjrank:341} has shown, one can build up any square matrix of
full rank by applying elementary operators to~$I_n$.
Section~\ref{eigen:310.40} will put the rule~(\ref{eigen:310:26}) to
good use.

\subsection{The determinant of a singular matrix}
\label{eigen:310.37}
\index{singular matrix!determinant of}
\index{determinant!zero}

Equation~(\ref{eigen:310:25}) gives elementary operators the power to
alter a matrix's determinant almost arbitrarily---almost arbitrarily,
but not quite.  What an $n \times n$ elementary operator%
\footnote{
  That is, an elementary operator which honors an $n \times n$ active
  region.  See \S~\ref{matrix:180.35}.
}
cannot do is to
change an $n \times n$ matrix's determinant to or from zero.  Once zero,
a determinant remains zero under the action of elementary operators.
Once nonzero, always nonzero.  Elementary operators being reversible
have no power to breach this barrier.

Another thing $n \times n$ elementaries cannot do according to
\S~\ref{gjrank:340.20} is to change an $n \times n$ matrix's rank.
Nevertheless, such elementaries can reduce any $n \times n$ matrix
reversibly to~$I_r$, where $r \le n$ is the matrix's rank, by the
Gauss-Jordan algorithm of \S~\ref{gjrank:341}.
Equation~(\ref{eigen:310:18}) has that the $n \times n$ determinant
of~$I_r$ is zero if $r<n$, so it follows that the $n\times n$
determinant of every rank-$r$ matrix is similarly zero if $r<n$; and
complementarily that the $n\times n$ determinant of a rank-$n$ matrix is
never zero.  Singular matrices always have zero determinants; full-rank
square matrices never do.  One can evidently tell the singularity or
invertibility of a square matrix from its determinant alone.

\subsection{The determinant of a matrix product}
\label{eigen:310.40}
\index{determinant!of a product}
\index{determinant!product of two}
\index{product!of determinants}
\index{product!determinant of}

Sections~\ref{eigen:310.35} and~\ref{eigen:310.37} suggest the useful
rule that
\bq{eigen:310:30}
  \det AB = \det A\det B.
\eq
To prove the rule, we consider three distinct cases.

The first case is that~$A$ is singular.  In this case,~$B$ acts as a
column operator on~$A$, whereas according to \S~\ref{gjrank:340.10} no
operator has the power to promote~$A$ in rank.  Hence the product~$AB$
is no higher in rank than~$A$, which says that~$AB$ is no
less singular than~$A$, which implies that~$AB$ like~$A$ has a null
determinant.  Evidently~(\ref{eigen:310:30}) holds in the first case.

The second case is that~$B$ is singular.  The proof here resembles that
of the first case.

The third case is that neither matrix is singular.  Here, we use
Gauss-Jordan to decompose both matrices into sequences of elementary
operators and rank-$n$ identity matrices, for which
\bqb
  \det AB &=&
    \det\left\{
      \left[A\right]
      \left[B\right]
    \right\}
  \\&=& \det\left\{
      \left[ \left(\prod T\right) I_n \left(\coprod T \right) \right]
      \left[ \left(\prod T\right) I_n \left(\coprod T \right) \right]
    \right\}
  \\&=&
    \left(\prod \det T\right) \det I_n \left(\coprod \det T\right)
    \left(\prod \det T\right) \det I_n \left(\coprod \det T\right)
  \\&=&
    \det\left[ \left(\prod T\right) I_n \left(\coprod T \right)\right]
    \det\left[ \left(\prod T\right) I_n \left(\coprod T \right)\right]
  \\&=&
    \det A
    \det B,
\eqb
which is a schematic way of pointing out in light
of~(\ref{eigen:310:26}) merely that since~$A$ and~$B$ are products of
identity matrices and elementaries, the determinant of the product is
the product of the determinants.

So it is that~(\ref{eigen:310:30}) holds in all three cases, as was to
be demonstrated.  \emph{The determinant of a matrix product is the
product of the matrix determinants.}

\subsection{Determinants of inverse and unitary matrices}
\label{eigen:310.60}
\index{inverse!determinant of}
\index{unitary matrix!determinant of}
\index{determinant!of a matrix inverse}
\index{determinant!of a unitary matrix}

From~(\ref{eigen:310:30}) it follows that
\bq{eigen:310:61}
  \det A^{-1} = \frac{1}{\det A}
\eq
because $A^{-1}A = I_n$ and $\det I_n = 1$.

From~(\ref{eigen:310:19}) it follows that if~$Q$ is a unitary matrix
(\S~\ref{mtxinv:465}), then
\bq{eigen:310:64}
  \begin{split}
    \det Q^{*} \det Q &= 1, \\
    \left|\det Q\right| &= 1.
  \end{split}
\eq
This reason is that $
  \left|\det Q\right|^2 =
  (\det Q)^{*}(\det Q) =
  \det Q^{*}\det Q =
  \det Q^{*}Q =
  \det Q^{-1}Q =
  \det I_n = 1
$.

\subsection{Inverting the square matrix by determinant}
\label{eigen:310.50}
\index{matrix!inversion of}
\index{inversion!by determinant}
\index{determinant!inversion by}

\index{inversion!symbolic}
The Gauss-Jordan algorithm comfortably inverts concrete matrices of
moderate size, but swamps one in nearly interminable algebra when
\emph{symbolically} inverting general matrices larger than the~$A_2$ at
the section's head.  Slogging through the algebra to invert~$A_3$
symbolically nevertheless (the reader need not actually do this unless
he desires a long exercise), one quite incidentally discovers a clever
way to factor the determinant:
\bq{eigen:310:50}
  \begin{split}
    C^TA &= (\det A) I_n = AC^T; \\
    c_{ij} &\equiv \det R_{ij}; \\
    [R_{ij}]_{i'j'} &\equiv \begin{cases}
      1 & \mbox{if $i'=i$ and $j'=j$,} \\
      0 & \mbox{if $i'=i$ or  $j'=j$ but not both,} \\
      a_{i'j'} & \mbox{otherwise.}
    \end{cases}
  \end{split}
\eq
Pictorially,
\[
  R_{ij} = \mf{ccccccc}{
    &\vdots&\vdots&\vdots&\vdots&\vdots& \\
    \cdots & {*} & {*} & 0 & {*} & {*} & \cdots \\
    \cdots & {*} & {*} & 0 & {*} & {*} & \cdots \\
    \cdots &  0  &  0  & 1 &  0  &  0  & \cdots \\
    \cdots & {*} & {*} & 0 & {*} & {*} & \cdots \\
    \cdots & {*} & {*} & 0 & {*} & {*} & \cdots \\
    &\vdots&\vdots&\vdots&\vdots&\vdots&
  },
\]
same as~$A$ except in the $i$th row and $j$th column.  The matrix~$C$,
called the \emph{cofactor} of~$A$, then consists of the determinants of
the various~$R_{ij}$.

Another way to write~(\ref{eigen:310:50}) is
\bq{eigen:310:54}
  [C^TA]_{ij} = (\det A)\delta_{ij} = [AC^T]_{ij},
\eq
which comprises two cases.  In the case that $i=j$,
\[
  [AC^T]_{ij} = [AC^T]_{ii} = \sum_\ell a_{i\ell}c_{i\ell}
  = \sum_\ell a_{i\ell}\det R_{i\ell} = \det A
  = (\det A)\delta_{ij},
\]
wherein the equation $\sum_\ell a_{i\ell}\det R_{i\ell} = \det A$ states
that $\det A$, being a determinant, consists of several terms, each term
including one factor from each row of~$A$, where~$a_{i\ell}$ provides
the $i$th row and~$R_{i\ell}$ provides the other rows.%
\footnote{
  This is a bit subtle, but if you actually write out~$A_3$ and its
  cofactor~$C_3$ symbolically, trying~(\ref{eigen:310:54}) on them, then
  you will soon see what is meant.
}
In the case that $i\neq j$,
\[
  [AC^T]_{ij} = \sum_\ell a_{i\ell}c_{j\ell}
  = \sum_\ell a_{i\ell}\det R_{j\ell} = 0 = (\det A)(0) =
  (\det A)\delta_{ij},
\]
wherein $\sum_\ell a_{i\ell}\det R_{j\ell}$ is the determinant, not
of~$A$ itself, but rather of~$A$ with the $j$th row replaced by a copy
of the $i$th, which according to~(\ref{eigen:310:17}) evaluates to zero.
Similar equations can be written for $[C^TA]_{ij}$ in both cases.  The
two cases together prove~(\ref{eigen:310:54}), hence
also~(\ref{eigen:310:50}).

\index{Cramer's rule}
\index{Cramer, Gabriel (1704--1752)}
\index{matrix!large}
Dividing~(\ref{eigen:310:50}) by $\det A$, we have that%
\footnote{
  Cramer's rule \cite[\S~1.6]{Franklin}, of which the reader may have
  heard, results from applying~(\ref{eigen:310:58})
  to~(\ref{mtxinv:230:15}).  However, Cramer's rule is really
  nothing more than~(\ref{eigen:310:58}) in a less pleasing form, so
  this book does not treat Cramer's rule as such.
}
\bqa
  A^{-1}A &=&
    I_n \makebox[\arraycolsep]{}=\makebox[\arraycolsep]{} AA^{-1},
  \xn\\
  A^{-1} &=& \frac{C^T}{\det A}.
  \label{eigen:310:58}
\eqa
Equation~(\ref{eigen:310:58}) inverts a matrix by determinant.  In
practice, it inverts small matrices nicely, through about $4 \times 4$
dimensionality (the~$A_2^{-1}$ equation at the head of the section is
just eqn.~\ref{eigen:310:58} for $n=2$).  It inverts $5 \times 5$ and
even $6 \times 6$ matrices reasonably, too---especially with the help of
a computer to do the arithmetic.  Though~(\ref{eigen:310:58}) still
holds in theory for yet larger matrices, and though symbolically
it expresses the inverse of an abstract, $n \times n$ matrix concisely
whose entries remain unspecified, for concrete matrices much bigger than
$4 \times 4$ to $6 \times 6$ or so its several determinants begin to
grow too great and too many for practical calculation.  The Gauss-Jordan
technique (or even the Gram-Schmidt technique) is preferred to invert
concrete matrices above a certain size for this reason.%
\footnote{%
  For very large matrices, even the Gauss-Jordan grows impractical, due
  to compound floating-point rounding error and the maybe large but
  nonetheless limited quantity of available computer memory.  Iterative
  techniques,
  % diagn
  regrettably beyond this edition's scope,
  serve to invert such matrices approximately.%
}

% ----------------------------------------------------------------------

\section{Co\"incident properties}
\label{eigen:370}
\index{co\"incident properties of the square matrix}
\index{square matrix!co\"incident properties of}
\index{matrix!square, co\"incident properties of}

\index{exact arithmetic}
\index{arithmetic!exact}
Chapters~\ref{matrix}, \ref{gjrank} and~\ref{mtxinv}, plus this chapter up
to the present point, have discovered several co\"incident properties of
the invertible $n \times n$ square matrix.  One does not feel the full
impact of the co\"incidence when these properties are left scattered
across the long chapters; so, let us gather and summarize the properties
here.  A square, $n \times n$ matrix evidently has either all of the
following properties or none of them, never some but not others.
\bi
  \item
    The matrix is invertible
    (\S~\ref{mtxinv:220}).
  \item
    Its rows are linearly independent
    (\S\S~\ref{gjrank:335} and~\ref{gjrank:341.11}).
  \item
    Its columns are linearly independent
    (\S~\ref{gjrank:340.25}).
  \item
    Its columns address the same space the columns of~$I_n$ address, and
    its rows address the same space the rows of~$I_n$ address
    (\S~\ref{gjrank:340.60}).
  \item
    The Gauss-Jordan algorithm reduces it to~$I_n$
    (\S~\ref{gjrank:341.10}).
    (In this, per \S~\ref{gjrank:340.20}, the choice of pivots does not
    matter.)
  \item
    Decomposing it, the Gram-Schmidt algorithm achieves a fully square,
    unitary, $n \times n$ factor~$Q$ (\S~\ref{mtxinv:460.30}).
  \item
    It has full rank $r=n$
    (\S~\ref{gjrank:340.25}).
  \item
    The linear system $A\ve x = \ve b$ it represents has a unique
    $n$-element solution~$\ve x$, given any specific $n$-element
    driving vector~$\ve b$
    (\S~\ref{mtxinv:230}).
  \item
    The determinant $\det A \neq 0$
    (\S~\ref{eigen:310.37}).
  \item
    None of its eigenvalues is zero
    (\S~\ref{eigen:410}, below).
\ei
The square matrix which has one of these properties, has all of them.
The square matrix which lacks one, lacks all.  Assuming exact
arithmetic, a square matrix is either invertible, with all that that
implies, or singular; never both.  The distinction between invertible
and singular matrices is theoretically as absolute as (and is indeed
analogous to) the distinction between nonzero and zero scalars.

\index{matrix!ill-conditioned}
\index{condition}
\index{ill-conditioned matrix}
Whether the distinction is always useful is another matter.  Usually the
distinction is indeed useful, but a matrix can be \emph{almost} singular
just as a scalar can be almost zero.  Such a matrix is known, among
other ways, by its unexpectedly small determinant.  Now it is true: in
exact arithmetic, a nonzero determinant, no matter how small, implies a
theoretically invertible matrix.  Practical matrices however often have
entries whose values are imprecisely known; and even when they don't,
the computers that invert them tend to do arithmetic imprecisely in
floating-point.  Matrices which live on the hazy frontier between
invertibility and singularity resemble the infinitesimals of
\S~\ref{drvtv:210.001}.  They are called \emph{ill-conditioned}
matrices.  Section~\ref{eigen:470} develops the topic.

% ----------------------------------------------------------------------

\section{The eigenvalue itself}
\label{eigen:410}
\index{eigenvalue}

\index{characteristic polynomial}
\index{polynomial!characteristic}
We stand ready at last to approach the final major agent of matrix arithmetic,
the \emph{eigenvalue.} Suppose a square, $n \times n$ matrix~$A$, a
nonzero $n$-element vector
\bq{eigen:410:05}
  \ve v = I_n\ve v \neq 0,
\eq
and a scalar~$\lambda$, together such that
\bq{eigen:eigen}
  A\ve v = \lambda\ve v,
\eq
or in other words such that $A\ve v = \lambda I_n\ve v$.  If so, then
\bq{eigen:410:10}
  [A-\lambda I_n]\ve v = 0.
\eq
Since~$I_n\ve v$ is nonzero, the last equation is true if and only
if the matrix $[A-\lambda I_n]$ is singular---which in light of
\S~\ref{eigen:310.37} is to demand that
\bq{eigen:eigdet}
  \det(A-\lambda I_n) = 0.
\eq
The left side of~(\ref{eigen:eigdet}) is an $n$th-order polynomial
in~$\lambda$, the \emph{characteristic polynomial,} whose~$n$ roots are
the \emph{eigenvalues}%
\footnote{
  An example:
  \bqb
    A &=& \mf{cc}{
      2 & 0 \\
      3 &-1
    }, \\
    \det(A-\lambda I_n) &=& \det \mf{cc}{
      2-\lambda & 0 \\
      3 &-1-\lambda
    }
    \\&=& (2-\lambda)(-1-\lambda) - (0)(3)
    \\&=& \lambda^2 - \lambda - 2 = 0, \\
    \lambda &=& \mbox{$-1$ or~$2$}.
  \eqb
}
of the matrix~$A$.

\index{eigenvector}
What is an eigenvalue, really?  An eigenvalue is a scalar a matrix
resembles under certain conditions.  When a matrix happens to operate
on the right \emph{eigenvector}~$\ve v$, it is all the same whether one
applies the entire matrix or just the eigenvalue to the vector.  The
matrix scales the eigenvector by the eigenvalue without otherwise
altering the vector, changing the vector's magnitude but not its
direction.  The eigenvalue alone takes the place of the whole, hulking
matrix.  This is what~(\ref{eigen:eigen}) means.  Of course it works
only when~$\ve v$ happens to be the right \emph{eigenvector,} which
\S~\ref{eigen:420} discusses.

Observe incidentally that the characteristic polynomial of an $n\times
n$ matrix always enjoys full order~$n$ regardless of the matrix's rank.
The reason lies in the determinant $\det(A-\lambda I_n)$, which
comprises exactly~$n!$ determinant-terms (we say ``determinant-terms''
rather than ``terms'' here only to avoid confusing the determinant's
terms with the characteristic polynomial's), only one of which,
$(a_{11}-\lambda)(a_{22}-\lambda)\cdots(a_{nn}-\lambda)$, gathers
elements straight down the main diagonal of the matrix $[A-\lambda
I_n]$.  When multiplied out, this main-diagonal determinant-term
evidently contributes a $(-\lambda)^n$ to the characteristic polynomial,
whereas none of the other determinant-terms finds enough factors
of~$\lambda$ to reach order~$n$.  (If unsure, take your pencil and just
calculate the characteristic polynomials of the $3\times 3$
matrices~$I_3$ and~$0$.  You will soon see what is meant.)

\index{eigenvalue!zero}
On the other hand, nothing prevents $\lambda = 0$.
When $\lambda = 0$, (\ref{eigen:eigdet}) makes $\det A=0$, which as we have
said is the sign of a singular matrix.  Zero eigenvalues and singular
matrices always travel together.  \emph{Singular matrices each have at
least one zero eigenvalue; nonsingular matrices never do.}

\index{eigenvalue!of a matrix inverse}
The eigenvalues of a matrix's inverse are the inverses of the matrix's
eigenvalues.  That is,
\bq{eigen:410:30}
  \lambda'_j\lambda_j = 1 \ \ %
  \mbox{for all $1\le j\le n$ if $A'A = I_n = AA'$.}
\eq
The reason behind~(\ref{eigen:410:30}) comes from answering the
question: if~$A\ve v_j$ scales~$\ve v_j$ by the factor~$\lambda_j$, then
what does $A'A\ve v_j = I\ve v_j$ do to~$\ve v_j$?

Naturally one must solve~(\ref{eigen:eigdet})'s $n$th-order polynomial
to locate the actual eigenvalues.  One solves it by the same techniques
by which one solves any polynomial: the quadratic
formula~(\ref{alggeo:240:quad}); the cubic and quartic methods of
chapter~\ref{cubic}; the Newton-Raphson iteration~(\ref{drvtv:NR}).  On the
other hand, the determinant~(\ref{eigen:eigdet}) can be impractical to
expand for a large matrix;
here iterative techniques\footnote{%
  % diagn
  Such iterative techniques are regrettably not treated by this
  edition.%
}
help.\footnote{%
  The inexpensive~\cite{Franklin} also covers the topic competently and
  readably.%
}

% ----------------------------------------------------------------------

\section{The eigenvector}
\label{eigen:420}
\index{eigenvector}

It is an odd fact that~(\ref{eigen:410:10}) and~(\ref{eigen:eigdet})
reveal the eigenvalues~$\lambda$ of a square matrix~$A$ while obscuring
the associated \emph{eigenvectors}~$\ve v$.  Once one has calculated an
eigenvalue, though, one can feed it back to calculate the associated
eigenvector.  According to~(\ref{eigen:410:10}), the eigenvectors are
the $n$-element vectors for which
\[
  [A-\lambda I_n]\ve v = 0,
\]
which is to say that the eigenvectors are the vectors of the kernel
space of the degenerate matrix $[A-\lambda I_n]$---which one can
calculate (among other ways) by the Gauss-Jordan kernel
formula~(\ref{mtxinv:245:kernel}) or by a method
exploiting~(\ref{mtxinv:450:perp}).

\index{eigensolution}
An eigenvalue and its associated eigenvector, taken together, are
sometimes called an \emph{eigensolution.}

% ----------------------------------------------------------------------

\section{Eigensolution facts}
\label{eigen:422}
\index{eigensolution}

Many useful or interesting mathematical facts concern the eigensolution,
among them the following.
\bi
  \item
    \index{shifting an eigenvalue}
    \index{eigenvalue!shifting of}
    \emph{If the eigensolutions of~$A$ are $(\lambda_j, \ve v_j)$, then
    the eigensolutions of $A+\alpha I_n$ are $(\lambda_j+\alpha, \ve
    v_j)$.}  The eigenvalues move over by~$\alpha I_n$ while the
    eigenvectors remain fixed.  This is seen by adding~$\alpha
    \ve v_j$ to both sides of the definition $A\ve v_j = \lambda \ve
    v_j$.
  \item
    \index{eigensolution!of a matrix inverse}
    \index{eigenvector!of a matrix inverse}
    \emph{A matrix and its inverse share the same eigenvectors with
    inverted eigenvalues.}  Refer to~(\ref{eigen:410:30}) and its
    explanation in \S~\ref{eigen:410}.
  \item
    \index{eigenvector!linearly independent}
    \index{eigenvalue!distinct}
    \index{coefficient!nontrivial}
    \emph{Eigenvectors corresponding to distinct eigenvalues are always
    linearly independent of one another.}  To prove this fact, consider
    several independent eigenvectors $\ve v_1, \ve v_2, \ldots, \ve
    v_{k-1}$ respectively with distinct eigenvalues $\lambda_1,
    \lambda_2, \ldots, \lambda_{k-1}$, and further consider another
    eigenvector~$\ve v_k$ which might or might not be independent but
    which too has a distinct eigenvalue~$\lambda_k$.  Were~$\ve v_k$
    dependent, which is to say, did nontrivial coefficients~$c_j$ exist
    such that
    \[
      \ve v_k  = \sum_{j=1}^{k-1} c_j \ve v_j,
    \]
    then left-multiplying the equation by $A-\lambda_kI_n$ would yield
    \[
      0 = \sum_{j=1}^{k-1} (\lambda_j-\lambda_k)c_j \ve v_j,
    \]
    impossible since the $k-1$ eigenvectors are independent.  Thus~$\ve
    v_k$ too is independent, whereupon by induction from a start case of
    $k=1$ we conclude that there exists no dependent eigenvector with a
    distinct eigenvalue.
  \item
    \index{repeated eigenvalue}
    \index{eigenvalue!repeated}
    \emph{If an $n \times n$ square matrix~$A$ has~$n$ independent
    eigenvectors} (which is always so if the matrix has~$n$ distinct
    eigenvalues and often so even otherwise), \emph{then any
    $n$-element vector can be expressed as a unique linear combination
    of the eigenvectors.}  This is a simple consequence of the fact that
    the $n \times n$ matrix~$V$ whose columns are the several
    eigenvectors~$\ve v_j$ has full rank $r=n$.  Unfortunately, some
    matrices with repeated eigenvalues also have repeated
    eigenvectors---as for example, curiously,%
    \footnote{\cite{math21b}}
    $[ 1\;0 ; 1\;1 ]^{T}$, whose double eigenvalue $\lambda = 1$ has the
    single eigenvector $[1\;0]^{T}$.
    Section~\ref{eigen:520.30} speaks of matrices of the last kind.
  \item
    \index{eigensolution!impossibility to share, given independent eigenvectors}
    \emph{An $n \times n$ square matrix whose eigenvectors are linearly
    independent of one another cannot share all eigensolutions with any
    other $n \times n$ square matrix.}
    This fact proceeds from the last point, that every $n$-element
    vector~$\ve x$ is a unique linear combination of independent
    eigenvectors.  Neither of the two proposed matrices~$A_1$ and~$A_2$
    could scale any of the eigenvector components of~$\ve x$ differently
    than the other matrix did, so $A_1\ve x - A_2\ve x = (A_1 - A_2)\ve
    x = 0$ for all~$\ve x$, which in turn is possible only if $A_1 =
    A_2$.
  \item
    \index{positive definiteness}
    \index{nonnegative definiteness}
    \emph{A positive definite matrix has only real, positive
    eigenvalues.  A nonnegative definite matrix has only real,
    nonnegative eigenvalues.}  Were it not so, then $\ve v^{*} A \ve v =
    \lambda \ve v^{*} \ve v$ (in which $\ve v^{*} \ve v$ naturally is a
    positive real scalar) would violate the criterion for positive or
    nonnegative definiteness.  See \S~\ref{mtxinv:320.34}.
  \item
    \index{eigensolution!count of}
    \emph{Every $n \times n$ square matrix has at least one eigensolution}
    if $n > 0$, because according to the fundamental theorem of
    algebra~(\ref{noth:320:50}) the matrix's characteristic
    polynomial~(\ref{eigen:eigdet}) has at least one root, an
    eigenvalue, which by definition would be no eigenvalue if it had no
    eigenvector to scale, and for which~(\ref{eigen:410:10}) necessarily
    admits at least one nonzero solution~$\ve v$ because its matrix
    $A-\lambda I_n$ is degenerate.
\ei

% ----------------------------------------------------------------------

\section{Diagonalization}
\label{eigen:423}
\index{diagonalization}
\index{decomposition!diagonal}
\index{decomposition!eigenvalue}
\index{diagonal decomposition}
\index{eigenvalue decomposition}

\index{eigenvalue matrix}
\index{matrix!eigenvalue}
Any $n \times n$ matrix with~$n$ independent eigenvectors (which class
per \S~\ref{eigen:422} includes, but is not limited to, every $n \times
n$ matrix with~$n$ distinct eigenvalues) can be \emph{diagonalized} as
\bq{eigen:diag}
  A = V \Lambda V^{-1},
\eq
where
\[
  \Lambda = \mf{ccccc}{
    \lambda_1 & 0 & \cdots & 0 & 0 \\
    0 & \lambda_2 & \cdots & 0 & 0 \\
    \vdots & \vdots & \ddots & \vdots & \vdots \\
    0 & 0 & \cdots & \lambda_{n-1} & 0 \\
    0 & 0 & \cdots & 0 & \lambda_n
  }
\]
is an otherwise empty $n \times n$ matrix with the eigenvalues of~$A$
set along its main diagonal and where
\[
  V = \left[
    \br{ccccc}
      \ve v_1 & \ve v_2 & \cdots & \ve v_{n-1} & \ve v_n
    \er
  \right]
\]
is an $n \times n$ matrix whose columns are the eigenvectors of~$A$.
This is so because the identity $A\ve v_j = \ve v_j \lambda_j$
holds for all $1 \le j \le n$; or, expressed more concisely, because the
identity
\bq{eigen:423:25}
  AV = V\Lambda
\eq
holds (reason: the $j$th column of the
product~$AV$ is~$A\ve v_j$, whereas the $j$th column of~$\Lambda$
having just the one element acts to scale $V$'s $j$th column only).
The matrix~$V$ is invertible because its columns the eigenvectors are
independent, from which~(\ref{eigen:diag}) follows.
Equation~(\ref{eigen:diag}) is called the \emph{eigenvalue
decomposition,} the \emph{diagonal decomposition} or the
\emph{diagonalization} of the square matrix~$A$.

One might object that we had shown only how to compose some
matrix $V\Lambda V^{-1}$ with the correct eigenvalues and independent
eigenvectors, but had failed to show that the matrix was actually~$A$.
However, we need not show this, because \S~\ref{eigen:422} has
already demonstrated that two matrices with the same eigenvalues and
independent eigenvectors are in fact the same matrix, whereby the
product $V\Lambda V^{-1}$ can be nothing other than~$A$.

\index{eigenvector!independent}
\index{eigenvalue!distinct}
\index{diagonalizability}
An $n \times n$ matrix with~$n$ independent
eigenvectors (which class, again, includes every $n \times n$ matrix
with~$n$ distinct eigenvalues and also includes many matrices with fewer)
is called a \emph{diagonalizable} matrix.
Besides factoring a diagonalizable matrix by~(\ref{eigen:diag}), one can
apply the same formula to compose a diagonalizable matrix with desired
eigensolutions.

The diagonal matrix $\mopx{diag}\{\ve x\}$ of~(\ref{matrix:325:diag}) is
trivially diagonalizable as $\mopx{diag}\{\ve x\}=I_n\mopx{diag}\{\ve
x\}I_n$.

\index{matrix!raised to a complex power}
It is a curious and useful fact that
\[
  A^2 = (V\Lambda V^{-1})(V\Lambda V^{-1}) = V\Lambda^2 V^{-1}
\]
and by extension that
\bq{eigen:423:30}
  A^k = V\Lambda^k V^{-1}
\eq
for any diagonalizable matrix~$A$.  The diagonal matrix~$\Lambda^k$ is
nothing more than the diagonal matrix~$\Lambda$ with each element
individually raised to the $k$th power, such that
\[
  \left[ \Lambda^k \right]_{ij} = \delta_{ij} \lambda_{j}^k.
\]
Changing $z \la k$ implies the generalization%
\footnote{
  It may not be clear however according to~(\ref{cexp:230:33}) which
  branch of~$\lambda_j^z$ one should choose at each index~$j$,
  especially if~$A$ has negative or complex eigenvalues.
}
\settoheight\tlj{\scriptsize $k$}
\bq{eigen:423:35}
  \begin{split}
    A^z &= V\Lambda^z V^{-1}, \\
    \left[ \Lambda^{\rule{0em}{\tlj}z} \right]_{ij} &= \delta_{ij} \lambda_{j}^z,
  \end{split}
\eq
good for any diagonalizable~$A$ and complex~$z$.

\index{matrix!nondiagonalizable versus singular}
\index{singular matrix!versus a nondiagonalizable matrix}
\index{nondiagonalizable matrix!versus a singular matrix}
\index{eigenvector!repeated}
\index{eigensolution!repeated}
\index{repeated eigenvalue}
Nondiagonalizable matrices are troublesome and interesting.  The
nondiagonalizable matrix vaguely resembles the singular matrix in that
both represent edge cases and can be hard to handle numerically; but the
resemblance ends there, and a matrix can be either without being the
other.  The $n \times n$ null matrix for example is singular but still
diagonalizable.  What a nondiagonalizable matrix is, is, in essence, a
matrix with a repeated eigensolution: the same eigenvalue with the same
eigenvector, twice or more.  More formally, a nondiagonalizable matrix
is a matrix with an $n$-fold eigenvalue whose corresponding eigenvector
space fewer than~$n$ eigenvectors fully characterize.
Section~\ref{eigen:520.30} will have more to say about the
nondiagonalizable matrix.

% ----------------------------------------------------------------------

\section{Remarks on the eigenvalue}
\label{eigen:429}

\index{dominant eigenvalue}
\index{eigenvalue!dominant}
Eigenvalues and their associated eigenvectors stand among the principal
causes that one should go to such considerable trouble to develop matrix theory as
we have done in recent chapters.  The idea that a matrix resembles a
humble scalar in the right circumstance is powerful.  Among the reasons
for this is that a matrix can represent an iterative process, operating
repeatedly on a vector~$\ve v$ to change it first to~$A \ve v$, then
to~$A^2 \ve v$, $A^3 \ve v$ and so on.  The \emph{dominant eigenvalue}
of~$A$, largest in magnitude, tends then to transform~$\ve v$ into the
associated eigenvector, gradually but relatively eliminating all other
components of~$\ve v$.  Should the dominant eigenvalue have greater than
unit magnitude, it destabilizes the iteration; thus one can sometimes
judge the stability of a physical process indirectly by examining the
eigenvalues of the matrix which describes it.  Then there is the edge
case of the nondiagonalizable matrix, which matrix surprisingly covers
only part of its domain with eigenvectors.  All this is fairly deep
mathematics.  It brings an appreciation of the matrix for reasons which
were anything but apparent from the outset of chapter~\ref{matrix}.

Remarks continue in \S\S~\ref{eigen:520.30} and~\ref{eigen:900}.

% ----------------------------------------------------------------------

\section{Matrix condition}
\label{eigen:470}
\index{matrix!condition of}
\index{condition}

\index{eigenvalue!dominant}
\index{dominant eigenvalue}
The largest in magnitude of the several eigenvalues of a diagonalizable
operator~$A$, denoted here~$\lambda_\mr{max}$, tends to dominate the
iteration~$A^k\ve x$.  Section~\ref{eigen:429} has
named~$\lambda_\mr{max}$ the \emph{dominant eigenvalue} for this reason.

\index{normalization}
\index{unit magnitude}
\index{magnitude!unit}
\index{magnitude!of an eigenvalue}
\index{iteration}
\index{magnification}
\index{eigenvalue!magnitude of}
\index{eigenvalue!zero}
\index{eigenvalue!small}
\index{eigenvalue!large}
\index{matrix!singular}
\index{singular matrix}
One sometimes finds it convenient to normalize a dominant eigenvalue by
defining a new operator $A' \equiv A/\left|\lambda_\mr{max}\right|$,
whose own dominant eigenvalue
$\lambda_\mr{max}/\left|\lambda_\mr{max}\right|$ has unit magnitude.  In
terms of the new operator, the iteration becomes $A^k\ve
x=\left|\lambda_\mr{max}\right|^kA'^k\ve x$, leaving one free to carry
the magnifying effect $\left|\lambda_\mr{max}\right|^k$ separately if
one prefers to do so.  However, the scale factor
$1/\left|\lambda_\mr{max}\right|$ scales all eigenvalues equally; thus,
if $A$'s eigenvalue of \emph{smallest} magnitude is
denoted~$\lambda_\mr{min}$, then the corresponding eigenvalue of~$A'$ is
$\lambda_\mr{min}/\left|\lambda_\mr{max}\right|$.  If zero, then both
matrices according to \S~\ref{eigen:410} are singular; if nearly zero,
then both matrices are ill conditioned.

\index{ill-conditioned matrix}
\index{matrix!ill-conditioned}
\index{imprecise quantity}
\index{floating-point number}
\index{skepticism}
\index{exact arithmetic}
\index{arithmetic!exact}
Such considerations lead us to define the \emph{condition} of a
diagonalizable matrix quantitatively as%
\footnote{
  \cite{deSturler-lecture}
}
\bq{eigen:470:10}
  \kappa \equiv \left| \frac{
    \lambda_\mr{max}
  }{
    \lambda_\mr{min}
  } \right|,
\eq
by which
\bq{eigen:470:20}
  \kappa \ge 1
\eq
is always a real number of no less than unit magnitude.  For best
invertibility, $\kappa = 1$ would be ideal (it would mean that all
eigenvalues had the same magnitude), though in practice quite a broad
range of~$\kappa$ is usually acceptable.  Could we always work in exact
arithmetic, the value of~$\kappa$ might not interest us much as long as
it stayed finite; but in computer floating point, or where the elements
of~$A$ are known only within some tolerance, infinite~$\kappa$ tends to
emerge imprecisely rather as large $\kappa \gg 1$.  An
\emph{ill-conditioned} matrix by definition%
\footnote{
  There is of course no definite boundary, no particular edge value
  of~$\kappa$, less than which a matrix is well conditioned, at and
  beyond which it turns ill-conditioned; but you knew that already.  If
  I tried to claim that a matrix with a fine $\kappa=3$ were ill
  conditioned, for instance, or that one with a wretched $\kappa =
  2^\mr{0x18}$ were well conditioned, then you might not credit
  me---but the mathematics nevertheless can only give the number; it
  remains to the mathematician to interpret it.
}
is a matrix of large $\kappa \gg 1$.  The applied mathematician
handles such a matrix with due skepticism.

\index{error!in the solution to a linear system}
\index{solution!error in}
Matrix condition so defined turns out to have another useful
application.  Suppose that a diagonalizable matrix~$A$ is precisely
known but that the corresponding driving vector~$\ve b$ is not.  If
\[
  A(\ve x + \delta\ve x)= \ve b + \delta\ve b,
\]
where~$\delta\ve b$ is the error in~$\ve b$ and~$\delta\ve x$ is the
resultant error in~$\ve x$, then one should like to bound the
ratio $\left|\delta\ve x\right|/\left|\ve x\right|$ to ascertain the
reliability of~$\ve x$ as a solution.
Transferring~$A$ to the equation's right side,
\[
  \ve x + \delta\ve x= A^{-1}(\ve b + \delta\ve b).
\]
Subtracting $\ve x=A^{-1}\ve b$ and taking the magnitude,
\[
  \left|\delta\ve x\right| = \left|A^{-1}\,\delta\ve b\right|.
\]
Dividing by $\left|\ve x\right| = \left|A^{-1}\ve b\right|$,
\[
  \frac{\left|\delta\ve x\right|}{\left|\ve x\right|}
    = \frac{\left|A^{-1}\,\delta\ve b\right|}{\left|A^{-1}\ve b\right|}.
\]
The quantity $\left|A^{-1}\,\delta\ve b\right|$ cannot
exceed $\left|\lambda_\mr{min}^{-1}\,\delta\ve b\right|$.  The quantity
$\left|A^{-1}\ve b\right|$ cannot fall short
of $\left|\lambda_\mr{max}^{-1}\ve b\right|$.  Thus,
\[
  \frac{\left|\delta\ve x\right|}{\left|\ve x\right|}
    \le \frac{\left|\lambda_\mr{min}^{-1}\,\delta\ve
    b\right|}{\left|\lambda_\mr{max}^{-1}\ve b\right|}
    =
    \left|\frac{\lambda_\mr{max}}{\lambda_\mr{min}}\right|
    \frac{\left|\delta\ve b\right|}{\left|\ve b\right|}.
\]
That is,
\bq{eigen:470:30}
  \frac{\left|\delta\ve x\right|}{\left|\ve x\right|}
    \le \kappa \frac{\left|\delta\ve b\right|}{\left|\ve b\right|}.
\eq

\index{scalar!condition of}
\index{condition!of a scalar}
Condition, incidentally, might technically be said to apply to scalars as
well as to matrices, but ill condition remains a property of matrices
alone.  According to~(\ref{eigen:470:10}), the condition of every
nonzero scalar is happily $\kappa = 1$.

% ----------------------------------------------------------------------

\section{The similarity transformation}
\label{eigen:505}
\index{similarity transformation}

\index{basis}
\index{vector!building of from basis vectors}
Any collection of vectors assembled into a matrix can serve as a
\emph{basis} by which other vectors can be expressed.  For example, if
the columns of
\[
  B = \mf{rr}{
    1 & -1 \\
    0 &  2 \\
    0 &  1
  }
\]
are regarded as a basis, then the vector
\[
  B \mf{r}{
    5 \\ 1
  } = 5\mf{r}{
    1 \\ 0 \\ 0
  } + 1\mf{r}{
    -1 \\ 2 \\ 1
  } = \mf{r}{
    4 \\ 2 \\ 1
  }
\]
is $(5,1)$ in the basis~$B$: five times the first basis vector plus once
the second.  The basis provides the units from which other vectors can
be built.

\index{basis!complete}
\index{complete basis}
\index{basis!converting to and from}
Particularly interesting is the $n \times n$, invertible \emph{complete
basis}~$B$, in which the~$n$ basis vectors are independent and address
the same full space the columns of~$I_n$ address.  If
\[
  \ve x = B \ve u
\]
then~$\ve u$ represents~$\ve x$ in the basis~$B$.  Left-multiplication
by~$B$ evidently converts out of the basis.  Left-multiplication
by~$B^{-1}$,
\[
  \ve u = B^{-1} \ve x,
\]
then does the reverse, converting into the basis.  One can therefore
convert any operator~$A$ to work within a complete basis~$B$ by the
successive steps
\bqb
  A\ve x &=& \ve b, \\
  AB\ve u &=& \ve b, \\
  \mbox{}[B^{-1}AB]\ve u &=& B^{-1}\ve b,
\eqb
by which the operator $B^{-1}AB$ is seen to be the operator~$A$, only
transformed to work within the basis%
\footnote{
  The reader may need to ponder the basis concept a while to grasp it,
  but the concept is simple once grasped and little purpose would be
  served by dwelling on it here.  Basically, the idea is that one can
  build the same vector from alternate building blocks, not only from
  the standard building blocks~$\ve e_1$, $\ve e_2$, $\ve e_3$,
  etc.---except that the right word for the relevant ``building block''
  is \emph{basis vector.}  The books~\cite{Hefferon} and~\cite{Lay}
  introduce the basis more gently; one might consult one of those if
  needed.
}$\mbox{}^,$%
\footnote{
  The professional matrix literature sometimes distinguishes by typeface
  between the matrix~$B$ and the basis~$\textsf{B}$ its columns
  represent.  Such semantical distinctions seem a little too fine for
  applied use, though.  This book just uses~$B$.
}%
~$B$.

\index{similarity}
\index{unitary similarity}
\index{unitary transformation}
The conversion from~$A$ into $B^{-1}AB$ is called a \emph{similarity
transformation.}  If~$B$ happens to be unitary (\S~\ref{mtxinv:465}),
then the conversion is also called a \emph{unitary transformation.}  The
matrix $B^{-1}AB$ the transformation produces is said to be
\emph{similar} (or, if~$B$ is unitary, \emph{unitarily similar}) to the
matrix~$A$.  We have already met the similarity transformation in
\S\S~\ref{matrix:321} and~\ref{gjrank:337}.  Now we have the theory to
appreciate it properly.

Probably the most important property of the similarity transformation is
that it alters no eigenvalues.  That is, if
\[
  A \ve x = \lambda \ve x,
\]
then, by successive steps,
\bqa
  B^{-1}A(BB^{-1})\ve x &=& \lambda B^{-1} \ve x, \xn\\
  \mbox{}[B^{-1}AB]\ve u &=& \lambda \ve u. \label{eigen:505:50}
\eqa
\emph{The eigenvalues of~$A$ and the similar $B^{-1}AB$ are the same}
for any square, $n \times n$ matrix~$A$ and any invertible, square, $n
\times n$ matrix~$B$.

% ----------------------------------------------------------------------

\section{The Schur decomposition}
\label{eigen:520}
\index{decomposition!Schur}
\index{Schur decomposition}
\index{Schur, Issai (1875--1941)}

The \emph{Schur decomposition} of an arbitrary, $n \times n$ square
matrix~$A$ is
\bq{eigen:schur}
  A = QU_SQ^{*},
\eq
where~$Q$ is an $n \times n$ unitary matrix whose inverse, as for any
unitary matrix (\S~\ref{mtxinv:465}), is $Q^{-1} = Q^{*}$; and
where~$U_S$ is a general upper triangular matrix which can have any
values (even zeros) along its main diagonal.  The Schur decomposition is
slightly obscure, is somewhat tedious to derive and is of limited use in
itself, but serves a theoretical purpose.%
\footnote{
  The alternative is to develop the interesting but difficult
  \emph{Jordan canonical form,} which for brevity's sake this chapter
  prefers to omit.
}
We derive it here for this reason.

\subsection{Derivation}
\label{eigen:520.10}

Suppose that%
\footnote{
  This subsection assigns various capital Roman letters to represent the
  several matrices and submatrices it manipulates.  Its choice of
  letters except in~(\ref{eigen:schur}) is not standard and carries no
  meaning elsewhere.  The writer had to choose some letters and these
  are ones he chose.

  This footnote mentions the fact because good mathematical style
  avoid assigning letters that already bear a conventional meaning in a
  related context (for example, this book avoids writing $A\ve x = \ve
  b$ as $T\ve e = \ve i$, not because the latter is wrong but because it
  would be extremely confusing).  The Roman alphabet provides only
  twenty-six capitals, though, of which this subsection uses too many to
  be allowed to reserve any.  See appendix~\ref{greek}\@.
}
(for some reason, which will shortly grow clear) we have a
matrix~$B$ of the form
\settowidth\tla{$C \equiv Q^{*}BQ$}
\settowidth\tlc{$,$}
\bq{eigen:520:21}
  \makebox[\tla][r]{$B$} = \mf{ccccccccc}{
    \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
    \cdots &{*}&{*}&{*}&{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 &{*}&{*}&{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 &{*}&{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 &{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 &{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 &{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 &{*}&{*}&{*}& \cdots \\
    &\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
  }\makebox[\tlc][l]{$,$}
\eq
where the $i$th row and $i$th column are depicted at center.
Suppose further that we wish to transform~$B$ not only similarly but
unitarily into
\bq{eigen:520:22}
  \makebox[\tla][r]{$C \equiv W^{*}BW$} = \mf{ccccccccc}{
    \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
    \cdots &{*}&{*}&{*}&{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 &{*}&{*}&{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 &{*}&{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 &{*}&{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 &{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 &{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 & 0 &{*}&{*}& \cdots \\
    &\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
  }\makebox[\tlc][l]{$,$}
\eq
where~$W$ is an $n\times n$ unitary matrix, and where we do not mind if
any or all of the~${*}$ elements change in going from~$B$ to~$C$ but we
require zeros in the indicated spots.  Let~$B_o$ and~$C_o$ represent the
$(n-i) \times (n-i)$ submatrices in the lower right corners respectively
of~$B$ and~$C$, such that
\bq{eigen:520:25}
  \begin{split}
    B_o & \equiv I_{n-i}H_{-i}BH_{i}I_{n-i}, \\
    C_o & \equiv I_{n-i}H_{-i}CH_{i}I_{n-i},
  \end{split}
\eq
where~$H_k$ is the shift operator of \S~\ref{matrix:340}.
Pictorially,
\[
  B_o = \mf{cccc}{
    {*}&{*}&{*}& \cdots \\
    {*}&{*}&{*}& \cdots \\
    {*}&{*}&{*}& \cdots \\
    \vdots&\vdots&\vdots&\ddots
  },
  \ \ %
  C_o = \mf{cccc}{
    {*}&{*}&{*}& \cdots \\
     0 &{*}&{*}& \cdots \\
     0 &{*}&{*}& \cdots \\
    \vdots&\vdots&\vdots&\ddots
  }.
\]

Equation~(\ref{eigen:520:22}) seeks an $n \times n$ unitary matrix~$W$ to
transform the matrix~$B$ into a new matrix $C \equiv W^{*}BW$ such
that~$C$ fits the form~(\ref{eigen:520:22}) stipulates.  The question
remains as to whether a unitary~$W$ exists that satisfies the form and
whether for general~$B$ we can discover a way to calculate it.  To
narrow the search, because we need not find every~$W$ that satisfies the
form but only one such~$W$, let us look first for a~$W$ that fits the
restricted template
\bq{eigen:520:30}
  W = I_{i} + H_{i}W_oH_{-i}
  = \mf{ccccccccc}{
    \ddots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\\
    \cdots & 1 & 0 & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 1 & 0 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 1 & 0 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 1 & 0 & 0 & 0 & \cdots \\
    \cdots & 0 & 0 & 0 & 0 &{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 &{*}&{*}&{*}& \cdots \\
    \cdots & 0 & 0 & 0 & 0 &{*}&{*}&{*}& \cdots \\
    &\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots
  },
\eq
which contains a smaller, $(n-i)\times(n-i)$ unitary submatrix~$W_o$ in
its lower right corner and resembles~$I_n$ elsewhere.  Beginning
from~(\ref{eigen:520:22}), we have by successive, reversible steps that
\bqb
  C &=& W^{*} B W
  \\&=&
    (I_{i} + H_{i}W_o^{*}H_{-i})
    (B)
    (I_{i} + H_{i}W_oH_{-i})
  \\&=&
    I_{i}BI_{i}
    + I_{i}BH_{i}W_oH_{-i}
    + H_{i}W_o^{*}H_{-i}BI_{i}
  \\&&\ \ \ \ \mbox{}
    + H_{i}W_o^{*}H_{-i}BH_{i}W_oH_{-i}.
\eqb
The unitary submatrix~$W_o$ has only $n-i$ columns and $n-i$ rows, so
$I_{n-i}W_o = W_o = W_oI_{n-i}$.  Thus,\footnote{%
  The algebra is so thick that, even if one can logically follow it, one
  might nonetheless wonder how the writer had thought to write it.
  However, much of the algebra consists of crop-and-shift operations
  like $H_iI_{n-i}$ which, when a sample matrix is sketched on a sheet
  of paper, are fairly easy to visualize.  Indeed, the whole derivation
  is more visual than the inscrutable symbols let on.  The writer
  had the visuals in mind.%
}
\bqb
  C &=&
    I_{i}BI_{i}
    + I_{i}BH_{i}W_oI_{n-i}H_{-i}
    + H_{i}I_{n-i}W_o^{*}H_{-i}BI_{i}
  \\&&\ \ \ \ \mbox{}
    + H_{i}I_{n-i}W_o^{*}I_{n-i}H_{-i}BH_{i}I_{n-i}W_oI_{n-i}H_{-i}
  \\&=&
    I_{i}[B]I_{i}
    + I_{i}[BH_{i}W_oH_{-i}](I_n-I_i)
    + (I_n-I_i)[H_{i}W_o^{*}H_{-i}B]I_{i}
  \\&&\ \ \ \ \mbox{}
    + (I_n-I_i)[H_{i}W_o^{*}B_oW_oH_{-i}](I_n-I_i),
\eqb
where the last step has used~(\ref{eigen:520:25}) and the
identity~(\ref{matrix:340:61}).  The four terms on the equation's right,
each term with rows and columns neatly truncated, represent the four
quarters of $C \equiv W^{*} B W$---upper left, upper right, lower left
and lower right, respectively.  The lower left term is null because
\bqb
  (I_n-I_i)[H_{i}W_o^{*}H_{-i}B]I_i &=&
    (I_n-I_i)[H_{i}W_o^{*}I_{n-i}H_{-i}BI_i]I_i
  \\&=&
    (I_n-I_i)[H_{i}W_o^{*}H_{-i}][(I_n-I_i)BI_i]I_i
  \\&=&
    (I_n-I_i)[H_{i}W_o^{*}H_{-i}][0]I_i = 0,
\eqb
leaving
\bqb
  C &=&
    I_{i}[B]I_{i}
    + I_{i}[BH_{i}W_oH_{-i}](I_n-I_i)
  \\&&\ \ \ \ \mbox{}
    + (I_n-I_i)[H_{i}W_o^{*}B_oW_oH_{-i}](I_n-I_i).
\eqb
But the upper left term makes the upper left areas of~$B$ and~$C$ the
same, and the upper right term does not bother us because we have not
restricted the content of $C$'s upper right area.  Apparently any
$(n-i)\times(n-i)$ unitary submatrix~$W_o$ whatsoever
obeys~(\ref{eigen:520:22}) in the lower left, upper left and upper
right.

That leaves the lower right.  Left- and
right-multiplying~(\ref{eigen:520:22}) by the truncator
$(I_n-I_i)$ to focus solely on the lower right area, we have the
reduced requirement that
\bq{eigen:520:38}
  (I_n-I_i)C(I_n-I_i) =
    (I_n-I_i)W^{*}BW(I_n-I_i).
\eq
Further left-multiplying by~$H_{-i}$, right-multiplying by~$H_i$,
and applying the identity~(\ref{matrix:340:61}) yields that
\bqb
  I_{n-i}H_{-i}CH_{i}I_{n-i}
    = I_{n-i}H_{-i}W^{*}BWH_{i}I_{n-i};
\eqb
or, substituting from~(\ref{eigen:520:25}), that
\[
  C_o = I_{n-i}H_{-i}W^{*}BWH_{i}I_{n-i}.
\]
Expanding~$W$ per~(\ref{eigen:520:30}),
\[
  C_o = I_{n-i}H_{-i}(I_{i} + H_{i}W_o^{*}H_{-i})
    B(I_{i} + H_{i}W_oH_{-i})H_{i}I_{n-i};
\]
or, since $I_{n-i}H_{-i}I_{i} = 0 = I_iH_{i}I_{n-i}$,
\bqb
  C_o &=& I_{n-i}H_{-i}(H_{i}W_o^{*}H_{-i})
    B(H_{i}W_oH_{-i})H_{i}I_{n-i}
  \\&=&
    I_{n-i}W_o^{*}H_{-i}
    BH_{i}W_oI_{n-i}
  \\&=&
    W_o^{*}I_{n-i}H_{-i}BH_{i}I_{n-i}W_o.
\eqb
Per~(\ref{eigen:520:25}), this has that
\bq{eigen:520:40}
  C_o = W_o^{*}B_oW_o.
\eq
The steps from~(\ref{eigen:520:38}) to~(\ref{eigen:520:40}) are
reversible, so the latter is as good a way to state the reduced
requirement as the former is.  To achieve a unitary transformation of
the form~(\ref{eigen:520:22}), therefore, it suffices to
satisfy~(\ref{eigen:520:40}).

The increasingly well-stocked armory of matrix theory we now have to
draw from makes satisfying~(\ref{eigen:520:40}) possible as follows.
Observe per \S~\ref{eigen:422} that every square matrix has at least one
eigensolution.  Let $(\lambda_o,\ve v_o)$ represent an eigensolution
of~$B_o$---\emph{any} eigensolution of~$B_o$---with~$\ve v_o$ normalized
to unit magnitude.  Form the broad, $(n-i)\times(n-i+1)$ matrix
\[
  F \equiv \left[
    \br{cccccc}
      \ve v_o & \ve e_1 & \ve e_2 & \ve e_3 & \cdots & \ve e_{n-i}
    \er
  \right].
\]
Decompose~$F$ by the Gram-Schmidt technique of \S~\ref{mtxinv:460.30},
choosing $p=1$ during the first instance of the algorithm's
step~\ref{mtxinv:461:s20} (though choosing any permissible~$p$
thereafter), to obtain
\[
  F = Q_F R_F.
\]
Noting that the Gram-Schmidt algorithm orthogonalizes only rightward,
observe that the first column of the $(n-i) \times (n-i)$ unitary
matrix~$Q_F$ remains simply the first column of~$F$, which is the unit
eigenvector~$\ve v_o$:
\[
  \left[ Q_F \right]_{*1} = Q_F \ve e_1 = \ve v_o.
\]
Transform~$B_o$ unitarily by~$Q_F$ to define the new matrix
\[
  G \equiv Q_F^{*} B_o Q_F,
\]
then transfer factors to reach the equation
\[
  Q_F G Q_F^{*} = B_o.
\]
Right-multiplying by $Q_F \ve e_1 = \ve v_o$ and noting that
$B_o\ve v_o = \lambda_o\ve v_o$, observe that
\[
  Q_F G \ve e_1 = \lambda_o\ve v_o.
\]
Left-multiplying by~$Q_F^{*}$,
\[
  G \ve e_1 = \lambda_o Q_F^{*}\ve v_o.
\]
Noting that the Gram-Schmidt process has rendered orthogonal to~$\ve
v_o$ all columns of~$Q_F$ but the first, which is~$\ve v_o$, observe that
\[
  G \ve e_1 = \lambda_o Q_F^{*}\ve v_o = \lambda_o \ve e_1 = \mf{c}{
    \lambda_o \\ 0 \\ 0 \\ \vdots
  },
\]
which means that
\[
  G = \mf{cccc}{
    \lambda_o&{*}&{*}& \cdots \\
     0 &{*}&{*}& \cdots \\
     0 &{*}&{*}& \cdots \\
    \vdots&\vdots&\vdots&\ddots
  },
\]
which fits the very form~(\ref{eigen:520:25}) the submatrix~$C_o$ is
required to have.  Conclude therefore that
\bq{eigen:520:50}
  \begin{split}
    W_o &= Q_F, \\
    C_o &= G,
  \end{split}
\eq
where~$Q_F$ and~$G$ are as this paragraph develops, together
constitute a valid choice for~$W_o$ and~$C_o$,
satisfying the reduced requirement~(\ref{eigen:520:40}) and thus also
the original requirement~(\ref{eigen:520:22}).

Equation~(\ref{eigen:520:50}) completes a failsafe technique to
transform unitarily any square matrix~$B$ of the
form~(\ref{eigen:520:21}) into a square matrix~$C$ of the
form~(\ref{eigen:520:22}).  Naturally the technique can be applied
recursively as
\bq{eigen:520:53}
  B|_{i=i'} = C|_{i=i'-1}, \ \ 1 \le i' \le n,
\eq
because the form~(\ref{eigen:520:21}) of~$B$ at $i=i'$ is nothing
other than the form~(\ref{eigen:520:22}) of~$C$ at $i=i'-1$.  Therefore,
if we let
\bq{eigen:520:56}
  B|_{i=0} = A,
\eq
then it follows by induction that
\bq{eigen:520:57}
  B|_{i=n} = U_S,
\eq
where per~(\ref{eigen:520:21}) the matrix~$U_S$ has the general upper
triangular form the Schur decomposition~(\ref{eigen:schur}) requires.
Moreover, because the product of unitary matrices according
to~(\ref{mtxinv:465:20}) is itself a unitary matrix, we have that
\bq{eigen:520:60}
  Q = \coprod_{i'=0}^{n-1} \left( W|_{i=i'} \right),
\eq
which along with~(\ref{eigen:520:57}) accomplishes the Schur
decomposition.

\subsection{The nondiagonalizable matrix}
\label{eigen:520.30}
\index{nondiagonalizable matrix}
\index{matrix!nondiagonalizable}

\index{triangular matrix}
\index{general triangular matrix}
\index{diagonal!main}
\index{main diagonal}
\index{matrix!main diagonal of}
\index{characteristic polynomial}
\index{polynomial!characteristic}
The characteristic equation~(\ref{eigen:eigdet}) of
the general upper triangular matrix~$U_S$ is
\[
 \det( U_S - \lambda I_n ) = 0.
\]
Unlike most determinants, this determinant brings only the one term
\[
  \det( U_S - \lambda I_n ) = \prod_{i=1}^n ( u_{Sii} - \lambda ) = 0
\]
whose factors run straight down the main diagonal, where the
determinant's $n!-1$ other terms are all zero because each of them
includes at least one zero factor from below the main diagonal.%
\footnote{
  The determinant's definition in \S~\ref{eigen:310} makes the following
  two propositions equivalent: (i)~that a determinant's term which
  includes one or more factors above the main diagonal also includes one
  or more factors below; (ii)~that the only permutor that marks no
  position below the main diagonal is the one which also marks no
  position above.  In either form, the proposition's truth might seem
  less than obvious until viewed from the proper angle.

  Consider a permutor~$P$.  If~$P$ marked no position below the main
  diagonal, then it would necessarily have $p_{nn}=1$, else the
  permutor's bottom row would be empty which is not allowed.  In the
  next-to-bottom row, $p_{(n-1)(n-1)}=1$, because the $n$th column is
  already occupied.  In the next row up, $p_{(n-2)(n-2)}=1$; and so on,
  thus affirming the proposition.
}
Hence no element above the main diagonal of~$U_S$ even influences the
eigenvalues, which apparently are
\bq{eigen:520:70}
  \lambda_i = u_{Sii},
\eq
the main-diagonal elements.

According to~(\ref{eigen:505:50}), similarity transformations
preserve eigenvalues.
The Schur decomposition~(\ref{eigen:schur}) is in fact
a similarity transformation; and, as we have seen, every matrix~$A$ has
a Schur decomposition.  If therefore
\[
  A = QU_SQ^{*},
\]
then \emph{the eigenvalues of~$A$ are just the values along the main
diagonal of~$U_S$.}%
\footnote{
  An unusually careful reader might worry that~$A$ and~$U_S$ had the
  same eigenvalues with different multiplicities.  It would be
  surprising if it actually were so; but, still, one would like to give
  a sounder reason than the participle ``surprising.''  Consider however
  that
  \bqb
    \lefteqn{A - \lambda I_n = QU_SQ^{*} - \lambda I_n
    = Q[U_S - Q^{*}(\lambda I_n)Q]Q^{*}} &&
    \\&&\ \ \ \ \ \ \ \ \ \ \ \ = Q[U_S - \lambda(Q^{*}I_nQ)]Q^{*}
    = Q[U_S - \lambda I_n]Q^{*}.
  \eqb
  According to~(\ref{eigen:310:30}) and~(\ref{eigen:310:64}),
  this equation's determinant is
  \[
    \det[A - \lambda I_n]
    = \det\{Q[U_S - \lambda I_n]Q^{*}\}
    = \det Q \det [U_S - \lambda I_n] \det Q^{*}
    = \det [U_S - \lambda I_n],
  \]
  which says that~$A$ and~$U_S$ have not only the same eigenvalues
  but also the same characteristic polynomials, and thus further the
  same eigenvalue multiplicities.
}

One might think that the Schur decomposition offered an easy way to
calculate eigenvalues, but it is less easy than it first appears because
one must calculate eigenvalues to reach the Schur decomposition in the
first place.  Whatever practical merit the Schur decomposition might
have or lack, however, it brings at least the theoretical benefit
of~(\ref{eigen:520:70}): every square matrix without exception has a
Schur decomposition, whose triangular factor~$U_S$ openly lists all
eigenvalues along its main diagonal.

\index{perturbation}
\index{eigenvalue!perturbed}
This theoretical benefit pays when some of the~$n$ eigenvalues of an $n
\times n$ square matrix~$A$ repeat.  By the Schur decomposition, one can
construct a second square matrix~$A'$, as near as desired to~$A$ but
having~$n$ distinct eigenvalues, simply by perturbing the main diagonal
of~$U_S$ to%
\footnote{
  Equation~(\ref{matrix:325:diag}) defines the $\mopx{diag}\{\cdot\}$
  notation.
}
\bqa
    U_S' &\equiv& U_S + \ep\mopx{diag}\{\ve u\},
  \label{eigen:520:75}\\
    u_{i'} &\neq& u_i \ \ \mbox{if $\lambda_{i'} = \lambda_i$},
  \xn
\eqa
where $|\ep| \ll 1$ and where~$\ve u$ is an arbitrary vector that meets
the criterion given.  Though infinitesimally near~$A$, the modified
matrix $A' = QU_S'Q^{*}$ unlike~$A$ has~$n$ (maybe infinitesimally)
distinct eigenvalues.  With sufficient toil, one might analyze such
perturbed eigenvalues and their associated eigenvectors similarly as
\S~\ref{inttx:260.20} has analyzed perturbed poles.

\index{repeated eigenvalue}
\index{repeated eigensolution}
\index{eigenvalue!repeated}
\index{eigenvector!repeated}
\index{eigensolution!repeated}
Equation~(\ref{eigen:520:75}) brings us to the nondiagonalizable matrix
of the subsection's title.  Section~\ref{eigen:423} and its
diagonalization formula~(\ref{eigen:diag}) diagonalize any matrix with
distinct eigenvalues and even any matrix with repeated eigenvalues but
distinct eigenvectors, but fail where eigenvectors repeat.
Equation~(\ref{eigen:520:75}) separates eigenvalues, and thus also
eigenvectors---for according to \S~\ref{eigen:422} eigenvectors of
distinct eigenvalues never depend on one another---permitting a
nonunique but still sometimes usable form of diagonalization in the
limit $\ep \ra 0$ even when the matrix in question is strictly nondiagonalizable.

\index{characteristic polynomial}
\index{polynomial!characteristic}
The finding that every matrix is arbitrarily nearly diagonalizable
illuminates a question the chapter has evaded up to the present point.
The question: does a $p$-fold root in the characteristic
polynomial~(\ref{eigen:eigdet}) necessarily imply a $p$-fold
eigenvalue in the corresponding matrix?
The existence of the nondiagonalizable matrix casts a shadow of doubt
until one realizes that every nondiagonalizable matrix is arbitrarily
nearly diagonalizable---and, better, is arbitrarily nearly
diagonalizable with distinct eigenvalues.  If you claim that a matrix
has a triple eigenvalue and someone disputes the claim, then you can
show him a nearly identical matrix with three infinitesimally distinct
eigenvalues.  That is the essence of the idea.  We will leave the
answer in that form.

\index{generalized eigenvector}
\index{eigenvector!generalized}
\index{Hessenberg, Gerhard (1874--1925)}
\index{Hessenberg matrix}
Generalizing the nondiagonalizability concept leads one eventually to
the ideas of the \emph{generalized eigenvector}%
\footnote{\cite[chapter~7]{Friedberg-IS}}
(which solves the higher-order linear system $[A-\lambda I]^k\ve v=0$)
and the \emph{Jordan canonical form,}%
\footnote{\cite[chapter~5]{Franklin}}
which together roughly track the sophisticated conventional
pole-separation technique of \S~\ref{inttx:260.50}.
Then there is a kind of sloppy Schur form called a Hessenberg form which
allows content in~$U_S$ along one or more subdiagonals just beneath the
main diagonal.  One could profitably propose and prove any number of
useful theorems concerning the nondiagonalizable matrix and its
generalized eigenvectors, or concerning the eigenvalue problem%
\footnote{\cite{Wilkinson}}
more broadly, in more and less rigorous ways,
but for the time being we will let the matter rest there.

% ----------------------------------------------------------------------

\section{The Hermitian matrix}
\label{eigen:550}
\index{Hermitian matrix}
\index{matrix!Hermitian}
\index{Hermite, Charles (1822--1901)}

\index{adjoint}
\index{self-adjoint matrix}
\index{matrix!self-adjoint}
An $m \times m$ square matrix~$A$ that is its own adjoint,
\bq{eigen:550:10}
  A^{*} = A,
\eq
is called a \emph{Hermitian} or \emph{self-adjoint} matrix.  Properties
of the Hermitian matrix include that
\bi
  \item
    \index{eigenvalue!real}
    its eigenvalues are real,
  \item
    \index{eigenvector!orthogonal}
    its eigenvectors corresponding to distinct eigenvalues lie
    orthogonal to one another, and
  \item
    \index{diagonalizable matrix}
    \index{matrix!diagonalizable}
    it is unitarily diagonalizable (\S\S~\ref{mtxinv:465}
    and~\ref{eigen:423}) such that
    \bq{eigen:550:30}
      A = V \Lambda V^{*}.
    \eq
\ei

That the eigenvalues are real is proved by
letting $(\lambda, \ve v)$ represent an eigensolution of~$A$ and
constructing the product $\ve v^{*}A\ve v$, for which
\[
  \lambda^{*} \ve v^{*}\ve v
  =
  \ve (A\ve v)^{*}\ve v
  =
  \ve v^{*}A\ve v
  =
  \ve v^{*}(A\ve v)
  =
  \lambda \ve v^{*}\ve v.
\]
That is,
\[
  \lambda^{*}
  =
  \lambda,
\]
which naturally is possible only if~$\lambda$ is real.

That eigenvectors corresponding to distinct eigenvalues lie
orthogonal to one another is proved%
\footnote{\cite[\S~8.1]{Lay}}
by letting $(\lambda_1, \ve v_1)$ and $(\lambda_2, \ve v_2)$ represent
eigensolutions of~$A$ and constructing the product $\ve v_2^{*}A\ve
v_1$, for which
\[
  \ve \lambda_2^{*} \ve v_2^{*}\ve v_1
  =
  \ve (A\ve v_2)^{*}\ve v_1
  =
  \ve v_2^{*}A\ve v_1
  =
  \ve v_2^{*}(A\ve v_1)
  =
  \ve \lambda_1 \ve v_2^{*}\ve v_1.
\]
That is,
\[
  \mbox{$\lambda_2^{*} = \lambda_1$\ \ or\ \ $\ve v_2^{*}\ve v_1 = 0$.}
\]
But according to the last paragraph all eigenvalues are real; the
eigenvalues~$\lambda_1$ and~$\lambda_2$ are no exceptions.  Hence,
\[
  \mbox{$\lambda_2 = \lambda_1$\ \ or\ \ $\ve v_2^{*}\ve v_1 = 0$.}
\]

To prove the last hypothesis of the three needs first some definitions
as follows.  Given an $m \times m$ matrix~$A$, let the~$s$ columns of
the $m \times s$ matrix~$V_o$ represent the~$s$ independent eigenvectors
of~$A$ such that (i)~each column has unit magnitude and~(ii) columns
whose eigenvectors share the same eigenvalue lie orthogonal to one
another.  Let the $s \times s$ diagonal matrix~$\Lambda_o$ carry the
eigenvalues on its main diagonal such that
\[
  AV_o = V_o\Lambda_o,
\]
where the distinction between the matrix~$\Lambda_o$ and the full
eigenvalue matrix~$\Lambda$ of~(\ref{eigen:diag}) is that the
latter always includes a $p$-fold eigenvalue~$p$ times, whereas the
former includes a $p$-fold eigenvalue only as many times as the
eigenvalue enjoys independent eigenvectors.  Let the $m-s$ columns of
the $m \times (m-s)$ matrix~$V_o^\perp$ represent the complete
orthogonal complement (\S~\ref{mtxinv:450}) to~$V_o$---perpendicular to
all eigenvectors, each column of unit magnitude---such that
\[
  V_o^{\perp{*}}V_o = 0 \ \ \mbox{and}\ \ V_o^{\perp{*}}V_o^\perp = I_{m-s}.
\]
Recall from \S~\ref{eigen:422} that $s \neq 0$ but $0 < s \le m$ because
every square matrix has at least one eigensolution.  Recall from
\S~\ref{eigen:423} that $s=m$ if and only if~$A$ is diagonalizable.%
\footnote{
  A concrete example: the invertible but nondiagonalizable matrix
  \[
    A = \mf{rrrr}{-1&0&0&0\\-6&5&\frac 5 2&-\frac 5 2\\0&0&5&0\\0&0&0&5}
  \]
  has a single eigenvalue at $\lambda=-1$ and a triple eigenvalue
  at $\lambda = 5$, the latter of whose eigenvector space is
  fully characterized by two eigenvectors rather than three such that
  \settowidth\tli{\footnotesize $-\frac 1{\sqrt 2}$}
  \settowidth\tlj{\footnotesize $\frac 1{\sqrt 2}$}
  \[
    V_o = \mf{ccc}{\frac 1{\sqrt 2}&0&0\\\frac 1{\sqrt 2}&1&0\\
    0&0&\frac 1{\sqrt 2}\\0&0&\frac 1{\sqrt 2}},
    \ \ %
    \Lambda_o = \mf{rrr}{-1&0&0\\0&5&0\\0&0&5},
    \ \ %
    V_o^\perp = \mf{c}{
      \makebox[\tli][r]{\makebox[\tlj][c]{$0$}}\\
      \makebox[\tli][r]{\makebox[\tlj][c]{$0$}}\\
      \makebox[\tli][r]{\makebox[\tlj][r]{$\frac{1}{\sqrt 2}$}}\\
      \makebox[\tli][r]{\makebox[\tlj][r]{$-\frac{1}{\sqrt 2}$}}
    }.
  \]
  The orthogonal complement~$V_o^\perp$ supplies the missing vector, not
  an eigenvector but perpendicular to them all.

  In the example, $m=4$ and $s=3$.

  All vectors in the example are reported with unit magnitude.  The two
  $\lambda = 5$ eigenvectors are reported in mutually orthogonal form,
  but notice that eigenvectors corresponding to distinct eigenvalues
  need not be orthogonal when~$A$ is not Hermitian.
}

With these definitions in hand, we can now prove by contradiction that
all Hermitian matrices are diagonalizable, falsely supposing a
nondiagonalizable Hermitian matrix~$A$, whose~$V_o^\perp$ (since~$A$ is
supposed to be nondiagonalizable, implying that $s<m$) would have at
least one column.  For such a matrix~$A$, $s \times (m-s)$ and $(m-s)
\times (m-s)$ auxiliary matrices~$F$ and~$G$ necessarily would exist
such that
\[
  AV_o^\perp = V_o F + V_o^\perp G,
\]
not due to any unusual property of the product~$AV_o^\perp$ but for the
mundane reason that the columns of~$V_o$ and~$V_o^\perp$ together by
definition addressed the space of \emph{all} $m$-element
vectors---including the columns of~$AV_o^\perp$.  Left-multiplying
by~$V_o^{*}$, we would have by successive steps that
\bqb
  V_o^{*}AV_o^\perp &=& V_o^{*}V_o F + V_o^{*}V_o^\perp G, \\
  (AV_o)^{*}V_o^\perp &=& I_s F + V_o^{*}V_o^\perp G, \\
  (V_o\Lambda_o)^{*}V_o^\perp &=& F + V_o^{*}V_o^\perp G, \\
  \Lambda_o^{*}V_o^{*}V_o^\perp &=& F + V_o^{*}V_o^\perp G, \\
  \Lambda_o^{*}(0) &=& F + (0)G, \\
  0 &=& F,
\eqb
where we had relied on the assumption that~$A$ were Hermitian and thus
that, as proved above, its distinctly eigenvalued eigenvectors
lay orthogonal to one
another; in consequence of which $A^{*}=A$ and $V_o^{*}V_o = I_s$.

The finding that $F=0$ reduces the~$AV_o^\perp$ equation above to read
\[
  AV_o^\perp = V_o^\perp G.
\]
In the reduced equation the matrix~$G$ would have at least one
eigensolution, not due to any unusual property of~$G$ but because
according to \S~\ref{eigen:422} every square matrix, $1 \times 1$ or
larger, has at least one eigensolution.  Let $(\mu,\ve w)$ represent an
eigensolution of~$G$.  Right-multiplying by the $(m-s)$-element vector
$\ve w \neq 0$, we would have by successive steps that
\bqb
  AV_o^\perp\ve w &=& V_o^\perp G\ve w, \\
  A(V_o^\perp\ve w) &=& \mu (V_o^\perp\ve w).
\eqb
The last equation claims that $(\mu, V_o^\perp\ve w)$ were an
eigensolution of~$A$, when we had supposed that all of $A$'s
eigenvectors lay in the space addressed by the columns of~$V_o$, and thus by
construction did not lie in the space addressed by the columns of~$V_o^\perp$.  The
contradiction proves false the assumption that gave rise to it.  The
assumption: that a nondiagonalizable Hermitian~$A$ existed.  We conclude
that all Hermitian matrices are diagonalizable---and conclude
further that they are \emph{unitarily} diagonalizable
on the ground that their eigenvectors lie orthogonal to one another---as
was to be demonstrated.

Having proven that all Hermitian matrices are diagonalizable and have
real eigenvalues and orthogonal eigenvectors, one wonders whether the
converse holds: are all diagonalizable matrices with real eigenvalues and
orthogonal eigenvectors Hermitian?  To show that they are, one can
construct the matrix described by the diagonalization
formula~(\ref{eigen:diag}),
\[
  A = V \Lambda V^{*},
\]
where $V^{-1} = V^{*}$ because this~$V$ is unitary
(\S~\ref{mtxinv:465}).  The equation's adjoint is
\[
  A^{*} = V \Lambda^{*} V^{*}.
\]
But all the eigenvalues here are real, which means that $\Lambda^{*} =
\Lambda$ and the right sides of the two equations are the same.
That is, $A^{*} = A$ as was to be demonstrated.  \emph{All
diagonalizable matrices with real eigenvalues and orthogonal
eigenvectors are Hermitian.}

This section brings properties that simplify many kinds of
matrix analysis.  The properties demand a Hermitian matrix, which might
seem a severe and unfortunate restriction---except that one can
left-multiply any exactly determined linear system $C\ve x = \ve d$
by~$C^{*}$ to get the equivalent Hermitian system
\bq{eigen:550:50}
  [C^{*}C]\ve x = [C^{*}\ve d],
\eq
in which $A=C^{*}C$ and $\ve b=C^{*}\ve d$, for which the properties
obtain.%
\footnote{
  The device~(\ref{eigen:550:50}) worsens a matrix's condition and may
  be undesirable for this reason, but it works in theory at least.
}

% ----------------------------------------------------------------------

\section{The singular-value decomposition}
\label{eigen:600}
\index{singular-value decomposition}
\index{decomposition!singular-value}

% The following used to be included in the text.
% The formulation is here retained, commented out.
%[This section is a new draft.  The author requests error reports to
%$\langle\texttt{thb@}$
%% bad break
%$\texttt{derivations.org}\rangle$ if the date on
%the book's title page is less than a year old.]

Occasionally an elegant idea awaits discovery, overlooked, almost in
plain sight.  If the unlikely thought occurred to you to take the square
root of a matrix, then the following idea is one you might discover.%
\footnote{\cite[``Singular value decomposition,'' 14:29, 18~Oct.\ 2007]{wikip}}

\index{singular-value matrix}
\index{matrix!singular-value}
\index{singular value}
Consider the $n \times n$ product~$A^{*}A$ of a tall or square,
$m \times n$ matrix~$A$ of full column rank
\[
  r = n \le m
\]
and its adjoint~$A^{*}$.  The product~$A^{*}A$ is invertible according
to \S~\ref{mtxinv:320.30}; is positive definite according to
\S~\ref{mtxinv:320.34}; and, since $(A^{*}A)^{*} = A^{*}A$, is clearly
Hermitian according to \S~\ref{eigen:550}; thus is
unitarily diagonalizable according to~(\ref{eigen:550:30}) as
\bq{eigen:600:10}
  A^{*}A = V\Lambda V^{*}.
\eq
Here, the $n \times n$ matrices~$\Lambda$ and~$V$ represent respectively
the eigenvalues and eigenvectors not of~$A$ but of the product~$A^{*}A$.
Though nothing requires the product's eigenvectors to be real, because
the product is positive definite \S~\ref{eigen:422} does
require all of its eigenvalues to be real and moreover positive---which
means among other things that the eigenvalue matrix~$\Lambda$ has full
rank.  That the eigenvalues, the diagonal elements of~$\Lambda$, are
real and positive is a useful fact; for just as a real, positive scalar
has a real, positive square root, so equally has~$\Lambda$ a real,
positive square root under these conditions.  Let the symbol $\Sigma =
\sqrt{\Lambda}$ represent the $n \times n$ real, positive square root
of the eigenvalue matrix~$\Lambda$ such that
\bqa
  \Lambda &=& \Sigma^{*}\Sigma,
  \label{eigen:600:15}\\
  \Sigma^{*} = \Sigma &=& \mf{ccccc}{
    +\sqrt{\lambda_1} & 0 & \cdots & 0 & 0 \\
    0 & +\sqrt{\lambda_2} & \cdots & 0 & 0 \\
    \vdots & \vdots & \ddots & \vdots & \vdots \\
    0 & 0 & \cdots & +\sqrt{\lambda_{n-1}} & 0 \\
    0 & 0 & \cdots & 0 & +\sqrt{\lambda_n}
  }, \xn
\eqa
where the \emph{singular values} of~$A$ populate $\Sigma$'s diagonal.
Applying~(\ref{eigen:600:15}) to~(\ref{eigen:600:10}) then yields that
\bq{eigen:600:20}
  \begin{split}
    A^{*}A &= V\Sigma^{*}\Sigma V^{*}, \\
    V^{*}A^{*}AV &= \Sigma^{*}\Sigma.
  \end{split}
\eq

Now consider the $m \times m$ matrix~$U$ such that
\bq{eigen:600:SVD}
  \begin{split}
    AV\Sigma^{-1} &= UI_n, \\
    AV &= U\Sigma, \\
    A &= U\Sigma V^{*}.
  \end{split}
\eq
Substituting~(\ref{eigen:600:SVD})'s second line
into~(\ref{eigen:600:20})'s second line gives the equation
\[
  \Sigma^{*}U^{*}U\Sigma = \Sigma^{*}\Sigma;
\]
but $\Sigma\Sigma^{-1} = I_n$, so left- and right-multiplying
respectively by~$\Sigma^{-*}$ and~$\Sigma^{-1}$ leaves that
\[
  I_nU^{*}UI_n = I_n,
\]
which says neither more nor less than that the first~$n$ columns of~$U$
are orthonormal.  Equation~(\ref{eigen:600:SVD}) does not constrain the
last $m-n$ columns of~$U$, leaving us free to make them anything we
want.  Why not use Gram-Schmidt to make them orthonormal, too, thus
making~$U$ a unitary matrix?  If we do this, then the surprisingly
simple~(\ref{eigen:600:SVD}) constitutes the \emph{singular-value
decomposition} of~$A$.

If~$A$ happens to have broad shape then we can decompose~$A^{*}$,
instead, so this case poses no special trouble.
Apparently every full-rank matrix
has a singular-value decomposition.

But what of the matrix of less than full rank $r < n$?  In this case the
product~$A^{*}A$ is singular and has only $s < n$ nonzero eigenvalues
(it may be that $s = r$, but this is irrelevant to the proof at hand).
However, if the~$s$ nonzero eigenvalues are arranged first in~$\Lambda$,
then~(\ref{eigen:600:SVD}) becomes
\bq{eigen:600:SVDr}
  \begin{split}
    AV\Sigma^{-1} &= UI_s, \\
    AV &= U\Sigma, \\
    A &= U\Sigma V^{*}.
  \end{split}
\eq
The product~$A^{*}A$ is nonnegative definite in this case and
$\Sigma\Sigma^{-1} = I_s$, but the reasoning is otherwise the same as
before.  Apparently every matrix of less than full rank has a
singular-value decomposition, too.

If~$A$ happens to be an invertible square matrix, then the
singular-value decomposition evidently inverts it as
\bq{eigen:600:SVDinv}
  A^{-1} = V\Sigma^{-1} U^{*}.
\eq

% ----------------------------------------------------------------------

\section{General remarks on the matrix}
\label{eigen:900}

Chapters~\ref{matrix} through~\ref{eigen} have derived the uncomfortably
bulky but---%
% bad break
in\-cred\-i\-bly---%
approximately minimal knot of theory one needs to grasp the matrix
properly and to use it with moderate versatility.  As far as the writer
knows, no one has yet discovered a satisfactory way to untangle
the knot.  The choice to learn the basic theory of the matrix is almost
an all-or-nothing choice; and how many scientists and engineers would
rightly choose the ``nothing'' if the matrix did not serve so very many
applications as it does?  Since it does serve so very many, the ``all''
it must be.%
\footnote{
  Of course, one might avoid true understanding and instead work by memorized rules.
  That is not always a bad plan, really; but if that were \emph{your}
  plan then it seems spectacularly unlikely that you would be reading a
  footnote buried beneath the further regions of the hinterland of
  Chapter~\ref{eigen} in such a book as this.
}
Applied mathematics brings nothing else quite like it.

% diagn: References in this paragraph may need updating and adding once
% the book is more fully written.
\index{projector}
\index{matrix!projection}
These several matrix chapters have not covered every topic they might.
The topics they omit fall roughly into two classes.  One is the class of
more advanced and more specialized matrix theory, about which we will
have more to say in a moment.  The other is the class of basic matrix
theory these chapters do not happen to use.  The essential agents of
matrix analysis---multiplicative associativity, rank, inversion,
pseudoinversion, the kernel, the orthogonal complement,
orthonormalization, the eigenvalue, diagonalization and so on---are the
same in practically all books on the subject, but the way the agents are
developed differs.  This book has chosen a way that needs some tools
like truncators other books omit, but does not need other tools like
projectors other books%
\footnote{
  Such as \cite[\S~3.VI.3]{Hefferon}, a lengthy but well-knit tutorial
  this writer recommends.
}
include.  What has given these chapters their hefty bulk is not so much
the immediate development of the essential agents as the preparatory
development of theoretical tools used to construct the essential agents,
yet most of the tools are of limited interest in themselves; it is the
agents that matter.  Tools like the projector not used here tend to be
omitted here or deferred to later chapters, not because they are
altogether useless but because they are not used \emph{here} and because
the present chapters are already too long.  The reader who understands the
Moore-Penrose pseudoinverse and/or the Gram-Schmidt process reasonably
well can after all pretty easily figure out how to construct a projector
without explicit instructions thereto, should the need arise.%
\footnote{
  Well, since we have brought it up (though only as an example of tools
  these chapters have avoided bringing up), briefly: a projector is a
  matrix that flattens an arbitrary vector~$\ve b$ into its nearest
  shadow~$\tilde{\ve b}$ within some restricted subspace.  If the
  columns of~$A$ represent the subspace, then~$\ve x$
  represents~$\tilde{\ve b}$ in the subspace basis iff $A\ve x =
  \tilde{\ve b}$, which is to say that $A\ve x \approx \ve b$, whereupon
  $\ve x = A^\dagger\ve b$.  That is, per~(\ref{mtxinv:psinv}),
  \[
    \tilde{\ve b} = A\ve x = AA^\dagger\ve b
    = [BC][C^{*}(CC^{*})^{-1}(B^{*}B)^{-1}B^{*}] \ve b
    = B(B^{*}B)^{-1}B^{*} \ve b,
  \]
  in which the matrix $B(B^{*}B)^{-1}B^{*}$ is the projector.  Thence it
  is readily shown that the deviation $\ve b - \tilde{\ve b}$ lies
  orthogonal to the shadow~$\tilde{\ve b}$.  More broadly defined, any
  matrix~$M$ for which $M^2=M$ is a projector.  One can approach the
  projector in other ways, but there are two ways at least.
}

\index{Cholesky, Andr\'e-Louis (1875--1918)}
Paradoxically and thankfully, more advanced and more specialized matrix
theory though often harder tends to come in smaller, more manageable
increments: the Cholesky decomposition, for instance; or the
conjugate-gradient algorithm.  The theory develops endlessly.  From the
present pause one could proceed directly to such topics.  However, since
this is the \emph{first} proper pause these several matrix chapters have
afforded, since the book is \emph{Derivations of Applied Mathematics}
rather than \emph{Derivations of Applied Matrices,} maybe we ought to
take advantage to change the subject.

