\chapter{Probability}
\label{prob}
\index{probability}

\index{statistics}
Of all mathematical fields of study, none may be so counterintuitive
and yet so widely applied as that of probability---whether
as \emph{probability} in the technical term's conventional, restricted
meaning or as probability in its expanded or inverted guise as
\emph{statistics.}\footnote{%
  The nomenclature is slightly unfortunate.  Were statistics called
  ``inferred probability'' or ``probabilistic estimation'' the name
  would suggest something like the right taxonomy.  Actually, the
  nomenclature is fine once you know what it means, but on first
  encounter it provokes otherwise unnecessary footnotes like this one.

  Statistics (singular noun) the expanded mathematical discipline---as
  opposed to the statistics (plural noun) mean and standard deviation of
  \S~\ref{prob:070}---as such lies mostly beyond this book's scope, but
  the chapter will have at least a little to say about it in
  \S~\ref{prob:200}.%
}
The untrained mind seems to rebel against the concept.
Nevertheless, sustained reflection upon the concept gradually reveals
a fascinating mathematical landscape.

\index{thumb}
\index{uncertainty}
As calculus is the mathematics of change, so \emph{probability} is the
mathematics of uncertainty.  If I tell you that my thumb is three inches
long, I likely do not mean that it is exactly three inches.  I mean that
it is about three inches.  Quantitatively, I might report the length
as~$3.0 \pm 0.1$ inches, thus indicating not only the length but the
degree of uncertainty in the length.  Probability in its guise as
\emph{statistics} is the mathematics which produces, analyzes and
interprets such quantities.

\index{U.S.\ male}
\index{height}
More obviously statistical is a report that, say, the
average~25-year-old U.S.\ male is~$69 \pm 3$ inches tall, inferred from
actual measurements\footnote{\cite[Tables~6 and~12]{CDC}}
of some number $N>1$ of~25-year-old U.S.\ males.
Deep mathematics underlie such a report, for the report implies among
other things that a little over
two-thirds---$(1/\sqrt{2\pi})\int_{-1}^{1}\exp(-\tau^2/2)\,d\tau \approx
\mbox{0x0.AEC5}$, to be precise---of a typical, randomly chosen sample
of~25-year-old U.S.\ males ought to be found to have heights
between~66 and~72 inches.

\index{game of chance}
\index{card}
\index{deck of cards}
Probability is also met in games of chance and in models of systems
which---from the model's point of view---logically resemble games of
chance, and in this setting probability is not statistics.  The reason
it is not is that its mathematics in this case is based not on
observation but on a teleological assumption of some kind, often an
assumption of symmetry such as that no face of a die or card from a deck
ought to turn up more often than another.  Entertaining so plausible an
assumption, if you should draw three cards at random from a
standard 52-card\footnote{Decimal notation is used here.} deck,
the deck comprising four cards each of thirteen
ranks, then there would be some finite probability---which is
$(3/51)(2/50) = 1/425$---that the three cards drawn would share the
same rank (why?).  If I should however shuffle the deck, draw three
cards off the top, and look at the three cards without showing them to
you, all before inviting you to draw three, then the probability that
your three would share the same rank were again $1/425$ (why?).  On the
other hand, if before you drew I let you \emph{peek} at my three hidden cards,
and you saw that my three hidden cards
were ace, queen and ten, then this knowledge alone must
slightly lower your estimate of the probability that your three would
subsequently share the same rank to $(40/49)(3/48)(2/47)
+ (9/49)(2/48)(1/47) \approx 1/428$ (again, why?).

\index{trial}
\index{expectation}
That the probability should be $1/425$ suggests that one would draw
three of the same rank once in~425 tries.  That is, were I to
shuffle~425 decks and you to draw three cards from each, then
for you to draw three of the same rank from just one of the~425
decks would be expected.
Nevertheless, despite expectations, you \emph{might} draw three of the
same rank from two, three or four decks, or from none at all---so what
does a probability of $1/425$ really mean?  The answer is that it
means something like this: were I to shuffle~425 \emph{million} decks
then you would draw three of the same rank from very nearly~1.0 million
decks---almost certainly not from as many as~1.1 million nor as few
as~0.9 million.  It means that the ratio of the number of
three-of-the-same-rank events to the number of trials must converge
exactly upon $1/425$ as the number of trials tends toward infinity.

See also \S~\ref{drvtv:220}.

% It should not really be necessary to tie \emph{a~priori,} but in a
% text like this, if it breaks after the "a," the italic "a" looks too
% much like a variable $a$.
\index{climate}
\index{die}
\index{probability!empirical}
\index{probability!\emph{a priori}}
\index{empirical probability}
\index{\emph{a priori} probability}
If unsure, consider this.  Suppose that during six days in an unfamiliar
climate, in a place you had never before visited, it rained twice.  Then
suppose that during six throws of a six-sided die, a single pip came up
twice.  What would you conclude about the climate?  What would you
conclude about the die?  See, these are different cases.

Regarding the climate, the best one can do might be to suppose
\emph{empirically} that, on average, it rained two days out of every
six; whereas one should probably assume \emph{a~priori} that, on average,
a single pip were to come up one throw out of every six.  For the die,
one would regard the two-throw observation to represent but a random
fluctuation.

Cases of either kind can be quantitatively analyzed.  This chapter
mostly (though not solely) analyzes cases of the \emph{a~priori}
kind, cases like that of the die.

Other than by the brief introduction you are reading, this book is not
well placed to offer a gentle tutorial in probabilistic thought.%
\footnote{%
  R.~W.\ Hamming's \cite{Hamming} ably fills such a role.%
}
What it does offer, in the form of the present chapter, is the discovery
and derivation of some of the essential mathematical functions of
probability theory, plus a brief investigation of these
functions' principal properties and typical use.

% ----------------------------------------------------------------------

\section{Definitions and basic concepts}
\label{prob:050}
\index{probability!definitions pertaining to}

\index{probability}
\index{probability density function}
\index{density function, probability}
\index{PDF (probability density function)}
\index{distribution}
\index{cumulative distribution function}
\index{distribution function, cumulative}
\index{CDF (cumulative distribution function)}
\index{quantile}
\index{random variable}
\index{variable!random}
A \emph{probability} is the chance that a trial of some kind will result
in some specified event of interest.  Conceptually,
\[
  P_{\mr{event}} \equiv \lim_{N\ra\infty} \frac{N_{\mr{event}}}{N},
\]
where~$N$ and~$N_{\mr{event}}$ are the numbers respectively of trials
and events.  A \emph{probability density function} (PDF) or
\emph{distribution} is a function $f(x)$ defined such that
\bq{prob:050:10}
  \begin{split}
    P_{ba} &= \int_{a}^{b} f(x) \,dx, \\
    1 &= \int_{-\infty}^{\infty} f(x) \,dx, \\
    0 &\le f(x), \ \ \Im[f(x)] = 0.
  \end{split}
\eq
where the event of interest is that the \emph{random variable}~$x$
fall%
\footnote{
  This sentence and the rest of the section condense somewhat lengthy
  tracts of an introductory collegiate statistics textbook
  like~\cite{Walpole/Myers}\cite{Alder/Roessler}\cite{Lindgren}\cite{Rosenkrantz},
  among others.  If the sentence and section make little sense to you
  then so likely will the rest of the chapter, but any statistics text
  you might find conveniently at hand should fill the gap---which is less
  a mathematical gap than a conceptual one.  Or, if defiant, you can
  stay here and work through the concepts on your own.
}
within the interval%
\footnote{
  We might as well have expressed the interval $a < x < b$ as $a \le x
  \le b$ or even as $a \le x < b$, except that such notational niceties
  would distract from the point the notation means to convey.  The
  notation in this case is not really interested in the bounding points
  themselves.  If \emph{we} are interested in the bounding points, as for
  example we would be if $f(x) = \delta(x)$ and $a=0$, then we can
  always write in the style of $P_{(0^{-})b}$, $P_{(0^{+})b}$,
  $P_{(a-\ep)(b+\ep)}$, $P_{(a+\ep)(b-\ep)}$ or the like.  We can even be
  most explicit and write in the style of $P\{a \le x \le b\}$, often
  met in the literature.
}
$a < x < b$ and~$P_{ba}$ is the probability of this event.  The
corresponding \emph{cumulative distribution function} (CDF) is
\bq{prob:CDF}
  F(x) \equiv \int_{-\infty}^{x} f(\tau) \,d\tau,
\eq
where
\bq{prob:050:020}
  \begin{split}
    0 &= F(-\infty), \\
    1 &= F(\infty), \\
    P_{ba} &= F(b) - F(a).
  \end{split}
\eq
The \emph{quantile} $F^{-1}(\cdot)$ inverts the CDF $F(x)$ such that
\bq{prob:quantile}
  F^{-1}[F(x)] = x,
\eq
generally calculable by a Newton-Raphson iteration~(\ref{drvtv:NR}) if
by no other means.

\index{probability!that both of two independent events will occur}
\index{probability density function!of a sum of random variables}
\index{convolution}
It is easy enough to see that the product
\bq{prob:050:30}
  P = P_1P_2
\eq
of two probabilities composes the single probability that not just one
but both of two independent events will occur.  Harder to see, but just
as important, is that the convolution
\bq{prob:050:40}
  f(x) = f_1(x) \ast f_2(x)
\eq
of two probability density functions composes the single probability
density function of the sum of two random variables
\bq{prob:050:41}
    x=x_1+x_2,
\eq
where, per Table~\ref{fouri:110:tbl41},
\[
  f_2(x) \ast f_1(x) = f_1(x) \ast f_2(x) \equiv
  \int_{-\infty}^{\infty} f_1\left(\frac{x}{2}-\tau\right)
      f_2\left(\frac{x}{2}+\tau\right) \,d\tau.
\]
That is, if you think about it in a certain way, the probability that $a
< x_1 + x_2 < b$ cannot but be
\bqb
  P_{ba}
  &=&
    \lim_{\ep \ra 0^{+}} \sum_{k=-\infty}^{\infty}
    \bigg\{
      \bigg[
        \int_{a-k\ep}^{b-k\ep} f_1(x) \,dx
      \bigg]
      \bigg[
        \int_{(k-1/2)\ep}^{(k+1/2)\ep} f_2(x) \,dx
      \bigg]
    \bigg\}
  \\&=&
    \lim_{\ep \ra 0^{+}} \sum_{k=-\infty}^{\infty}
    \bigg\{
      \bigg[
        \int_{a}^{b} f_1(x-k\ep) \,dx
      \bigg]
      \bigg[
        \ep f_2(k\ep)
      \bigg]
    \bigg\}
  \\&=&
    \int_{-\infty}^{\infty} \bigg[
        \int_{a}^{b} f_1(x-\tau) \,dx
    \bigg] f_2(\tau) \,d\tau
  \\&=&
    \int_{a}^{b} \left[ \int_{-\infty}^{\infty} f_1(x-\tau) f_2(\tau) \,d\tau \right] \,dx
  \\&=&
    \int_{a}^{b} \left[ \int_{-\infty}^{\infty}
      f_1\left(\frac x2 - \tau\right)
      f_2\left(\frac x2 + \tau\right) \,d\tau \right] \,dx,
\eqb
which in consideration of~(\ref{prob:050:10})
implies~(\ref{prob:050:40}).

% ----------------------------------------------------------------------

\section{The statistics of a distribution}
\label{prob:070}
\index{statistic}

\index{mean}
\index{standard deviation}
\index{$\mu$}
\index{$\sigma$}
\index{expected value}
\index{$\langle\cdot\rangle$}
A probability density function $f(x)$ describes a distribution
whose \emph{mean}~$\mu$ and \emph{standard deviation}~$\sigma$ are
defined such that
\bq{prob:stat}
  \begin{split}
    \mu \equiv \langle x \rangle &= \int_{-\infty}^{\infty} f(x) x \,dx, \\
    \sigma^2 \equiv \left\langle ( x - \langle x \rangle )^2 \right\rangle &=
      \int_{-\infty}^{\infty} f(x) (x-\mu)^2 \,dx;
  \end{split}
\eq
where~$\langle\cdot\rangle$ indicates the \emph{expected value} of the
quantity enclosed, defined as the first line of~(\ref{prob:stat})
suggests.  The mean~$\mu$ is just the distribution's average, about
which a random variable should center.  The standard deviation~$\sigma$
measures a random variable's typical excursion from the mean. The mean
and standard deviation are \emph{statistics} of the distribution.%
\footnote{
  Other statistics than the mean and standard deviation are possible,
  but these two are the most important ones and are the two this book
  will treat.
}
When for example the chapter's introduction proposed that the average~25-year-old
U.S.\ male were~$69 \pm 3$ inches tall, it was saying that his height
could quantitatively be modeled as a random variable drawn from a
distribution whose statistics are $\mu = 69$ inches and $\sigma =
3$ inches.

The first line of~(\ref{prob:stat}), defining~$\mu$, might seem obvious
enough, but one might ask why $\sigma$ had not instead been defined to
be $\langle | x - \langle x \rangle | \rangle$.  Would that not have
been more obvious?  One answer is\footnote{The writer does not know the
original, historical answer.} that, yes, it might have been more obvious
but it would not have been analytic (\S\S~\ref{alggeo:225.3}
and~\ref{taylor:320}).  Another answer is that one likes to regard long
excursions from the mean more seriously than short ones.  A third answer
is that the second line of~(\ref{prob:stat}) comports with the elegant
mathematics of least squares and Moore-Penrose (\S~\ref{mtxinv:320}).
Whatever the answer,~(\ref{prob:stat}) is the definition conventionally
used.

% ----------------------------------------------------------------------

\section{The sum of random variables}
\label{prob:074}
\index{random variable!sum of}

The statistics of the sum of two random variables $x=x_1+x_2$ are of
interest.  For the mean, substituting~(\ref{prob:050:40}) into the first
line of~(\ref{prob:stat}),
\bqb
  \mu
  &=&
    \int_{-\infty}^{\infty} [f_1(x) \ast f_2(x)] x \,dx
  \\&=&
    \int_{-\infty}^{\infty} \int_{-\infty}^{\infty}
      f_1\left(\frac x2-\tau\right) f_2\left(\frac x2+\tau\right)
      \,d\tau\,x\,dx
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \int_{-\infty}^{\infty}
      f_1(x-\tau) x \,dx\,d\tau
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \int_{-\infty}^{\infty}
      f_1(x)(x+\tau) \,dx\,d\tau
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \left[
      \int_{-\infty}^{\infty}
      f_1(x)x \,dx
      +\tau\int_{-\infty}^{\infty}
      f_1(x) \,dx
    \right]\,d\tau
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) [\mu_1 + \tau] \,d\tau
  \\&=&
    \mu_1\int_{-\infty}^{\infty} f_2(\tau) \,d\tau
    +\int_{-\infty}^{\infty} f_2(\tau) \tau \,d\tau.
\eqb
That is,
\bq{prob:070:20}
  \mu = \mu_1 + \mu_2,
\eq
which is no surprise, but at least it is nice to know that our
mathematics is working as it should.  The standard deviation of the sum
of two random variables is such that, substituting~(\ref{prob:050:40})
into the second line of~(\ref{prob:stat}),
\bqb
  \sigma^2
  &=&
    \int_{-\infty}^{\infty} [f_1(x) \ast f_2(x)] (x-\mu)^2 \,dx
  \\&=&
    \int_{-\infty}^{\infty} \int_{-\infty}^{\infty}
      f_1\left(\frac x2-\tau\right) f_2\left(\frac x2+\tau\right)
      \,d\tau\,(x-\mu)^2\,dx.
\eqb
Applying~(\ref{prob:070:20}),
\bqb
  \sigma^2
  &=&
    \int_{-\infty}^{\infty} \int_{-\infty}^{\infty}
      f_1\left(\frac x2-\tau\right) f_2\left(\frac x2+\tau\right)
      \,d\tau\,(x-\mu_1-\mu_2)^2\,dx
  \\&=&
    \int_{-\infty}^{\infty} \int_{-\infty}^{\infty}
      f_1\left(\frac{x+\mu_1}2-\tau\right) f_2\left(\frac{x+\mu_1}2+\tau\right)
      \,d\tau\,(x-\mu_2)^2\,dx
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \int_{-\infty}^{\infty}
      f_1(x+\mu_1-\tau) (x-\mu_2)^2 \,dx\,d\tau
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \int_{-\infty}^{\infty}
      f_1(x) [(x-\mu_1)+(\tau-\mu_2)]^2 \,dx\,d\tau
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \bigg\{
      \int_{-\infty}^{\infty} f_1(x) (x-\mu_1)^2 \,dx
  \\&&\ \ \ \ \ \ \ \ \mbox{}
      +2(\tau-\mu_2)\int_{-\infty}^{\infty} f_1(x) (x-\mu_1) \,dx
  \\&&\ \ \ \ \ \ \ \ \mbox{}
      +(\tau-\mu_2)^2\int_{-\infty}^{\infty} f_1(x) \,dx
    \bigg\}\,d\tau
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \bigg\{
      \int_{-\infty}^{\infty} f_1(x) (x-\mu_1)^2 \,dx
  \\&&\ \ \ \ \ \ \ \ \mbox{}
      +2(\tau-\mu_2)\int_{-\infty}^{\infty} f_1(x) x \,dx
  \\&&\ \ \ \ \ \ \ \ \mbox{}
      +(\tau-\mu_2)(\tau-\mu_2-2\mu_1)\int_{-\infty}^{\infty} f_1(x) \,dx
    \bigg\}\,d\tau.
\eqb
Applying~(\ref{prob:stat}) and~(\ref{prob:050:10}),
\bqb
  \sigma^2
  &=&
    \int_{-\infty}^{\infty} f_2(\tau) \left\{
      \sigma_1^2 + 2(\tau-\mu_2)\mu_1
      +(\tau-\mu_2)(\tau-\mu_2-2\mu_1)
    \right\}\,d\tau
  \\&=&
    \int_{-\infty}^{\infty} f_2(\tau) \left\{
      \sigma_1^2 + (\tau-\mu_2)^2
    \right\}\,d\tau
  \\&=&
    \sigma_1^2 \int_{-\infty}^{\infty} f_2(\tau) \,d\tau
    +\int_{-\infty}^{\infty} f_2(\tau) (\tau-\mu_2)^2 \,d\tau.
\eqb
Applying~(\ref{prob:stat}) and~(\ref{prob:050:10}) again,
\bq{prob:070:30}
  \sigma^2 = \sigma_1^2 + \sigma_2^2.
\eq
If this is right---as indeed it is---then the act of adding random
variables together not only adds the means of the variables' respective
distributions according to~(\ref{prob:070:20}) but also, according
to~(\ref{prob:070:30}), adds the squares of the standard deviations.  It
follows inductively that, if~$N$ independent instances $x_1, x_2, \ldots,
x_N$ of a random variable are drawn from the same distribution
$f_o(x_k)$, the distribution's statistics being~$\mu_o$ and~$\sigma_o$,
then the statistics of their sum $x = \sum_{k=1}^N x_k = x_1 + x_2 +
\cdots + x_N$ are
\bq{prob:070:40}
  \begin{split}
    \mu &= N\mu_o, \\
    \sigma &= \left(\sqrt N\right)\sigma_o.
  \end{split}
\eq

% ----------------------------------------------------------------------

\section{The transformation of a random variable}
\label{prob:080}
\index{random variable!transformation of}
\index{transformation!of a random variable}

\index{distribution!conversion between two}
\index{inverse!of a function}
\index{function!inverse of}
\index{suaveness}
If~$x_o$ is a random variable obeying the distribution $f_o(x_o)$ and
$g(\cdot)$ is some invertible function whose inverse
per~(\ref{alggeo:250:30}) is styled $g^{-1}(\cdot)$, then
\[
  x \equiv g(x_o)
\]
is itself a random variable obeying the distribution
\bq{prob:080:10}
  f(x) = \left. \frac{f_o(x_o)}{\left|dg/dx_o\right|} \right|_{x_o = g^{-1}(x)}.
\eq
Another, suaver way to write the same thing is as that
\bq{prob:080:11}
  f(x) \left|dx\right| = f_o(x_o) \left|dx_o\right|.
\eq
Either way, this is almost obvious if seen from just the right
perspective, but can in any case be supported symbolically by
\[
  \int_{a}^{b} f_o(x_o) \,dx_o
  = \left| \int_{g(a)}^{g(b)} f_o(x_o) \frac{dx_o}{dx} \,dx \right|
  = \int_{g(a)}^{g(b)} f_o(x_o) \left|\frac{dx_o}{dg}\right| \,dx
\]
since, on the other hand
\[
  \int_{a}^{b} f_o(x_o) \,dx_o
  = \int_{g(a)}^{g(b)} f(x) \,dx.
\]

\index{random variable!scaling of}
\index{scaling}
\index{probability density function!flattening of}
\index{flattening}
One of the most frequently useful transformations is the simple
\bq{prob:080:48}
  x \equiv g(x_o) \equiv \alpha x_o,
  \ \ \mbox{$\Im(\alpha) = 0$.}
\eq
For this, evidently $dg/dx_o = \alpha$ or $dx = \alpha dx_o$, so according
to~(\ref{prob:080:10}) or~(\ref{prob:080:11})
\bq{prob:080:49}
  f(x) = \frac{1}{\left|\alpha\right|} f_o\left(\frac{x}{\alpha}\right).
\eq
If $\mu_o=0$ and $\sigma_o=1$, then $\mu=0$ and,
applying~(\ref{prob:080:11}) in train of~(\ref{prob:stat}),
\[
  \sigma^2
    = \int_{-\infty}^{\infty} f(x) x^2 \,dx
    = \int_{-\infty}^{\infty} f_o(x_o) (\alpha x_o)^2 \,dx_o
    = \alpha^2;
\]
whereby $\sigma=|\alpha|$ and, if $\alpha > 0$, one can rewrite the
transformed PDF as
\bq{prob:080:50}
    f(x) = \frac{1}{\sigma} f_o\left(\frac{x}{\sigma}\right)\ %
    \mbox{and $\mu = 0$, if $\mu_o = 0$ and $\sigma_o = 1$.}
\eq
Assuming null mean,~(\ref{prob:080:50}) states that the act of scaling a
random variable flattens out the variable's distribution and scales its
standard deviation, all by the same factor---which, naturally, is what
one would expect such an act to do.

% ----------------------------------------------------------------------

\section{The normal distribution}
\label{prob:100}
\index{normal distribution}
\index{distribution!normal}

\index{$\Omega$ as the Gaussian pulse}
Combining the ideas of \S\S~\ref{prob:074} and~\ref{prob:080} can lead one
to ask whether a zero-mean distribution does not exist for which, when
independent random variables drawn from it are added together, \emph{the
sum obeys the same distribution,} only the standard deviations
differing.  More precisely, the ideas can lead one to seek a distribution
\[
  \mbox{$f_o(x_o)$: $\mu_o = 0$, $\sigma_o = 1$;}
\]
for which, if~$x_1$ and~$x_2$ are random variables drawn respectively
from the distributions
\[
  \begin{split}
    f_1(x_1) &= \frac{1}{\sigma_1} f_o\left(\frac{x_1}{\sigma_1}\right), \\
    f_2(x_2) &= \frac{1}{\sigma_2} f_o\left(\frac{x_2}{\sigma_2}\right),
  \end{split}
\]
as~(\ref{prob:080:50}) suggests, then
\[
  x = x_1 + x_2
\]
is by construction a random variable drawn from the distribution
\[
  f(x) = \frac{1}{\sigma} f_o\left(\frac{x}{\sigma}\right),
\]
where per~(\ref{prob:070:30}),
\[
  \sigma^2 = \sigma_1^2 + \sigma_2^2.
\]
There are several distributions one might try, but eventually the
Gaussian pulse $\Omega(x_o)$ of \S\S~\ref{fours:095} and~\ref{fouri:130},
\bq{prob:normdist}
  \Omega(x) = \frac{\exp\left(-x^2/2\right)}{\sqrt{2\pi}},
\eq
recommends itself.  This works.  The distribution $f_o(x_o) =
\Omega(x_o)$ meets our criterion.

\subsection{Proof}
\label{prob:100.10}
\index{normal distribution!proof of}
\index{distribution!normal, proof of}

To prove that the distribution $f_o(x_o) = \Omega(x_o)$ meets our
criterion we shall have first to show that it is indeed a distribution
according to~(\ref{prob:050:10}).  Especially, we shall have to
demonstrate that
\[
  \int_{-\infty}^{\infty} \Omega(x_o) \,dx_o = 1.
\]
Fortunately as it happens we have already demonstrated this fact
in an earlier chapter, while working on Fourier transforms, as
(\ref{fouri:130:60}).  The function $\Omega(x_o)$ had interested us
during the earlier chapter because it is an analytic function that
autotransforms, so now in this chapter we observe that, since $\Omega(x_o)$
evidently meets the other demands
of~(\ref{prob:050:10}), $\Omega(x_o)$ is apparently indeed also a proper
distribution, whatever its other properties might be.
That $\mu_o = 0$ for $\Omega(x_o)$ is obvious by symmetry.  That
$\sigma_o = 1$ is shown by
\bqb
  \lefteqn{
    \sigma^2 \equiv
    \int_{-\infty}^{\infty} \Omega(x_o) x_o^2 \,dx_o
  } \qquad\quad &&
  \\ &=&
    \frac{1}{\sqrt{2\pi}}
    \int_{-\infty}^{\infty} \exp\left(-\frac{x_o^2}{2}\right) x_o^2 \,dx_o
  \\&=&
    \frac{1}{\sqrt{2\pi}}
    \int_{-\infty}^{\infty}
    \left\{\bigg[\mbox{$\ds -x_o$}\bigg]\bigg[\mbox{$\ds -x_o\exp\left(-\frac{x_o^2}{2}\right)dx_o$}\bigg]\right\}
  \\&=&
    -\frac{1}{\sqrt{2\pi}}
    \int_{x_o=-\infty}^{\infty} x_o \,d\left[\exp\left(-\frac{x_o^2}{2}\right)\right]
  \\&=&
    \makebox[-3pt][c]{}\left.-\frac{x_o \exp\left(-x_o^2/2\right)}{\sqrt{2\pi}}\right|_{-\infty}^{\infty}
    +\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}
    \exp\left(-\frac{x_o^2}{2}\right) \,dx_o
  \\&=&
    0 + \int_{-\infty}^{\infty} \Omega(x_o) \,dx_o,
\eqb
the integration via the by-parts method of \S~\ref{inttx:230},
the result according to
% bad break
(\ref{fouri:130:60}) that
\bq{prob:100:10}
  \sigma^2 \equiv \int_{-\infty}^{\infty} \Omega(x_o) x_o^2 \,dx_o = 1,
\eq
implying that $\sigma=1$ as was to be shown.  Now having justified the
assertions that $\Omega(x_o)$ is a proper distribution and that its
statistics are $\mu_o
= 0$ and $\sigma_o = 1$, all that remains to be proved
per~(\ref{prob:050:40}) is that
\bq{prob:100:20}
  \begin{split}
    \left[\frac{1}{\sigma_1}\Omega\left(\frac{x_o}{\sigma_1}\right)\right]
    \ast \left[\frac{1}{\sigma_2}\Omega\left(\frac{x_o}{\sigma_2}\right)\right]
    &= \frac{1}{\sigma}\Omega\left(\frac{x_o}{\sigma}\right),
  \\
    \sigma_1^2 + \sigma_2^2 &= \sigma^2,
  \end{split}
\eq
which is to prove that the sum of Gaussian random variables is itself
Gaussian.  We will prove it in the Fourier domain of chapter~\ref{fouri} as
follows.  According to Tables~\ref{fouri:110:tbl20}, \ref{fouri:110:tbl41}
and~\ref{fouri:120:tbl25}, and to~(\ref{prob:normdist}),
\bqb
  \lefteqn{
    \left[\frac{1}{\sigma_1}\Omega\left(\frac{x_o}{\sigma_1}\right)\right]
    \ast \left[\frac{1}{\sigma_2}\Omega\left(\frac{x_o}{\sigma_2}\right)\right]
  }
  &&\\&=&
    \mathcal{F}^{-1}\left\{
      \left(\sqrt{2\pi}\right)
      \mathcal F \left[\frac{1}{\sigma_1}\Omega\left(\frac{x_o}{\sigma_1}\right)\right]
      \mathcal F \left[\frac{1}{\sigma_2}\Omega\left(\frac{x_o}{\sigma_2}\right)\right]
    \right\}
  \\&=&
    \mathcal{F}^{-1}\left\{
      \left(\sqrt{2\pi}\right)
      \Omega(\sigma_1 x_o)
      \Omega(\sigma_2 x_o)
    \right\}
  \\&=&
    \mathcal{F}^{-1}\left\{
      \frac 1{\sqrt{2\pi}} \exp\left[-\frac{\sigma_1^2 x_o^2}2\right]
        \exp\left[-\frac{\sigma_2^2 x_o^2}2\right]
    \right\}
  \\&=&
    \mathcal{F}^{-1}\left\{
      \frac 1{\sqrt{2\pi}} \exp\left[-\frac{\left(\sigma_1^2+\sigma_2^2\right) x_o^2}2\right]
    \right\}
  \\&=&
    \mathcal{F}^{-1}\left\{
    \Omega\left[\left(\sqrt{\sigma_1^2+\sigma_2^2}\right)x_o\right]
    \right\}
  \\&=&
    \frac{1}{\sqrt{\sigma_1^2+\sigma_2^2}}\,
      \Omega\left(\frac{x_o}{\sqrt{\sigma_1^2+\sigma_2^2}}\right),
\eqb
the last line of which is~(\ref{prob:100:20}) in other notation, thus
completing the proof.

\subsection{Plots and remarks}
\label{prob:100.40}

\index{Gauss, Carl Friedrich (1777--1855)}
\index{Gaussian pulse}
\index{pulse!Gaussian}
\index{normal distribution}
\index{distribution!normal}
\index{Gaussian distribution}
\index{distribution!Gaussian}
\index{bell curve}
In the Fourier context of chapter~\ref{fouri} one usually names
$\Omega(\cdot)$ the \emph{Gaussian pulse,} as we have seen.  The function
$\Omega(\cdot)$ turns out to be even more prominent in probability theory
than in Fourier theory, however, and in a probabilistic context it
usually goes by the name of the \emph{normal distribution.}  This is
what we will call $\Omega(\cdot)$ through the rest of the present
chapter.  Alternate conventional names include those of the
\emph{Gaussian distribution} and the \emph{bell curve} (the Greek
capital~$\Omega$ vaguely, accidentally resembles a bell, as does the
distribution's plot, and we will not be too proud to take advantage of
the accident, so that is how you can remember it if you like).  By
whichever name, Fig.~\ref{prob:normdist-fig} plots the normal
distribution $\Omega(\cdot)$ and its cumulative distribution
function~(\ref{prob:CDF}).
\begin{figure}
  \caption[The normal distribution and its CDF.]{%
    The normal distribution $\Omega(x) \equiv
    (1/\sqrt{2\pi})\exp(-x^2/2)$ and its cumulative distribution
    function $F_\Omega(x) = \int_{-\infty}^{x} \Omega(\tau) \,d\tau$.%
  }
  \label{prob:normdist-fig}
  \bc
    \settowidth\tla{\small$1$}
    \nc\xxxab{5.4}
    \nc\xxya{-0.3}
    \nc\xxyb{1.7}
    \nc\xxyc{2.3}
    \nc\fxa{-6.0} \nc\fxb{6.0}
    \nc\fya{-4.6} \nc\fyb{2.6}
    \nc\xxticlength{0.15}
    \nc\xxscale{1.7}
    \nc\xxfigoffset{3.8}
    \nc\xxaxes{%
      {%
        \psset{linewidth=0.5pt}%
        \psline(-\xxxab,0)(\xxxab,0)%
        \psline(0,\xxya)(0,\xxyb)%
        \uput[r](\xxxab,0){$x$}%
        \uput[u](0,\xxyb){$\Omega(x)$}%
      }%
    }
    \nc\xxaxess{%
      {%
        \psset{linewidth=0.5pt}%
        \psline(-\xxxab,0)(\xxxab,0)%
        \psline(0,\xxya)(0,\xxyc)%
        \psline[linestyle=dashed](-\xxxab,\xxscale)(\xxxab,\xxscale)%
        \uput[r](\xxxab,0){$x$}%
        \uput[u](0,\xxyc){$F_\Omega(x)$}%
      }%
    }
    \begin{pspicture}(\fxa,\fya)(\fxb,\fyb)
      %\psframe[linewidth=0.5pt,dimen=outer](\fxa,\fya)(\fxb,\fyb)
      %\localscalebox{1.0}{1.0}
      \small
      \rput(0,0){
         \xxaxes
         \psplot[linewidth=2.0pt,plotpoints=300]{-5.10}{5.10}{
           /twopi 6.28318530717959 def
           /e 2.71828182845905 def
           /scale 1.7 def
           e x scale div dup mul 2.0 div neg exp twopi sqrt div scale mul
         }
         \uput[ur](0,0.67820){$1/\sqrt{2\pi}$}
         \rput(\xxscale,0){\psline[linewidth=0.5pt](0,0)(0,0.41135)}
         \rput(-\xxscale,0){\psline[linewidth=0.5pt](0,0)(0,0.41135)}
         \uput[d](\xxscale,0){$1$}
         \uput[d](-\xxscale,0){\makebox[\tla][r]{$-1$}}
      }
      \rput(0,-\xxfigoffset){
         \xxaxess
         \psplot[linewidth=2.0pt,plotpoints=300]{-5.10}{5.10}{
           /twopi 6.28318530717959 def
           /e 2.71828182845905 def
           /scale 1.7 def
           x scale div dup dup mul neg exch dup
           1 1 15 {2.0 mul 4 1 roll 2 index mul 3 index div dup
           4 index 1 add div 3 2 roll add exch 4 3 roll pop} for
           pop exch pop twopi sqrt div 0.5 add scale mul
         }
         \uput[ur](0,\xxscale){$1$}
         \rput(\xxscale,0){\psline[linestyle=dashed,linewidth=0.5pt](0,0)(0,\xxscale)}
         \rput(-\xxscale,0){\psline[linestyle=dashed,linewidth=0.5pt](0,0)(0,\xxscale)}
         \uput[d](\xxscale,0){$1$}
         \uput[d](-\xxscale,0){\makebox[\tla][r]{$-1$}}
      }
    \end{pspicture}
  \ec
\end{figure}%

\index{normal distribution!cumulative distribution function of}
\index{cumulative distribution function!of the normal distribution}
\index{cumulative distribution function!numerical calculation of}
\index{cumulative normal distribution function}
Regarding the cumulative normal distribution function, one way to
calculate it numerically is to integrate the normal distribution's
Taylor series term by term.  As it happens, \S~\ref{inttx:450} has
worked a similar integral as an example, so this section will not
repeat the details, but the result is that
\bqa
  F_\Omega(x_o) = \int_{-\infty}^{x_o} \Omega(\tau) \,d\tau
  &=&
    \frac 12 + \frac{1}{\sqrt{2\pi}}
      \sum_{k=0}^{\infty} \frac{(-)^kx_o^{2k+1}}{(2k+1)2^k k!}
  \xn\\&=&
    \frac 12 + \frac{x_o}{\sqrt{2\pi}} \sum_{k=0}^{\infty} \frac{1}{2k+1}
      \prod_{j=1}^k \frac{-x_o^2}{2j}.
  \label{prob:100:40}
  \ \ \ \ %
\eqa
Unfortunately, this Taylor series---though always theoretically
correct---is practical only for small and moderate $\left|x_o\right|
\lesssim 1$.  For $\left|x_o\right| \gg 1$, see \S~\ref{prob:750}.

\index{distribution!default}
The normal distribution tends to be the default distribution in applied
mathematics.  When one lacks a reason to do otherwise, one models a
random quantity as a normally distributed random variable.
Section~\ref{prob:300} tells more.

\subsection{Motive}
\label{prob:100.60}
\index{motive!to posit the normal distribution}
\index{normal distribution!motive to posit}

\index{position}
Equation~(\ref{prob:normdist}) seems almost unfair to posit.  Once the
equation has been posited, the proof follows, the proof validating the
position; so the logic is valid, but why posit the equation in the
first place?\footnote{This subsection is optional reading for the benefit of
the curious.  You can skip it without burdening the rest of the book.}

One answer is that the equation~(\ref{prob:normdist}) is not really all
that obscure.  To study expressions that resemble $\exp(-x^2/2)$ for
their own sakes is neither unreasonable nor especially unlikely.  Some
mathematician or other must probably, eventually have thought to try
such an expression against the logic of \S~\ref{prob:100.10}.  One he
had tried it and had shown us his result, we would know to posit it.

The last paragraph's answer is actually a pretty good answer.  We should not be
embarrassed to give it.  Much of mathematics goes that way, after all.

Nevertheless, an alternate answer is known.  Suppose that~$N$ coins are tossed
%\footnote{%
%  Will the tossed coin remain familiar to future generations of
%  readers?  Probably so, but the expression has the ring of approaching
%  obsolescence even as it is written.  Nonetheless, ``tossed coin''
%  remains current at this writing.
%
%  A certain degree of obsolescence is a thing readers of old books must
%  always encounter, isn't it?  Whether or not you would still speak of a
%  tossed coin in your day, the image it conveys ought to be clear enough.
%
%  There is a regrettable tendency among too many readers of any
%  generation to smile callowly at the antiquities of old authors.  Such
%  readers are apt to forget that, during the old authors' generation,
%  \emph{they had yet older authors to smile at, too.}  Why, during the old
%  author's generation, they even had older authors more recent than,
%  say, the scribes of the Sumerian clay tablets, hard though that may be
%  to believe.  Maybe they even had a few older authors who, for all their
%  failure to anticipate the discoveries, trends and fashions of
%  centuries to come, are smarter and wiser than you.
%
%  Old authors are hardly a new thing.
%
%  If you think your generation exceptional, well, not only are you likely
%  mistaken but worse, you imagine even your \emph{mistake} to be new!
%  No, such conceit is as old as the hills.  Now, between the two of us,
%  across the gulf of years, who is smiling at whom, eh?  I never knew
%  you, but see the arrow of time: it points both ways.%
%}
and that~$2m$ is the number of heads in excess of the number of tails (for
example, if~6 heads and~2 tails, then $2m=6-2=4$ and $N=6+2=8$).
According to the combinatorics of \S~\ref{drvtv:220},
\[
  f(m) = \cmb{N}{[N+2m]/2} = \frac{N!/[(N-2m)/2]!}{[(N+2m)/2]!}
\]
computes the probability that~$m$ will have a given value.

Since we are merely motivating, we need not be precise, so approximately,
\bqb
    \lefteqn{\frac{d}{dm}\ln f(m) = \frac{df/dm}{f(m)}} &&
    \\&\approx& \frac{f(m+1)-f(m-1)}{2f(m)}
    \\&\approx& \frac 1 2 \left[\frac{f(m+1)}{f(m)} - \frac{f(m-1)}{f(m)}\right]
    \\&\approx& \frac 1 2 \left[\frac{(N-2m)/2}{(N+2m+2)/2} - \frac{(N+2m)/2}{(N-2m+2)/2}\right]
    \\&\approx& \frac 1 2 \left[\frac{1-2m/N}{1+(2m+2)/N} - \frac{1+2m/N}{1-(2m+2)/N}\right]
    \\&\approx& \frac 1 2 \left[\frac{(-8m-4)/N}{1-[(2m+2)/N]^2}\right]
    \approx \frac 1 2 \left[\frac{-8m-4}N\right]\left[1+\left(\frac{2m+2}N\right)^2\right]
    \\&\approx& \frac 1 2 \left[\frac{-8m}N\right] = -\frac{4m}{N}.
\eqb
Changing $x \la m$ and $\alpha \la 4/N$,
\[
  \frac{d}{dx}\ln f(x) = \frac{df/dx}{f(x)} \approx -\alpha x.
\]
A function that does this is
\[
  f(x) \approx C\exp(-\alpha x^2/2),
\]
which motivates~(\ref{prob:normdist}).

% ----------------------------------------------------------------------

\section{Inference of statistics}
\label{prob:200}
\index{inference of statistics}
\index{statistic!inference of}
\index{estimation of statistics}

\index{sample}
\index{instance}
Suppose that several, concrete instances of a random
variable---the instances collectively called a \emph{sample}---were
drawn from a distribution $f(x)$ and presented to you, but that you were
not told the shape of $f(x)$.  Could you infer the shape?

The answer is that you could infer the shape with passable accuracy
provided that the number~$N$ of instances were large.  Typically however
one will be prepared to make some assumption about the shape such as
that
\bq{prob:200:20}
  f(x) = \mu + \frac{1}{\sigma}\Omega\left(\frac{x}{\sigma}\right),
\eq
which is to assume that~$x$ were normally distributed with unknown
statistics~$\mu$ and~$\sigma$.  The problem then becomes to infer the
statistics from the sample.

\subsection{Inference of the mean}
\label{prob:200.20}
\index{mean!inference of}

In the absence of additional information, one can hardly suppose
much regarding the mean other than that
\bq{prob:200:mean}
  \mu \approx \frac{1}{N}\sum_k x_k.
\eq
One infers the mean to be the average of the instances observed.

\subsection{An imputed ensemble}
\label{prob:200.25}
\index{ensemble}
\index{imputed ensemble}

One might na\"ively think to infer a standard deviation in much the same
way as \S~\ref{prob:200.20} has inferred a mean, except that to
calculate the standard deviation directly according to~(\ref{prob:stat})
would implicate our imperfect
estimate~(\ref{prob:200:mean}) of the mean.  If we wish to estimate the
standard deviation accurately from the sample then we shall have to
proceed more carefully than that.

Section~\ref{prob:200.30} will estimate the standard deviation after
the subsection you are reading has prepared the ground on which to do
it.  To prepare the ground, let us now define the shifted random variable
\[
  u \equiv x - \mu_{\mr{true}}
\]
in lieu of the random variable~$x$, where~$\mu_{\mr{true}}$ \emph{is not
the estimated mean} of~(\ref{prob:200:mean}) but is the true, unknown
mean of the hidden distribution $f(x)$---such that an instance~$u_k$ of
the random variable~$u$ is in no way independent of, but is rather
wholly dependent on, the corresponding instance~$x_k$ of the random
variable~$x$; but also, paradoxically, such that the exact value
of~$u_k$ remains unknown even if the exact value of~$x_k$ is known.
And why does the exact value of~$u_k$ remain unknown?  It remains
unknown because the separation~$\mu_{\mr{true}}$ (which from the present
perspective is no random variable but a fixed number) between~$u_k$
and~$x_k$ remains unknown.  At any rate, the distribution of~$u$ is
\[
  f_u(u) \equiv f(u+\mu_{\mr{true}}),
\]
a distribution which by construction \emph{is known to have zero true
mean,}
\[
  \langle u \rangle = 0,
\]
even though the standard deviation~$\sigma_\mr{true}$ the two
distributions $f(x)$ and $f_u(u)$ share remains unknown.
%(Naturally, we do not know---we shall never know---the actual
%value of the true mean~$\mu_{\mr{true}}$ of the hidden distribution
%$f[\cdot]$, any more than we shall ever know the actual
%shape of $f[\cdot]$ and $f_u[\cdot]$,
%but this lack of knowledge will not prevent us from
%representing~$\mu_{\mr{true}}$ symbolically during analysis.)

Statistical reasoning is tricky, isn't it?  No?  Quite straightforward,
you say?  Good, let us continue.

Regarding not any particular sample of~$N$ instances but a conceptually
infinite \emph{ensemble} of samples, each sample consisting of~$N$ instances,
two identities the standard-deviational analysis of \S~\ref{prob:200.30}
will be able to use are that
\[
  \begin{split}
    \left\langle \sum_k u_k^2 \right\rangle &= N\sigma_{\mr{true}}^2, \\
    \left\langle {\sum_k}^2 u_k \right\rangle &= N\sigma_{\mr{true}}^2,
  \end{split}
\]
where~$\sigma_{\mr{true}}$ is (as the notation suggests and as an
earlier paragraph has observed) the
true, unknown standard deviation of the hidden distribution $f(x)$ and
thus also of the shifted distribution $f_u(u)$.
The first of the two identities is merely a statement of the leftward part of
(\ref{prob:stat})'s second line with respect to the distribution
$f_u(u)$ whose mean $\langle u \rangle = 0$ is, as we said, known to be
zero despite that the distribution itself remains unknown.
The second of the two identities
considers the sum $\sum_k u_k$ itself as a
random variable whose mean again is zero but whose standard
deviation~$\sigma_\Sigma$ according to~(\ref{prob:070:30}) is such that
$\sigma_\Sigma^2 = N\sigma_{\mr{true}}^2$.

\index{observation}
Admittedly, one might wonder how we can
speak sensibly of an ensemble when no
concrete ensemble is to be observed.  Observation, after all, sees
only the one sample of~$N$ instances.  However, we have assumed that a
hidden distribution $f(x)$ exists and that the several instances~$x_k$,
which \emph{are} observed, have been drawn from it.  Our assumption
might be \emph{wrong,} of course---in general this is very difficult to
judge---but we have assumed it and the assumption has
consequences.  Among the consequences is that $f(x)$ possesses
statistics~$\mu_{\mr{true}}$ and~$\sigma_{\mr{true}}$.
We do not know---we shall never know---the right values of these
statistics; but our assumption implies that they do exist and do have
values, values one can and should write symbols to represent.

Section~\ref{prob:200.30} will employ the symbols, next.

\subsection{Inference of the standard deviation}
\label{prob:200.30}
\index{standard deviation!inference of}

With the definitions and identities of \S~\ref{prob:200.25} in hand, let
us construct from the available sample the quantity
\bqb
  \left(\sigma'\right)^2
  \equiv
    \frac 1 N \sum_k\left(
      x_k - \frac 1 N \sum_\ell x_\ell
    \right)^2,
\eqb
modeled on~(\ref{prob:stat}).  Evidently,
\[
  \lim_{N\ra\infty} \sigma' = \sigma_{\mr{true}}.
\]
However, unlike the~$\sigma_{\mr{true}}$, the~$\sigma'$ is a quantity we
can actually compute from an observed sample.  Let the sample consist of
$N > 1$ instances.  By successive steps,
\bqb
  \left(\sigma'\right)^2
  &=&
    \frac 1 N \sum_k\left(
      [u_k+\mu_{\mr{true}}] - \frac 1 N \sum_\ell [u_\ell+\mu_{\mr{true}}]
    \right)^2
  \\&=&
    \frac 1 N \sum_k\left(
      u_k - \frac 1 N \sum_\ell u_\ell
    \right)^2
  \\&=& \frac 1 N
    \sum_k\left(
      u_k^2 - \frac 2 N u_k \sum_\ell u_\ell
      + \frac 1{N^2} {\sum_\ell}^2 u_\ell
    \right)
  \\&=&
    \frac 1 N \sum_k u_k^2
    -\frac 2{N^2} {\sum_k} u_k {\sum_\ell} u_\ell
    +\frac 1{N^2} {\sum_\ell}^2 u_\ell
  \\&=&
    \frac 1 N \sum_k u_k^2
    -\frac 2{N^2} {\sum_k}^2 u_k
    +\frac 1{N^2} {\sum_k}^2 u_k
  \\&=&
    \frac 1 N \sum_k u_k^2
    -\frac 1{N^2} {\sum_k}^2 u_k,
\eqb
the expected value of which over a conceptually infinite
ensemble of samples (each sample consisting---as explained by
\S~\ref{prob:200.25}---of an equal number~$N>1$ of
instances) is
\[
  \left\langle \left(\sigma'\right)^2 \right\rangle
  =
    \frac 1 N \left\langle \sum_k u_k^2 \right\rangle
    -\frac 1{N^2} \left\langle {\sum_k}^2 u_k \right\rangle.
\]
Applying the identities of \S~\ref{prob:200.25},
\[
  \left\langle \left(\sigma'\right)^2 \right\rangle
  = \sigma_{\mr{true}}^2 - \frac{\sigma_{\mr{true}}^2}{N}
  = \frac{N-1}{N} \sigma_{\mr{true}}^2,
\]
from which
\[
  \sigma_{\mr{true}}^2 = \frac{N}{N-1} \left\langle \left(\sigma'\right)^2 \right\rangle.
\]
Because the expectation $\langle(\sigma')\rangle$ is not a quantity
whose value we know, we can only suppose that $\langle(\sigma')^2\rangle
\approx (\sigma')^2$, whereby
\[
  \sigma_{\mr{true}}^2 \approx \frac{N}{N-1} \left(\sigma'\right)^2.
\]
Substituting the definition of $(\sigma')^2$ into the last equation and
changing symbols $\sigma \la \sigma_{\mr{true}}$, we have that
\bq{prob:200:stdev}
  \sigma^2 \approx
    \frac 1{N-1} \sum_k\left(
      x_k - \frac 1 N \sum_\ell x_\ell
    \right)^2.
\eq
This~$\sigma^2$ is apparently a little greater (though, provided
that~$N$ is sufficiently large, not much greater) than a na\"ive
assumption that~$\sigma$ equalled~$\sigma'$ would have supposed.

Notice according~(\ref{prob:200:stdev}) that~$\sigma$,
unlike~$\sigma'$, infers a standard deviation only when the sample
includes at least two instances!  Indeed,~$\sigma$ is sensible to
do so, for the one case in which a na\"ive analysis
were right would be when the true mean $\mu_{\mr{true}}$ were for some
reason \emph{a~priori} exactly known, leaving only the standard
deviation to be inferred.  In such a case,
\bq{prob:200:37}
  \sigma^2 \approx
    \frac 1 N \sum_k\left(
      x_k - \mu_{\mr{true}}
    \right)^2.
\eq

\index{statistic!sample}
\index{sample statistic}
The estimates~(\ref{prob:200:mean}) and~(\ref{prob:200:stdev}) are known
as \emph{sample statistics.}  They are the statistics one imputes to an
unknown distribution based on the incomplete information a sample of $N
> 1$ instances affords.

\subsection{Correlation and its inference}
\label{prob:200.40}
\index{correlation}
\index{correlation!inference of}

\index{independence}
\index{dependence}
\index{correlation coefficient}
\index{Pfufnik, Gorbag~J.}
The chapter you are reading has made some assumptions, not all of which
it has explicitly stated, or at any rate not all of which it has fully
developed.  One assumption the chapter has made is that instances of its
random variables have been \emph{independent.}
In statistical work however one must sometimes handle \emph{correlated}
quantities like the height and weight of a~25-year-old
U.S.\ male---for, obviously, if I point to some 25-year-old over
there and say, ``That's Pfufnik.  The average is~187 pounds, but he
weighs~250!'' then your estimate of his probable height will change,
because height and weight are not independent but correlated.  The
conventional statistical measure%
\footnote{\cite[\S~9.9]{Walpole/Myers}\cite[eqns.~12-6 and~12-14]{Alder/Roessler}}
of the correlation of a sample of~$N$ pairs $(x_k,y_k)$ of data, such as the
$([\mbox{height}]_k,[\mbox{weight}]_k)$ of the example, is the
\emph{correlation coefficient}
\bq{prob:070:60}
  r \equiv \frac{
    \sum_k (x_k-\mu_x) (y_k-\mu_y)
  }{
    \sqrt{\sum_k (x_k-\mu_x)^2 \sum_k (y_k-\mu_y)^2}
  },
\eq
a unitless quantity whose value is~$\pm 1$---the~$\pm 1$ indicating perfect
% bad break
cor\-re\-la\-tion---when
$y_k = x_k$ or even $y_k = a_1 x_k + a_0$; but
whose value should be near zero when the paired data are unrelated.  See
Fig.~\ref{mtxinv:320:fig1} for another example of the kind of paired
data in whose correlation one might be interested: in the figure, the
correlation would be~$+1$ if the points fell all right on the line.
(Beware that the conventional correlation coefficient of
eqn.~\ref{prob:070:60} can overstate the relationship between paired
data when~$N$ is small.  Consider for instance that $r=\pm 1$ always
when $N=2$.  The coefficient as given is nevertheless conventional.)

That $r = \pm 1$ when
$
  y_k = a_1 x_k + a_0
$
is seen by observing that
\bqb
  y_k-\mu_y
  &=&
  (a_1) \left[ x_k-\frac{\mu_y-a_0}{a_1} \right]
  =
  (a_1) \left[ x_k-\frac{\left(\sum_\ell y_\ell\right)/N-a_0}{a_1} \right]
  \\&=&
  (a_1) \left[ x_k-\frac{\left(\sum_\ell a_1x_\ell\right)/N}{a_1} \right]
  =
  (a_1) \left[ x_k-\mu_x \right],
\eqb
which, when substituted into~(\ref{prob:070:60}), yields the stipulated
result.

\subsection{Remarks}
\label{prob:200.90}

If further elaborated, the mathematics of statistics rapidly grows much
more complicated.  The book will not pursue the matter further but will
mention that the kinds of questions that arise can, among others, involve the
statistics of the statistics themselves, treating the \emph{statistics} as
random variables.  Section~\ref{prob:200.30} has done this just a bit.
The book will avoid doing more of it.

Such questions confound two, separate uncertainties:
the uncertainty inherent by definition~(\ref{prob:050:10}) in a random
variable even were the variable's distribution precisely known; and the
uncertain knowledge of the distribution.%
\footnote{
  The subtle mathematical implications of this far exceed the scope of
  the present book but are developed to one degree or another in
  numerous collegiate statistics texts of
  which \cite{Bulmer}\cite{Walpole/Myers}\cite{Alder/Roessler}\cite{Lindgren}\cite{Rosenkrantz}
  are representative examples.
}
Fortunately, if $N \gg 1$, then one can usually tear the two uncertainties
from one another without undue violence to accuracy, pretending that one
knew the unknown statistics~$\mu$ and~$\sigma$ to have exactly the
values~(\ref{prob:200:mean}) and~(\ref{prob:200:stdev}) respectively
calculate for them, supposing that the distribution were the
normal~(\ref{prob:200:20}), and modeling on this basis.

\index{Student, a statistician}
\index{Bulmer, M.~G.}
\index{Fisher, R.~A.\ (1890--1962)}
Unfortunately, that $N \gg 1$ is not so for many samples of practical
interest.  As the biologist M.~G.\ Bulmer recounts,
\begin{quote}
  `Student' had found, however, that in his practical work for Guinness'
  brewery he was often forced to deal with samples far too small for the
  customary large sample approximations to be applicable.  It was
  gradually {r}{e}{a}{l}{i}{s}{e}{d} after the publication of his paper,
  and of R.~A.\ Fisher's papers on other problems in small sample
  theory, that if the sample were large enough the answer to any
  question one might ask would be obvious, and that it was only in the
  case of small and moderate-sized samples that any statistical problem
  arose.~\cite[chapter~9]{Bulmer}
\end{quote}
Notwithstanding, the book you are reading will delve no further into the
matter, but will turn attention back to probabilistic topics.  It is
worth noting before we turn, however, that it took a pair of
biologists---`Student' and Fisher---to broaden the relevant mathematics.
Professional mathematicians might never have discovered the right
direction in which to explore on their own.  See
\S\S~\ref{conclu:athwart} and~\ref{conclu:klein}.

% ----------------------------------------------------------------------

\section{The random walk and its consequences}
\label{prob:300}
\index{random walk}
\index{walk, random}

\index{attempt, all-or-nothing}
\index{all-or-nothing attempt}
This section analyzes the simple but oft-encountered statistics of a series
of all-or-nothing attempts.

\subsection{The random walk}
\label{prob:300.10}
\index{random walk}
\index{walk, random}

\index{Sands, Matthew}
\index{Feynman, Richard~P.\ (1918-1988)}
\index{lecture}
\index{coin}
Matthew Sands gave a famous lecture~\cite[\S~I:6]{Feynman} to freshmen in
physics, on probability, on behalf of Richard~P. Feynman at Caltech in the fall
of~1961.  The lecture is a classic and is recommended to every reader
who can conveniently lay hands on a
copy---recommended among other reasons because the lecture lends needed context
to the rather abstruse mathematics this chapter has presented to the
present point.  One section of the lecture begins, ``There is [an]
interesting problem in which the idea of probability is required.  It is
the problem of the `random walk.'  In its simplest version, we imagine a
`game' in which a `player' starts at the point [$D=0$] and at each
`move' is required to take a step \emph{either} forward (toward [$+D$])
\emph{or} backward (toward [$-D$]).  The choice is to be made
\emph{randomly,} determined, for example, by the toss of a coin.  How
shall we describe the resulting motion?''

\index{stray}
\index{likely stray}
Sands goes on to observe that, though one cannot guess whether the
``player'' will have gone forward or backward after~$N$ steps---and,
indeed, that in the absence of other information one must expect
that $\langle D_N \rangle = 0$, indicating zero expected
net progress---``[one has] the feeling
that as~$N$ increases, [the `player'] is likely to have strayed farther
from the starting point.''  Sands is right, but if $\langle D_N \rangle$
is not a suitable measure of this ``likely stray,'' so to speak, then
what would be?

The measure $\langle\left| D_N \right|\rangle$ might recommend itself
but this, being nonanalytic (\S\S~\ref{alggeo:225.3}
and~\ref{taylor:320}), proves inconvenient in practice (you can try it if
you like).  Fortunately, an alternate, analytic measure, $\langle
D_N^2 \rangle$, presents itself.  The success of the least-squares
technique of \S~\ref{mtxinv:320} encourages us to try it.
When tried, the alternate, analytic measure prospers.

% diagn: review this fairly heavily revised paragraph
\index{index of stray}
Section~\ref{prob:070} has actually already introduced $\langle D_N^2
\rangle$ in another guise as~$\sigma^2$ (the~$\sigma^2$ in this context
being the standard deviation of an ensemble of a conceptually infinite
number of instances of~$D_N$, each instance being the sum of~$N$ random
steps).
The squared distance~$D_N^2$ is nonnegative, a quality necessary to a
good index of stray.  The squared distance~$D_N^2$ and its
expectance $\langle D_N^2 \rangle$ are easy to calculate
and, comparatively, also convenient to use.  Moreover, scientists and
engineers have long been used to accepting such quantities and
equivalents like~$\sigma^2$ as statistical characterizations.
We will use them for these reasons among others.

In his lecture, Sands observes that, if the symbol $D_{N-1}$ represents
the ``player's'' position after $N-1$ steps, if next step is~$\pm 1$ in
size, then the ``player's'' position after~$N$ steps must be
$D_N = D_{N-1} \pm 1$.  The expected value $\langle D_N \rangle = 0$ is
uninteresting as we said, but the expected value $\langle D_N^2 \rangle$
is interesting.  And what is this expected value?  Sands finds two
possibilities: either the ``player'' steps forward on his $N$th step, in
which case
\[
  \big\langle D_N^2 \big\rangle = \big\langle (D_{N-1} + 1)^2 \big\rangle
  = \big\langle D_{N-1}^2 \big\rangle + 2\big\langle D_{N-1} \big\rangle + 1;
\]
or he steps backward on his $N$th step, in which case
\[
  \big\langle D_N^2 \big\rangle = \big\langle (D_{N-1} - 1)^2 \big\rangle
  = \big\langle D_{N-1}^2 \big\rangle - 2\big\langle D_{N-1} \big\rangle + 1.
\]
Since forward and backward are equally likely, the actual expected value
must be the average
\[
  \left\langle D_N^2 \right\rangle = \left\langle D_{N-1}^2 \right\rangle + 1
\]
of the two possibilities.  Evidently, the expected value increases
by~$1$ with each step.  Thus by induction, since $\langle D_0^2 \rangle
= 0$,
\[
  \left\langle D_N^2 \right\rangle = N.
\]

Observe that the PDF of a single step~$x_k$ is $f_o(x_o) =
[\delta(x_o+1) + \delta(x_o-1)]/2$, where $\delta(\cdot)$ is the Dirac
delta of Fig.~\ref{integ:670:fig-d}; and that the corresponding
statistics are $\mu_o = 0$ and $\sigma_o = 1$.  The PDF of~$D_N$ is more
complicated (though not especially hard to calculate in view of
\S~\ref{drvtv:220}), but its statistics are evidently $\mu_N = 0$ and
$\sigma_N = \sqrt N$, agreeing with~(\ref{prob:070:40}).

\subsection{Consequences}
\label{prob:300.50}
\index{random walk!consequences of}
\index{walk, random!consequences of}

\index{real-estate agent}
\index{house}
\index{sales}
An important variation of the random walk comes with the distribution
\bq{prob:300:50}
  f_o(x_o) = (1-p_o)\delta(x_o) + p_o\delta(x_o-1),
\eq
which describes or governs an act whose probability of success is~$p_o$.
This distribution's statistics according to~(\ref{prob:stat}) are such
that
\bq{prob:300:55}
  \begin{split}
    \mu_o &= p_o, \\
    \sigma_o^2 &= (1-p_o)p_o.
  \end{split}
\eq
As an example of the use,\footnote{Decimal notation is again here
employed.  Maybe, abstractly, $2\pi \approx \mbox{0x6.487F}$, but even a
hexadecimal enthusiast is unlikely to numeralize real-estate sales in
hexadecimal.}
consider a real-estate agent who expects to
sell one house per~10 times he shows a house to a prospective buyer:
$p_o=1/10=0.10$.  The agent's expected result from a single showing,
according to~(\ref{prob:300:55}), is to sell $\mu_o \pm \sigma_o = 0.10
\pm 0.30$ of a house.  The agent's expected result from $N = 400$
showings, according to~(\ref{prob:070:40}), is to sell $\mu \pm \sigma =
N\mu_o \pm \left(\sqrt N\right)\sigma_o = 40.0 \pm 6.0$ houses.  Such a
conclusion, of course, is valid only to the extent to which the model is
valid---which in a real-estate agent's case might be \emph{not very}---but
that nevertheless is how the mathematics of it work.

\index{normal distribution!convergence toward}
As the number~$N$ of attempts grows large one finds that the
distribution $f(x)$ of the number of successes begins more and more to
take on the bell-shape of Fig.~\ref{prob:normdist-fig}'s normal
distribution.  Indeed, this makes sense, for one would expect the
aforementioned real-estate agent to enjoy a relatively high probability of
selling~39, 40 or~41 houses but a low probability to sell~10 or~70.
Of course, not all distributions that make~39, 40 or~41 more
likely than~10 or~70 are normal; but the logic of
\S~\ref{prob:100} does suggest that, if there \emph{were} a shape toward
which such a distribution tended as~$N$ increased, then that shape could
hardly be other than the shape of the normal distribution.  We will
leave the argument in that form.\footnote{%
  Admittedly, the argument, which supposes that all (or at least most)
  aggregate PDFs must tend toward some common shape as~$N$ grows large,
  is somewhat specious, or at least unrigorous---though on the other
  hand it is hard to imagine any plausible conclusion other than the
  correct one the argument reaches---but one might construct an alternate
  though tedious argument toward the normal distribution on the
  pattern of \S~\ref{prob:100.60} or on another pattern.  To fill in the
  tedious details is left as an exercise to the interested (penitent?)
  reader.  The author confesses that he prefers the specious argument of
  the narrative.%
}

\index{distribution!default}
For such reasons, applications tend to approximate sums of several
random variables as though the sums were normally distributed; and,
moreover, tend to impute normal distributions to random variables whose
true distributions are unnoticed, uninteresting or unknown.  In the
theory and application of probability, the normal distribution is the
master distribution, the distribution of last resort, often the only
distribution tried.  The banal suggestion, ``When unsure, go normal!''
usually prospers in probabilistic work.

% ----------------------------------------------------------------------

\section{Other distributions}
\label{prob:400}

Many distributions other than the normal one of
Fig.~\ref{prob:normdist-fig} are possible.  This section will name a few
of the most prominent.

\subsection{The uniform distribution}
\label{prob:400.10}
\index{uniform distribution}
\index{distribution!uniform}

\index{computer!pseudorandom-number generator}
\index{pseudorandom number}
The \emph{uniform distribution} can be defined in any of several forms, but
the conventional form is
\bq{prob:400:10}
  f(x) = \Pi\left(x-\frac 12\right) =
  \begin{cases}
    1 &\mbox{if $0 \le x < 1$,} \\
    0 &\mbox{otherwise.}
  \end{cases}
\eq
where $\Pi(\cdot)$ is the square pulse of Fig.~\ref{fours:095:fig1}.
Besides sometimes being useful in its own right, this is also the
distribution a computer's pseudorandom-number generator obeys.  One can
extract normally distributed (\S~\ref{prob:100}) or Rayleigh-distributed
(\S~\ref{prob:400.30}) random variables from it by the Box-Muller
transformation of \S~\ref{prob:410}.

\subsection{The exponential distribution}
\label{prob:400.20}
\index{exponential distribution}
\index{distribution!exponential}

\index{bearing}
\index{mechanical bearing}
\index{retail establishment}
\index{customer}
\index{failure of a mechanical part}
The \emph{exponential distribution} is
%\footnote{
%  Unlike the section's other subsections, this one explicitly includes
%  the mean~$\mu$ in the expression~(\ref{prob:400:20}) of its
%  distribution.  The inclusion of~$\mu$ here is admittedly inconsistent.
%  The reader who prefers to do so can mentally set $\mu = 1$ and read
%  the section in that light.  However, in typical applications the
%  entire point of choosing the exponential distribution may be to
%  specify~$\mu$, or to infer it.  The exponential distribution is
%  inherently ``$\mu$-focused,'' so to speak.  The author prefers to
%  leave the~$\mu$ in the expression for this reason.
%}
\bq{prob:400:20}
  f(x) = \frac{u(t)}{\mu}\exp\left(-\frac{x}{\mu}\right),
\eq
the $u(t)$ being Heaviside's unit step~(\ref{integ:670:10}).
The distribution's mean is
\[
  \frac 1 \mu \int_0^\infty \exp\left( -\frac x \mu \right) x\,dx
  =
    \left.
      - \exp\left( -\frac x \mu \right) ( x + \mu )
    \right|_0^\infty
  = \mu
\]
as advertised and its standard deviation is such that
\bqb
  \sigma^2
  &=&
    \frac 1 \mu \int_0^\infty \exp\left( -\frac x \mu \right) (x-\mu)^2\,dx
  \\&=&
    \left.
      - \exp\left( -\frac x \mu \right) ( x^2 + \mu^2 )
    \right|_0^\infty
\eqb
(the integration by the method of unknown coefficients of
\S~\ref{inttx:240} or, quicker, by Table~\ref{inttx:470:tbl}), which
implies that
\bq{prob:400:25}
  \sigma = \mu.
\eq
The exponential's CDF~(\ref{prob:CDF}) and
quantile~(\ref{prob:quantile}) are evidently
\bq{prob:400:28}
  \begin{split}
    F(x) &= 1 - \exp\left( -\frac{x}{\mu} \right), \ \ x \ge 0;\\
    F^{-1}(v) &= -\mu\ln(1-v).
  \end{split}
\eq
Among other effects, the exponential distribution models the delay until
some imminent event like a mechanical bearing's failure or the
arrival of a retail establishment's next customer.

\subsection{The Poisson distribution}
\label{prob:400.25}
\index{Poisson distribution}
\index{distribution!Poisson}
\index{Poisson, Sim\'eon Denis (1781--1840)}

The Poisson distribution is\footnote{\cite[chapter~6]{Bulmer}}
\bq{prob:400:p25}
  f(x) = \exp(-\mu)\sum_{k=0}^\infty\frac{\mu^x\delta(x-k)}{x!}.
\eq
It comes from the consideration of a large number $N \gg 1$ of
individually unlikely trials, each trial having a probability~$0 < \ep
\ll 1$ of success, such that the expected number of successes is $\mu =
\ep N$.
\begin{itemize}
  \item The chance that no trial will succeed is evidently
    \[
        \lim_{\eta \ra 0^{+}} \int_{-\eta}^\eta f(x)dx = (1-\ep)^N
        \approx \exp(-\ep N) = \exp(-\mu).
    \]
  \item The chance that exactly one trial will succeed is
    \[
      \begin{split}
        \lim_{\eta \ra 0^{+}} \int_{1-\eta}^{1+\eta} f(x)dx &= \cmb{N}{1}(\ep)(1-\ep)^{N-1}
        \\&\approx \ep N\exp(-\ep N) = \mu\exp(-\mu).
      \end{split}
    \]
  \item The chance that exactly two trials will succeed is
    \[
      \begin{split}
        \lim_{\eta \ra 0^{+}} \int_{2-\eta}^{2+\eta} f(x)dx &= \cmb{N}{2}(\ep^2)(1-\ep)^{N-2}
        \\&\approx \frac{(\ep N)^2}{2!}\exp(-\ep N) = \frac{\mu^2\exp(-\mu)}{2!}.
      \end{split}
    \]
  \item And so on.
\end{itemize}
In the limit as $N \ra \infty$ and $\ep \ra 0^{+}$, the product $\mu =
\ep N$ remaining finite, the approximations become exact
and~(\ref{prob:400:p25}) results.

Integrating~(\ref{prob:400:p25}) to check,
\bqb
  \int_{-\infty}^{\infty} f(x) dx
    &=& \exp(-\mu)\int_{-\infty}^{\infty}\sum_{k=0}^\infty\frac{\mu^x\delta(x-k)}{x!} dx
    \\&=& \exp(-\mu)\sum_{k=0}^\infty\int_{-\infty}^{\infty}\frac{\mu^x\delta(x-k)}{x!} dx
    \\&=& \exp(-\mu)\sum_{k=0}^\infty\frac{\mu^k}{k!}
    = \exp(-\mu)\exp(\mu) = 1,
\eqb
as~(\ref{prob:050:10}) requires.

Compared to the exponential distribution (\S~\ref{prob:400.20}), the
Poisson distribution serves to model for example the number of customers
to arrive at a retail establishment during the next hour.

\subsection{The Rayleigh distribution}
\label{prob:400.30}
\index{Rayleigh distribution}
\index{distribution!Rayleigh}
\index{Rayleigh, John Strutt, 3rd baron (1842--1919)}

\index{missile}
The \emph{Rayleigh distribution} is a generalization of the normal
distribution for position in a plane.  Let each of the~$x$ and~$y$
coordinates be drawn independently from a normal distribution of zero
mean and unit standard deviation, such that
\bqb
  dP
  &\equiv&
    \left[ \Omega(x) \,dx \right]
    \left[ \Omega(y) \,dy \right]
  \\&=&
    \frac{1}{2\pi}
    \exp\left( -\frac{x^2+y^2}{2} \right) \,dx\,dy
  \\&=&
    \frac{1}{2\pi}
    \exp\left( -\frac{\rho^2}{2} \right) \rho\,d\rho\,d\phi,
\eqb
whence
\bqb
  P_{ba}
  &\equiv&
    \int_{\phi=-\pi}^\pi \int_{\rho=a}^b dP
  \\&=&
    \frac{1}{2\pi}
    \int_{-\pi}^\pi \int_a^b
    \exp\left( -\frac{\rho^2}{2} \right) \rho\,d\rho\,d\phi
  \\&=&
    \int_a^b
    \exp\left( -\frac{\rho^2}{2} \right) \rho\,d\rho,
\eqb
which implies the distribution
\bq{prob:400:30}
  f(\rho) = u(\rho) \rho \exp\left( -\frac{\rho^2}{2} \right).
\eq
This is the Rayleigh distribution.  That it is a proper distribution
according to~(\ref{prob:050:10}) is proved by evaluating the integral
\bq{prob:400:32}
  \int_0^{\infty} f(\rho) \,d\rho = 1
\eq
using part of the method of \S~\ref{fouri:130}.
Rayleigh's CDF~(\ref{prob:CDF}) and quantile~(\ref{prob:quantile}) are
evidently\footnote{\cite[\S~5.2]{Papoulis}}
\bq{prob:400:34}
  \begin{split}
    F(\rho) &= 1 - \exp\left( -\frac{\rho^2}{2} \right), \ \ \rho \ge 0;\\
    F^{-1}(v) &= \sqrt{-2\ln(1-v)}.
  \end{split}
\eq
The Rayleigh distribution models among others the distance~$\rho$ by
which a missile might miss its target.

\index{azimuth}
Incidentally, there is nothing in the mathematics to favor any
particular value of~$\phi$ over another,~$\phi$ being the azimuth at
which the missile misses, for the integrand $\exp(-\rho^2/2) \rho
\,d\rho\,d\phi$ includes no~$\phi$.  The azimuth~$\phi$ must by
symmetry therefore be uniformly distributed.

Rayleigh's mean and standard deviation are computed
via~(\ref{prob:stat}) to be
\bq{prob:400:35}
  \begin{split}
    \mu &= \frac{\sqrt{2\pi}}{2}, \\
    \sigma^2 &= 2 - \frac{2\pi}{4}.
  \end{split}
\eq
by
\[
  \mu = \int_0^\infty \rho^2 \exp\left( -\frac{\rho^2}{2} \right)
  \,d\rho = \frac{\sqrt{2\pi}}{2}
\]
(compare eqn.~\ref{prob:100:10}, observing however that the present
integral integrates over only half the domain) and
\bqb
  \sigma^2 &=& \int_0^\infty \left(\rho - \frac{\sqrt{2\pi}}{2}\right)^2
    \rho \exp\left( -\frac{\rho^2}{2} \right) \,d\rho
  \\&=& \int_0^\infty
    \rho^3 \exp\left( -\frac{\rho^2}{2} \right) \,d\rho
  \\&&\quad\mbox{} -\sqrt{2\pi}\int_0^\infty
    \rho^2 \exp\left( -\frac{\rho^2}{2} \right) \,d\rho
  \\&&\quad\mbox{} +\frac{2\pi}{4}\int_0^\infty
    \rho \exp\left( -\frac{\rho^2}{2} \right) \,d\rho
  \\&=& \int_0^\infty
    \rho^3 \exp\left( -\frac{\rho^2}{2} \right) \,d\rho
    - \frac{2\pi}{2} + \frac{2\pi}{4}
  \\&=& -\int_{\rho=0}^\infty
    \rho^2 \,d\left[\exp\left( -\frac{\rho^2}{2} \right)\right]
    - \frac{2\pi}{4}
  \\&=&
    \makebox[-3pt][c]{}\left.-\rho^2 \exp\left( -\frac{\rho^2}{2}\right)\right|_0^\infty
    +\int_{\rho=0}^\infty
    \exp\left( -\frac{\rho^2}{2} \right) \,d\left[\rho^2\right]
    - \frac{2\pi}{4}
  \\&=&
    0+2\int_0^\infty
    \rho\exp\left( -\frac{\rho^2}{2} \right) \,d\rho
    - \frac{2\pi}{4}
  = 2 - \frac{2\pi}{4}.
\eqb

\subsection{The Maxwell distribution}
\label{prob:400.40}
\index{Maxwell distribution}
\index{distribution!Maxwell}
\index{Maxwell, James Clerk (1831--1879)}

\index{particle}
\index{air}
The \emph{Maxwell distribution} extends the Rayleigh from two to three
dimensions.  Maxwell's derivation closely resembles Rayleigh's, with the
difference that Maxwell uses all three of~$x$, $y$ and~$z$ and then
transforms to spherical rather than cylindrical coordinates.  The
distribution which results, the Maxwell distribution, is
\bq{prob:400:40}
  f(r) = \frac{2r^2}{\sqrt{2\pi}} \exp\left( -\frac{r^2}{2} \right),
  \ \ r \ge 0,
\eq
which models among others the speed at which an air molecule might
\linebreak % bad break
travel.%
\footnote{\cite[eqn.~I:40.7]{Feynman}}
%\footnote{
%  The section has omitted several reasonably well-known distributions.
%  As the section ends it will pass these over, except one, without
%  special notice.  The one is the \emph{chi-square} distribution
%  \cite[chapter~13]{Adler}.  The reason the section has omitted the
%  chi-square is that the principal application of the chi-square lies in
%  ``hypothesis testing,'' an application of statistical inference this
%  book does not treat.
%}

%\subsection{The log-normal distribution}
%\label{prob:400.50}
%\index{log-normal distribution}
%\index{distribution!log-normal}
%
%In the \emph{log-normal distribution,} it is not~$x$ but
%\bq{prob:400:51}
%  y \equiv \frac{\ln x}{\alpha}
%\eq
%that is normally distributed, a fairly common case.  Setting $x =
%g(y) \equiv \exp \alpha y$ and $f_o(y) = \Omega(y)$
%in~(\ref{prob:080:10}), one can express the log-normal distribution in
%the form%
%\footnote{\cite[eqn.~5-11]{Papoulis}}
%\bq{prob:400:50}
%  f(x) = \frac{1}{\alpha x}\Omega\left(\frac{\ln x}{\alpha}\right).
%\eq

% ----------------------------------------------------------------------

\section{The Box-Muller transformation}
\label{prob:410}
\index{Box-Muller transformation}
\index{transformation!Box-Muller}
\index{Box, G.~E.~P.\ (1919--2013)}
\index{Muller, Mervin~E.\ (1928--)}
% Retain this note: on May 13, 2017, Dr. Muller replied to T. H. Black's
% email, indicating that his date of birth is June 1, 1928.

\index{distribution!conversion between two}
\index{quantile!use of to convert between distributions}
The quantiles~(\ref{prob:400:28}) and~(\ref{prob:400:34}) imply easy
conversions from the uniform distribution to the exponential and
Rayleigh.  Unfortunately, we lack a quantile formula for the normal
distribution.  However, we can still convert uniform to normal by way of
Rayleigh as follows.

Section~\ref{prob:400.30} has associated the
Rayleigh distribution with
the distance~$\rho$ by which a missile misses its target, the~$x$
and~$y$ coordinates of the missile's impact each being normally
distributed over equal standard deviations.
Section~\ref{prob:400.30} has further drawn out the
uniform distribution of the impact's azimuth~$\phi$.  Because
we know Rayleigh's quantiles, we are able to convert a pair of
instances~$u$ and~$v$ of a uniformly distributed random variable to
Rayleigh's distance and azimuth by\footnote{%
  One can eliminate a little trivial arithmetic by appropriate changes
  of variable in~(\ref{prob:410:20}) like $u' \la 1-u$, but to
  do so saves little computational time and makes the derivation harder
  to understand.  Still, the interested reader might complete the
  improvement as an exercise.%
}
\bq{prob:410:20}
  \begin{split}
    \rho &= \sqrt{-2\ln(1-u)}, \\
    \phi &= (2\pi)\left(v - \frac 1 2\right).
  \end{split}
\eq
But for the reason just given,
\bq{prob:410:25}
  \begin{split}
    x &= \rho\cos\phi, \\
    y &= \rho\sin\phi,
  \end{split}
\eq
must then constitute two independent instances of a normally distributed
random variable with $\mu = 0$ and $\sigma = 1$.  Evidently, though we
lack an easy way to convert a single uniform instance to a single normal
instance, we can convert a \emph{pair} of uniform instances to a pair of
normal instances.  Equations~(\ref{prob:410:20}) and~(\ref{prob:410:25})
are the \emph{Box-Muller transformation.}%
\footnote{\cite{Box/Muller}\cite{EWW}}

% ----------------------------------------------------------------------

\section{The normal CDF at large arguments}
\label{prob:750}
\index{normal distribution!cumulative distribution function of}
\index{cumulative distribution function!of the normal distribution}
\index{large-argument form}

\index{rounding error}
\index{error, rounding}
The Taylor series~(\ref{prob:100:40}) in theory correctly calculates the
normal CDF $F_\Omega(x)$, an entire function, for any argument~$x$.  In
practice however---consider the Taylor series
\[
  1 - F_\Omega(6) \approx
    - \mbox{0x0.8000}
    + \mbox{0x2.64C6}
    - \mbox{0xE.5CA7}
    + \mbox{0x4D.8DEC}
    - \cdots
\]
Not promising, is it?  Using a computer's standard, \texttt{double}-type
floating-point arithmetic, this calculation fails, swamped by rounding
error.

\index{integration!by parts}
\index{residual}
\index{semiconvergent series}
\index{asymptotic series}
\index{series!semiconvergent}
\index{series!asymptotic}
One can always calculate in greater precision,%
\footnote{\cite{libmpfi}}
of course, asking the computer to carry extra bits; and, actually, this
is not necessarily a bad approach.  There remain however several reasons
one might prefer a more efficient formula.
\bi
  \item
    One might wish to evaluate the CDF at thousands or millions
    of points, not just one.  At some scale, even with a computer, the
    calculation grows expensive.
  \item
    \index{embedded device}
    One might wish to evaluate the CDF on a low-power ``embedded
    device.''
  \item
    \index{embedded control}
    \index{aircraft}
    One might need to evaluate the CDF under a severe time constraint
    measured in microseconds, as in aircraft control.
  \item
    \index{computer}
    \index{pencil}
    \index{slide rule}
    Hard though it might be for some to imagine, one might actually wish
    to evaluate the CDF with a pencil!  Or with a slide rule.  (Besides
    that one might not have a suitable electronic computer conveniently
    at hand, that electronic computers will never again be scarce is a
    proposition whose probability the author is not prepared to
    evaluate.)
  \item
    \index{efficiency}
    \index{prudence}
    \index{elegance}
    The mathematical method by which a more efficient formula is derived
    is most instructive.%
    \footnote{
      Such methods prompt one to wonder how much useful mathematics our
      civilization should have forgone had Leonhard Euler (1707--1783),
      Carl Friedrich Gauss (1777--1855) and other hardy mathematical
      minds of the past had computers to lean on.
    }
  \item
    One might regard a prudent measure of elegance, even in
    applications, to be its own reward.
\ei
Here is the method.%
\footnote{\cite[\S~2.2]{Lebedev}}
Beginning from
\bqb
  1 - F_\Omega(x)
  &=&
    \frac{1}{\sqrt{2\pi}}
      \int_x^\infty \exp\left( -\frac{\tau^2}{2} \right) \,d\tau
  \\&=&
    \frac{1}{\sqrt{2\pi}} \left\{
      -\int_{\tau=x}^\infty \frac{ d\left[ e^{-\tau^2/2} \right] }{\tau}
    \right\}
\eqb
and integrating by parts,
\bqb
  1 - F_\Omega(x)
  &=&
    \frac{1}{\sqrt{2\pi}} \left\{
      \frac{e^{-x^2/2}}{x}
      - \int_{x}^\infty \frac{ e^{-\tau^2/2} \,d\tau }{\tau^2}
    \right\}
  \\&=&
    \frac{1}{\sqrt{2\pi}} \left\{
      \frac{e^{-x^2/2}}{x}
      + \int_{\tau=x}^\infty \frac{ d\left[ e^{-\tau^2/2} \right] }{\tau^3}
    \right\}.
\eqb
Integrating by parts again,
\bqb
  1 - F_\Omega(x)
  &=&
    \frac{1}{\sqrt{2\pi}} \left\{
      \frac{e^{-x^2/2}}{x}
      - \frac{e^{-x^2/2}}{x^3}
      + 3 \int_{x}^\infty \frac{ e^{-\tau^2/2} \,d\tau }{\tau^4}
    \right\}
  \\&=&
    \frac{1}{\sqrt{2\pi}} \left\{
      \frac{e^{-x^2/2}}{x}
      - \frac{e^{-x^2/2}}{x^3}
      - 3 \int_{\tau=x}^\infty \frac{ d\left[ e^{-\tau^2/2} \right] }{\tau^5}
    \right\}.
\eqb
Integrating by parts repeatedly,
\bqb
  1 - F_\Omega(x)
  &=&
    \frac{1}{\sqrt{2\pi}} \Bigg\{
      \frac{e^{-x^2/2}}{x}
      - \frac{e^{-x^2/2}}{x^3}
      + \frac{3e^{-x^2/2}}{x^5}
      - \cdots
  \\&&\ \ \ \ \ \ \ \ \ \ \ \ \mbox{}%
      + \frac{(-)^{n-1}(2n-3)!!e^{-x^2/2}}{x^{2n-1}}
  \\&&\ \ \ \ \ \ \ \ \ \ \ \ \mbox{}%
      + (-)^n(2n-1)!! \int_{x}^\infty \frac{ e^{-\tau^2/2} \,d\tau}{\tau^{2n}}
    \Bigg\},
\eqb
in which the convenient notation
\bq{prob:750:05}
  \index{\char 33 \char 33}
  \index{factorial!\char 33 \char 33 -style}
  m!! \equiv
  \settowidth\tla{$\prod_{j=1}^{(m+1)/2} (2j-1)$}
  \begin{cases}
    \prod_{j=1}^{(m+1)/2} (2j-1)
      = (m)(m-2)\cdots(5)(3)(1) &\mbox{for odd~$m$,} \\
    \makebox[\tla][l]{$\prod_{j=1}^{m/2} (2j)$}
      = (m)(m-2)\cdots(6)(4)(2) &\mbox{for even~$m$,}
  \end{cases}
\eq
is introduced.\footnote{\cite[Exercise~2.2.15]{Andrews}}
The last expression for $1 - F_\Omega(x)$ is better written,
\bqa
    1 - F_\Omega(x) &=& \frac{\Omega(x)}{x} [ S_n(x) + R_n(x) ],
  \label{prob:750:20}\\
    S_n(x) &\equiv&
      \sum_{k=0}^{n-1} \left[ \prod_{j=1}^k \frac{2j-1}{-x^2} \right]
      = \sum_{k=0}^{n-1} \frac{(-)^k(2k-1)!!}{x^{2k}},
  \xn\\
    R_n(x) &\equiv& (-)^n(2n-1)!! x \int_{x}^\infty
      \frac{ e^{(x^2-\tau^2)/2} \,d\tau}{\tau^{2n}}.
  \xn
\eqa
The series $S_n(x)$ is an \emph{asymptotic series,} also
called a \emph{semiconvergent series.}%
\footnote{\label{prob:750fn1}%
  \cite[\S~1.4.1]{Andrews}%
}
So long as
$x \gg 1$, the first several terms of the series will
successively shrink in magnitude but, no matter how great the
argument~$x$ might be, eventually the terms will insist on growing
again, growing without limit.  Unlike a Taylor series, $S_\infty(x)$
diverges for all~$x$.

\index{truncation}
\index{series!truncation of}
\index{error bound}
\index{term!finite number of}
\index{partial sum}
\index{sum!partial}
\index{cumulative distribution function!estimation of}
Fortunately, nothing requires us to let $n \ra \infty$, and we remain
free to choose~$n$ strategically as we like---for instance to exclude
from~$S_n$ the series' least term in magnitude and all the terms
following.  So excluding leaves us with the problem of evaluating the
integral~$R_n$, but see:
\bqb
  \left|R_n(x)\right|
  &\le&
    (2n-1)!! \left|x\right| \int_{x}^\infty
      \left| \frac{ e^{(x^2-\tau^2)/2} \,d\tau}{\tau^{2n}} \right|
  \\&\le&
    \frac{(2n-1)!!}{\left|x\right|^{2n}} \int_{x}^\infty
      \left| e^{(x^2-\tau^2)/2} \tau\,d\tau \right|,
\eqb
because $\left|x\right| \le \left|\tau\right|$, so
$\left|x\right|^{2n+1} \le \left|\tau\right|^{2n+1}$.
Changing $\xi^2 \la \tau^2 - x^2$, whereby $\xi \,d\xi = \tau \,d\tau$,
\[
  \left|R_n(x)\right|
  \le
    \frac{(2n-1)!!}{\left|x\right|^{2n}} \int_{0}^\infty
      \left| e^{-\xi^2/2} \xi\,d\xi \right|.
\]
Using~(\ref{prob:400:30}) and~(\ref{prob:400:32}),
\bq{prob:750:25}
  \left|R_n(x)\right|
  \le
    \frac{(2n-1)!!}{\left|x\right|^{2n}},
    \ \ \Im(x) = 0,\ \Re(x) > 0,
\eq
which in view of~(\ref{prob:750:20}) has that the
magnitude $\left|R_n\right|$ of the error due to truncating the series
after~$n$ terms does not exceed the magnitude of the first omitted term.
Equation~(\ref{prob:750:20}) thus provides the efficient means we have
sought to estimate the CDF accurately for large arguments.

% ----------------------------------------------------------------------

\section{Asymptotic series}
\label{prob:760}
\index{asymptotic series}
\index{series!asymptotic}

\index{truncation}
\index{series!truncation of}
Section~\ref{prob:750} has incidentally introduced the \emph{asymptotic
series} and has shown how to treat it.

Asymptotic series are strange.  They diverge, but only after
approaching a sum of interest.  Some asymptotic series approach
the sum of interest quite closely, and moreover do so in such a way that
the closeness---that is, the error in the sum---can with sufficient
effort be quantified.  The error in the sum of the asymptotic series of
\S~\ref{prob:750} has been found not to exceed the magnitude of the
first omitted term; and though one may have to prove it specially for
each such series, various series one encounters in practice tend to
respect bounds of the same kind.

\index{semiconvergent series}
\index{series!semiconvergent}
As \S~\ref{prob:750} has noted, asymptotic series are sometimes
alternately called \emph{semiconvergent series.}\footnote{See
footnote~\ref{prob:750fn1}.}

An ordinary, convergent series is usually preferable to
an asymptotic series, of course, especially in the subdomain near the
convergent series' expansion point (\S~\ref{taylor:317}).
However, a convergent series is not always available; and, even when it
is, its expansion point may lie so distant that the series becomes
numerically impractical to total.

An asymptotic series can fill the gap.

Aside from whatever practical applications an asymptotic series can fill,
this writer finds the topic of asymptotic series fascinating.  The
topic is curious, is it not?  How can a \emph{divergent series} reach a
definite total?  The answer seems to be: it cannot reach a definite
total but can draw arbitrarily close to one.  In~(\ref{prob:750:20})
and~(\ref{prob:750:25}) for example, the larger the argument, the
closer the draw.  It is a paradox yet, surprisingly, it works.

Asymptotic series arise in the study and application of special functions,
including (as we have seen) the $\Omega(\cdot)$ of the present chapter.
For this reason and maybe others, the applied mathematician will exercise and
exploit asymptotic series from time to time.

% ----------------------------------------------------------------------

\section{The normal quantile}
\label{prob:775}
\index{normal distribution!quantile of}
\index{quantile!of the normal distribution}

\index{Newton-Raphson iteration}
Though no straightforward quantile formula to satisfy~(\ref{prob:quantile})
for the normal distribution seems to be known, nothing prevents one from
calculating the quantile via the Newton-Raphson
iteration~(\ref{drvtv:NR})%
\footnote{
  When implementing numerical algorithms like these on the computer one
  should do it intelligently.  For example, if $F_\Omega(x_k)$ and~$v$ are
  both likely to be close to~1, then do not ask the computer to calculate
  and/or store these quantities.  Rather, ask it to calculate and/or
  store $1-F_\Omega(x_k)$ and $1-v$.  Then, when~(\ref{prob:775:10}) instructs
  you to calculate a quantity like $F_\Omega(x_k)-v$, let the computer instead
  calculate $[1-v]-[1-F_\Omega(x_k)]$, which is arithmetically no different but
  numerically, on the computer, much more precise.
}
\bq{prob:775:10}
  \begin{split}
    x_{k+1} &= x_k - \frac{F_\Omega(x_k)-v}{\Omega(x_k)}, \\
    F_\Omega^{-1}(v) &= \lim_{k \ra \infty} x_k, \\
    x_0 &= 0,
  \end{split}
\eq
where $F_\Omega(x)$ is as given by~(\ref{prob:100:40})
and/or~(\ref{prob:750:20}) and where $\Omega(x)$ is, as usual, as given
by~(\ref{prob:normdist}).  The shape of the normal CDF as seen in
Fig.~\ref{prob:normdist-fig} on
page~\pageref{prob:normdist-fig}---curving downward traveling right from
$x=0$, upward when traveling left---evidently guarantees convergence
per Fig.~\ref{drvtv:270:fig1}, page~\pageref{drvtv:270:fig1}.

\index{lazy convergence}
\index{convergence!lazy}
\index{bracket}
\index{accuracy}
\index{na\"ivet\'e}
In the large-argument limit,
\[
  \begin{split}
    1-v &\ll 1, \\
    x &\gg 1;
  \end{split}
\]
so, according to~(\ref{prob:750:20}),
\[
  F_\Omega(x) \approx 1 - \frac{\Omega(x)}{x}
    \left( 1 - \frac{1}{x^2} + \cdots \right).
\]
Substituting this into~(\ref{prob:775:10}) yields, by successive steps,
\bqb
  x_{k+1} &\approx&
    x_k - \frac{1}{\Omega(x_k)}\left[
      1 - v - \frac{\Omega(x_k)}{x_k}
      \left( 1 - \frac{1}{x_k^2} + \cdots \right)
    \right]
  \\&\approx&
    x_k - \frac{1-v}{\Omega(x_k)}
      + \frac{1}{x_k} - \frac{1}{x_k^3} + \cdots
  \\&\approx&
    x_k - \frac{\left(\sqrt{2\pi}\right)(1-v)}{1 - x_k^2/2 + \cdots}
      + \frac{1}{x_k} - \frac{1}{x_k^3} + \cdots
  \\&\approx&
    x_k - \bigg(\sqrt{2\pi}\bigg)\bigg(1-v\bigg)\bigg(1 + \frac{x_k^2}{2} + \cdots \bigg)
      + \frac{1}{x_k} - \frac{1}{x_k^3} + \cdots
  \\&\approx&
    x_k - \bigg(\sqrt{2\pi}\bigg)\bigg(1-v\bigg)
      + \frac{1}{x_k} + \cdots,
\eqb
suggesting somewhat lazy, but usually acceptable convergence in domains
of typical interest (the convergence might be unacceptable if, for
example, $x>\mbox{0x40}$, but the writer has never encountered an
application of the normal distribution $\Omega[x]$ or its incidents at
such large values of~$x$).  If unacceptable, various stratagems might be
tried to accelerate the Newton-Raphson, or---if you have no need to
impress anyone with the pure elegance of your technique but only want
the right answer reasonably fast---you might just search for the root in
the na\"ive way, trying $F_\Omega(2^0)$, $F_\Omega(2^1)$,
$F_\Omega(2^2)$ and so on until identifying a bracket $F_\Omega(2^{k-1}) < v
\le F_\Omega(2^k)$; then dividing the
bracket in half, then in half again, then again and again until
satisfied with the accuracy thus achieved, or until the bracket were
strait enough for you to set~$x_0$ to the bracket's lower (not upper)
limit and to switch over to~(\ref{prob:775:10}) which performs well when
it starts close enough to the root.  In truth, though not always
stylish, the normal quantile of a real argument is
relatively quick, easy and accurate to calculate once you
have~(\ref{prob:normdist}), (\ref{prob:100:40}) and~(\ref{prob:750:20})
in hand, even when the performance of~(\ref{prob:775:10}) might not
quite suit.  You only must remain a little flexible as to the
choice of technique.%
\footnote{See also \S~\ref{taylor:316.80}.}

%% ----------------------------------------------------------------------
%
%\section{Statistical mechanics: the ideal gas}
%\label{prob:900}
%\index{statistical mechanics}
%\index{mechanics}
%\index{ideal gas}
%\index{gas}
%
%\index{air}
%\index{probability!application of}
%\index{application}
%\index{physical application}
%The mechanics of air offer a lovely application of probability theory.
%As with most physical applications of mathematics, this application
%begins with a simplified, idealized model.
%
%\index{collision}
%\index{common velocity}
%\index{differential path}
%\index{path}
%\index{differential velocity}
%\index{velocity!common and differential}
%\index{particle}
%\index{rigidity}
%\index{momentum}
%\index{conservation of momentum}
%\index{law}
%Suppose that air were an \emph{ideal gas,} consisting of a very large
%number of tiny, elastic, rigid, spherical, moving particles, the
%particles together occupying a negligible fraction of the volume through
%which they travel or within which they are confined, the volume thus
%being mostly empty.  Suppose two such air particles, masses~$m_1$
%and~$m_2$, velocities~$\vu u_1$ and~$\vu u_2$, on a collision course.
%One can regard each velocity as the sum of two, separate velocities,
%\bq{prob:900:220}
%  \begin{split}
%    \ve u_1 &= \ve u_{c} + \ve u_{1d}, \\
%    \ve u_2 &= \ve u_{c} + \ve u_{2d}, \\
%    \ve u_c &\equiv \frac{ m_1 \ve u_1 + m_2 \ve u_2  }{ m_1 + m_2 },
%  \end{split}
%\eq
%where~$\ve u_c$ is the particles' shared \emph{common velocity} and
%where~$\ve u_{1d}$ and~$\ve u_{2d}$ are the particles' respective
%\emph{differential velocities.}  That the particles are \emph{elastic}
%implies by definition that, if~$\ve u_1'$ and~$\ve u_2'$ represent the
%particles' respective velocities after the collision,
%\bq{prob:900:225}
%  \begin{split}
%    u_d' &= u_d, \\
%    \ve u_d &\equiv \ve u_2 - \ve u_1 = \ve u_{2d} - \ve u_{1d}, \\
%    \ve u_d' &\equiv \ve u_2' - \ve u_1' = \ve u_{2d}' - \ve u_{1d}',
%  \end{split}
%\eq
%which is to assert that such particles diverge after a collision as fast
%as they converge before it.  The physical \emph{law of
%conservation of momentum,} which the model accepts as a postulate,
%demands that the collision not alter the quantity $m_1 \ve u_1 + m_2 \ve
%u_2$.  In our present notation we will style this demand as
%\bq{prob:900:230}
%  \ve u_c' = \ve u_c.
%\eq
%Rearranging and combining the lines of~(\ref{prob:900:220}) and
%using the symbol~$\ve u_d$ from~(\ref{prob:900:225}),
%\bq{prob:900:235}
%  \begin{split}
%    \ve u_{1d} &= -\frac{ m_2 }{ m_1 + m_2 } \ve u_d, \\
%    \ve u_{2d} &= +\frac{ m_1 }{ m_1 + m_2 } \ve u_d,
%  \end{split}
%\eq
%which says among other things that the paths the particles
%\emph{differentially} travel run parallel to one another.  For
%completeness' sake, let us write here in the manner
%of~(\ref{prob:900:220}) that
%\bq{prob:900:221}
%  \begin{split}
%    \ve u_1' &= \ve u_{c} + \ve u_{1d}', \\
%    \ve u_2' &= \ve u_{c} + \ve u_{2d}'; \\
%  \end{split}
%\eq
%and in the manner of~(\ref{prob:900:235}) that
%\bq{prob:900:236}
%  \begin{split}
%    \ve u_{1d}' &= -\frac{ m_2 }{ m_1 + m_2 } \ve u_d', \\
%    \ve u_{2d}' &= +\frac{ m_1 }{ m_1 + m_2 } \ve u_d';
%  \end{split}
%\eq
%since per~(\ref{prob:900:230}) the collision cannot alter the particles'
%shared common velocity and since, otherwise, the same physical
%considerations apply after the collision as before.  Let us also note
%that~(\ref{prob:900:225}), (\ref{prob:900:235}) and~(\ref{prob:900:236})
%together imply that
%\bq{prob:900:238}
%  \begin{split}
%    u_{1d}' &= u_{1d}, \\
%    u_{2d}' &= u_{2d}.
%  \end{split}
%\eq
%
%\index{angle of collision}
%\index{collision!angle of}
%Imagine at the moment of collision a line connecting the two particles'
%centers.  Recall that, differentially, the particles travel along
%parallel paths; which is to say that the particles share a single
%differential direction of travel~$\vu u_d$.  Let~$\alpha$ be the angle
%between the differential direction~$\vu u_d$ and the line connecting the
%centers, an angle we will call the \emph{angle of collision,} such that
%$\alpha = 0$ would imply a direct, head-on collision and $\alpha =
%2\pi/4$, a merely grazing collision.  The particles' rigidity and
%spherical shapes then imply the \emph{angle of deflection}
%\bq{prob:900:240}
%  \begin{split}
%    \theta &= \frac{2\pi}{2} - 2\alpha, \\
%    \cos\theta &\equiv \vu u_d \cdot \vu u_d'.
%  \end{split}
%\eq
%As it happens, under the foregoing conditions the PDF of~$\theta$ will
%turn out to be
%\bq{prob:900:245}
%  f_\theta(\theta) = \frac{\sin\theta}{2},
%\eq
%a fact we will prove momentarily.  What we wish to notice now is
%that~(\ref{prob:900:245}) implies, remarkably, that \emph{collision
%unbiasedly randomizes the differential direction of separation~$\vu
%u_d'$}.  To see the implication, imagine a large sphere of radius~$r$,
%centered on the point of collision, moving along with the particles at
%the common rate~$\ve u_c$ (we note incidentally, in advance of the next
%paragraph, that $r \gg \rho_{\mr{max}}$, but we've not defined the
%latter symbol yet).  Which region of the sphere's surface is the one
%particle or the other most likely to exit through?  The answer is that,
%inasmuch $dS = 2\pi r \sin\theta \,d\theta$ is a differential area of
%the sphere's surface,~(\ref{prob:900:245}) implies that no region of the
%surface is preferred.  With respect to the differential direction of
%travel, the collision of ideal particles apparently is a perfect
%randomizer.  This is true even though, according
%to~(\ref{prob:900:225}), (\ref{prob:900:235}) and~(\ref{prob:900:236}),
%such a collision does not alter the particles' differential speeds at
%all.
%
%\index{inertial frame of reference}
%\index{frame of reference, inertial}
%The foregoing however relies on the assertion, which we will now
%support, that~(\ref{prob:900:245}) were true.  We shall find the
%assertion easier to support if first we adopt the \emph{inertial frame
%of reference} of a nonaccelerating observer who moves along with the
%particles at the common rate~$\ve u_c$, an observer who regards himself
%to be standing at rest---and, thus, from whose point of view $\ve u_c =
%0$ and the differential velocities are the only velocities there are.
%Adopting the observer's perspective,%
%\footnote{
%  The inertial frame of reference is a powerful engine for the
%  generation of physical insight
%  % d-i-a-g-n
%  (and an equally powerful engine for the generation of metaphysical
%  confusion, when the inertial frame is mistaken for a teleological
%  principle, but let that pass~\cite{Feser}),
%  because physical modeling usually postulates, and physical observation
%  generally supports, that \emph{no inertial frame of reference is
%  privileged}---which, loosely speaking, is to assert that every
%  nonaccelerating observer is equally justified in regarding himself to
%  stand at rest, defining the velocities of objects under observation in
%  this respect.  The book you are reading only touches on the concept,
%  but you will find much more about it in a good introductory physics
%  book like \cite{Halliday/Resnick} or \cite{Feynman}.
%}
%let the symbol~$\rho_{\mr{max}}$ represent the sum of the radii of the
%two particles---which is the greatest distance that can possibly
%separate the particles' parallel paths if the particles are to
%collide---and sketch a circle of radius~$\rho_{\mr{max}}$, perpendicular
%to~$\vu u_d$, centered on particle~1 at the moment of collision.  If the
%particles are indeed to collide, particle~2's path must pass
%(perpendicularly) through the circle, but there seems to be no
%particular reason the path should prefer one region of
%the circle over another to pass through.  Considering that $dS = 2\pi \rho
%\,d\rho$ is a differential area of the circle, we surmise therefore that
%\[
%  f_\rho(\rho) = \frac{2\rho}{\rho_{\mr{max}}^2},
%  \ \ 0 \le \rho < \rho_{\mr{max}}.
%\]
%But $\rho = \rho_{\mr{max}}\sin\alpha$, so
%\[
%  f_\rho(\rho) = \frac{2\sin\alpha}{\rho_{\mr{max}}}.
%\]
%Multiplying by $d\rho = \rho_{\mr{max}}\cos\alpha \,d\alpha$ and
%applying~(\ref{prob:080:11}),
%\[
%  f_\rho(\rho) \,d\rho = 2\sin\alpha\cos\alpha = \sin 2\alpha \,d\alpha,
%\]
%in which we have applied a trigonometric identity of
%Table~\ref{trig:275:table}.  According then to~(\ref{prob:080:11}),
%\[
%  f_\alpha(\alpha) = \sin 2\alpha,
%\]
%which since $\sin 2\alpha = \sin [(2\pi/2) - \theta] = \sin\theta$
%and $\left|d\alpha\right| = \left|d\theta\right|\!/2$
%implies~(\ref{prob:900:245}) that was to be proved.
%
%\index{conjecture}
%Leaving the observer's inertial frame of reference and returning to our
%original perspective, we should like to deduce from---or at least to
%impute and test according to---the foregoing considerations the
%distribution of speeds of the particles in an ideal gas.  Deduction is
%hard.  Imputation is a matter of intelligent guessing, but after a
%patient intellectual labor of several false tries we might at last
%impute the not implausible distribution
%\bq{prob:900:520}
%  f_{u_{\ell i}}(u_{\ell i}) =
%  \Omega\left( \frac{m_\ell^{1/2} u_{\ell i}}{\hat\sigma_u} \right),
%\eq
%not for the particles' speeds as such but for
%the rectangular components~%
%$u_{1x}$,
%$u_{1y}$,
%$u_{1z}$,
%$u_{2x}$,
%$u_{2y}$ and~%
%$u_{2z}$
%of the particles' velocities,
%in which $\ell=1,2,$ stands for the $\ell$th particle and, in the
%Einstein notation of \S~\ref{vector:240}, $i=x,y,z,$ indicates one of
%the three cardinal directions; and furthermore in which~$\hat\sigma_u$ is a
%constant peculiar to the gas in question ($\hat\sigma_u$ is
%related to the gas' temperature, but we'll not develop the relationship
%here).  The conjecture will turn out to be right.  Really interested in
%the particles' speeds, though, we observe according to
%\S~\ref{prob:400.40} and its~(\ref{prob:400:40})
%that~(\ref{prob:900:520}) implies that the speeds~$u_1$ and~$u_2$ must
%obey the Maxwell distribution
%\bq{prob:900:530}
%  f_{u_\ell}(u_\ell) = \frac{2(m_\ell^{1/2} u_\ell/\hat\sigma_u)^2}{\sqrt{2\pi}}
%    \exp\left[ -\frac{ (m_\ell^{1/2} u_\ell/\hat\sigma_u)^2 }{2} \right],
%\eq
%the particles traveling in unbiasedly random directions.
%Either~(\ref{prob:900:520}) or
%% b-a-d b-r-e-a-k
%(\ref{prob:900:530})
%thus suffices to state the conjecture.  We will
%prefer~(\ref{prob:900:530}).
%
%The conjecture~(\ref{prob:900:530}) however remains but a conjecture.
%To prove it---in the spirit of the demonstration \S~\ref{prob:100} has
%earlier given of~(\ref{prob:100:20})---we will demonstrate that
%collisions of particles do not alter the distribution.  That is, we
%will demonstrate that~(\ref{prob:900:530}) implies that
%\bq{prob:900:540}
%  f_{u_\ell'}(\cdot) =
%  f_{u_\ell}(\cdot),
%\eq
%as follows.  According to~(\ref{prob:900:221}), $\ve u_\ell' = \ve u_{c} +
%\ve u_{\ell d}'$.  However, we already learned as~(\ref{prob:900:238}) that
%$u_{\ell d}' = u_{\ell d}$ and we have already seen that \emph{collision
%unbiasedly randomizes the differential direction of separation;} so we
%have now that
%\[
%  \ve u_\ell' = \ve u_{c} + \vu a u_{\ell d},
%\]
%where~$\vu a$ represents the unbiased random direction.  From this, for
%$\ve u_1'$ by successive steps,
%\bqb
%  \ve u_1'
%  &=&
%    \frac{ m_1 \ve u_1 + m_2 \ve u_2 }{ m_1 + m_2 }
%    - \vu a
%    \frac{ m_2 }{ m_1 + m_2 }
%    \left| \ve u_2 - \ve u_1 \right|
%  \\&=&
%    \frac{
%      m_1 \ve u_1 + m_2\left(
%        \ve u_2 - \vu a \left| \ve u_2 - \ve u_1 \right|
%      \right)
%    }{ m_1 + m_2 }
%\eqb
%From this. for $u_{1z}'$ by successive steps,
%\bqb
%  u_{1z}'
%  &=&
%    \frac{
%      m_1 u_{1z} + m_2\left(
%        u_{2z} - a_z \left| \ve u_2 - \ve u_1 \right|
%      \right)
%    }{ m_1 + m_2 }
%\eqb

