% Permission is granted to copy, distribute and/or modify this document
% under the terms of the GNU Free Documentation License, Version 1.3
% or any later version published by the Free Software Foundation;
% with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
% A copy of the license is included in the section entitled "GNU
% Free Documentation License".
%
% Written (C) 2012 Heiko Strathmann

\chapter{Statistical Testing}
This chapter describes \shogun{}'s framework for statistical hypothesis testing. We begin by giving a brief outline of the problem setting in section \ref{sec:hypothesis_testing_into}. Then, we describe methods for two-sample testing for independence testing.

Methods for two-sample testing currently consist of tests based on the \emph{Maximum Mean Discrepancy}, section \ref{sec:mmd_into}. There are two types of tests available, a quadratic time test, which is described in section \ref{sec:mmd_quadratic}; and a linear time test, which is described in section \ref{sec:mmd_linear}. Both come in various flavours.

Independence testing is currently based in the \emph{Hilbert Schmidt Independence Criterion}, which is described in section \ref{sec:hsic_test} along with a test using it.

\section{Statistical Hypothesis Testing}
\label{sec:hypothesis_testing_into}

To set the context, we here briefly describe statistical hypothesis testing. Informally, one defines a hypothesis on a certain domain and then uses a statistical test to check whether this hypothesis is true. Formally, the goal is to reject a so-called \emph{null-hypothesis} $H_0$, which is the complement of an \emph{alternative-hypothesis} $H_A$. 

To distinguish the hypothesises, a test statistic is computed on sample data. Since sample data is finite, this corresponds to sampling the true distribution of the test statistic. There are two different distributions of the test statistic -- one for each hypothesis. The \emph{null-distribution} corresponds to test statistic samples under the model that $H_0$ holds; the \emph{alternative-distribution} corresponds to test statistic samples under the model that $H_A$ holds.

In practice, one tries to compute the quantile of the test statistic in the null-distribution. In case the test statistic is in a high quantile, i.e.\ it is unlikely that the null-distribution has generated the test statistic -- the null-hypothesis $H_0$ is rejected.

There are two different kinds of errors in hypothesis testing:
\begin{itemize}
\item A \emph{type I error} is made when $H_0: p=q$ is wrongly rejected. That is, the tests says that the samples are from different distributions when they are not.
\item A \emph{type II error} is made when $H_A: p=q$ is wrongly accepted. That is, the tests says that the samples are from same distributions when they are from the same.
\end{itemize}
A so called \emph{consistent} test achieves zero type II error for a fixed type I error.

To decide whether to reject $H_0$, one could set a threshold, say at the 95\% quantile of the null-distribution, and reject $H_0$ when the test statistic lies below that threshold. This means that the chance that the samples were generated under $H_0$ are 5\%. We call this number the test power $\alpha$ (in this case $\alpha=0.05$). It is an upper bound on the probability for a type I error. An alternative way is simply to compute the quantile of the test statistic in the null-distribution, the so-called \emph{p-value}, and to compare the p-value against a desired test power, say $\alpha=0.05$, by hand. The advantage of the second method is that one not only gets a binary answer, but also an upper bound on the type I error.

In order to construct a two-sample test, the null-distribution of the test statistic has to be approximated. One way of doing this for any two-sample test is called \emph{bootstrapping}:
\begin{algorithm}
Inputs are:
\begin{itemize}
 \item $X,Y$, sets of samples from $p,q$ of size $m,n$ respectively
\end{itemize}
Output is:
\begin{itemize}
 \item One sample from null-distribution. Simply repeat for more samples.
\end{itemize}

 \begin{algorithmic}[1]
\STATE{$Z \gets \{X,Y\}$}
\STATE{$\hat{Z}=\{\hat{z}_1,...,\hat{z}_{m+n}\}\gets \text{randperm}(Z)$} \qquad (generate a random ordering)
\STATE{$\hat{X}\gets \{\hat{z}_1,...\hat{z}_m\}$}
\STATE{$\hat{Y}\gets \{\hat{z}_{m+1},...\hat{z}_{m+n}\}$}
\RETURN{Test statistic for $\hat{X},\hat{Y}$}
\end{algorithmic}
\caption{Bootstrapping a null-distribution.}
\label{alg:bootstrapping}
\end{algorithm}

Bootstrapping is a useful technique to create ground-truth samples from a null-distribution. However, it is rather costly because the statistic has to be re-computed for every sample. More details will be given when individual tests are described.

\subsection*{Interface for Statistical Testing}
\shogun{} implements statistical testing in the abstract class \shogunclass{CTestStatistic}.
\begin{itemize}
\item Test statistics can be computed with \texttt{compute\_statistic}.
\item P-values for a given statistic can be computed via \texttt{compute\_p\_value}. Results depend on method that is set for approximating null-distribution.
\item Statistic thresholds for a given p-value can be computed via \texttt{compute\_threshold}. Results depend on method that is set for approximating null-distribution.
\item A number of samples can be drawn from the null-distribution using bootstrapping via \texttt{bootstrap\_null}. This will call \texttt{compute\_statistic} a certain number of times while underlying data is modified in such way that the null hypothesis $H_0:p=q$ is true.
\item A complete two-sample test can be computed using \texttt{perform\_test}. There are two different versions of this method:
\begin{itemize}
\item  One without parameters which computes the statistic and returns a p-value for it. This p-value can be used to (not) reject the null hypothesis.
\item One which has a test level $\alpha$ as a parameter and returns \texttt{true} if the null hypothesis is rejected and \texttt{false} otherwise. Obviously, this method is just a simple wrapper of the above one.
\end{itemize}
\end{itemize}

The \texttt{perform\_test} methods are convenience wrappers for \texttt{compute\_statistic} and \texttt{compute\_p\_value}. However, in certain cases, it might be possible to compute statistic and p-value in the same loop which is more efficient. In subclasses of \shogunclass{CTestStatistic}, the first one might be overwritten in order to implement this -- if possible. See class documentations for availability. If an  efficient implementation is not existent, \texttt{perform\_test} simply is a wrapper for  \texttt{compute\_statistic} and \texttt{compute\_p\_value}.

\input{parts/algorithms/statistical_tests-mmd}
\input{parts/algorithms/statistical_tests-hsic}
