\documentclass[11pt,english]{article}
\usepackage{mathpazo}
\usepackage[a4paper]{geometry}
\geometry{verbose,tmargin=3cm,bmargin=3cm,lmargin=3cm,rmargin=3cm}
\usepackage{array}
\usepackage{booktabs}
\usepackage{multirow}

\newcommand{\noun}[1]{\textsc{#1}}
%% Because html converters don't know tabularnewline
\providecommand{\tabularnewline}{\\}

\usepackage{babel}
\begin{document}

\title{The extra package\\
(a collection of various convenience functions for hansl programming) }

\date{October 2018, release 0.5, this is still small but growing... }

\author{The \noun{gretl} team\thanks{Currently co-ordinated by Sven Schreiber.}}

\maketitle
\tableofcontents{}

\section{Usage}

This package is intended for hansl scripting, not for gretl's GUI.
(But of course other contributed function packages that make use of
functions in extra.gfn can provide GUI access for themselves.)

The usual one-time requirement is to do \texttt{pkg install extra.zip}
to get a copy on the local system (or install it via gretl's graphical
mechanism), and then in the respective hansl script have a line \texttt{include
extra.gfn}.

Note that functions that are exact lookalikes of Matlab functions
do not live here, but would go into the \texttt{matlab\_utilities}
package.

\section{Functions working without a dataset}

\subsection{commute}

Arguments: \texttt{matrix A, int m, int n} (optional), \texttt{bool
post} (optional)

\noindent Return type: \texttt{matrix}

Returns $A$ premultiplied by $K_{mn}$ (the commutation matrix; more
efficient than explicit multiplication). In particular, \texttt{commute(vec(B),
rows(B), cols(B))} gives $vec(B')$. The optional argument $n$ defaults to
$m$ (giving $K_{mm}=K_{m}$). If the optional arg \texttt{post} is
non-zero, then does post-multiplication ($A\times K_{mn}$). 

\subsection{eliminate}

Arguments: \texttt{matrix vecA}

Each column of the input vecA is assumed to come from the operation vec(A) 
on a square matrix, thus rows(vecA) must be a square number.
Returns vech(A), which is the result of pre-multiplying vec(A) with the 
"elimination" matrix $L_m$.
If vecA has several columns, each column is 
treated separately as described above (and the results stacked side-by-side).

\noindent Return type: \texttt{matrix}


\subsection{duplicate}

Arguments: \texttt{matrix vechA}

The input is a vector assumed to come from an operation like vech(A).
Returns vec(A), which is the result of pre-multiplying vech(A) with the 
"duplication" matrix $D_m$. If vechA has several columns, each column is 
treated separately as described above (and the results stacked side-by-side).

\noindent Return type: \texttt{matrix}

\subsection{nearPSD}

Arguments: \texttt{matrix} pointer \texttt{{*}m}, \texttt{scalar epsilon}
(optional) 

\noindent Return type: \texttt{scalar}

Forces the matrix $m$ into the positive semi-definite region. Algorithm
ported from ``DomPazz'' in Stackoverflow,
apparently mimicking the nearPD() function in R. Because of re-scaling
(to correlation matrix), the \texttt{epsilon} criterion value should
implicitly apply to the correlation-based eigenvalues. The return
value 0 or 1 indicates whether \texttt{m} was altered or not. 

\subsection{scores2x2}

Arguments: \texttt{matrix in, bool verbose} (optional)

\noindent Return type: \texttt{matrix}

Computes some standard score measures for a $2\times 2$ contingency
table of the form:

\begin{tabular}{cccc}
\toprule 
 &  & \multicolumn{2}{c}{Observed}\tabularnewline
 &  & 1 & 0\tabularnewline
\midrule
\multirow{2}{*}{Predicted} & 1 & h(its) & f(alse)\tabularnewline
 & 0 & m(iss) & z(eros)\tabularnewline
\bottomrule
\end{tabular}

and $n=h+f+m+z$ (total observations). Returns a column vector with
the following elements:
\begin{enumerate}
\item POD / prob of detection = $h/(h+m)$
\item POFD / prob of false detection = $f/(f+z)$ 
\item HR / hit rate = $(h+z)/n$ 
\item FAR / false alarm rate = $f/(h+f)$ 
\item CSI / critical success index = $h/(h+f+m)$
\item OR / odds ratio = $h*z/(f*m)$
\item BIAS / bias score =$(h+f)/(h+m)$
\item TSS / true skill stat =$POD-POFD$\\
The TSS is also known as the Hanssen-Kuipers score, and is = $h/(h+m)-f/(f+z)$.
\item HSS / Heidke skill score = $2*(h*z-f*m)/((h+m)*(m+z)+(h+f)*(f+z))$
\item ETS / equitable threat score = $(h*z-f*m)/((f+m)*n+(h*z-f*m))$
\item PRC / precision = $h/(h+f)$
\item FSC / F-Score = $2*(PRC*POD)/(PRC+POD)$\\
The F-Score can also be expressed as $2*h/(1+h+m)$.
\end{enumerate}
The input is always sanitized by taking the upper 2x2 part, using
absolute values, and integer-ization. Warnings are issued if \texttt{verbose}
is 1. 

\subsection{sepstr2arr}

% Arguments: \texttt{string in, string sep }(optional)

% \noindent Return type: \texttt{strings} array

Transforms comma-separated string to an array of strings. 
(Retired as of v0.5, use the extended functionality of gretl's built-in 
strsplit function.)
%(The comma
%as separation character can be overridden with the sep argument, but
%only the first character is used.) 

\subsection{truncnorm}

Arguments: \texttt{int n, scalar m, scalar sigma, scalar below, scalar
above}

\noindent Return type: \texttt{matrix}

Generates $n$ truncated normal random values. Specify mean \texttt{m}
and std.\,dev. \texttt{sigma}, and the left/right truncation values
\texttt{below} and \texttt{above}. (Pass NA for any one of them to
skip the respective truncation.) Returns a col vector of values.

\subsection{zeroifclose}

Arguments: \texttt{matrix} pointer \texttt{{*}m}, \texttt{scalar thresh}
(optional) 

\noindent Return type: \texttt{scalar}

Sets elements of \texttt{m} to zero if they are really close. The
return value 0 or 1 indicates whether \texttt{m} was altered or not.

\subsection{WSRcritical}

Arguments: \texttt{int n, scalar prob }(optional)\texttt{, bool forcenorm}
(optional)

Concerns the distribution of Wilcoxon's signed rank test statistic for
\texttt{n} trials (at least 4). Tries to find the critical values
(low/hi) where the two-sided area to the outside is as close as
possible to the given \texttt{prob} (default: 0.05). (Note that
``outside'' means including the critical
values themselves in the exact/discrete case.) If we end up in the
interior region not covered by the exact table (for \texttt{prob} far
away from 0 and also from 1), we fall back to the normal
approximation. Returned is col vector \{low; hi; epv\}, where epv is
the actual probability mass (close to \texttt{prob} but not equal in
general for small samples). 'low' and 'hi' can be non-integers in the
normal approximation case. The normal approximation instead of the
exact table values can be enforced with the \texttt{forcenorm}
argument (default: zero, do not enforce).

\noindent Return type: \texttt{matrix}

See also the sister function WSRpvalue.

\subsection{WSRpvalue}

Arguments: \texttt{int n, scalar W, bool forcenorm} (optional)

Concerns the distribution of Wilcoxon's signed rank test statistic for
\texttt{n} trials (at least 4), returns $P(X\geq W)$. In the interior
region not covered by the exact table, the true value is $\geq$ 12.5\%
(and $\leq$87.5\%) according to the table used,\footnote{Source of the
  table: Wilfrid J Dixon and Frank J. Massey, Jr., Introduction to
  Statistical Analysis, 2nd ed. (New York: McGraw-Hill, 1957), pp.
  443-444.} so typically based on such values H0 would not be
rejected. We fall back to the normal approximation in this region. In
the extreme outer regions not explicitly covered by the table, the
deviation from 0 or 1 will be smaller than 0.5\% = 0.005. We return
values 0.001 or 0.999 as an approximation here. The test statistic
\texttt{W} should usually be an integer, but in case of bindings it
could be fractional as well; in this case we also fall back to the
normal approximation.

The normal approximation instead of the exact table values can be
enforced with the \texttt{forcenorm} argument (default: zero, do not
enforce).

\noindent Return type: \texttt{scalar}

See also the sister function WSRcritical.

\subsection{powerset}

Arguments: \texttt{strings S}

Computes the powerset of the input S, i.e. all possible combinations 
of the string elements in S. (Including the empty set / 
empty string "".) Each combination yields one string in the output 
array. Being a set, the ordering is not defined and arbitrary. 

\noindent Return type: \texttt{strings} (array)

\section{Functions requiring a dataset}

\subsection{gap\_filler}

Arguments: \texttt{series x, int method }(optional)\texttt{ }

\noindent Return type: \texttt{series}

Simple convenience function to crudely get rid of missing values
interspersed between valid observations. Apart from the first argument
(series) accepts an integer parameter as second argument, whose
meaning is: 0: do nothing, leave the gaps; 1: NAs are replaced with
previous observations; 2: NAs are replaced with a linear
interpolation. Returns the filled series.

Note that the function only replaces NAs between valid observations;
therefore, if the origin series has missing values at the beginning or
the end of the sample, they will be in the returned series too.

\subsection{winsor}

Arguments: \texttt{series x, scalar p} (optional), \texttt{scalar
phi} (optional) 

\noindent Return type: \texttt{series}

Returns a trimmed (``winsorized'') version
of the series, where outliers are replaced with implicit threshold
values. Truncation quantiles are determined according to relative
tail frequencies \texttt{p} and \texttt{phi}. Default lower and upper
frequencies are 0.05, but re-settable with \texttt{p}. Pass \texttt{phi}
in addition to \texttt{p} for an asymmetric trimming, then \texttt{p}
determines only the lower frequency and \texttt{phi} the upper. 

\section{Authors}
\begin{itemize}
\item gap\_filler, commute, eliminate, duplicate, truncnorm, powerset: Jack Lucchetti
\item nearPSD, zeroifclose, scores2x2, WSRcritical, WSRpvalue:
Sven Schreiber 
\item winsor: Sven Schreiber, original code JoshuaHe
\end{itemize}

\section{Changelog }
\begin{itemize}
\item October 2018: 0.5, fix small commute bug; retire sepstr2arr; add powerset,
  eliminate, duplicate
\item February 2018: 0.41, allow non-integer input in WSRpvalue
\item January 2018: 0.4, add WSRcritical, WSRpvalue
\item December 2017: 0.3, add scores2x2; switch to pdf help document
\item September 2017: 0.2, add winsor 
\item July 2017: initial release
\end{itemize}

\end{document}
