\lstset{language=transcript}

\section{Introduction: How Cylc Works}
\label{HowCylcWorks}

\subsection{Scheduling Forecast Suites}
\label{SchedulingForecastSuites}

Environmental forecasting suites generate forecast products from a
potentially large group of interdependent scientific models and
associated data processing tasks. They are constrained by availability
of external driving data: typically one or more tasks will wait on real
time observations and/or model data from an external system, and these
will drive other downstream tasks, and so on. The dependency diagram for
a single forecast cycle point in such a system is a {\em Directed Acyclic
Graph} as shown in Figure~\ref{fig-dep-one} (in our terminology, a {\em
forecast cycle point} is comprised of all tasks with a common {\em cycle
point}, which is the nominal analysis time or start time of the forecast
models in the group). In real time operation processing will consist of
a series of distinct forecast cycle points that are each initiated, after a
gap, by arrival of the new cycle point's external driving data.

From a job scheduling perspective task execution order in such a system
must be carefully controlled in order to avoid dependency violations.
Ideally, each task should be queued for execution at the instant its
last prerequisite is satisfied; this is the best that can be done even
if queued tasks are not able to execute immediately because of resource
contention.

\subsection{EcoConnect}
\label{EcoConnect}

Cylc was developed for the EcoConnect Forecasting System at NIWA
(National Institute of Water and Atmospheric Research, New Zealand).
EcoConnect takes real time atmospheric and stream flow observations, and
operational global weather forecasts from the Met Office (UK), and uses
these to drive global sea state and regional data assimilating weather
models, which in turn drive regional sea state, storm surge, and
catchment river models, plus tide prediction, and a large number of
associated data collection, quality control, preprocessing,
post-processing, product generation, and archiving tasks.\footnote{Future
plans for EcoConnect include additional deterministic regional weather
forecasts and a statistical ensemble.} The global sea state forecast
runs once daily. The regional weather forecast runs four times daily but
it supplies surface winds and pressure to several downstream models that
run only twice daily, and precipitation accumulations to catchment river
models that run on an hourly cycle assimilating real time stream flow
observations and using the most recently available regional weather
forecast. EcoConnect runs on heterogeneous distributed hardware,
including a massively parallel supercomputer and several Linux servers.


\subsection{Dependence Between Tasks}

\subsubsection{Intra-cycle Dependence}
\label{IntracycleDependence}

Most dependence between tasks applies within a single forecast cycle
point. Figure~\ref{fig-dep-one} shows the dependency diagram for a single
forecast cycle point of a simple example suite of three forecast models
({\em a, b,} and {\em c}) and three post processing or product generation
tasks ({\em d, e} and {\em f}). A scheduler capable of handling this
must manage, within a single forecast cycle point, multiple parallel
streams of execution that branch when one task generates output for
several downstream tasks, and merge when one task takes input from several
upstream tasks.

\begin{figure}
    \begin{center}
        \includegraphics[width=6cm]{graphics/png/orig/dep-one-cycle.png}
    \end{center}
    \caption[A single cycle point dependency graph for a simple suite]
    {\scriptsize
    The dependency graph for a single forecast cycle point of a simple
    example suite. Tasks {\em a, b,} and {\em c} represent forecast models,
    {\em d, e} and {\em f} are post processing or product generation
    tasks, and {\em x} represents external data that the upstream
    forecast model depends on.}
    \label{fig-dep-one}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/timeline-one.png}
    \end{center}
    \caption[A single cycle point job schedule for real time operation]
    {\scriptsize
    The optimal job schedule for two consecutive cycle points of our
    example suite during real time operation, assuming that all tasks
    trigger off upstream tasks finishing completely. The horizontal
    extent of a task bar represents its execution time, and the vertical
    blue lines show when the external driving data becomes available.}
    \label{fig-time-one}
\end{figure}

Figure~\ref{fig-time-one} shows the optimal job schedule for two
consecutive cycle points of the example suite in real time operation, given
execution times represented by the horizontal extent of the task bars.
There is a time gap between cycle points as the suite waits on new external
driving data. Each task in the example suite happens to trigger off
upstream tasks {\em finishing}, rather than off any intermediate output
or event; this is merely a simplification that makes for clearer
diagrams.

\begin{figure}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/dep-two-cycles-linked.png}
    \end{center}
    \caption[What if the external driving data is available early?]{\scriptsize If
    the external driving data is available in advance, can we start
    running the next cycle point early?}
    \label{fig-dep-two-linked}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=6cm]{graphics/png/orig/timeline-one-c.png}
    \end{center}
    \caption[Attempted overlap of consecutive single-cycle-point job
    schedules]{\scriptsize A naive attempt to overlap two consecutive cycle
    points using the single-cycle-point dependency graph. The red shaded
    tasks will fail because of dependency violations (or will not be able to
    run because of upstream dependency violations).}
    \label{fig-overlap}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/timeline-one-a.png}
    \end{center}
    \caption[The only safe multi-cycle-point job schedule?]
    {\scriptsize The best that can be done {\em in general} when
    inter-cycle dependence is ignored.}
    \label{fig-job-no-overlap}
\end{figure}

Now the question arises, what happens if the external driving data for
upcoming cycle points is available in advance, as it would be after a
significant delay in operations, or when running a historical case
study?  While the forecast model {\em a} appears to depend only on the
external data {\em x} at this stage of the discussion, in fact it would
typically also depend on its own previous instance for the model {\em
background state} used in initializing the new forecast. Thus, as
alluded to in Figure~\ref{fig-dep-two-linked}, task {\em a} could in
principle start
as soon as its predecessor has finished. Figure~\ref{fig-overlap}
shows, however, that starting a whole new cycle point at this point is
dangerous - it results in dependency violations in half of the tasks in
the example suite. In fact the situation could be even worse than this
- imagine that task {\em b} in the first cycle point is delayed for some
reason {\em after} the second cycle point has been launched. Clearly we must
consider handling inter-cycle dependence explicitly or else agree not to
start the next cycle point early, as is illustrated in
Figure~\ref{fig-job-no-overlap}.

\subsubsection{Inter-Cycle Dependence}
\label{InterCyclePointDependence}

Forecast models typically depend on their own most recent previous
forecast for background state or restart files of some kind (this is
called {\em warm cycling}) but there can also be inter-cycle dependence
between different tasks. In an atmospheric forecast analysis suite, for
instance, the weather model may generate background states for observation
processing and data-assimilation tasks in the next cycle point as well as for
the next forecast model run. In real time operation inter-cycle
dependence can be ignored because it is automatically satisfied when one cycle
point finishes before the next begins. If it is not ignored it drastically
complicates the dependency graph by blurring the clean boundary between
cycle points. Figure~\ref{fig-dep-multi} illustrates the problem for our
simple example suite assuming minimal inter-cycle dependence: the warm
cycled models ($a$, $b$, and $c$) each depend on their own previous instances.

For this reason, and because we tend to see forecasting suites in terms of
their real time characteristics, other metaschedulers have ignored
inter-cycle dependence and are thus restricted to running entire cycle
points in sequence at all times. This does not affect normal real time
operation but it can be a serious impediment when advance availability of
external driving data makes it possible, in principle, to run some tasks from
upcoming cycle points before the current cycle point is finished - as was
suggested at the end of the previous section. This can occur, for instance,
after operational delays (late arrival of external data, system maintenance,
etc.) and to an even greater extent in historical case studies and parallel
test suites started behind a real time operation. It can be a serious problem
for suites that have little downtime between forecast cycle points and
therefore take many cycle points to catch up after a delay. Without taking
account of inter-cycle dependence, the best that can be done, in
general, is to reduce the gap between cycle points to zero as shown in
Figure~\ref{fig-job-no-overlap}. A limited crude overlap of the single cycle
point job schedule may be possible for specific task sets but the allowable
overlap may change if new tasks are added, and it is still dangerous: it
amounts to running different parts of a dependent system as if they were not
dependent and as such it cannot be guaranteed that some unforeseen delay in
one cycle point, after the next cycle point has begun, (e.g.\ due to resource
contention or task failures) won't result in dependency violations.

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/dep-multi-cycle.png}
    \end{center}
    \caption[The complete multi-cycle-point dependency graph]
    {\scriptsize The complete dependency graph for the example suite, assuming
    the least possible inter-cycle dependence: the forecast models ($a$,
    $b$, and $c$) depend on their own previous instances. The dashed arrows
    show connections to previous and subsequent forecast cycle points.}
    \label{fig-dep-multi}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=6cm]{graphics/png/orig/timeline-two-cycles-optimal.png}
    \end{center}
    \caption[The optimal two-cycle-point job schedule]
    {\scriptsize The optimal two cycle job schedule when the next cycle's driving data is available in
    advance, possible in principle when inter-cycle dependence is
    handled explicitly.}
    \label{fig-optimal-two}
\end{figure}

Figure~\ref{fig-optimal-two} shows, in contrast to
Figure~\ref{fig-overlap}, the optimal two cycle point job schedule obtained by
respecting all inter-cycle dependence. This assumes no delays due to
resource contention or otherwise - i.e.\ every task runs
as soon as it is ready to run. The scheduler running
this suite must be able to adapt dynamically to external conditions
that impact on multi-cycle-point scheduling in the presence of
inter-cycle dependence or else, again, risk bringing the system down
with dependency violations.

\begin{figure}
    \begin{center}
        \includegraphics[width=12cm]{graphics/png/orig/timeline-three.png}
    \end{center}
    \caption[Comparison of job schedules after a delay]{\scriptsize Job
    schedules for the example suite after a delay of almost one whole
    forecast cycle point, when inter-cycle dependence is
    taken into account (above the time axis), and when it is not
    (below the time axis). The colored lines indicate the time that
    each cycle point is delayed, and normal ``caught up'' cycle points
    are shaded gray.}
    \label{fig-time-three}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/timeline-two.png}
    \end{center}
    \caption[Optimal job schedule when all external data is
    available]{\scriptsize Job schedules for the example suite in case study
    mode, or after a long delay, when the external driving data are
    available many cycle points in advance. Above the time axis is the optimal
    schedule obtained when the suite is constrained only by its true
    dependencies, as in Figure \ref{fig-dep-two-linked}, and underneath
    is the best that can be done, in general, when inter-cycle
    dependence is ignored.}
    \label{fig-time-two}
\end{figure}

To further illustrate the potential benefits of proper inter-cycle
dependency handling, Figure~\ref{fig-time-three} shows an operational
delay of almost one whole cycle point in a suite with little downtime between
cycle points. Above the time axis is the optimal schedule that is possible in
principle when inter-cycle dependence is taken into account, and below
it is the only safe schedule possible {\em in general} when it is ignored.
In the former case, even the cycle point immediately after the delay is hardly
affected, and subsequent cycle points are all on time, whilst in the latter
case it takes five full cycle points to catch up to normal real time
operation.

%Note that simply overlapping the single cycle point schedules of
%Figure~\ref{fig-time-one} from the same start point would have resulted
%in dependency violation by task {\em c}.

Similarly, Figure~\ref{fig-time-two} shows example suite job schedules
for an historical case study, or when catching up after a very long
delay; i.e.\ when the external driving data are available many cycle
points in advance. Task {\em a}, which as the most upstream forecast
model is likely to be a resource intensive atmosphere or ocean model,
has no upstream dependence on co-temporal tasks and can therefore run
continuously, regardless of how much downstream processing is yet to be
completed in its own, or any previous, forecast cycle point (actually,
task {\em a} does depend on co-temporal task {\em x} which waits on the
external driving data, but that returns immediately when the data is
available in advance, so the result stands). The other forecast models
can also cycle continuously or with a short gap between, and some
post processing tasks, which have no previous-instance dependence, can
run continuously or even overlap (e.g.\ {\em e} in this case). Thus,
even for this very simple example suite, tasks from three or four
different cycle points can in principle run simultaneously at any given
time.

In fact, if our tasks are able to trigger off internal outputs of
upstream tasks (message triggers) rather than waiting on full completion,
then successive instances of the forecast models could overlap as well (because
model restart outputs are generally completed early in the forecast) for an
even more efficient job schedule.

%Finally, we note again that a good job scheduler should be able to
%dynamically adapt to delays in any part of the suite due to resource
%contention, varying run times, or anything else that will inevitably
%modify the depicted job schedules.

\subsection{The Cylc Scheduling Algorithm}
\label{TheCylcSchedulingAlgorithm}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/task-pool.png}
    \end{center}
    \caption[The cylc task pool]{\scriptsize How cylc sees a suite, in
    contrast to the multi-cycle-point dependency graph of
    Figure~\ref{fig-dep-multi}.
    Task colors represent different cycle points, and the small squares
    and circles represent different prerequisites and outputs. A task
    can run when its prerequisites are satisfied by the outputs
    of other tasks in the pool.}
    \label{fig-task-pool}
\end{figure}

Cylc manages a pool of proxy objects that represent the real tasks in a
suite. Task proxies know how to run the real tasks that they represent,
and they receive progress messages from the tasks as they run (usually
reports of completed outputs). There is no global cycling mechanism to
advance the suite; instead individual task proxies have their own
private cycle point and spawn their own successors when the time is
right. Task proxies are self-contained - they know their own
prerequisites and outputs but are not aware of the wider suite.
Inter-cycle dependence is not treated as special, and the task pool can
be populated with tasks with many different cycle points. The task pool
is illustrated in Figure~\ref{fig-task-pool}. {\em Whenever any task
changes state due to completion of an output, every task checks to see
if its own prerequisites have been satisfied.}
%\footnote{In fact this dependency negotiation goes through a broker
%object (rather than every task literally checking every other task)
%which scales as $n$ (rather than $n^2$) where $n$ is the number of task
%proxies in the pool.}
In effect, cylc gets a pool of tasks to self-organize by negotiating
their own dependencies so that optimal scheduling, as described in the
previous section, emerges naturally at run time.

%\pagebreak
\section{Cylc Screenshots}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.8\textwidth]{graphics/png/orig/gcylc-graph-and-dot-views.png}
    \end{center}
\caption[gcylc graph and dot views]{\scriptsize gcylc graph and dot views.}
\label{fig-gcylc-1}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.8\textwidth]{graphics/png/orig/gcylc-text-view.png}
    \end{center}
\caption[gcylc text view]{\scriptsize gcylc text view.}
\label{fig-gcylc-2}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/gscan.png}
    \end{center}
\caption[gscan multi-suite state summary GUI]{\scriptsize gscan multi-suite state summary GUI.}
\label{fig-gscan}
\end{figure}


\begin{figure}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/ecox-1.png}
    \end{center}
\caption[A large-ish suite graphed by cylc]{\scriptsize A large-ish suite graphed by cylc.}
\label{fig-ecox-1}
\end{figure}

% dump floats
\clearpage

%\pagebreak

\section{Required Software}
\label{Requirements}

\begin{myitemize}
    \item {\bf Cylc}, the version associated with this document is: \input{cylc-version.txt}. % generated each time by doc/process
        \newline Cylc can be downloaded from from GitHub: \url{https://cylc.github.com/cylc}
    \item {\bf OS: A Linux or Unix variant (including, reportedly, Apple OS X).}
    \item {\bf The Python Language, version 2.6 or later, but not 3.x yet}.
        \newline \url{https://python.org}
    \item {\bf sqlite (a server-less, zero-configuration, SQL database
        engine}). The Python API to sqlite is part of the Python standard
        library. The command line interface is likely included in your Linux
        distribution already. Cylc generates an sqlite database for each suite
        as it runs, to record task events and status.
        \newline \url{http://sqlite.org}
\end{myitemize}

The following packages are technically optional as you can construct
and run cylc suites without dependency graphing, the gcylc GUI, or
template processing {\em but this is not recommended, and without
Jinja2 you will not be able to run many of the example suites}:

\begin{myitemize}
    \item {\bf PyGTK}, a Python wrapper for the GTK+ GUI toolkit,
        required for the gcylc GUI.
        PyGTK is included in most Linux Distributions.
        \newline \url{http://www.pygtk.org}
    \item {\bf Graphviz} (latest tested 2.28.0) and {\bf Pygraphviz}
        (latest tested 1.1), a graph layout engine and a Python
        interface to it.
        \newline \url{http://www.graphviz.org}
        \newline \url{http://pygraphviz.github.io/}
    \item {\bf Jinja2}, a template processor for Python (latest tested:
        2.6). Jinja2 allows use of general programming constructs in
        suite definitions.
        \newline \url{http://jinja.pocoo.org/docs}
\end{myitemize}

If you use a binary package manager to install graphviz you may also
need a couple of {\em devel} packages for the pygraphviz build:

\begin{myitemize}
    \item python-devel
    \item graphviz-devel
\end{myitemize}

This user guide can be generated from the \LaTeX source by
running \lstinline=make= in the top level cylc directory after download.
The following \TeX packages are required (but note that the exact
packages required may be somewhat OS or distribution-dependent):

\begin{myitemize}
    \item texlive
    \item texlive-tocloft
    \item texlive-framed
    \item texlive-preprint (for fullpage.sty, formerly in tetex)
\end{myitemize}
And for HTML versions of the User Guide:

\begin{myitemize}
    \item texlive-tex4ht
    \item ImageMagick (for image scaling)
\end{myitemize}

Note: Older versions of cylc require Pyro-3 (Python Remote Objects). Current
version of cylc includes a modified version of Pyro 3.16 in its distribution.
Pyro is used for network communication between server processes (cylc suites)
and client programs (running tasks, the gcylc GUI, user commands). \newline
\url{https://pypi.python.org/pypi/Pyro/}

Finally, cylc makes heavy use of Python {\em ordered dictionary}
data structures. Significant speedup in parsing large suites can be had
by installing the
fast C-coded \lstinline=ordereddict= module by Anthon van der Neut:
\begin{myitemize}
    \item {\bf ordereddict} (latest tested 0.4.5) \newline
        \url{https://anthon.home.xs4all.nl/Python/ordereddict/}
\end{myitemize}
{\em This module is currently included with cylc} under
\lstinline=$CYLC_DIR/ext=, and is built by the top level cylc Makefile.
If you install the resulting library appropriately cylc will
automatically use it in place of a slower Python implementation of
the ordered dictionary structure.

\subsection{Known Version Compatibility Issues}

Cylc should run ``out of the box'' on recent Linux distributions.

\subsubsection{Apple Mac OSX}

It has been reported that cylc runs fine on OSX 10.6 SnowLeopard, but on
OSX 10.7 Lion there is an issue with constructing proper FQDNs (Fully
Qualified Domain Names) that requires a change to the DNS service.
Here's how to solve the problem:

\begin{myitemize}
    \item Edit \lstinline=/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist= \newline
        by adding \lstinline=<string>-AlwaysAppendSearchDomains</string>= after line 16:
\begin{lstlisting}
<key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/mDNSResponder</string>
    <string>-launchd</string>
        <string>-AlwaysAppendSearchDomains</string>
  </array>
\end{lstlisting}
    \item Now unload and reload the mDNSResponder service:
\begin{lstlisting}
% sudo launchctl unload -w
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
% sudo launchctl load -w
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
\end{lstlisting}
\end{myitemize}


\subsection{Other Software Used Internally By Cylc}

Cylc has incorporated a custom-modified version the {\em xdot} graph
viewer (\url{https://github.com/jrfonseca/xdot.py}, LGPL license).


\section{Installation}
\label{Installation}

\subsection{Install The External Dependencies}

First install graphviz, Pygraphviz, Jinja2, \TeX, and ImageMagick
using the package manager on your system if possible; otherwise download
the packages manually and follow their native installation documentation.
On a modern Linux system, this is very easy. For example, to install
cylc-5.1.0 on the Fedora 18 Linux distribution:

\lstset{language=transcript}
\begin{lstlisting}
shell$ yum install graphviz       # (2.28)
shell$ yum install graphviz-devel # (for pgraphviz build)
shell$ yum install python-devel   # (ditto)

# TeX packages, and ImageMagick, for generating the Cylc User Guide:
shell$ yum install texlive
shell$ yum install texlive-tex4ht
shell$ yum install texlive-tocloft
shell$ yum install texlive-framed
shell$ yum install texlive-preprint
shell$ yum install ImageMagick

# Python packages:
shell$ easy_install pyro   # (3.16)
shell$ easy_install Jinja2 # (2.6)
shell$ easy_install pygraphviz

# (sqlite 3.7.13 already installed on the system)
\end{lstlisting}

If you do not have root access on your intended cylc host machine and
cannot get a sysadmin to do this at system level, see~\ref{LocalInstall} for
tips on installing everything to a local user account.

Now check that everything other than the \LaTeX packages is
installed properly:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc check-software
Checking for Python >= 2.6 ... found 2.7.3 ... ok
Checking for non-Python packages:
 + Graphviz ... ok
 + sqlite ... ok
Checking for Python packages:
 + Jinja2 ... ok
 + pygraphviz ... ok
 + pygtk ... ok
\end{lstlisting}
If this command reports any errors then the packages concerned are not
installed, not in the system Python search path, or (for a local
install) not present in your \lstinline=$PYTHONPATH= variable.

\subsection{Install Cylc}
\label{InstallCylc}

Cylc installs into a normal user account, as an unpacked release tarball
or a git repository clone. See the \lstinline=INSTALL= file in the source
tree for instructions (also listed in~\ref{INSTALL}).

\subsubsection{Create A Site Config File}

Site and user global config files define some important parameters that affect
all suites, some of which may need to be customized for your site.
See~\ref{SiteAndUserConfiguration} for how to generate an initial site file and
where to install it. All legal site and user global config items are defined
in~\ref{SiteRCReference}.

\subsubsection{Configure Site Environment on Job Hosts}

If your users submit task jobs to hosts other than the hosts they use to run
their suites, you should ensure that the job hosts have the correct environment
for running cylc. A cylc suite generates task job scripts that normally invoke
\lstinline=bash=. The job will attempt to source the first of these files it
finds to set up its environment:

\begin{myitemize}
\item \lstinline=${HOME}/.cylc/job-init-env.sh=
\item \lstinline=${CYLC_DIR}/conf/job-init-env.sh=
\item \lstinline=${CYLC_DIR}/conf/job-init-env-default.sh=
\end{myitemize}

The \lstinline=${CYLC_DIR}/conf/job-init-env-default.sh= file is provided in
the cylc distribution, and will attempt to source \lstinline=/etc/profile= and
\lstinline=${HOME}/.profile=. If this behaviour is not desirable, you should
override it by adding a \lstinline=${CYLC_DIR}/conf/job-init-env.sh= file and
populate it with the appropriate contents.

\subsection{Automated Tests}
\label{RTAST}

The cylc test battery is primarily intended for developers to check that
changes to the source code don't break existing functionality. Note that
some test failures can be expected to result from suites timing out,
even if nothing is wrong, if you run too many tests in parallel. See
\lstinline=cylc test-battery --help=.

\subsection{Local User Installation}
\label{LocalInstall}

It is possible to install cylc and all of its software prerequisites
under your own user account. Cylc itself is already designed to be
installed into a normal user account, just follow the instructions above
in~\ref{InstallCylc}. For the other packages, depending on the
installation method used for each, it is just a matter of learning
how to change the default install path prefix from, for example,
\lstinline=/usr/local= to \lstinline=$HOME/installed/usr/local= and
then ensuring that the resulting local package paths are set properly
in your \lstinline=PYTHONPATH= environment variable.

\subsubsection{Some Guidelines}

\lstset{language=transcript}

\begin{myitemize}
\item For \lstinline=python setup.py install= installation:
\begin{lstlisting}
shell$ python setup.py install --prefix=/my/local/install/path
\end{lstlisting}
\item For building graphviz from source:
\begin{lstlisting}
shell$ ./configure --prefix=/my/local/install/path --with-qt=no
shell$ make
shell$ make install
\end{lstlisting}
The graphviz build reportedly may fail on systems that do not have QT
installed, hence the \lstinline@./configure --with-qt=no@ option above.
The graphviz lib and include locations are required when installing
Pygraphviz.
\item The pygraphviz \lstinline=setup.py= file may need to be edited
to point at your local graphviz library and include directories:
\begin{lstlisting}
# setup.py lines 31 and 32
library_path='/path/to/local/graphviz/lib'
include_path='/path/to/local/graphviz/include'
\end{lstlisting}
\item Note that when using \lstinline=python setup.py= or
\lstinline=easy_install= the local install location may need to exist
and it may be need to be present in \lstinline=PYTHONPATH= {\em before}
you initiate the install process.
\end{myitemize}

Finally, check that everything (other than \LaTeX for document
processing) is installed:

\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc check-software
Checking for Python >= 2.6 ... found 2.7.3 ... ok
Checking for non-Python packages:
 + Graphviz ... ok
 + sqlite ... ok
Checking for Python packages:
 + Jinja2 ... ok
 + pygraphviz ... ok
 + pygtk ... ok
\end{lstlisting}
If this command reports any errors then the packages concerned are not
installed, not in the system Python search path, or (for a local
install) not present in your \lstinline=$PYTHONPATH= variable.

\subsection{Upgrading To New Cylc Versions}

Upgrading is just a matter of unpacking the new cylc release. Successive
cylc releases can be installed in parallel as suggested in the
\lstinline=INSTALL= file (\ref{INSTALL}).

\section{{\em Cycling} And {\em Cycle Points} In Cylc}

There are several reasons why an atmospheric model (say) may need to be cycled:
\begin{myitemize}
    \item In real time forecasting it may need to wait on new real time data
        before each run.
    \item Job run length limits on compute hosts may require long jobs
        to be split into multiple shorter jobs. (This also makes
        incremental post-processing easier.)
\end{myitemize}
This section defines the terminology associated with cycling, and describes the
two ways to make a cylc workflow that cycles a model and associated jobs.

\subsection{Cycle Points}

In cylc terminology, a {\em cycle point} is a particular
date-time (or integer) point in a sequence. If a task has an hourly date-time
cycle that starts at 2014-01-01T00 and ends at 2014-01-02T00, the first cycle
point will be 2014-01-01T01, the second will be 2014-01-01T02, and so on.

You may be accustomed to the idea that a forecasting suite has a global (in the
suite-wide, not geographic, sense) cycle point that only advances when all
tasks in the current cycle point have finished (or even when a particular wall
clock time is reached, in real time operation). As explained in the
Introduction, that is not how cylc works. Rather, each task has a private cycle
point and can advance independently. The workflow is constrained only by the
dependencies of individual tasks, which may cross over to other cycle points.
It may still be convenient at times, however, to refer to the ``current cycle
point'', the ``previous cycle point'', or the ``next cycle point'' and so
forth, with reference to a particular task, or in the sense of all tasks that
``belong to'' a particular cycle point. But keep in mind that the members of
these groups may not be present simultaneously in the running suite - different
tasks may pass through the ``current cycle point'' at different times as the
suite evolves.

\subsection{Ways Of Cycling}
\label{Ways Of Cycling}

\subsubsection{Iterative Cycling}

Cylc's classic cycling mode, as described in the Introduction, can be termed
{\em iterative cycling.} The scheduler continually extends the workflow to
future cycle points as the suite runs, and each instance of a repeated job is
represented by the same logical task with a different cycle point.  For
example, a program \lstinline=model.exe= could be executed by a task
\lstinline@model@ that runs repeatedly for a series of cycle points
\lstinline=2016-01-01, 2016-01-02, 2016-01-03, ...=

This is the most powerful and flexible way of running cycling jobs in cylc, and
it is really the only way to run large operational suites that need to continue
indefinitely. The cycling is configured with general standards-based (ISO 8601)
{\em recurrence expressions} for any date-time sequence. Multiple cycling
sequences can be used at once. See Section~\ref{ConfiguringScheduling}.

\subsubsection{Parameterized Cycling}

{\em Parameterized cycling} is alternative cycling mode for workflows that are
{\em finite in extent and not too large}. Parameter expansion (or Jinja2 loops)
can be used to generate the entire workflow from start to finish, at start-up.
Once this is done, there is no sequence of cycle points in the sense defined
above, and every single instance of a repeated job is represented by a
different logical task with a different task name.  Functionally this is
entirely equivalent to defining every task instance separately in the suite
definition; parameter expansion is just a very efficient way to do that.
For example, a program \lstinline=model.exe= could be executed by each
of a series of tasks called
\lstinline=model-run0, model-run1, model-run2, ...=, where the integer
parameter values in the task names can be multiplied by an interval
(e.g.\ \lstinline=P1D= for one day) to get to the correct ``cycle point'' for
the job.

This is generally not as flexible or powerful as iterative cycling, and it has
several significant disadvantages: it is necessarily bounded (i.e.\ it can't go
on indefinitely), the difficult date-time arithmetic is not done automatically,
and at run time the full extent of the workflow (every single task instance
from start to finish) is visible at all times.  However, for smaller suites of
relatively short duration, it may be useful.  See Section~\ref{Parameterized
Cycling}.

\subsubsection{Mixed Cycling}

The two cycling approaches can be combined within a single suite.  In a daily
iterative cycling suite, for instance, long daily model runs could be split
into multiple shorter steps or chunks by parameterized cycling. (However, you
should first consider simply using a shorter iterative cycle).

\section{Global (Site, User) Configuration Files}
\label{SiteAndUserConfiguration}

Cylc site and user global configuration files contain settings that affect all
suites. Some of these, such as the range of network ports used by cylc,
should be set at site level,
\lstset{language=transcript}
\begin{lstlisting}
# cylc site global config file
/path/to/cylc/conf/global.rc
# Deprecated path to cylc site global config file
/path/to/cylc/conf/siterc/site.rc
\end{lstlisting}
Others, such as the preferred text editor for suite definitions,
can be overridden by users,
\lstset{language=transcript}
\begin{lstlisting}
# cylc user global config file
~/.cylc/global.rc
# Deprecated cylc user global config file
~/.cylc/user.rc
\end{lstlisting}

The \lstinline=cylc get-site-config= command retrieves current
global settings consisting of cylc defaults overridden by site settings,
if any, overridden by user settings, if any. To generate an
initial site or user global config file:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc get-site-config > $HOME/.cylc/global.rc
\end{lstlisting}
Settings that do not need to be changed should be deleted or commented
out of user global config files so that they don't override future changes to
the site file.

Legal items, values, and system defaults are documented in
(\ref{SiteRCReference}).

%\pagebreak
\section{Tutorial}
\label{Tutorial}

This section provides a hands-on tutorial introduction to basic cylc
suite preparation and control. A number of features are not yet touched
on by the tutorial examples, however, so please also read the rest of
the User Guide.

\subsection{User Config File}

Some global parameters affecting cylc's behaviour are defined in a
{\em site global config file}, and can be customized per user in
{\em user global config files}. For example, to choose the text editor
invoked by cylc on suite definitions:
\lstset{language=suiterc}
\begin{lstlisting}
# $HOME/.cylc/global.rc
[editors]
    terminal = vim
    gui = gvim -f
\end{lstlisting}

\begin{myitemize}
\item For more on site and user global config files, including how to generate
one at first, see~\ref{SiteAndUserConfiguration} and~\ref{SiteRCReference}.
\end{myitemize}

\subsubsection{Configure Environment on Job Hosts}

If you submit task jobs to hosts other than the hosts you use to run your
suites, you may need to customise the environment for
running cylc. A cylc suite generates task job scripts that normally invoke
\lstinline=bash=. The job will attempt to source the first of these files it
finds to set up its environment:

\begin{myitemize}
\item \lstinline=${HOME}/.cylc/job-init-env.sh=
\item \lstinline=${CYLC_DIR}/conf/job-init-env.sh=
\item \lstinline=${CYLC_DIR}/conf/job-init-env-default.sh=
\end{myitemize}

The \lstinline=${CYLC_DIR}/conf/job-init-env-default.sh= file is provided in
the cylc distribution, and will attempt to source \lstinline=/etc/profile= and
\lstinline=${HOME}/.profile=. If this behaviour is not desirable, your site
administrator who installed cylc should have overridden it by adding a
\lstinline=${CYLC_DIR}/conf/job-init-env.sh= file and populate it with the
appropriate contents. If customisation is still required, you can add your own
\lstinline=${HOME}/.cylc/job-init-env.sh= file and populate it with the
appropriate contents.

\subsection{User Interfaces}
\label{CUI}

Cylc has command line (CLI) and graphical (GUI) user interfaces.
To access them {\em do not simply put the bin directory of the current release
the shell search path for users.} Instead, as described in the cylc INSTALL
file (\ref{INSTALL}), configure the provided multi-version wrapper script to
point to install location for any or all cylc releases. This allows existing
suites to continue running at old cylc versions if necessary.

\subsubsection{Command Line Interface (CLI)}

The command line interface is unified under a single top level
\lstinline=cylc= command that provides access to many sub-commands
and their help documentation.
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc help       # Top level command help.
shell$ cylc run --help # Example command-specific help.
\end{lstlisting}

\begin{myitemize}
    \item Command help transcripts are also printed in~\ref{CommandReference},
        and can be accessed from the \lstinline=cylc gui= Help menu.
    \item Cylc suites are {\em scriptable} because the error status returned by
        all commands can be relied on.
\end{myitemize}

\subsubsection{Graphical User Interface (GUI)}

The cylc GUI covers the same functionality as the CLI, but it has more
sophisticated suite monitoring capability. It can start and stop suites, or
connect to suites that are already running; in either case, shutting down the
GUI does not affect the suite itself.
\lstset{language=transcript}
\begin{lstlisting}
shell$ gcylc & # or:
shell$ cylc gui & # Use the File menu to switch to specific suite.
shell$ cylc gscan & # Scan GUI for multiple running suites.
\end{lstlisting}
Clicking on a suite in the scan GUI, shown in
Figure~\ref{fig-gscan}, opens a gcylc instance for it.

\subsection{Suite Definitions}

Cylc suites are defined by extended-INI format \lstinline=suite.rc=
files (the main file format extension is section nesting). These reside
in {\em suite definition directories} that may also contain a
\lstinline=bin= directory and any other suite-related files.

\begin{myitemize}
\item For more on the suite definition file format, see~\ref{SuiteDefinition}
    and~\ref{SuiteRCReference}.
\end{myitemize}

\subsection{Suite Name Registration}

Suite registration associates a name with a suite definition directory,
in a simple database. Cylc commands that parse suite definition files
can take the file path or the suite name as input; commands that
interact with running suites have to target the suite by name.
\lstset{language=transcript}
\begin{lstlisting}
# target a suite by file path:
shell$ cylc validate /path/to/my/suite/suite.rc
shell$ cylc graph /path/to/my/suite/suite.rc
# register a name for a suite:
shell$ cylc register my.suite /path/to/my/suite/
# target a suite by name:
shell$ cylc graph my.suite
shell$ cylc validate my.suite
shell$ cylc run my.suite
shell$ cylc stop my.suite
# etc.
\end{lstlisting}

\begin{myitemize}
\item For more on registering, copying, deleting, and renaming suites,
see \lstinline=cylc db --help= (see~\ref{SuiteRegistration}
and~\ref{SuiteStorageEtc}).
\end{myitemize}

\subsection{Suite Passphrases}
\label{tutPassphrases}

A suite-specific passphrase file is automatically generated in the suite
definition directory at registration time. It is loaded by the suite daemon at
start-up and used to authenticate all client connections. Clients on the suite
host account automatically load the passphrase from the suite definition
directory. The passphrase is also installed automatically in the suite run
directory on hosts for running task jobs. On other accounts that do not share
the same HOME directory with the suite host account and do not have
non-interactive SSH access to the suite host account, it is possible to install
the passphrase manually. (A client will attempt to install the passphrase
automatically via non-interactive SSH.)

Possession of a suite passphrase gives full control and full read-access to the
suite. Without it, client access is determined by the suite's public access
privilege level.

For more on connection authentication, suite passphrases, and public access,
see~\ref{ConnectionAuthentication}.

\subsection{Import The Example Suites}
\label{ImportTheExampleSuites}

Run the following command to import cylc's example suites to a chosen
directory location and register them for use under the {\em examples}
name group:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc import-examples $TMPDIR examples
\end{lstlisting}
(first check that \lstinline=$TMPDIR= is defined in your environment, or
else use a different location). List the newly registered tutorial
suites using the \lstinline=cylc print= command:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc db print examples.tutorial -y
examples.tutorial.oneoff.jinja2   /tmp/examples/tutorial/oneoff/jinja2
examples.tutorial.cycling.two     /tmp/examples/tutorial/cycling/two
examples.tutorial.cycling.three   /tmp/examples/tutorial/cycling/three
examples.tutorial.oneoff.remote   /tmp/examples/tutorial/oneoff/remote
# ...
\end{lstlisting}
See \lstinline=cylc db print --help= for other display options. The
tree-form display shows how hierarchical suite names can be used to
organize related suites nicely (suite names do not have to be related
to their source directory paths, although they are in this case):
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc db pr --tree -x examples.tutorial
examples
 `-tutorial
   `-cycling
   | | ...
   | |-two        Two cycling tasks with inter-cycle dependence.
   | | ...
   `-oneoff
     |-retry      A task with automatic retry on failure
     |-remote     Hello World! on a remote host
     | ...
     |-basic      The cylc Hello World! suite
     `-jobsub     Hello World! by 'at' job submission
\end{lstlisting}

\subsection{Rename The Imported Tutorial Suites}
\
Rename (re-register) the tutorial suites to make their names a bit
shorter:
\begin{lstlisting}
$ cylc rereg examples.tutorial tut
REREGISTER examples.tutorial.oneoff.jinja2 to tut.oneoff.jinja2
#...
shell$ cylc db print -x tut
tut.oneoff.external   Hello World! from an external task script
# ...
\end{lstlisting}

\subsection{Suite Validation}

Suite definitions can be validated against the \lstinline=suite.rc=
file format specification to detect many types of error without running
the suite.
\lstset{language=transcript}
\begin{lstlisting}
# pass:
shell$ cylc validate tut.oneoff.basic
Valid for cylc-6.0.0
shell$ echo $?
0
# fail:
shell$ cylc validate my.bad.suite
'Illegal item: [scheduling]special tusks'
shell$ echo $?
1
\end{lstlisting}

\subsection{Hello World in Cylc}

\hilight{ suite: \lstinline=tut.oneoff.basic= }
\vspace{3mm}

Here's the traditional {\em Hello World} program rendered as a cylc
suite:
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/oneoff/basic/suite.rc}
\lstset{language=transcript}

Cylc suites feature a clean separation of scheduling configuration,
which determines {\em when} tasks are ready to run; and runtime
configuration, which determines {\em what} to run (and {\em where} and
{\em how} to run it) when a task is ready. In this example the
\lstinline=[scheduling]= section defines a single task called
\lstinline=hello= that triggers immediately when the suite starts
up. When the task finishes the suite shuts down. That this is a
{\em dependency graph} will be more obvious when more tasks are added.
Under the \lstinline=[runtime]= section the
\lstinline=script= item defines a simple inlined
implementation for \lstinline=hello=: it sleeps for ten seconds,
then prints \lstinline=Hello World!=, and exits. This ends up in a {\em
job script} generated by cylc to encapsulate the task (below) and,
thanks to some defaults designed to allow quick
prototyping of new suites, it is submitted to run as a background job on
the suite host. In fact cylc even provides a default task implementation
that makes the entire \lstinline=[runtime]= section technically optional:
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/oneoff/minimal/suite.rc}
\lstset{language=transcript}
(the resulting {\em dummy task} just prints out some identifying
information and exits).

\subsection{Editing Suites}

The text editor invoked by cylc on suite definitions is determined
by cylc site and user global config files, as shown above in~\ref{CUI}.
Check that you have renamed the tutorial examples suites as described
just above and open the {\em Hello World} suite definition in your text
editor:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc edit tut.oneoff.basic # in-terminal
shell$ cylc edit -g tut.oneoff.basic & # or GUI
\end{lstlisting}
Alternatively, start gcylc on the suite:
\lstset{language=transcript}
\begin{lstlisting}
shell$ gcylc tut.oneoff.basic &
\end{lstlisting}
and choose {\em Suite } \textrightarrow {\em Edit} from the menu.

The editor will be invoked from within the suite definition directory for easy
access to other suite files (in this case there are none). There are syntax
highlighting control files for several text editors under
\lstinline=/path/to/cylc/conf/=; see in-file comments for installation
instructions.

\subsection{Running Suites}
\label{RunningSuitesCLI}

\subsubsection{CLI}
Run \lstinline=tut.oneoff.basic= using the \lstinline=cylc run= command.
As a suite runs detailed timestamped information is written to a {\em suite
log} and progress can be followed with cylc's suite monitoring tools (below).
By default a running suite daemonizes after printing a short message so that
you can exit the terminal or even log out without killing the suite:

\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc run tut.oneoff.basic
************************************************************************
*                     The Cylc Suite Engine [6.0.0]                    *
*             Copyright (C) 2008-2016 NIWA              *
*                                                                      *
* This program comes with ABSOLUTELY NO WARRANTY.  It is free software;*
* you are welcome to redistribute it under certain conditions. Details:*
*          `cylc license conditions'; `cylc license warranty'          *
************************************************************************

Suite Info:
 + Name: tut.oneoff.basic
 + PID:  29338
 + Port: 7766
 + Logs: /home/oliverh/cylc-run/tut.oneoff.basic/log/suite/{log,out,err}

To see if this suite is still running:
 * cylc scan
 * cylc ping -v tut.oneoff.basic
 * ps -fu $USER | grep 'cylc-run .* tut.oneoff.basic'

To run in non-daemon mode use --no-detach or --debug.
For more information see 'cylc --help' and the User Guide.
\end{lstlisting}

If you're quick enough (this example only takes 10-15 seconds to run) the
\lstinline=cylc scan= command will detect the running suite:
\begin{lstlisting}
shell$ cylc scan
tut.oneoff.basic oliverh oliverh-34403DL 7766

# wait 10-15 seconds ...

shell$ cylc scan
# (nothing: suite finished)
\end{lstlisting}

Using the \lstinline=--no-detach= option prevents the suite daemonizing,
but only warnings and errors, if any, are printed to the terminal.
The \lstinline=--debug= option, however, causes (among other things) task job
submission commands to to be printed stdout:

\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc run --debug tut.oneoff.basic
************************************************************************
*                     The Cylc Suite Engine [6.0.0]                    *
*             Copyright (C) 2008-2016 NIWA              *
*                                                                      *
* This program comes with ABSOLUTELY NO WARRANTY.  It is free software;*
* you are welcome to redistribute it under certain conditions. Details:*
*          `cylc license conditions'; `cylc license warranty'          *
************************************************************************
cylc job-submit /home/oliverh/cylc-run/tut.oneoff.basic/log/job/1/hello/01/job
'Finished'
Suite shutting down at 2014-08-14T13:26:52+12
DONE
\end{lstlisting}
When a task is ready to run a {\em job script} is generated to run it. The
command printed just above is how cylc executes background jobs on the
suite host (this is the default), retrieves the job process ID, and directs the
output to log files in standard locations. For the log file locations that can
be seen above, in \lstinline=job/1/hello/01/= the first \lstinline=1/= is the
{\em cycle point} of the task \lstinline=hello= (for a non-cycling task
this is just the integer 1); and the final \lstinline=01/= directory is
its {\em submission number} (tasks can be made to retry on failure, manually
or automatically, in which case the submission number increments).

The suite shuts down automatically once all tasks have succeeded.

\subsubsection{GUI}

The cylc GUI can start and stop suites, or (re)connect to suites that
are already running:
\begin{lstlisting}
shell$ cylc gui tut.oneoff.basic &
\end{lstlisting}
Use the tool bar {\em Play} button, or the {\em Control}
\textrightarrow {\em Run} menu item, to run the suite again.
You may want to alter the suite definition slightly to make the task
take longer to run. Try right-clicking on the \lstinline=hello= task
to view its output logs. The relative merits of the three {\em suite
views} - dot, text, and graph - will be more apparent later when we
have more tasks. Closing the GUI does not affect the suite itself.

\subsection{Discovering Running Suites}

Suites that are currently running can be detected with command line or
GUI tools:
\begin{lstlisting}
# list currently running suites and their port numbers:
shell$ cylc scan
tut.oneoff.basic oliverh oliverh-34403DL 7766

# GUI summary view of running suites:
shell$ cylc gscan &
\end{lstlisting}

The scan GUI is shown in Figure~\ref{fig-gscan}; clicking on a suite in it
opens gcylc.

\subsection{Task Identifiers}

At run time, task instances are identified by {\em name}, which is
determined entirely by the suite definition, and a {\em cycle point} which is
usually a date-time or an integer:
\lstset{language=transcript}
\begin{lstlisting}
foo.20100808T00Z   # a task with a date-time cycle point
bar.1              # a task with an integer cycle point (could be non-cycling)
\end{lstlisting}
Non-cycling tasks usually just have the cycle point \lstinline=1=, but this
still has to be used to target the task instance with cylc commands.

\subsection{Job Submission: How Tasks Are Executed}

\hilight{ suite: \lstinline=tut.oneoff.jobsub= }
\vspace{3mm}

Task {\em job scripts} are generated by cylc to wrap the task implementation
specified in the suite definition (environment, script, etc.) in
error trapping code, messaging calls to report task progress back to the suite
daemon, and so forth. Job scripts are written to the {\em suite job log
directory} where they can be viewed alongside the job output logs. They
can be accessed at run time by right-clicking on the task in the cylc GUI, or
printed to the terminal:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc cat-log tut.oneoff.basic hello.1
\end{lstlisting}
This command can also print the suite log (and stdout and stderr for suites
in daemon mode) and task stdout and stderr logs (see
\lstinline=cylc cat-log --help=).
A new job script can also be generated on the fly for inspection:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc jobscript tut.oneoff.basic hello.1
\end{lstlisting}

Take a look at the job script generated for \lstinline=hello.1= during
the suite run above. The custom scripting should be clearly visible
toward the bottom of the file.

The \lstinline=hello= task in the first tutorial suite defaults to
running as a background job on the suite host. To submit it to the Unix
\lstinline=at= scheduler instead, configure its job submission settings
as in \lstinline=tut.oneoff.jobsub=:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[job]]]
            batch system = at
\end{lstlisting}
If you run the suite (first check that the {\em at daemon}
\lstinline=atd= is running on the suite host) you'll see a slightly different
job submission command printed to stdout:
\lstset{language=transcript}
\begin{lstlisting}
 cylc run --debug tut.oneoff.jobsub
************************************************************************
*        The Cylc Suite Engine [6.0.0alpha1-294-ge08560-dirty]         *
*             Copyright (C) 2008-2016 NIWA              *
*                                                                      *
* This program comes with ABSOLUTELY NO WARRANTY.  It is free software;*
* you are welcome to redistribute it under certain conditions. Details:*
*          `cylc license conditions'; `cylc license warranty'          *
************************************************************************
cylc job-submit /home/oliverh/cylc-run/tut.oneoff.jobsub/log/job/1/hello/01/job
'Finished'
Suite shutting down at 2014-08-14T14:14:43+12
DONE
\end{lstlisting}

Cylc supports a number of different job submission methods. Tasks
submitted to external batch queuing systems like \lstinline=at=,
\lstinline=PBS=, \lstinline=SLURM=, \lstinline=Moab=, or
\lstinline=LoadLeveler=, are displayed as {\em submitted} in the cylc GUI until
they actually start executing.

\begin{myitemize}
\item For more on task job scripts, see~\ref{JobScripts}.
\item For more on job submission methods, see~\ref{AvailableMethods}.
\end{myitemize}

\subsection{Locating Suite And Task Output}

If the \lstinline=--no-detach= option is not used, suite stdout and
stderr will be directed to the suite run directory along with the
time-stamped suite log file, and task job scripts and job logs
(task stdout and stderr). The default suite run directory location is
\lstinline=$HOME/cylc-run=:

\lstset{language=transcript}
\begin{lstlisting}
shell$ tree $HOME/cylc-run/tut.oneoff.basic/
|-- cylc-suite-private.db # private suite run database
|-- cylc-suite-public.db  # public suite run database
|-- cylc-suite.db         # symbolic link to public suite run database
|-- cylc-suite-env        # suite environment file
|-- share                 # suite share directory (not used in this example)
|-- work                  # task work space (sub-dirs are deleted if not used)
|    `-- 1                   # task cycle point directory (or 1)
|        `-- hello              # task work directory (deleted if not used)
|-- log                   # suite log directory
|   `-- suite                # suite daemon log directory
|   |   |-- err                 # suite daemon stderr log (daemon mode only)
|   |   |-- out                 # suite daemon stdout log (damon mode only)
|   |   `-- log                 # suite daemon event log (timestamped info)
|   `-- job                  # task job log directory
|       `-- 1                   # task cycle point directory (or 1)
|           `-- hello              # task name
|               |-- 01                # task submission number
|               |   |-- job              # task job script
|               |   `-- job-activity.log # task job activity log
|               |   |-- job.err          # task stderr log
|               |   |-- job.out          # task stdout log
|               |   `-- job.status       # task status file
|               `-- NN -> 01          # symlink to latest submission number
\end{lstlisting}
The suite run database files, suite environment file,
and task status files are used internally by cylc. Tasks execute in
private \lstinline=work/= directories that are deleted automatically
if empty when the task finishes. The suite
\lstinline=share/= directory is made available to all tasks (by
\lstinline=$CYLC_SUITE_SHARE_DIR=) as a common share space. The task submission
number increments from 1 if a task retries on failure; this is used a
sub-directory of the log tree to avoid overwriting log files from earlier
job submissions.

The top level run directory location can be changed in site and user
config files if necessary, and the suite share and work locations can be
configured separately because of the potentially larger disk space
requirement.

Task job logs can be viewed by right-clicking on tasks in the gcylc
GUI (so long as the task proxy is live in the suite), manually
accessed from the log directory (of course), or printed to the terminal
with the \lstinline=cylc cat-log= command:
\lstset{language=transcript}
\begin{lstlisting}
# suite logs:
shell$ cylc cat-log    tut.oneoff.basic           # suite event log
shell$ cylc cat-log -o tut.oneoff.basic           # suite stdout log
shell$ cylc cat-log -e tut.oneoff.basic           # suite stderr log
# task logs:
shell$ cylc cat-log    tut.oneoff.basic hello.1   # task job script
shell$ cylc cat-log -o tut.oneoff.basic hello.1   # task stdout log
shell$ cylc cat-log -e tut.oneoff.basic hello.1   # task stderr log
\end{lstlisting}
\begin{myitemize}
    \item For a web-based interface to suite and task logs (and much more),
        see {\em Rose} in~\ref{SuiteStorageEtc}.
    \item For more on environment variables supplied to tasks,
    such as \lstinline=$CYLC_SUITE_SHARE_DIR=, see~\ref{TaskExecutionEnvironment}.
\end{myitemize}

\subsection{Remote Tasks}
\label{RemoteTasks}

\hilight{ suite: \lstinline=tut.oneoff.remote= }
\vspace{3mm}

The \lstinline=hello= task in the first two tutorial suites defaults to
running on the suite host. To make it run on a remote host instead
change its runtime configuration as in \lstinline=tut.oneoff.remote=:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
\end{lstlisting}

For remote task hosting to work several requirements must be satisfied:
\begin{myitemize}

\item Passwordless ssh must be enabled from the suite host account to
the task host account, for task job submission.

\item Your shell initialization (.profile, .bashrc, .cshrc, etc) on the remote
host must not produce any standard output as it may confuse commands such as
scp. See \url{http://www.openssh.com/faq.html#2.9} for more information.

\item Networking settings must allow communication {\em back} from
the task host to the suite host, either by network ports (Pyro) or ssh,
unless the last-resort one way {\em task polling} communication method
is used.

\item Cylc must be installed on the task host. Other software dependencies like
graphviz and Jinja2 are not required there.

\item Any files needed by a remote task must be installed on the task
host. In this example there is nothing to install because the
implementation of \lstinline=hello= is inlined in the suite definition
and thus ends up entirely contained within the task job script.

\end{myitemize}

If your username is different on the task host the
\lstinline=[[[remote]]]= section also supports an
\lstinline@owner=username@ item, or your \lstinline=$HOME/.ssh/config=
file can be configured for username translation.

If you configure a task host according to the requirements above and run
the suite again in debug mode, you'll see that the job submission command
printed to suite stdout is now considerably more complicated. That's because it
has to create remote log directories, source login scripts on the remote host
to ensure cylc is visible there, send the task job script over, and submit it
to run there by the configured job submission method:

\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc run --debug tut.oneoff.remote
# ...
ssh -oBatchMode=yes -oConnectTimeout=10 wrh-1.niwa.co.nz\
'CYLC_VERSION='"'"'6.1.0'"'"\
' "/opt/cylc/bin/cylc" '"'"'job-submit'"'"\
' --remote-mode "/home/oliverh/cylc-run/tut.oneoff.remote/log/job/1/hello/01/job"'
\end{lstlisting}
(Don't be intimidated by this - it is really quite straightforward and would
appear much simpler if the job log path was shorter!). Remote task job logs are
saved to the suite run directory on the task host, not on the suite host,
although they can be retrieved by right-clicking on the task in the GUI. If you
want the job logs pulled back to the suite host automatically, you can set
"retrieve job logs=True" under the "[[[remote]]]" section:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
            retrieve job logs = True
\end{lstlisting}

The suite will attempt to \lstinline=rsync= the job logs once from the remote
host each time a task job completes. E.g. if the job file is
\lstinline=~/cylc-run/tut.oneoff.remote/log/job/1/hello/01/job=, anything under
\lstinline=~/cylc-run/tut.oneoff.remote/log/job/1/hello/01/= will be retrieved.

Some batch systems have considerable delays between the time when the job
completes and when it writes the job logs in its normal location. If this is
the case, you can configure an initial delay and retry delays for job log
retrieval by setting some delays. E.g.:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
            retrieve job logs = True
            # Retry after 10 seconds, 1 minute and 3 minutes
            retrieve job logs retry delays = PT10S, PT1M, PT3M
\end{lstlisting}

Finally, if the disk space of the suite host is limited, you may want to set
\lstinline@[[[remote]]]retrieve job logs max size=SIZE@. The value of SIZE can
be anything that is accepted by the \lstinline@--max-size=SIZE@ option of the
\lstinline=rsync= command. E.g.:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
            retrieve job logs = True
            # Don't get anything bigger than 10MB
            retrieve job logs max size = 10M
\end{lstlisting}

It is worth noting that cylc uses the existence of a job's \lstinline=job.out=
or \lstinline=job.err= in the local file system to indicate a successful job
log retrieval. If \lstinline=retrieve job logs max size=SIZE= is set and both
\lstinline=job.out= and \lstinline=job.err= are bigger than \lstinline=SIZE=
then cylc will consider the retrieval as failed. If retry delays are specified,
this will trigger some useless (but harmless) retries. If this occurs
regularly, you should try the following:

\begin{myitemize}
\item Reduce the verbosity of STDOUT or STDERR from the task.
\item Redirect the verbosity from STDOUT or STDERR to an alternate log file.
\item Adjust the size limit with tolerance to the expected size of STDOUT or STDERR.
\end{myitemize}

\begin{myitemize}
\item For more on remote tasks see~\ref{RunningTasksOnARemoteHost}

\item For more on task communications, see~\ref{TaskComms}.

\item For more on suite passphrases and authentication,
    see~\ref{tutPassphrases} and~\ref{ConnectionAuthentication}.
\end{myitemize}


\subsection{Task Triggering}

\hilight{ suite: \lstinline=tut.oneoff.goodbye= }
\vspace{3mm}

To make a second task called \lstinline=goodbye= trigger after
\lstinline=hello= finishes successfully, return to the original
example, \lstinline=tut.oneoff.basic=, and change the suite graph
as in \lstinline=tut.oneoff.goodbye=:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "hello => goodbye"
\end{lstlisting}
or to trigger it at the same time as \lstinline=hello=,
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "hello & goodbye"
\end{lstlisting}
and configure the new task's behaviour under \lstinline=[runtime]=:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[goodbye]]
        script = "sleep 10; echo Goodbye World!"
\end{lstlisting}

Run \lstinline=tut.oneoff.goodbye= and check the output from the new
task:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cat ~/cylc-run/tut.oneoff.goodbye/log/job/1/goodbye/01/job.out
  # or
shell$ cylc cat-log -o tut.oneoff.goodbye goodbye.1
JOB SCRIPT STARTING
cylc (scheduler - 2014-08-14T15:09:30+12): goodbye.1 started at 2014-08-14T15:09:30+12
cylc Suite and Task Identity:
  Suite Name  : tut.oneoff.goodbye
  Suite Host  : oliverh-34403dl.niwa.local
  Suite Port  : 7766
  Suite Owner : oliverh
  Task ID     : goodbye.1
  Task Host   : oliverh-34403DL
  Task Owner  : oliverh
  Task Try No.: 1

Goodbye World!
cylc (scheduler - 2014-08-14T15:09:40+12): goodbye.1 succeeded at 2014-08-14T15:09:40+12
JOB SCRIPT EXITING (TASK SUCCEEDED)
\end{lstlisting}

\subsubsection{Task Failure And Suicide Triggering}

\hilight{ suite: \lstinline=tut.oneoff.suicide= }
\vspace{3mm}

Task names in the graph string can be qualified with a state indicator
to trigger off task states other than success:
\lstset{language=suiterc}
\lstset{language=suiterc}
\begin{lstlisting}
    graph = """
 a => b        # trigger b if a succeeds
 c:submit => d # trigger d if c submits
 e:finish => f # trigger f if e succeeds or fails
 g:start  => h # trigger h if g starts executing
 i:fail   => j # trigger j if i fails
            """
\end{lstlisting}

A common use of this is to automate recovery from known modes of failure:
\lstset{language=suiterc}
\begin{lstlisting}
    graph = "goodbye:fail => really_goodbye"
\end{lstlisting}
i.e. if task \lstinline=goodbye= fails, trigger another task that
(presumably) really says goodbye.

Failure triggering generally requires use of {\em suicide triggers} as
well, to remove the recovery task if it isn't required (otherwise it
would hang about indefinitely in the waiting state):
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """hello => goodbye
            goodbye:fail => really_goodbye
         goodbye => !really_goodbye # suicide"""
\end{lstlisting}
This means if \lstinline=goodbye= fails, trigger
\lstinline=really_goodbye=; and otherwise, if \lstinline=goodbye=
succeeds, remove \lstinline=really_goodbye= from the suite.

Try running \lstinline=tut.oneoff.suicide=, which also configures
the \lstinline=hello= task's runtime to make it fail, to see how this
works.
\begin{myitemize}
    \item For more on suite dependency graphs see~\ref{ConfiguringScheduling}.
    \item For more on task triggering see~\ref{TriggerTypes}.
\end{myitemize}

\subsection{Runtime Inheritance}

\hilight{ suite: \lstinline=tut.oneoff.inherit= }
\vspace{3mm}

The \lstinline=[runtime]= section is actually a {\em multiple
inheritance} hierarchy. Each subsection is a {\em namespace} that
represents a task, or if it is inherited by other namespaces, a {\em
family}. This allows common configuration to be factored out of related
tasks very efficiently.
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/oneoff/inherit/suite.rc}
The \lstinline=[root]= namespace provides defaults for all tasks in the suite.
Here both tasks inherit \lstinline=script= from \lstinline=root=, which they
customize with different values of the environment variable
\lstinline=$GREETING=. Note that inheritance from \lstinline=root= is
implicit; from other parents an explicit \lstinline@inherit = PARENT@
is required, as shown below.

\begin{myitemize}
\item For more on runtime inheritance, see~\ref{NIORP}.
\end{myitemize}

\subsection{Triggering Families}

\hilight{ suite: \lstinline=tut.oneoff.ftrigger1= }
\vspace{3mm}

Task families defined by runtime inheritance can also be used as
shorthand in graph trigger expressions. To see this, consider two
``greeter'' tasks that trigger off another task \lstinline=foo=:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => greeter_1 & greeter_2"
\end{lstlisting}
If we put the common greeting functionality of \lstinline=greeter_1=
and \lstinline=greeter_2= into a special \lstinline=GREETERS= family,
the graph can be expressed more efficiently like this:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => GREETERS"
\end{lstlisting}
i.e.\ if \lstinline=foo= succeeds, trigger all members of
\lstinline=GREETERS= at once. Here's the full suite with runtime
hierarchy shown:
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/oneoff/ftrigger1/suite.rc}

(Note that we recommend given ALL-CAPS names to task families to help
distinguish them from task names. However, this is just a convention).

Verbose validation shows the family member substitution done
when the suite definition is parsed:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc val -v tut.oneoff.ftrigger1
...
Graph line substitutions occurred:
  IN: foo => GREETERS
  OUT: foo => greeter_1 & greeter_2
...
\end{lstlisting}

Experiment with the \lstinline=tut.oneoff.ftrigger1= suite to see
how this works.

\subsection{Triggering Off Families}

\hilight{ suite: \lstinline=tut.oneoff.ftrigger2= }
\vspace{3mm}

Tasks (or families) can also trigger {\em off} other families, but
in this case we need to specify what the trigger means in terms of
the upstream family members. Here's how to trigger another task
\lstinline=bar= if all members of \lstinline=GREETERS= succeed:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """foo => GREETERS
            GREETERS:succeed-all => bar"""
\end{lstlisting}
Verbose validation in this case reports:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc val -v tut.oneoff.ftrigger2
...
Graph line substitutions occurred:
  IN: GREETERS:succeed-all => bar
  OUT: greeter_1:succeed & greeter_2:succeed => bar
...
\end{lstlisting}
Cylc ignores family member qualifiers like \lstinline=succeed-all= on
the right side of a trigger arrow, where they don't make sense, to
allow the two graph lines above to be combined in simple cases:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => GREETERS:succeed-all => bar"
\end{lstlisting}

Any task triggering status qualified by \lstinline=-all= or
\lstinline=-any=, for the members, can be used with a family trigger.
For example, here's how to trigger \lstinline=bar= if all members
of \lstinline=GREETERS= finish (succeed or fail) and any of them them
succeed:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """foo => GREETERS
    GREETERS:finish-all & GREETERS:succeed-any => bar"""
\end{lstlisting}
(use of \lstinline@GREETERS:succeed-any@ by itself here would trigger
\lstinline=bar= as soon as any one member of \lstinline=GREETERS=
completed successfully). Verbose validation now begins to show how
family triggers can simplify complex graphs, even for this tiny
two-member family:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc val -v tut.oneoff.ftrigger2
...
Graph line substitutions occurred:
  IN: GREETERS:finish-all & GREETERS:succeed-any => bar
  OUT: ( greeter_1:succeed | greeter_1:fail ) & \
       ( greeter_2:succeed | greeter_2:fail ) & \
       ( greeter_1:succeed | greeter_2:succeed ) => bar
...
\end{lstlisting}

Experiment with \lstinline=tut.oneoff.ftrigger2= to see how this
works.

\begin{myitemize}
\item For more on family triggering, see~\ref{FamilyTriggers}.
\end{myitemize}

\subsection{Suite Visualization}

\lstset{language=suiterc}
You can style dependency graphs with an optional
\lstinline=[visualization]= section, as shown in
\lstinline=tut.oneoff.ftrigger2=:
\lstset{language=suiterc}
\begin{lstlisting}
[visualization]
    default node attributes = "style=filled"
    [[node attributes]]
        foo = "fillcolor=#6789ab", "color=magenta"
        GREETERS = "fillcolor=#ba9876"
        bar = "fillcolor=#89ab67"
\end{lstlisting}

To display the graph in an interactive viewer:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc graph tut.oneoff.ftrigger2 &    # dependency graph
shell$ cylc graph -n tut.oneoff.ftrigger2 & # runtime inheritance graph
\end{lstlisting}
It should look like Figure~\ref{fig-tut-hello-multi} (with the
GREETERS family node expanded on the right).
\begin{figure}
    \begin{center}
        \includegraphics[height=0.3\textheight]{graphics/png/orig/tut-hello-multi-1.png}
        \hspace{20mm}
        \includegraphics[height=0.3\textheight]{graphics/png/orig/tut-hello-multi-2.png}
        \hspace{20mm}
        \includegraphics[height=0.3\textheight]{graphics/png/orig/tut-hello-multi-3.png}
    \end{center}
    \caption{The {\em tut.oneoff.ftrigger2} dependency and runtime inheritance graphs}
\label{fig-tut-hello-multi}
\end{figure}

Graph styling can be applied to entire families at once, and custom
``node groups'' can also be defined for non-family groups.


\subsection{External Task Scripts}

\hilight{ suite: \lstinline=tut.oneoff.external= }
\vspace{3mm}

The tasks in our examples so far have all had inlined implementation, in
the suite definition, but real tasks often need to call external
commands, scripts, or executables. To try this, let's return to the
basic Hello World suite and cut the implementation of the task
\lstinline=hello= out to a file \lstinline=hello.sh= in the suite
bin directory:
\lstset{language=bash}
\lstinputlisting{../examples/tutorial/oneoff/external/bin/hello.sh}
Make the task script executable, and change the \lstinline=hello= task
runtime section to invoke it:
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/oneoff/external/suite.rc}

If you run the suite now the new greeting from the external task script
should appear in the \lstinline=hello= task stdout log. This works
because cylc automatically adds the suite bin directory to
\lstinline=$PATH= in the environment passed to tasks via their job
scripts. To execute scripts (etc.) located elsewhere you can
refer to the file by its full file path, or set \lstinline=$PATH=
appropriately yourself (this could be done via
\lstinline=$HOME/.profile=, which is sourced at the top of the task job
script, or in the suite definition itself).

Note the use of \lstinline=set -e= above to make the script abort on
error. This allows the error trapping code in the task job script to
automatically detect unforeseen errors.

\subsection{Cycling Tasks}

\hilight{ suite: \lstinline=tut.cycling.one= }
\vspace{3mm}

So far we've considered non-cycling tasks, which finish without spawning
a successor.

Cycling is based around iterating through date-time or integer sequences. A
cycling task may run at each cycle point in a given sequence (cycle). For
example, a sequence might be a set of date-times every 6 hours starting from a
particular date-time. A cycling task may run for each date-time item (cycle
point) in that sequence.

There may be multiple instances of this type of task running in parallel, if
the opportunity arises and their dependencies allow it. Alternatively, a
sequence can be defined with only one valid cycle point - in that case, a task
belonging to that sequence may only run once.

Open the \lstinline=tut.cycling.one= suite:
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/cycling/one/suite.rc}
The difference between cycling and non-cycling suites is all in the
\lstinline=[scheduling]= section, so we will leave the
\lstinline=[runtime]= section alone for now (this will result in
cycling dummy tasks). Note that the graph is now defined under a new
section heading that makes each task under it have a succession of cycle points
ending in $00$ or $12$ hours, between specified initial and final cycle
points (or indefinitely if no final cycle point is given), as shown in
Figure~\ref{fig-tut-one}.

\begin{figure}
    \begin{center}
        %Q Image out of date now
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/tut-one.png}
    \end{center}
    \caption{The \lstinline=tut.cycling.one= suite}
\label{fig-tut-one}
\end{figure}

\lstset{language=transcript}

If you run this suite instances of \lstinline=foo= will spawn in parallel out
to the {\em runahead limit}, and each \lstinline=bar= will trigger off the
corresponding instance of \lstinline=foo= at the same cycle point. The
runahead limit, which defaults to a few cycles but is configurable, prevents
uncontrolled spawning of cycling tasks in suites that are not constrained by
clock triggers in real time operation.

Experiment with \lstinline=tut.cycling.one= to see how cycling tasks work.

\paragraph{ISO 8601 Date-Time Syntax}

The suite above is a very simple example of a cycling date-time workflow. More
generally, cylc comprehensively supports the ISO 8601 standard for date-time
instants, intervals, and sequences. Cycling graph sections can be specified
using full ISO 8601 recurrence expressions, but these may be simplified
by assuming context information from the suite - namely initial and final cycle
points. One form of the recurrence syntax looks like
\lstinline=Rn/start-date-time/period= (\lstinline=Rn= means run
\lstinline=n= times). In the example above, if the initial cycle point
is always at 00 or 12 hours then \lstinline=[[[T00,T12]]]= could be
written as \lstinline=[[[PT12H]]]=, which is short for
\lstinline=[[[R/initial-cycle-point/PT12H/]]]= - i.e.\ run every 12 hours
indefinitely starting at the initial cycle point. It is possible to add
constraints to the suite to only allow initial cycle points at 00 or 12 hours
e.g.

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 20130808T00
    initial cycle point constraints = T00, T12
\end{lstlisting}
\lstset{language=transcript}

\begin{myitemize}
    %Q Runahead factor now
    \item For a comprehensive description of ISO 8601 based date-time cycling,
        see~\ref{AdvancedCycling}
    \item For more on runahead limiting in cycling suites,
        see~\ref{RunaheadLimit}.
\end{myitemize}

\subsubsection{Inter-Cycle Triggers}
\label{TutInterCyclePointTriggers}

\hilight{ suite: \lstinline=tut.cycling.two= }
\vspace{3mm}

The \lstinline=tut.cycling.two= suite adds inter-cycle dependence
to the previous example:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        # Repeat with cycle points of 00 and 12 hours every day:
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
For any given cycle point in the sequence defined by the
cycling graph section heading, \lstinline=bar= triggers off
\lstinline=foo= as before, but now \lstinline=foo= triggers off its own
previous instance \lstinline=foo[-PT12H]=. Date-time offsets in
inter-cycle triggers are expressed as ISO 8601 intervals (12 hours
in this case). Figure~\ref{fig-tut-two} shows how this connects the cycling
graph sections together.
\begin{figure}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/tut-two.png}
    \end{center}
    \caption{The \lstinline=tut.cycling.two= suite}
\label{fig-tut-two}
\end{figure}

Experiment with this suite to see how inter-cycle triggers work.
Note that the first instance of \lstinline=foo=, at suite start-up, will
trigger immediately in spite of its inter-cycle trigger, because cylc
ignores dependence on points earlier than the initial cycle point.
However, the presence of an inter-cycle trigger usually implies something
special has to happen at start-up. If a model depends on its own previous
instance for restart files, for example, then some special process has to
generate the initial set of restart files when there is no previous cycle point
to do it. The following section shows one way to handle this in cylc suites.

\subsubsection{Initial Non-Repeating (R1) Tasks}
\label{initial-non-repeating-r1-tasks}
\hilight{ suite: \lstinline=tut.cycling.three= }
\vspace{3mm}

Sometimes we want to be able to run a task at the initial cycle point, but
refrain from running it in subsequent cycles. We can do this by writing an
extra set of dependencies that are only valid at a single date-time cycle
point. If we choose this to be the initial cycle point, these will only apply
at the very start of the suite.

The cylc syntax for writing this single date-time cycle point occurrence is
\lstinline=R1=, which stands for
\lstinline=R1/no-specified-date-time/no-specified-period=.
This is an adaptation of part of the ISO 8601 date-time standard's recurrence
syntax (\lstinline=Rn/date-time/period=) with some special context information
supplied by cylc for the \lstinline=no-specified-*= data.

The \lstinline=1= in the \lstinline=R1= means run once. As we've specified
no date-time, Cylc will use the initial cycle point date-time by default,
which is what we want. We've also missed out specifying the period - this is
set by cylc to a zero amount of time in this case (as it never
repeats, this is not significant).

For example, in \lstinline=tut.cycling.three=:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    cycle point time zone = +13
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep => foo"
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
\lstset{language=transcript}
This is shown in Figure~\ref{fig-tut-three}.

Note that the time zone has been set to \lstinline=+1300= in this case,
instead of UTC (\lstinline=Z=) as before. If no time zone or UTC mode was
set, the local time zone of your machine will be used in the cycle points.


At the initial cycle point, \lstinline=foo= will depend on
\lstinline=foo[-PT12H]= and also on \lstinline=prep=:
\lstset{language=suiterc}
\begin{lstlisting}
prep.20130808T0000+13 & foo.20130807T1200+13 => foo.20130808T0000+13
\end{lstlisting}
\lstset{language=transcript}

Thereafter, it will just look like e.g.:
\lstset{language=suiterc}
\begin{lstlisting}
foo.20130808T0000+13 => foo.20130808T1200+13
\end{lstlisting}
\lstset{language=transcript}

However, in our initial cycle point example, the dependence on
\lstinline=foo.20130807T1200+13= will be ignored, because that task's cycle
point is earlier than the suite's initial cycle point and so it cannot run.
This means that the initial cycle point dependencies for \lstinline=foo=
actually look like:
\lstset{language=suiterc}
\begin{lstlisting}
prep.20130808T0000+13 => foo.20130808T0000+13
\end{lstlisting}
\lstset{language=transcript}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/tut-three.png}
    \end{center}
    \caption{The \lstinline=tut.cycling.three= suite}
\label{fig-tut-three}
\end{figure}

\begin{myitemize}
    \item \lstinline=R1= tasks can also be used to make something special
        happen at suite shutdown, or at any single cycle point throughout the
        suite run. For a full primer on cycling syntax,
        see~\ref{AdvancedCycling}.
\end{myitemize}


\subsubsection{Integer Cycling}
\label{TutInteger}
\hilight{ suite: \lstinline=tut.cycling.integer= }
\vspace{3mm}

Cylc can do also do integer cycling for repeating workflows that are not
date-time based.

Open the \lstinline=tut.cycling.integer= suite, which is plotted in
Figure~\ref{fig-tut-int}.
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/cycling/integer/suite.rc}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.65\textwidth]{graphics/png/orig/tut-cyc-int.png}
    \end{center}
    \caption{The \lstinline=tut.cycling.integer= suite}
\label{fig-tut-int}
\end{figure}

The integer cycling notation is intended to look similar to the ISO 8601
date-time notation, but it is simpler for obvious reasons. The example suite
illustrates two recurrence forms,
\lstinline=Rn/start-point/period= and
\lstinline=Rn/period/stop-point=, simplified somewhat using suite context
information (namely the initial and final cycle points). The first form is
used to run one special task called \lstinline=start= at start-up, and for the
main cycling body of the suite; and the second form to run another special task
called \lstinline=stop= in the final two cycles. The \lstinline=P= character
denotes period (interval) just like in the date-time notation.
\lstinline=R/1/P2= would generate the sequence of points \lstinline=1,3,5,...=.

\begin{myitemize}
    \item For more on integer cycling, including a more realistic usage example
        see ~\ref{IntegerCycling}.
\end{myitemize}

\subsection{Jinja2}
\hilight{ suite: \lstinline=tut.oneoff.jinja2= }
\vspace{3mm}

Cylc has built in support for the Jinja2 template processor, which
allows us to embed code in suite definitions to generate the
final result seen by cylc.

The \lstinline=tut.oneoff.jinja2= suite illustrates two common
uses of Jinja2: changing suite content or structure based on the value
of a logical switch; and iteratively generating dependencies and runtime
configuration for groups of related tasks:
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/oneoff/jinja2/suite.rc}

To view the result of Jinja2 processing with the Jinja2 flag
\lstinline@MULTI@ set to \lstinline=False=:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc view --jinja2 --stdout tut.oneoff.jinja2
\end{lstlisting}
\lstset{language=suiterc}
\begin{lstlisting}
title = "A Jinja2 Hello World! suite"
[scheduling]
    [[dependencies]]
        graph = "hello"
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
\end{lstlisting}

And with \lstinline=MULTI= set to \lstinline=True=:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc view --jinja2 --stdout tut.oneoff.jinja2
\end{lstlisting}
\lstset{language=suiterc}
\begin{lstlisting}
title = "A Jinja2 Hello World! suite"
[scheduling]
    [[dependencies]]
        graph = "hello => BYE"
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
    [[BYE]]
        script = "sleep 10; echo Goodbye World!"
    [[ goodbye_0 ]]
        inherit = BYE
    [[ goodbye_1 ]]
        inherit = BYE
    [[ goodbye_2 ]]
        inherit = BYE
\end{lstlisting}

\subsection{Task Retry On Failure}

\hilight{ suite: \lstinline=tut.oneoff.retry= }
\vspace{3mm}

Tasks can be configured to retry a number of times if they fail.
An environment variable \lstinline=$CYLC_TASK_TRY_NUMBER= increments
from $1$ on each successive try, and is passed to the task to allow
different behaviour on the retry:
\lstset{language=suiterc}
\lstinputlisting{../examples/tutorial/oneoff/retry/suite.rc}

When a task with configured retries fails, its cylc task proxy goes
into the {\em retrying} state until the next retry delay is up, then it
resubmits. It only enters the {\em failed} state on a final definitive
failure.

Experiment with \lstinline=tut.oneoff.retry= to see how this works.

\subsection{Other Users' Suites}

If you have read access to another user's account (even on another host)
it is possible to use \lstinline=cylc monitor= to look at their suite's
progress without full shell access to their account. To do this, you
will need to copy their suite passphrase to
\lstset{language=transcript}
\begin{lstlisting}
    $HOME/.cylc/SUITE_OWNER@SUITE_HOST/SUITE_NAME/passphrase
\end{lstlisting}
(use of the host and owner names is optional here - see~\ref{passphrases})
{\em and} also retrieve the port number of the running suite from:
\begin{lstlisting}
    ~SUITE_OWNER/.cylc/ports/SUITE_NAME
\end{lstlisting}
Once you have this information, you can run
\begin{lstlisting}
shell$ cylc monitor --user=SUITE_OWNER --port=SUITE_PORT SUITE_NAME
\end{lstlisting}
to view the progress of their suite.

Other suite-connecting commands work in the same way too;
see~\ref{RemoteControl}.

\subsection{Searching A Suite}

The cylc suite search tool reports pattern matches in the suite definition
by line number, suite section, and file, even if the suite uses nested
include-files, and by file and line number for matches in suite bin
scripts:
\lstset{language=transcript}

\begin{lstlisting}
shell$ cylc search examples/admin/suite.rc OUTPUT_DIR

FILE: /home/hilary/cylc/examples/admin/suite.rc
   SECTION: [runtime]->[[root]]->[[[environment]]]
      (52):             OUTPUT_DIR = $WORKSPACE

FILE: /home/hilary/cylc/examples/admin/bin/A.sh
   (28): touch $OUTPUT_DIR/surface-winds-${CYLC_TASK_CYCLE_POINT}.nc
   (29): touch $OUTPUT_DIR/precipitation-${CYLC_TASK_CYCLE_POINT}.nc

#...
\end{lstlisting}

\subsection{Other Things To Try}

Almost every feature of cylc can be tested quickly and easily with a
simple dummy suite. You can write your own, or start from one of the
example suites in \lstinline=/path/to/cylc/examples= (see use of
\lstinline=cylc import-examples= above) - they all run ``out the box''
and can be copied and modified at will.

\begin{myitemize}

\item Change the suite runahead limit in a cycling suite.

\item Stop a suite mid-run with \lstinline=cylc stop=, and restart
it again with \lstinline=cylc restart=.

\item Hold (pause) a suite mid-run with \lstinline=cylc hold=,
    then modify the suite definition and \lstinline=cylc reload= it
    before using \lstinline=cylc release= to continue (you can also
    reload without holding).

\item Use the gcylc View menu to show the task state color key and
watch tasks in the \lstinline=task-states= example evolve
as the suite runs.

\item Manually re-run a task that has already completed or failed,
    with \lstinline=cylc trigger=.

\item Use an {\em internal queue} to prevent more than an alotted number
    of tasks from running at once even though they are ready -
   see~\ref{InternalQueues}.

\item Configure task event hooks to send an email, or shut the suite down,
    on task failure.

\end{myitemize}


\section{Suite Name Registration}
\label{SuiteRegistration}

Cylc commands target suites via names registered in a {\em suite name
database} located at \lstinline=$HOME/.cylc/REGDB/=. Suite names are
hierarchical like directory paths, allowing nested tree-like grouping,
but use the `.' character as a delimiter. This :
\begin{lstlisting}
shell$ cylc db print -t nwp
nwp
 |-oper
 | |-region1  Local Model Region1       /oper/nwp/suites/LocalModel/nested/Region1
 | `-region2  Local Model Region2       /oper/nwp/suites/LocalModel/nested/Region2
 `-test
   `-region1  Local Model TEST Region1  /home/hilary/suites/Regional/TESTS/Region1
\end{lstlisting}

Suite titles held in the name database are parsed from the suite
definition at the time of initial suite registration. If you change the
title later use \lstinline=cylc db refresh= to update the database.

Name groups are entirely virtual, they do not need to be
explicitly created before use, and they automatically disappear if all
tasks are removed from them. From the listing above, for example, to
move the suite \lstinline=nwp.oper.region2= into the
\lstinline=nwp.test= group:
\begin{lstlisting}
shell$ cylc db rereg nwp.oper.region2 nwp.test.region2
REREGISTER nwp.oper.region2 to nwp.test.region2
shell$ cylc db print -tx nwp
nwp
 |-oper
 | `-region1  Local Model Region1
 `-test
   |-region1  Local Model TEST Region1
   `-region2  Local Model Region2
\end{lstlisting}
And to move \lstinline=nwp.test.region2= into a new group \lstinline=nwp.para=:
\begin{lstlisting}
shell$ cylc db rereg nwp.test.region2 nwp.para.region2
REREGISTER nwp.test.region2 to nwp.para.region2
shell$ cylc db print -tx nwp
nwp
 |-oper
 | `-region1  Local Model Region1
 |-test
 | `-region1  Local Model TEST Region1
 `-para
   `-region2  Local Model Region2
\end{lstlisting}

Currently you cannot explicitly indicate a group name on the command
line by appending a dot character. Rather, in database operations such
as copy, reregister, or unregister, the identity of the source item
(group or suite) is inferred from the content of the database; and if
the source item is a group, so must the target be a group (or it will
be, in the case of an item that will be created by the operation). This
means that you cannot copy a single suite into a group that does not
exist yet unless you specify the entire target suite name.

\lstinline=cylc db register --help= shows a number of other examples.

\subsection{Suite Name Registration Commands}

On the command line, the  `database' (or `db') command category contains
commands to implement the aforementioned operations.

\lstset{language=usage}
\begin{lstlisting}
shell$ cylc db help
CATEGORY: db|database - Suite name registration, copying, deletion, etc.

Suite name registrations are held in a simple database $HOME/.cylc/REGDB
$ cat $HOME/.cylc/REGDB/my.suite
   title=my suite title
   path=/path/to/my/suite

HELP: cylc [db|database] COMMAND help,--help
  You can abbreviate db|database and COMMAND.
  The category db|database may be omitted.

COMMANDS:
  copy|cp ............. Copy a suite or a group of suites
  get-directory ....... Retrieve suite definition directory paths
  print ............... Print registered suites
  refresh ............. Report invalid registrations and update suite titles
  register ............ Register a suite for use
  reregister|rename ... Change the name of a suite
  unregister .......... Unregister and optionally delete suites
\end{lstlisting}

Groups of suites (at any level in the name hierarchy) can be deleted,
copied, imported, and exported; as well as individual suites. To do this,
just use suite group names as source and/or target for operations, as
appropriate. For instance, if a group \lstinline=foo.bar= contains the
suites \lstinline=foo.bar.baz= and \lstinline=foo.bar.qux=, you can copy
a single suite like this:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc copy foo.bar.baz boo $HOME/suites
\end{lstlisting}
(resulting in a new suite \lstinline=boo=); or the group like this:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc copy foo.bar boo $HOME/suites
\end{lstlisting}
(resulting in new suites \lstinline=boo.baz= and \lstinline=boo.qux=); or the group like this:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc copy foo boo $HOME/suites
\end{lstlisting}
(resulting in new suites \lstinline=boo.bar.baz= and \lstinline=boo.bar.qux=).
When suites are copied, the suite definition directories are copied into
a directory tree, under the target directory, that reflects the
suite name hierarchy. \lstinline=cylc copy --help= has some explicit examples.

The same functionality is also available by right-clicking on suites
or groups in the gcylc ``Open Registered Suite'' dialog.


%\pagebreak
\section{Suite Definition}
\label{SuiteDefinition}

Cylc suites are defined in structured, validated, {\em suite.rc} files
that concisely specify the properties of, and the relationships
between, the various tasks managed by the suite. This section of the
User Guide deals with the format and content of the suite.rc file,
including task definition. Task implementation - what's required of the
real commands, scripts, or programs that do the processing that the
tasks represent - is covered in~\ref{TaskImplementation}; and
task job submission - how tasks are submitted to run - is
in~\ref{TaskJobSubmission}.

\subsection{Suite Definition Directories}
\label{SuiteDefinitionDirectories}

A cylc {\em suite definition directory} contains:
\begin{myitemize}
    \item {\bf A suite.rc file}: this is the suite definition.
        \begin{myitemize}
            \item And any include-files used in it (see below; may be
                kept in sub-directories).
        \end{myitemize}
    \item {\bf A \lstinline=bin/= sub-directory} (optional)
        \begin{myitemize}
            \item For scripts and executables that implement, or are
                used by, suite tasks.
            \item Automatically added to \lstinline=$PATH= in task
                execution environments.
            \item Alternatively, tasks can call external
                commands, scripts, or programs; or they can be scripted
                entirely within the suite.rc file.
        \end{myitemize}
    \item {\bf A \lstinline=lib/python/= sub-directory} (optional)
        \begin{myitemize}
            \item For custom job submission modules
                (see~\ref{CustomJobSubmissionMethods})
                and local Python modules imported by custom Jinja2 filters
                (see~\ref{CustomJinja2Filters}).
        \end{myitemize}
    \item {\bf Any other sub-directories and files} - documentation,
        control files, etc. (optional)
        \begin{myitemize}
            \item Holding everything in one place makes proper suite
                revision control possible.
            \item Portable access to files here, for running tasks, is
                provided through
                \lstinline=$CYLC_SUITE_DEF_PATH=
                (see~\ref{TaskExecutionEnvironment}).
            \item Ignored by cylc, but the entire suite definition
                directory tree is copied when you copy a
                suite using cylc commands.

        \end{myitemize}
\end{myitemize}
A typical example:
\lstset{language=transcript}
\begin{lstlisting}
/path/to/my/suite   # suite definition directory
    suite.rc           # THE SUITE DEFINITION FILE
    bin/               # scripts and executables used by tasks
        foo.sh
        bar.sh
        ...
    # (OPTIONAL) any other suite-related files, for example:
    inc/               # suite.rc include-files
        nwp-tasks.rc
        globals.rc
        ...
    doc/               # documentation
    control/           # control files
    ancil/             # ancillary files
    ...
\end{lstlisting}

\subsection{Suite.rc File Overview}
\label{SuiteRCFile}

Suite.rc files are an extended-INI format with section nesting.

Embedded template processor expressions may also be used in the file, to
programatically generate the final suite definition seen by
cylc. Currently the Jinja2 template processor is supported
(\url{http://jinja.pocoo.org/docs}); see~\ref{Jinja2} for examples. In the
future cylc may provide a plug-in interface to allow use of other template
engines too.

\subsubsection{Syntax}
\label{Syntax}

The following defines legal suite.rc syntax:
\begin{myitemize}
    \item {\bf Items} are of the form \lstinline@item = value@.
    \item {\bf [Section]} headings are enclosed in square brackets.
    \item {\bf Sub-section [[nesting]]} is defined by repeated square brackets.
    \item Sections are {\bf closed} by the next section heading.
    \item {\bf Comments} (line and trailing) follow a hash character: \#
    \item {\bf List values} are comma-separated.
    \item {\bf Single-line string values} can be single-, double-, or un-quoted.
    \item {\bf Multi-line string values} are triple-quoted (using
        single or double quote characters).
    \item {\bf Boolean values} are capitalized: True, False.
    \item {\bf Leading and trailing whitespace} is ignored.
    \item {\bf Indentation} is optional but should be used for clarity.
    \item {\bf Continuation lines} follow a trailing backslash: \textbackslash
    \item {\bf Duplicate sections} add their items to those previously
        defined under the same section.
    \item {\bf Duplicate items} override, {\em except for dependency
        \lstinline=graph= strings, which are additive}.
    \item {\bf Include-files} \lstinline=%include inc/foo.rc= can be
        used as a verbatim inlining mechanism.
\end{myitemize}
Suites that embed Jinja2 code (see~\ref{Jinja2}) must
process to raw suite.rc syntax.

\subsubsection{Include-Files}

Cylc has native support for suite.rc include-files, which may help to
organize large suites. Inclusion boundaries are completely arbitrary -
you can think of include-files as chunks of the suite.rc file simply
cut-and-pasted into another file. Include-files may be included
multiple times in the same file, and even nested. Include-file paths
can be specified portably relative to the suite definition directory,
e.g.:
\lstset{language=suiterc}
\begin{lstlisting}
# include the file $CYLC_SUITE_DEF_PATH/inc/foo.rc:
%include inc/foo.rc
\end{lstlisting}

\paragraph{Editing Temporarily Inlined Suites}

Cylc's native file inclusion mechanism supports optional inlined
editing:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc edit --inline SUITE
\end{lstlisting}
The suite will be split back into its constituent include-files when you
exit the edit session. While editing, the inlined file becomes the
official suite definition so that changes take effect whenever you save
the file. See \lstinline=cylc prep edit --help= for more information.

\paragraph{Include-Files via Jinja2}

Jinja2 (\ref{Jinja2}) also has template inclusion functionality.

\subsubsection{Syntax Highlighting For Suite Definitions}
\label{SyntaxHighlighting}

\lstset{language=transcript}
Cylc comes with syntax files for a number of text editors:
\lstset{language=transcript}
\begin{lstlisting}
$CYLC_DIR/conf/cylc.vim     # vim
$CYLC_DIR/conf/cylc-mode.el # emacs
$CYLC_DIR/conf/cylc.lang    # gedit (and other gtksourceview programs)
$CYLC_DIR/conf/cylc.xml     # kate
\end{lstlisting}
Refer to comments at the top of each file to see how to use them.

\subsubsection{Gross File Structure}

Cylc suite.rc files consist of a suite title and description followed by
configuration items grouped under several top level section headings:

\begin{myitemize}
    \item {\bf [cylc] } - {\em non task-specific suite configuration}
    \item {\bf [scheduling] } - {\em determines when tasks are ready to run}
        \begin{myitemize}
            \item tasks with special behaviour, e.g. clock-trigger tasks
            \item the dependency graph, which defines the relationships
                between tasks
        \end{myitemize}
    \item {\bf [runtime] } - {\em determines how, where, and what to
        execute when tasks are ready}
        \begin{myitemize}
            \item script, environment, job submission, remote
                hosting, etc.
            \item suite-wide defaults in the {\em root} namespace
            \item a nested family hierarchy with common properties
                inherited by related tasks
        \end{myitemize}
    \item {\bf [visualization] } - suite graph styling
\end{myitemize}


\subsubsection{Validation}
\label{Validation}

Cylc suite.rc files are automatically validated against a specification
that defines all legal entries, values, options, and defaults. This
detects formatting errors, typographic errors, illegal items and illegal
values prior to run time. Some values are complex strings that require
further parsing by cylc to determine their correctness (this is also
done during validation). All legal entries are documented in the {\em
Suite.rc Reference} (\ref{SuiteRCReference}).

The validator reports the line numbers of detected errors. Here's an
example showing a section heading with a missing right bracket:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc validate my.suite
    [[special tasks]
'Section bracket mismatch, line 19'
\end{lstlisting}

If the suite.rc file uses include-files \lstinline=cylc view= will
show an inlined copy of the suite with correct line numbers
(you can also edit suites in a temporarily inlined state with
\lstinline=cylc edit --inline=).

Validation does not check the validity of chosen job submission methods.
%this is to allow users to extend cylc with their own job submission
%methods, which are by definition unknown to the suite.rc spec.

\subsection{Scheduling - Dependency Graphs}
\label{ConfiguringScheduling}

\lstset{language=suiterc}
The \lstinline=[scheduling]= section of a suite.rc file defines the
relationships between tasks in a suite - the information that allows
cylc to determine when tasks are ready to run. The most important
component of this is the suite dependency graph. Cylc graph notation
makes clear textual graph representations that are very concise because
sections of the graph that repeat at different hours of the day, say,
only have to be defined once. Here's an example with dependencies that
vary depending on the particular cycle point:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 20200401
    final cycle point = 20200405
    [[dependencies]]
        [[[T00,T06,T12,T18]]] # validity (hours)
            graph = """
A => B & C   # B and C trigger off A
A[-PT6H] => A  # Model A restart trigger
                    """
        [[[T06,T18]]] # hours
            graph = "C => X"
\end{lstlisting}
\lstset{language=transcript}
Figure~\ref{fig-dep-eg-1} shows the complete suite.rc listing alongside
the suite graph.
This is a complete, valid, runnable suite (it will use default
task runtime properties such as \lstinline=script=).

\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
title = "Dependency Example 1"
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20200401
    final cycle point = 20200405
    [[dependencies]]
        [[[T00,T06,T12,T18]]] # validity (hours)
            graph = """
A => B & C   # B and C trigger off A
A[-PT6H] => A  # Model A restart trigger
                    """
        [[[T06,T18]]] # hours
            graph = "C => X"
[visualization]
    initial cycle point = 20200401
    final cycle point = 20200401T06
    [[node attributes]]
        X = "color=red"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/dep-eg-1.png}
    \end{center}
\end{minipage}
\caption[Example Suite]{\scriptsize Example Suite}
\label{fig-dep-eg-1}
\end{figure}

\subsubsection{Graph String Syntax}

Multiline graph strings may contain:
\begin{myitemize}
    \item {\bf blank lines}
    \item {\bf arbitrary white space}
    \item {\bf internal comments:} following the \lstinline=#= character
    \item {\bf conditional task trigger expressions} - see below.
\end{myitemize}

\subsubsection{Interpreting Graph Strings}

Suite dependency graphs can be broken down into pairs in which the left
side (which may be a single task or family, or several that are
conditionally related) defines a trigger for the task or family on the
right. For instance the ``word graph'' {\em C triggers off B which
triggers off A} can be deconstructed into pairs {\em C triggers off B}
and {\em B triggers off A}. In this section we use only the default
trigger type, which is to trigger off the upstream task succeeding;
see~\ref{TriggerTypes} for other available triggers.

In the case of cycling tasks, the triggers defined by a graph string are
valid for cycle points matching the list of hours specified for the
graph section. For example this graph:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "A => B"
\end{lstlisting}
\lstset{language=transcript}
implies that B triggers off A for cycle points in which the hour matches $00$
or $12$.

To define inter-cycle dependencies, attach an offset indicator to the
left side of a pair:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "A[-PT12H] => B"
\end{lstlisting}
\lstset{language=transcript}
This means B[time] triggers off A[time-PT12H] (12 hours before) for cycle
points with hours matching $00$ or $12$. $time$ is implicit because this keeps
graphs clean and concise, given that the majority of tasks will typically
depend only on others with the same cycle point. Cycle point offsets can only
appear on the left of a pair, because a pairs define triggers for the right
task at cycle point $time$. However, \lstinline@A => B[-PT6H]@, which is
illegal, can be reformulated as a {\em future trigger}
\lstinline@A[+PT6H] => B@ (see~\ref{InterCyclePointTriggers}). It is also
possible to combine multiple offsets within a cycle point offset e.g.
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "A[-P1D-PT12H] => B"
\end{lstlisting}
\lstset{language=transcript}
This means that B[Time] triggers off A[time-P1D-PT12H] (1 day and 12 hours
before).

Triggers can be chained together. This graph:
\lstset{language=suiterc}
\begin{lstlisting}
    graph = """A => B  # B triggers off A
               B => C  # C triggers off B"""
\end{lstlisting}
is equivalent to this:
\begin{lstlisting}
    graph = "A => B => C"
\end{lstlisting}
\lstset{language=transcript}

{\em Each trigger in the graph must be unique} but {\em the same task
can appear in multiple pairs or chains}. Separately defined triggers
for the same task have an AND relationship. So this:
\lstset{language=suiterc}
\begin{lstlisting}
    graph = """A => X  # X triggers off A
               B => X  # X also triggers off B"""
\end{lstlisting}
\lstset{language=transcript}
\lstset{language=suiterc}
is equivalent to this:
\begin{lstlisting}
    graph = "A & B => X"  # X triggers off A AND B
\end{lstlisting}
\lstset{language=transcript}

In summary, the branching tree structure of a dependency graph can
be partitioned into lines (in the suite.rc graph string) of pairs
or chains, in any way you like, with liberal use of internal white space
and comments to make the graph structure as clear as possible.

\begin{lstlisting}
# B triggers if A succeeds, then C and D trigger if B succeeds:
    graph = "A => B => C & D"
# which is equivalent to this:
    graph = """A => B => C
               B => D"""
# and to this:
    graph = """A => B => D
               B => C"""
# and to this:
    graph = """A => B
               B => C
               B => D"""
# and it can even be written like this:
    graph = """A => B # blank line follows:

               B => C # comment ...
               B => D"""
\end{lstlisting}

\paragraph{Handling Long Graph Lines}

Long chains of dependencies can be split into pairs:
\begin{lstlisting}
    graph = "A => B => C"
# is equivalent to this:
    graph = """A => B
               B => C"""
# BUT THIS IS AN ERROR:
    graph = """A => B => # WRONG!
               C"""      # WRONG!
\end{lstlisting}
If you have very long task names, or long conditional trigger
expressions (below) then you can use the suite.rc line continuation
marker:
\begin{lstlisting}
    graph = "A => B \
    => C"  # OK
\end{lstlisting}
Note that a line continuation marker must be the final character on
the line; it cannot be followed by trailing spaces or a comment.

\lstset{language=transcript}

\subsubsection{Graph Types}
\label{GraphTypes}

A suite definition can contain multiple graph strings that are combined
to generate the final graph.

\paragraph{One-off (Non-Cycling)}

Figure~\ref{fig-test1} shows a small suite of one-off non-cycling
tasks; these all share a single cycle point (\lstinline=1=) and don't spawn
successors (once they're all finished the suite just exits). The integer
\lstinline=1= attached to each graph node is just an arbitrary label here.
\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
title = some one-off tasks
[scheduling]
    [[dependencies]]
        graph = "foo => bar & baz => qux"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=0.25\textwidth]{graphics/png/orig/test1.png}
    \end{center}
\end{minipage}
\caption[One-off (Non-Cycling) Tasks]{\scriptsize One-off (Non-Cycling) Tasks.}
\label{fig-test1}
\end{figure}

\paragraph{Cycling Graphs}

For cycling tasks the graph section heading defines a sequence of cycle points
for which the subsequent graph section is valid. Figure~\ref{fig-test2} shows
a small suite of cycling tasks.
\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
title = some cycling tasks
# (no dependence between cycle points)
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "foo => bar & baz => qux"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/test2.png}
    \end{center}
\end{minipage}
\caption[Cycling Tasks]{\scriptsize Cycling Tasks.}
\label{fig-test2}
\end{figure}

\subsubsection{Graph Section Headings}

Graph section headings define recurrence expressions, the graph within a graph
section heading defines a workflow at each point of the recurrence. For
example in the following scenario:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[ T06 ]]]  # A graph section heading
            graph = foo => bar
\end{lstlisting}
\lstset{language=transcript}

\lstinline=T06= means "Run every day starting at 06:00 after the
initial cycle point". Cylc allows you to start (or end) at any particular
time, repeat at whatever frequency you like, and even optionally limit the
number of repetitions.

Graph section heading can also be used with integer cycling see
\ref{IntegerCycling}.

\paragraph{Syntax Rules}

Date-time cycling information is made up of a starting {\em date-time}, an
{\em interval}, and an optional {\em limit}.

The time is assumed to be in the local time zone unless you set
\lstinline=[cylc]cycle point time zone= or \lstinline=[cylc]UTC mode=. The
calendar is assumed to be the proleptic Gregorian calendar unless you set
\lstinline=[scheduling]cycling mode=.

The syntax for representations is based on the ISO 8601 date-time standard.
This includes the representation of {\em date-time}, {\em interval}. What we
define for cylc's cycling syntax is our own optionally-heavily-condensed form
of ISO 8601 recurrence syntax. The most common full form is:
\lstinline=R[limit?]/[date-time]/[interval]=. However, we allow omitting
information that can be guessed from the context (rules below). This means
that it can be written as:
\begin{lstlisting}
R[limit?]/[date-time]
R[limit?]//[interval]
[date-time]/[interval]
R[limit?] # Special limit of 1 case
[date-time]
[interval]
\end{lstlisting}

with example graph headings for each form being:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R5/T00 ]]]           # Run 5 times at 00:00 every day
[[[ R//PT1H ]]]          # Run every hour (Note the R// is redundant)
[[[ 20000101T00Z/P1D ]]] # Run every day starting at 00:00 1st Jan 2000
[[[ R1 ]]]               # Run once at the initial cycle point
[[[ 20000101T00Z ]]]     # Run once at 00:00 1st Jan 2000
[[[ P1Y ]]]              # Run every year
\end{lstlisting}

Note that \lstinline=T00= is an example of \lstinline=[date-time]=, with an
inferred 1 day period and no limit.

Where some or all {\em date-time} information is omitted, it is inferred to
be relative to the initial date-time cycle point. For example, \lstinline=T00=
by itself would mean the next occurrence of midnight that follows, or is, the
initial cycle point. Entering \lstinline=+PT6H= would mean 6 hours after the
initial cycle point. Entering \lstinline=-P1D= would mean 1 day before the
initial cycle point. Entering no information for the {\em date-time} implies
the initial cycle point date-time itself.

Where the {\em interval} is omitted and some (but not all) {\em date-time}
information is omitted, it is inferred to be a single unit above
the largest given specific {\em date-time } unit. For example, the largest
given specific unit in \lstinline=T00= is hours, so the inferred interval is
1 day (daily), \lstinline=P1D=.

Where the {\em limit} is omitted, unlimited cycling is assumed. This will be
bounded by the final cycle point's date-time if given.

Another supported form of ISO 8601 recurrence is:
\lstinline=R[limit?]/[interval]/[date-time]=. This form uses the
{\em date-time } as the end of the cycling sequence rather than the start.
For example, \lstinline=R3/P5D/20140430T06= means:
\begin{lstlisting}
20140420T06
20140425T06
20140430T06
\end{lstlisting}

This kind of form can be used for specifying special behaviour near the end of
the suite, at the final cycle point's date-time. We can also represent this in
cylc with a collapsed form:
\begin{lstlisting}
R[limit?]/[interval]
R[limit?]//[date-time]
[interval]/[date-time]
\end{lstlisting}

So, for example, you can write:
\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1//+P0D ]]]  # Run once at the final cycle point
[[[ R5/P1D ]]]    # Run 5 times, every 1 day, ending at the final
                  # cycle point
[[[ P2W/T00 ]]]   # Run every 2 weeks ending at 00:00 following
                  # the final cycle point
[[[ R//T00 ]]]    # Run every 1 day ending at 00:00 following the
                  # final cycle point
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Referencing The Initial And Final Cycle Points}
\label{referencing-the-initial-and-final-cycle-points}

For convenience the caret and dollar symbols may be used as shorthand for the
initial and final cycle points. Using this shorthand you can write:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1/^+PT12H ]]]  # Repeat once 12 hours after the initial cycle point
                    # R[limit]/[date-time]
                    # Equivalent to [[[ R1/+PT12H ]]]
[[[ R1/$ ]]]        # Repeat once at the final cycle point
                    # R[limit]/[date-time]
                    # Equivalent to [[[ R1//+P0D ]]]
[[[ $-P2D/PT3H ]]]  # Repeat 3 hourly starting two days before the
                    # [date-time]/[interval]
                    # final cycle point
\end{lstlisting}
\lstset{language=transcript}

Note that there can be multiple ways to write the same headings, for instance
the following all run once at the final cycle point:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1/P0Y ]]]      # R[limit]/[interval]
[[[ R1/P0Y/$ ]]]    # R[limit]/[interval]/[date-time]
[[[ R1/$ ]]]        # R[limit]/[date-time]
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Excluding Dates}
\label{excluding-dates}
\lstset{language=suiterc}

Datetimes can be excluded from a recurrence by an exclamation mark for example
\lstinline=[[[ PT1D!20000101 ]]]= means run daily except on the
first of January 2000.

This syntax can only be used to exclude one datetime from a recurrence. Note
that the \lstinline=^= and \lstinline=$= symbols (shorthand for the initial
and final cycle points) are both datetimes so \lstinline=[[[ T12!$-PT1D ]]]=
is valid.

If using a run limit in combination with an exclusion, the heading might not
run the number of times specified in the limit. For example in the following
suite \lstinline=foo= will only run once as its second run has been excluded.

\begin{lstlisting}
[scheduling]
    initial cycle point = 20000101T00Z
    final cycle point = 20000105T00Z
    [[dependencies]]
        [[[ R2/P1D!20000102 ]]]
            graph = foo
\end{lstlisting}
\lstset{language=transcript}

\paragraph{How Multiple Graph Strings Combine}
\label{HowMultipleGraphStringsCombine}

For a cycling graph with multiple validity sections for different
hours of the day, the different sections {\em add} to generate the
complete graph. Different graph sections can overlap (i.e.\ the same
hours may appear in multiple section headings) and the same tasks may
appear in multiple sections, but individual dependencies should be
unique across the entire graph. For example, the following graph defines
a duplicate prerequisite for task C:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            graph = "A => B => C"
        [[[T06,T18]]]
            graph = "B => C => X"
            # duplicate prerequisite: B => C already defined at T06, T18
\end{lstlisting}
\lstset{language=transcript}
This does not affect scheduling, but for the sake of clarity and brevity
the graph should be written like this:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            graph = "A => B => C"
        [[[T06,T18]]]
            # X triggers off C only at 6 and 18 hours
            graph = "C => X"
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Advanced Examples}
\label{AdvancedCycling}

The following examples show the various ways of writing graph headings in cylc.
\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1 ]]]         # Run once at the initial cycle point
[[[ P1D ]]]        # Run every day starting at the initial cycle point
[[[ PT5M ]]]       # Run every 5 minutes starting at the initial cycle
                   # point
[[[ T00/P2W ]]]    # Run every 2 weeks starting at 00:00 after the
                   # initial cycle point
[[[ +P5D/P1M ]]]   # Run every month, starting 5 days after the initial
                   # cycle point
[[[ R1/T06 ]]]     # Run once at 06:00 after the initial cycle point
[[[ R1/P0Y ]]]     # Run once at the final cycle point
[[[ R1/$ ]]]       # Run once at the final cycle point (alternative
                   # form)
[[[ R1/$-P3D ]]]   # Run once three days before the final cycle point
[[[ R3/T0830 ]]]   # Run 3 times, every day at 08:30 after the initial
                   # cycle point
[[[ R3/01T00 ]]]   # Run 3 times, every month at 00:00 on the first
                   # of the month after the initial cycle point
[[[ R5/W-1/P1M ]]] # Run 5 times, every month starting on Monday
                   # following the initial cycle point
[[[ T00!^ ]]]      # Run at the first occurrence of T00 that isn't the
                   # initial cycle point
[[[ PT1D!20000101 ]]]  # Run every day days excluding 1st Jan 2000
[[[ 20140201T06/P1D ]]]    # Run every day starting at 20140201T06
[[[ R1/min(T00,T06,T12,T18) ]]]  # Run once at the first instance
                                 # of either T00, T06, T12 or T18
                                 # starting at the initial cycle
                                 # point
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Advanced Starting Up}
\label{AdvancedStartingUp}

Dependencies that are only valid at the initial cycle point can be written
using the \lstinline=R1= notation (e.g. as
in~\ref{initial-non-repeating-r1-tasks}. For example:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep => foo"
        [[[T00]]]
            graph = "foo[-P1D] => foo => bar"
\end{lstlisting}
\lstset{language=transcript}

In the example above, \lstinline=R1= implies \lstinline=R1/20130808T00=, so
\lstinline=prep= only runs once at that cycle point (the initial cycle point).
At that cycle point, \lstinline=foo= will have a dependence on
\lstinline=prep= - but not at subsequent cycle points.

However, it is possible to have a suite that has multiple effective initial
cycles - for example, one starting at \lstinline=T00= and another starting
at \lstinline=T12=. What if they need to share an initial task?

Let's suppose that we add the following section to the suite example above:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep => foo"
        [[[T00]]]
            graph = "foo[-P1D] => foo => bar"
        [[[T12]]]
            graph = "baz[-P1D] => baz => qux"
\end{lstlisting}
\lstset{language=transcript}

We'll also say that there should be a starting dependence between
\lstinline=prep= and our new task \lstinline=baz= - but we still want to have
a single \lstinline=prep= task, at a single cycle.

We can write this using a special case of the \lstinline=task[-interval]= syntax -
if the interval is null, this implies the task at the initial cycle point.

For example, we can write our suite like~\ref{fig-test4}.

\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep"
        [[[R1/T00]]]
# ^ implies the initial cycle point:
     graph = "prep[^] => foo"
        [[[R1/T12]]]
# ^ is initial cycle point, as above:
     graph = "prep[^] => baz"
        [[[T00]]]
     graph = "foo[-P1D] => foo => bar"
        [[[T12]]]
     graph = "baz[-P1D] => baz => qux"
[visualization]
    initial cycle point = 20130808T00
    final cycle point = 20130810T00
    [[node attributes]]
        foo = "color=red"
        bar = "color=orange"
        baz = "color=green"
        qux = "color=blue"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/test4.png}
    \end{center}
\end{minipage}
\caption[Staggered Start Suite]{\scriptsize Staggered Start Suite}
\label{fig-test4}
\end{figure}

This neatly expresses what we want - a task running at the initial cycle point
that has one-off dependencies with other task sets at different cycles.

\begin{figure}
\begin{minipage}[h]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130808T18
    [[dependencies]]
        [[[R1]]]
            graph = "setup_foo => foo"
        [[[+PT6H/PT6H]]]
            graph = """
                foo[-PT6H] => foo
                foo => bar
            """
[visualization]
    initial cycle point = 20130808T00
    final cycle point = 20130808T18
    [[node attributes]]
        foo = "color=red"
        bar = "color=orange"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[h]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/test5.png}
    \end{center}
\end{minipage}
\caption[Restricted First Cycle Point Suite]{
    \scriptsize Restricted First Cycle Point Suite}
\label{fig-test5}
\end{figure}


A different kind of requirement is displayed in Figure \ref{fig-test5}.
Usually, we want to specify additional tasks and dependencies at the initial
cycle point. What if we want our first cycle point to be entirely special, with
some tasks missing compared to subsequent cycle points?

In Figure \ref{fig-test5}, \lstinline=bar= will not be run at the initial
cycle point, but will still run at subsequent cycle points.
\lstinline=[[[+PT6H/PT6H]]]= means start at \lstinline=+PT6H= (6 hours after
the initial cycle point) and then repeat every \lstinline=PT6H= (6 hours).

Some suites may have staggered start-up sequences where different tasks need
running once but only at specific cycle points, potentially due to differing
data sources at different cycle points with different possible initial cycle
points. To allow this cylc provides a \lstinline=min( )= function that can be
used as follows:

\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20100101T03
    [[dependencies]]
        [[[R1/min(T00,T12)]]]
            graph = "prep1 => foo"
        [[[R1/min(T06,T18)]]]
            graph = "prep2 => foo"
        [[[T00,T06,T12,T18]]]
            graph = "foo => bar"

\end{lstlisting}
\lstset{language=transcript}


In this example the initial cycle point is \lstinline=20100101T03=, so the
\lstinline=prep1= task will run once at \lstinline=20100101T12= and the
\lstinline=prep2= task will run once at \lstinline=20100101T06= as these are
the first cycle points after the initial cycle point in the respective
\lstinline=min( )= entries.


\paragraph{Integer Cycling}
\label{IntegerCycling}

In addition to non-repeating and date-time cycling workflows, cylc can do
integer cycling for repeating workflows that are not date-time based.

To construct an integer cycling suite, set
\lstinline@[scheduling]cycling mode = integer@, and specify integer values for
the initial and (optional) final cycle points. The notation for intervals,
offsets, and recurrences (sequences) is similar to the date-time cycling
notation, except for the simple integer values.

The full integer recurrence expressions supported are:
\begin{myitemize}
    \item \lstinline@Rn/start-point/interval # e.g. R3/1/P2@
    \item \lstinline@Rn/interval/end-point # e.g. R3/P2/9@
\end{myitemize}
But, as for date-time cycling, sequence start and end points can be omitted
where suite initial and final cycle points can be assumed. Some examples:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1 ]]]        # Run once at the initial cycle point
                  # (short for R1/initial-point/?)
[[[ P1 ]]]        # Repeat with step 1 from the initial cycle point
                  # (short for R/initial-point/P1)
[[[ P5 ]]]        # Repeat with step 5 from the initial cycle point
                  # (short for R/initial-point/P5)
[[[ R2//P2 ]]]    # Run twice with step 3 from the initial cycle point
                  # (short for R2/initial-point/P2)
[[[ R/+P1/P2 ]]]  # Repeat with step 2, from 1 after the initial cycle point
[[[ R2/P2 ]]]     # Run twice with step 2, to the final cycle point
                  # (short for R2/P2/final-point)
[[[ R1/P0 ]]]     # Run once at the final cycle point
                  # (short for R1/P0/final-point)
\end{lstlisting}

\subparagraph{Example}

The tutorial illustrates integer cycling in~\ref{TutInteger}, and
\lstinline=$CYLC_DIR/examples/satellite/= is a
self-contained example of a realistic use for integer cycling. It simulates
the processing of incoming satellite data: each new dataset arrives after a
random (as far as the suite is concerned) interval, and is labeled by an
arbitrary (as far as the suite is concerned) ID in the filename. A task called
\lstinline=get_data= at the top of the repeating workflow waits on the next
dataset and, when it finds one, moves it to a cycle-point-specific shared
workspace for processing by the downstream tasks. When \lstinline=get_data.1=
finishes, \lstinline=get_data.2= triggers and begins waiting for the next
dataset at the same time as the downstream tasks in cycle point 1 are
processing the first one, and so on. In this way multiple datasets can be
processed at once if they happen to come in quickly. A single shutdown task
runs at the end of the final cycle to collate results. The suite graph is
shown in Figure~\ref{fig-satellite}.

\begin{figure}
    \begin{center}
        \includegraphics[width=0.4\textwidth]{graphics/png/orig/satellite.png}
    \end{center}
    \caption{The \lstinline=examples/satellite= integer suite}
\label{fig-satellite}
\end{figure}

\subparagraph{Advanced Integer Cycling Syntax}

The same syntax used to reference the initial and final cycle points
introduced in~\ref{referencing-the-initial-and-final-cycle-points}) for
use with datetime cycling can also be used for integer cycling. For
example you can write:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1/^ ]]]     # Run once at the initial cycle point
[[[ R1/$ ]]]     # Run once at the final cycle point
[[[ R3/^/P2 ]]]  # Run three times with step two starting at the
                 # initial cycle point
\end{lstlisting}
\lstset{language=transcript}

Likewise the syntax introduced in~\ref{excluding-dates} for excluding
a particular point from a recurrence also works for integer cycling. For
example:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R/P4!8 ]]]       # Run with step 4, to the final cycle point
                     # but not at point 8
[[[ R3/3/P2!5 ]]]    # Run with step 2 from point 3 but not at
                     # point 5
[[[ R/+P1/P6!14 ]]]  # Run with step 6 from 1 step after the
                     # initial cycle point but not at point 14
\end{lstlisting}
\lstset{language=transcript}


\subsubsection{Trigger Types}
\label{TriggerTypes}

\lstset{language=suiterc}

Trigger type, indicated by {\em:type} after the upstream task (or family)
name, determines what kind of event results in the downstream task (or
family) triggering.

\paragraph{Success Triggers}

The default, with no trigger type specified, is to trigger off the
upstream task succeeding:
\begin{lstlisting}
# B triggers if A SUCCEEDS:
    graph = "A => B"
\end{lstlisting}
For consistency and completeness, however, the success trigger can be explicit:
\begin{lstlisting}
# B triggers if A SUCCEEDS:
    graph = "A => B"
# or:
    graph = "A:succeed => B"
\end{lstlisting}

\paragraph{Failure Triggers}

To trigger off the upstream task reporting failure:
\begin{lstlisting}
# B triggers if A FAILS:
    graph = "A:fail => B"
\end{lstlisting}
{\em Suicide triggers} can be used to remove task \lstinline=B= here if
\lstinline=A= does not fail, see~\ref{SuicideTriggers}.

\paragraph{Start Triggers}

To trigger off the upstream task starting to execute:
\begin{lstlisting}
# B triggers if A STARTS EXECUTING:
    graph = "A:start => B"
\end{lstlisting}
This can be used to trigger tasks that monitor other tasks once they
(the target tasks) start executing. Consider a long-running forecast model,
for instance, that generates a sequence of output files as it runs. A
postprocessing task could be launched with a start trigger on the model
(\lstinline@model:start => post@) to process the model output as it
becomes available. Note, however, that there are several alternative
ways of handling this scenario: both tasks could be triggered at the
same time (\lstinline@foo => model & post@), but depending on
external queue delays this could result in the monitoring task starting
to execute first; or a different postprocessing task could be
triggered off a message output for each data file
(\lstinline@model:out1 => post1@ etc.; see~\ref{MessageTriggers}), but this
may not be practical if the
number of output files is large or if it is difficult to add cylc
messaging calls to the model.

\paragraph{Finish Triggers}

To trigger off the upstream task succeeding or failing, i.e.\ finishing
one way or the other:
\begin{lstlisting}
# B triggers if A either SUCCEEDS or FAILS:
    graph = "A | A:fail => B"
# or
    graph = "A:finish => B"
\end{lstlisting}

\paragraph{Message Triggers}
\label{MessageTriggers}

Tasks can also trigger off custom output messages. These must be registered in
the \lstinline=[runtime]= section of the emitting task, and reported using the
\lstinline=cylc message= command in task scripting. The graph trigger notation
refers to the item name of the registered output message.
The example suite \lstinline=$CYLC_DIR/examples/message-triggers= illustrates
message triggering.

\lstset{language=suiterc}
\lstinputlisting{../examples/message-triggers/suite.rc}

\paragraph{Job Submission Triggers}

It is also possible to trigger off a task submitting, or failing to submit:
\begin{lstlisting}
# B triggers if A submits successfully:
    graph = "A:submit => B"
# D triggers if C fails to submit successfully:
    graph = "C:submit-fail => D"
\end{lstlisting}

A possible use case for submit-fail triggers: if a task goes into the
submit-failed state, possibly after several job submission retries,
another task that inherits the same runtime but sets a different job
submission method and/or host could be triggered to, in effect, run the
same job on a different platform.


\paragraph{Conditional Triggers}

AND operators (\lstinline=&=) can appear on both sides of an arrow. They
provide a concise alternative to defining multiple triggers separately:
\begin{lstlisting}
# 1/ this:
    graph = "A & B => C"
# is equivalent to:
    graph = """A => C
               B => C"""
# 2/ this:
    graph = "A => B & C"
# is equivalent to:
    graph = """A => B
               A => C"""
# 3/ and this:
    graph = "A & B => C & D"
# is equivalent to this:
    graph = """A => C
               B => C
               A => D
               B => D"""
\end{lstlisting}

OR operators (\lstinline=|=) which result in true conditional triggers,
can only appear on the left,\footnote{An OR
operator on the right doesn't make much sense: if ``B or C'' triggers
off A, what exactly should cylc do when A finishes?}
\begin{lstlisting}
# C triggers when either A or B finishes:
    graph = "A | B => C"
\end{lstlisting}

Forecasting suites typically have simple conditional
triggering requirements, but any valid conditional expression can be
used, as shown in Figure~\ref{fig-conditional}
(conditional triggers are plotted with open arrow heads).
\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
        graph = """
# D triggers if A or (B and C) succeed
A | B & C => D
# just to align the two graph sections
D => W
# Z triggers if (W or X) and Y succeed
(W|X) & Y => Z
                """
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/conditional-triggers.png}
    \end{center}
\end{minipage}
\caption[Conditional Triggers] {\scriptsize
Conditional triggers are plotted with open arrow heads.}
\label{fig-conditional}
\end{figure}

\paragraph{Suicide Triggers}
\label{SuicideTriggers}

Suicide triggers take tasks out of the suite. This can be used for
automated failure recovery. The suite.rc listing and accompanying
graph in Figure~\ref{fig-suicide} show how to define a chain of failure
recovery tasks
that trigger if they're needed but otherwise remove themselves from the
suite (you can run the {\em AutoRecover.async} example suite to see how
this works). The dashed graph edges ending in solid dots indicate
suicide triggers, and the open arrowheads indicate conditional triggers
as usual. Suicide triggers are ignored by default in the graph view, unless you toggle them on with {\em View} -> {\em Options} -> {\em Ignore Suicide Triggers}.

\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
title = automated failure recovery
description = """
Model task failure triggers diagnosis
and recovery tasks, which take themselves
out of the suite if model succeeds. Model
post processing triggers off model OR
recovery tasks.
              """
[scheduling]
    [[dependencies]]
        graph = """
pre => model
model:fail => diagnose => recover
model => !diagnose & !recover
model | recover => post
                """
[runtime]
    [[model]]
        # UNCOMMENT TO TEST FAILURE:
        # script = /bin/false
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/suicide.png}
    \end{center}
\end{minipage}
\caption[Automated failure recovery via suicide triggers] {\scriptsize
Automated failure recovery via suicide triggers.}
\label{fig-suicide}
\end{figure}

Note that multiple suicide triggers combine in the same way as other triggers, so this:
\begin{lstlisting}
foo => !baz
bar => !baz
\end{lstlisting}
is equivalent to this:
\begin{lstlisting}
foo & bar => !baz
\end{lstlisting}
i.e.\ both \lstinline=foo= and \lstinline=bar= must succeed for
\lstinline=baz= to be taken out of the suite. If you really want a task
to be taken out if any one of several events occurs then be careful to
write it that way:
\begin{lstlisting}
foo | bar => !baz
\end{lstlisting}

A word of warning on the meaning of ``bare suicide triggers''. Consider
the following suite:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => !bar"
\end{lstlisting}
Task \lstinline=bar= has a suicide trigger but no normal prerequisites
(a suicide trigger is not a task triggering prerequisite, it is a task
removal prerequisite) so this is entirely equivalent to:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
            foo & bar
           foo => !bar
                """
\end{lstlisting}
In other words both tasks will trigger immediately, at the same time,
and then \lstinline=bar= will be removed if \lstinline=foo= succeeds.

If an active task proxy (currently in the submitted or running states)
is removed from the suite by a suicide trigger, a warning will be logged.

\paragraph{Family Triggers}
\label{FamilyTriggers}

Families defined by the namespace inheritance hierarchy
(~\ref{NIORP}) can be used in the graph trigger whole groups of
tasks at the same time (e.g.\ forecast model ensembles and groups of
tasks for processing different observation types at the same time) and
for triggering downstream tasks off families as a whole. Higher level
families, i.e.\ families of families, can also be used, and are reduced
to the lowest level member tasks. Note that tasks can also trigger off
individual family members if necessary.

To trigger an entire task family at once:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => FAM"
[runtime]
    [[FAM]]    # a family (because others inherit from it)
    [[m1,m2]]  # family members (inherit from namespace FAM)
        inherit = FAM
\end{lstlisting}
This is equivalent to:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => m1 & m2"
[runtime]
    [[FAM]]
    [[m1,m2]]
        inherit = FAM
\end{lstlisting}

To trigger other tasks off families we have to specify whether
to triggering off {\em all members} starting, succeeding, failing,
or finishing, or off {\em any} members (doing the same). Legal family
triggers are thus:

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
      # all-member triggers:
    FAM:start-all => one
    FAM:succeed-all => one
    FAM:fail-all => one
    FAM:finish-all => one
      # any-member triggers:
    FAM:start-any => one
    FAM:succeed-any => one
    FAM:fail-any => one
    FAM:finish-any => one
                """
\end{lstlisting}

Here's how to trigger downstream processing after if one or more family
members succeed, but only after all members have finished (succeeded or
failed):

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
    FAM:finish-all & FAM:succeed-any => foo
                """
\end{lstlisting}

\paragraph{Writing Efficient Inter-Family Triggering}
\label{EfficientInterFamilyTriggering}

While cylc allows writing dependencies between two families it is important to
consider the number of dependencies this will generate. In the following
example, each member of \lstinline=FAM2= has dependencies pointing at all the
members of \lstinline=FAM1=.

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
    FAM1:succeed-any => FAM2
                """
\end{lstlisting}

Expanding this out, you generate \lstinline=N * M= dependencies, where
\lstinline=N= is the number of members of \lstinline=FAM1= and \lstinline=M= is
the number of members of \lstinline=FAM2=. This can result in high memory use
as the number of members of these families grows, potentially rendering the
suite impractical for running on some systems.

You can greatly reduce the number of dependencies generated in these situations
by putting dummy tasks in the graphing to represent the state of the family you
want to trigger off. For example, if \lstinline=FAM2= should trigger off any
member of \lstinline=FAM1= succeeding you can create a dummy task
\lstinline=FAM1_succeed_any_marker= and place a dependency on it as follows:

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
    FAM1:succeed-any => FAM1_succeed_any_marker => FAM2
                """
[runtime]
...
    [[FAM1_succeed_any_marker]]
        script = true
...
\end{lstlisting}

This graph generates only \lstinline=N + M= dependencies, which takes
significantly less memory and CPU to store and evaluate.

\paragraph{Inter-Cycle Triggers}
\label{InterCyclePointTriggers}

Typically most tasks in a suite will trigger off others in the same
cycle point, but some may depend on others with other cycle points.
This notably applies to warm-cycled forecast models, which depend on
their own previous instances (see below); but other kinds of inter-cycle
dependence are possible too.\footnote{In NWP forecast analysis
suites parts of the observation processing and data assimilation
subsystem will typically also depend on model background fields
generated by the previous forecast.} Here's how to express this
kind of relationship in cylc:
\begin{lstlisting}
[dependencies]
    [[PT6H]]
        # B triggers off A in the previous cycle point
        graph = "A[-PT6H] => B"
\end{lstlisting}
inter-cycle and trigger type (or message trigger) notation can be
combined:
\begin{lstlisting}
    # B triggers if A in the previous cycle point fails:
    graph = "A[-PT6H]:fail => B"
\end{lstlisting}

At suite start-up inter-cycle triggers refer to a previous cycle point
that does not exist. This does not cause the dependent task to wait
indefinitely, however, because cylc ignores triggers that reach back
beyond the initial cycle point. That said, the presence of an
inter-cycle trigger does normally imply that something special has to
happen at start-up. If a model depends on its own previous instance for
restart files, for instance, then an initial set of restart files has to be
generated somehow or the first model task will presumably fail with
missing input files. There are several ways to handle this in cylc
using different kinds of one-off (non-cycling) tasks that run at suite
start-up. They are illustrated in the Tutorial
(\ref{TutInterCyclePointTriggers}); to summarize here briefly:

\begin{myitemize}
    \item \lstinline=R1= tasks (recommended):
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[R1]]]
            graph = "prep"
        [[[R1/T00,R1/T12]]]
            graph = "prep[^] => foo"
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}

\end{myitemize}
\lstset{language=transcript}

\lstinline=R1=, or \lstinline=R1/date-time= tasks are the recommended way to
specify unusual start up conditions. They allow you to specify a clean
distinction between the dependencies of initial cycles and the dependencies
of the subsequent cycles.

Initial tasks can be used for real model cold-start processes, whereby a
warm-cycled model at any given cycle point can in principle have its inputs
satisfied by a previous instance of itself, {\em or} by an initial task with
(nominally) the same cycle point.

In effect, the \lstinline=R1= task masquerades as the previous-cycle-point trigger
of its associated cycling task. At suite start-up initial tasks will
trigger the first cycling tasks, and thereafter the inter-cycle trigger
will take effect.

If a task has a dependency on another task in a different cycle point, the
dependency can be written using the \lstinline=[offset]= syntax such as
\lstinline=[-PT12H]= in \lstinline@foo[-PT12H] => foo@. This means that
\lstinline=foo= at the current cycle point depends on a previous instance of
 \lstinline=foo= at 12 hours before the current cycle point. Unlike the
 cycling section headings (e.g. \lstinline=[[[T00,T12]]]=), dependencies
 assume that relative times are relative to the current cycle point, not the
 initial cycle point.

However, it can be useful to have specific dependencies on tasks at or near
the initial cycle point. You can switch the context of the offset to be
the initial cycle point by using the caret symbol: \lstinline=^=.

For example, you can write \lstinline=foo[^]= to mean foo at the initial
cycle point, and \lstinline=foo[^+PT6H]= to mean foo 6 hours after the initial
cycle point. Usually, this kind of dependency will only apply in a limited
number of cycle points near the start of the suite, so you may want to write
it in \lstinline=R1=-based cycling sections. Here's the example inter-cycle
\lstinline=R1= suite from above again.

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[R1]]]
            graph = "prep"
        [[[R1/T00,R1/T12]]]
            graph = "prep[^] => foo"
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
\lstset{language=transcript}

You can see there is a dependence on the initial \lstinline=R1= task
\lstinline=prep= for \lstinline=foo= at the first \lstinline=T00= cycle point,
and at the first \lstinline=T12= cycle point. Thereafter, \lstinline=foo= just
depends on its previous (12 hours ago) instance.

This is quite different to the non-preferred \lstinline=cold-start tasks=
way of working.

\paragraph{Special Sequential Tasks}
\label{SequentialTasks}

If a cycling task does not generate files required by its own successor,
then successive instances can run in parallel if the opportunity arises.
However, if such a task would interfere with its own siblings for
internal reasons (e.g.\ use of a hardwired non cycle dependent
temporary file or similar) then it can be forced to run sequentially.
This can be done with explicit inter-cycle triggers in the graph:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
or by declaring the task to be {\em sequential}:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[special tasks]]
        sequential = foo
    [[dependencies]]
        [[[T00,T12]]]
            graph = "foo => bar"
\end{lstlisting}

The {\em sequential} declaration also results in each instance of
\lstinline=foo= triggering off its own predecessor, exactly as in
the explicit version. The only difference is that implicit triggers will
not appear in graph visualizations. The implicit version can also be
considerably simpler when the task appears in multiple graph sections or
in a non-uniform cycling sequence: this suite:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[special tasks]]
        sequential = foo
    [[dependencies]]
        [[[T00,T03,T11]]]
            graph = "foo => bar"
\end{lstlisting}
is equivalent to this one:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T03,T11]]]
            graph = "foo => bar"
        [[[T00]]]
            graph = "foo[-PT13H] => foo"
        [[[T03]]]
            graph = "foo[-PT3H] => foo"
        [[[T11]]]
            graph = "foo[-PT8H] => foo"
\end{lstlisting}


\paragraph{Future Triggers}

Cylc also supports inter-cycle triggering off tasks ``in the future'' (with
respect to cycle point):
\begin{lstlisting}
[[dependencies]]
    [[[T00,T06,T12,T18]]]
        graph = """
    # A runs in this cycle:
            A
    # B in this cycle triggers off A in the next cycle.
            A[PT6H] => B
        """
\end{lstlisting}
(Recall that \lstinline=A[t+PT6H]= can run before B[t] because tasks in cylc
have private cycle points). Future triggers present a problem at the suite
shutdown rather than at start-up. Here, \lstinline=B= at the final cycle
point wants to trigger off an instance of \lstinline=A= that will never exist
because it is beyond the suite stop point. Consequently cylc prevents tasks
from spawning successors that depend on other tasks beyond the stop point.

\paragraph{Clock Triggers}
\label{ClockTriggerTasks}

In addition to depending on other tasks (and on external events -
see~\ref{ExternalTriggers}) tasks can depend on the wall clock: specifically,
they can trigger off a wall clock time expressed as an offset from their own
cycle point:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[special tasks]]
        clock-trigger = foo(PT2H)
    [[dependencies]]
        [[[T00]]]
            graph = foo
\end{lstlisting}
Here, \lstinline=foo[2015-08-23T00]= would trigger (other dependencies allowing)
when the wall clock time reaches \lstinline=2015-08-23T02=. Clock-trigger
offsets are normally positive, to trigger some time {\em after} the wall-clock
time is equal to task cycle point.

Clock-triggers have no effect on scheduling if the suite is running sufficiently
far behind the clock (e.g.\ after a delay, or because it is processing archived
historical data) that the trigger times, which are relative to task cycle
point, have already passed.

\paragraph{Clock-Expire Triggers}
\label{ClockExpireTasks}

Tasks can be configured to {\em expire} - i.e.\ to skip job submission and
enter the {\em expired} state - if they are too far behind the wall clock when
they become ready to run, and other tasks can trigger off this. As a possible
use case, consider a cycling task that copies the latest of a set of files to
overwrite the previous set: if the task is delayed by more than one cycle there
may be no point in running it because the freshly copied files will just be
overwritten immediately by the next task instance as the suite catches back up
to real time operation. Clock-expire tasks are configured like clock-trigger
tasks, with a date-time offset relative to cycle point (\ref{ClockExpireRef}).
The offset should be positive to make the task expire if the wall-clock time
has gone beyond the cycle point. Triggering off an expired task typically
requires suicide triggers to remove the workflow that runs if the task has not
expired. Here a task called \lstinline=copy= expires, and its downstream
workflow is skipped, if it is more than one day behind the wall-clock (see also
\lstinline=examples/clock-expire=):
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
   cycle point format = %Y-%m-%dT%H
[scheduling]
    initial cycle point = 2015-08-15T00
    [[special tasks]]
        clock-expire = copy(-P1D)
    [[dependencies]]
        [[[P1D]]]
            graph = """
        model[-P1D] => model => copy => proc
              copy:expired => !proc"""
\end{lstlisting}

\paragraph{External Triggers}
\label{ExternalTriggers}

In addition to depending on other tasks (and on the wall clock -
see~\ref{ClockTriggerTasks}) tasks can trigger off events reported by an
external system. For example, an external process could detect incoming data
on an ftp server, and then notify a suite containing a task to retrieve the
new data for processing. This is an alternative to long-running tasks that poll
for external events.

Note that cylc does not currently support triggering off ``filesystem events''
(e.g.\ \lstinline=inotify= on Linux). However, external watcher processes can
use filesystem events to detect triggering conditions, if that is appropriate,
before notifying a suite with our general external event system.

The external triggering process must call \lstinline=cylc ext-trigger= with the
name of the target suite, the message that identifies this type of event to the
suite, and an ID that distinguishes this particular event instance from others
(the name of the target task or its current cycle point is not required). The
event ID is just an arbitary string to cylc, but it typically identifies the
filename(s) of the latest dataset in some way. When the suite daemon receives
the external event notification it will trigger the next instance of any task
waiting on that trigger (whatever its cycle point) and then broadcast
(see~\ref{cylc-broadcast}) the event ID to the cycle point of the triggered
task as \lstinline=$CYLC_EXT_TRIGGER_ID=. Downstream tasks with the same cycle
point therefore know the new event ID too and can use it, if they need to, to
identify the same new dataset. In this way a whole workflow can be associated
with each new dataset, and multiple datasets can be processed in parallel if
they happen to arrive in quick succession.

An externally-triggered task must register the event it waits on in the suite
scheduling section:
\lstset{language=suiterc}
\begin{lstlisting}
# suite "sat-proc"
[scheduling]
    cycling mode = integer
    initial cycle point = 1
    [[special tasks]]
        external-trigger = get-data("new sat X data avail")
    [[dependencies]]
        [[[P1]]]
            graph = get-data => conv-data => products
\end{lstlisting}

Then, each time a new dataset arrives the external detection system should
notify the suite like this:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc ext-trigger sat-proc "new sat X data avail" passX12334a
\end{lstlisting}
where ``sat-proc'' is the suite name and ``passX12334a'' is the ID string for
the new event. The suite passphrase must be installed on triggering account.

Note that only one task in a suite can trigger off a particular external
message. Other tasks can trigger off the externally triggered task as required,
of course.

\lstinline=$CYLC_DIR/examples/satellite/ext-triggers/suite.rc= is a working
example of a simulated satellite processing suite.

External triggers are not normally needed in date-time cycling suites driven
by real time data that comes in at regular intervals. In these cases a data
retrieval task can be clock-triggered (and have appropriate retry intervals
supplied) to submit at the expected data arrival time, so little time if any
is wasted in polling. However, if the arrival time of the cycle-point-specific
data is highly variable, external triggering may be used with the cycle point
embedded in the message:
\lstset{language=suiterc}
\begin{lstlisting}
# suite "data-proc"
[scheduling]
    initial cycle point = 20150125T00
    final cycle point   = 20150126T00
    [[special tasks]]
        external-trigger = get-data("data arrived for $CYLC_TASK_CYCLE_POINT")
    [[dependencies]]
        [[[T00]]]
            graph = init-process => get-data => post-process
\end{lstlisting}

Once the variable-length waiting is finished, an external detection system
should notify the suite like this:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc ext-trigger data-proc "data arrived for 20150126T00" passX12334a
\end{lstlisting}
where ``data-proc'' is the suite name, the cycle point has replaced the
variable in the trigger string, and ``passX12334a'' is the ID string for
the new event. The suite passphrase must be installed on the triggering
account. In this case, the event will trigger for the second cycle point but
not the first because of the cycle-point matching.

\subsubsection{Model Restart Dependencies}
\label{ModelRestartDependencies}

Warm-cycled forecast models generate {\em restart files}, e.g.\ model
background fields, to initialize the next forecast. This kind of
dependence requires an inter-cycle trigger:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            graph = "A[-PT6H] => A"
\end{lstlisting}

If your model is configured to write out additional restart files
to allow one or more cycle points to be skipped in an emergency {\em do not
represent these potential dependencies in the suite graph} as they
should not be used under normal circumstances. For example, the
following graph would result in task \lstinline=A= erroneously
triggering off \lstinline=A[T-24]= as a matter of course, instead of
off \lstinline=A[T-6]=, because \lstinline=A[T-24]= will always
be finished first:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            # DO NOT DO THIS (SEE ACCOMPANYING TEXT):
            graph = "A[-PT24H] | A[-PT18H] | A[-PT12H] | A[-PT6H] => A"
\end{lstlisting}

\subsection{Runtime - Task Configuration}
\label{NIORP}

The \lstinline=[runtime]= section of a suite definition configures what
to execute (and where and how to execute it) when each task is ready to
run, in a {\em multiple inheritance hierarchy} of {\em
namespaces} culminating in individual tasks. This allows all common
configuration detail to be factored out and defined in one place.

Any namespace can configure any or all of the items defined in the
{\em Suite.rc Reference} (\ref{SuiteRCReference}).

Namespaces that do not explicitly inherit from others automatically
inherit from the {\em root} namespace (below).

Nested namespaces define {\em task families} that can be used in the
graph as convenient shorthand for triggering all member tasks at once,
or for triggering other tasks off all members at once -
see~\ref{FamilyTriggers}. Nested namespaces can be
progressively expanded and collapsed in the dependency graph viewer, and
in the gcylc graph and text views. Only the first parent of each
namespace (as for single-inheritance) is used for suite visualization
purposes.

\subsubsection{Namespace Names}

Namespace names may contain letters, digits, underscores, and hyphens.
%They may not contain colons, which would preclude use of suite
%suite names in shell \lstinline=$PATH= variables. The
%`.' character is the suite registration hierarchy delimiter (which
%separates suite groups and names, e.g.\
%my\_suites.test.foo).

Note that {\em task names need not be hardwired into task implementations}
because task and suite identity can be extracted portably from the task
execution environment supplied by the suite daemon
(\ref{TaskExecutionEnvironment}) - then to rename a task you can just change
its name in the suite definition.

\subsubsection{Root - Runtime Defaults}

The root namespace, at the base of the inheritance hierarchy,
provides default configuration for all tasks in the suite.
Most root items are unset by default, but some have default values
sufficient to allow test suites to be defined by dependency graph alone.
The {\em script} item, for example, defaults to code that
prints a message then sleeps for between 1 and 15 seconds and
exits. Default values are documented with each item in~\ref{SuiteRCReference}.
You can override the defaults or
provide your own defaults by explicitly configuring the root namespace.

\subsubsection{Defining Multiple Namespaces At Once}
\label{MultiTaskDef}

If a namespace section heading is a comma-separated list of names
then the subsequent configuration applies to each list member.
Particular tasks can be singled out at run time using the
\lstinline=$CYLC_TASK_NAME= variable.

As an example, consider a suite containing an ensemble of closely
related tasks that each invokes the same script but with a unique
argument that identifies the calling task name:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[ENSEMBLE]]
        script = "run-model.sh $CYLC_TASK_NAME"
    [[m1, m2, m3]]
        inherit = ENSEMBLE
\end{lstlisting}

For large ensembles Jinja2 template processing can be used to
automatically generate the member names and associated dependencies
(see~\ref{Jinja2}).

\subsubsection{Runtime Inheritance - Single}

The following listing of the {\em inherit.single.one} example suite
illustrates basic runtime inheritance with single parents.

\lstset{language=suiterc}
\lstinputlisting{../examples/inherit/single/one/suite.rc}
\lstset{language=transcript}

\subsubsection{Runtime Inheritance - Multiple}

If a namespace inherits from multiple parents the linear order of
precedence (which namespace overrides which) is determined by the
so-called {\em C3 algorithm} used to find the linear {\em method
resolution order} for class hierarchies in Python and several other
object oriented programming languages. The result of this should be
fairly obvious for typical use of multiple inheritance in cylc suites,
but for detailed documentation of how the algorithm works refer to the
official Python documentation here:
\lstinline=http://www.python.org/download/releases/2.3/mro/=.

The {\em inherit.multi.one} example suite, listed here, makes use of
multiple inheritance:

\lstset{language=suiterc}
\lstinputlisting{../examples/inherit/multi/one/suite.rc}
\lstset{language=transcript}

\lstinline=cylc get-suite-config= provides an easy way to check the result of
inheritance in a suite. You can extract specific items, e.g.:
\begin{lstlisting}
shell$ cylc get-suite-config --item '[runtime][var_p2]script' \
    inherit.multi.one
echo ``RUN: run-var.sh''
\end{lstlisting}
or use the \lstinline=--sparse= option to print entire namespaces
without obscuring the result with the dense runtime structure obtained
from the root namespace:
\begin{lstlisting}
shell$ cylc get-suite-config --sparse --item '[runtime]ops_s1' inherit.multi.one
script = echo ``RUN: run-ops.sh''
inherit = ['OPS', 'SERIAL']
[directives]
   job_type = serial
\end{lstlisting}

\paragraph{Suite Visualization And Multiple Inheritance}

The first parent inherited by a namespace is also used as the
collapsible family group when visualizing the suite. If this is not what
you want, you can demote the first parent for visualization purposes,
without affecting the order of inheritance of runtime properties:
\begin{lstlisting}
[runtime]
    [[BAR]]
        # ...
    [[foo]]
        # inherit properties from BAR, but stay under root for visualization:
        inherit = None, BAR
\end{lstlisting}


\subsubsection{How Runtime Inheritance Works}

The linear precedence order of ancestors is computed for each namespace
using the C3 algorithm. Then any runtime items that are explicitly
configured in the suite definition are ``inherited'' up the linearized
hierarchy for each task, starting at the root namespace: if a particular
item is defined at multiple levels in the hierarchy, the level nearest
the final task namespace takes precedence. Finally, root namespace
defaults are applied for every item that has not been configured in the
inheritance process (this is more efficient than carrying the full dense
namespace structure through from root from the beginning).

\subsubsection{Task Execution Environment}
\label{TaskExecutionEnvironment}

The task execution environment contains suite and task identity
variables provided by the suite daemon, and user-defined environment variables.
The environment is explicitly exported (by the task job script) prior to
executing the task \lstinline=script= (see~\ref{TaskJobSubmission}).

Suite and task identity are exported first, so that user-defined
variables can refer to them. Order of definition is preserved throughout
so that variable assignment expressions can safely refer to previously
defined variables.

Additionally, access to cylc itself is configured prior to the user-defined
environment, so that variable assignment expressions can make use of
cylc utility commands:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        [[[environment]]]
            REFERENCE_TIME = $( cylc util cycletime --offset-hours=6 )
\end{lstlisting}

\paragraph{User Environment Variables}

A task's user-defined environment results from its inherited
\lstinline=[[[environment]]]= sections:
\begin{lstlisting}
[runtime]
    [[root]]
        [[[environment]]]
            COLOR = red
            SHAPE = circle
    [[foo]]
        [[[environment]]]
            COLOR = blue  # root override
            TEXTURE = rough # new variable
\end{lstlisting}
This results in a task {\em foo} with
\lstinline@SHAPE=circle@,
\lstinline@COLOR=blue@, and
\lstinline@TEXTURE=rough@ in its environment.

\paragraph{Overriding Environment Variables}

When you override inherited namespace items the original parent
item definition is {\em replaced} by the new definition. This applies to
all items including those in the environment sub-sections which,
strictly speaking, are not ``environment variables'' until they are
written, post inheritance processing, to the task job script that
executes the associated task. Consequently, if you override an
environment variable you cannot also access the original parent value:
\begin{lstlisting}
[runtime]
    [[FOO]]
        [[[environment]]]
            COLOR = red
    [[bar]]
        inherit = FOO
        [[[environment]]]
            tmp = $COLOR        # !! ERROR: $COLOR is undefined here
            COLOR = dark-$tmp   # !! as this overrides COLOR in FOO.
\end{lstlisting}
The compressed variant of this, \lstinline@COLOR = dark-$COLOR@, is
also in error for the same reason. To achieve the desired result you
must use a different name for the parent variable:
\begin{lstlisting}
[runtime]
    [[FOO]]
        [[[environment]]]
            FOO_COLOR = red
    [[bar]]
        inherit = FOO
        [[[environment]]]
            COLOR = dark-$FOO_COLOR  # OK
\end{lstlisting}

\paragraph{Suite And Task Identity Variables}

The task identity variables provided to tasks by the suite daemon include:
\lstset{language=bash}
\begin{lstlisting}
$CYLC_TASK_ID                    # X.20110511T1800Z (e.g.)
$CYLC_TASK_NAME                  # X
$CYLC_TASK_CYCLE_POINT           # 20110511T1800Z
$CYLC_TASK_LOG_ROOT              # ~/cylc-run/foo.bar.baz/log/job/20110511T1800Z/X/01/job
$CYLC_TASK_NAMESPACE_HIERARCHY   # "root postproc X" (e.g.)
$CYLC_TASK_SUBMIT_NUMBER         # increments with every submit
$CYLC_TASK_TRY_NUMBER            # increments with automatic retry-on-fail
$CYLC_TASK_WORK_DIR              # task work directory (see below)
$CYLC_SUITE_SHARE_DIR            # suite (or task!) shared directory (see below)
\end{lstlisting}
And the suite identity variables are:
\begin{lstlisting}
$CYLC_SUITE_DEF_PATH   # $HOME/mysuites/baz (e.g.)
$CYLC_SUITE_NAME       # foo.bar.baz (e.g.)
$CYLC_SUITE_REG_PATH   # name translate to path: foo/bar/baz
$CYLC_SUITE_HOST       # orca.niwa.co.nz (e.g.)
$CYLC_SUITE_PORT       # 7766 (e.g.)
$CYLC_SUITE_OWNER      # hilary (e.g.)
\end{lstlisting}
Some of these variables are also used by cylc task messaging commands in
order to target the right task proxy object in the right suite.

\paragraph{Suite Share Directories}

A {\em suite share directory} is created automatically under the suite run
directory as a share space for tasks. The location is available to tasks as
\lstinline=$CYLC_SUITE_SHARE_DIR=. In a cycling suite, output files are
typically held in cycle point sub-directories of the suite share directory.

The top level share and work directory (below) location can be changed
(e.g.\ to a large data area) by a global config setting
(see~\ref{workdirectory}).

\paragraph{Task Work Directories}

Task job scripts are executed from within {\em work directories} created
automatically under the suite run directory. A task can get its own work
directory from \lstinline=$CYLC_TASK_WORK_DIR= (or simply \lstinline=$PWD= if
it does not \lstinline=cd= elsewhere at runtime). By default the location
contains task name and cycle point, to provide a unique workspace for every
instance of every task. This can be overridden in the suite definition,
however, to get several tasks to share the same work directory
(see~\ref{worksubdirectory}).

The top level work and share directory (above) location can be changed
(e.g.\ to a large data area) by a global config setting
(see~\ref{workdirectory}).


\paragraph{Other Cylc-Defined Environment Variables}

Initial and final cycle points, if supplied via the suite.rc file or the
command line, are passed to task execution environments as:
\begin{lstlisting}
$CYLC_SUITE_INITIAL_CYCLE_POINT
$CYLC_SUITE_FINAL_CYCLE_POINT
\end{lstlisting}
Tasks can use these to determine whether or not they are running
in the first or final cycles. Note however that \lstinline=R1= graph
sections are now the preferred way to get different behaviour at suite
start-up or shutdown.

\lstset{language=transcript}

\paragraph{Environment Variable Evaluation}

Variables in the task execution environment are not evaluated in the
shell in which the suite is running prior to submitting the task. They
are written in unevaluated form to the job script that is submitted by
cylc to run the task (\ref{JobScripts}) and are therefore
evaluated when the task begins executing under the task owner account
on the task host. Thus \lstinline=$HOME=, for instance, evaluates at
run time to the home directory of task owner on the task host.

\subsubsection{How Tasks Get Access To The Suite Directory}

Tasks can use \lstinline=$CYLC_SUITE_DEF_PATH= to access suite files on
the task host, and the suite bin directory is automatically added
\lstinline=$PATH=. If a remote suite definition directory is not
specified the local (suite host) path will be assumed with the local
home directory, if present, swapped for literal \lstinline=$HOME= for
evaluation on the task host.

\subsubsection{Remote Task Hosting}
\label{RunningTasksOnARemoteHost}

If a task declares an owner other than the suite owner and/or
a host other than the suite host, cylc will use non-interactive ssh to
execute the task on the \lstinline=owner@host= account by the configured
job submission method:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        [[[remote]]]
            host = orca.niwa.co.nz
            owner = bob
        [[[job]]]
            batch system = pbs
\end{lstlisting}
\lstset{language=transcript}
For this to work:
\begin{myitemize}
    \item non-interactive ssh/scp is required from the suite host to the task
    host accounts. Note: Your shell initialization (.profile, .bashrc, .cshrc,
    etc) on the remote host must not produce any standard output as it may
    confuse commands such as scp. See \url{http://www.openssh.com/faq.html#2.9}
    for more information.

    \item cylc must be installed on task hosts.
    \begin{myitemize}
        \item Optional software dependencies such as graphviz and
        Jinja2 are not needed on task hosts.
        \item If polling task communication is used, there is no other
        requirement.
        \item If SSH task communication is configured, non-interactive ssh is
        required from the task host to the suite host.
        \item If Pyro (default) task communication is configured, the task host
        should have access to the Pyro port on the suite host.
    \end{myitemize}
    \item the suite definition directory, or some fraction of its
        content, can be installed on the task host, if needed.
\end{myitemize}

To learn how to give remote tasks access to cylc,
see~\ref{HowTasksGetAccessToCylc}.

Tasks running on the suite host under another user account are treated as
remote tasks.

Remote hosting, like all namespace settings, can be declared globally in
the root namespace, or per family, or for individual tasks.

\paragraph{Dynamic Host Selection}

Instead of hardwiring host names into the suite definition you can
specify a shell command that prints a hostname, or an environment
variable that holds a hostname, as the value of the host config item.
See~\ref{DynamicHostSelection}.

\paragraph{Remote Task Log Directories}

Task stdout and stderr streams are written to log files in a
suite-specific sub-directory of the {\em suite run directory}, as
explained in~\ref{WhitherStdoutAndStderr}. For remote tasks
the same directory is used, but {\em on the task host}.
Remote task log directories, like local ones, are created on the fly, if
necessary, during job submission.

\subsection{Visualization}
\label{viso}

The visualization section of a suite definition is used to configure
suite graphing, principally graph node (task) and edge (dependency
arrow) style attributes. Tasks can be grouped for the purpose of
applying common style attributes. See~\ref{SuiteRCReference} for details.

\subsubsection{Collapsible Families In Suite Graphs}

\lstset{language=suiterc}
\begin{lstlisting}
[visualization]
    collapsed families = family1, family2
\end{lstlisting}
\lstset{language=transcript}

Nested families from the runtime inheritance hierarchy can be expanded
and collapsed in suite graphs and the gcylc graph view. All families
are displayed in the collapsed state at first, unless
\lstinline=[visualization]collapsed families= is used to single out
specific families for initial collapsing.

In the gcylc graph view, nodes outside of the main graph (such as the
members of collapsed families) are plotted as rectangular nodes to
the right if they are doing anything interesting (submitted, running,
failed).

Figure~\ref{fig-namespaces} illustrates successive expansion of nested task
families in the {\em namespaces} example suite.

\begin{figure}
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-2.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-3.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-4.png}
    \end{center}
\end{minipage}

\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-5.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-6.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-7.png}
    \end{center}
\end{minipage}
\caption[{\em namespaces} example suite graphs]{\scriptsize Graphs of the {\em
namespaces} example suite showing various states of expansion of the
nested namespace family hierarchy, from all families collapsed (top
left) through to all expanded (bottom right). This can also be done by
right-clicking on tasks in the gcylc graph view.}
\label{fig-namespaces}
\end{figure}

\subsection{Parameterized Tasks}
\label{Parameterized Tasks}

Cylc can automatically generate tasks and dependencies by expanding
parameterized task names over lists of parameter values. Uses for this
include:
\begin{myitemize}
    \item generating an ensemble of similar model runs
    \item generating chains of tasks to process similar datasets
    \item replicating an entire workflow, or part thereof, over several runs
    \item splitting a long model run into smaller steps or ``chunks``
        (parameterized cycling)
\end{myitemize}

{\em Note that this can be done with Jinja2 loops too (Section~\ref{Jinja2})
    but parameterization is much cleaner (nested loops can seriously reduce
the clarity of a suite definition).}

\subsubsection{Parameter Expansion}

Parameter values can be lists of strings, or integer ranges (with inclusive bounds):
\begin{lstlisting}
[cylc]
    [[parameters]]
        obs = ship, buoy, plane
        run = 1..5  # 1, 2, 3, 4, 5
\end{lstlisting}
Then angle brackets denote use of these parameters throughout the suite
definition. For the values above, this parameterized name:
\begin{lstlisting}
    model<run>  # for run = 1..2
\end{lstlisting}
expands to these concrete task names:
\begin{lstlisting}
    model_run1, model_run2
\end{lstlisting}
and this parameterized name:
\begin{lstlisting}
    proc<obs>  # for obs = ship, buoy, plane
\end{lstlisting}
expands to these concrete task names:
\begin{lstlisting}
    proc_ship, proc_buoy, proc_plane
\end{lstlisting}
By default, to avoid any ambiguity, the parameter name appears in the expanded
task names for integer values, but not for string values. For example,
\lstinline=model_run1= for \lstinline@run = 1@, but \lstinline=proc_ship= for
\lstinline@obs = ship@. However, the default expansion templates can be
overridden if need be:
\begin{lstlisting}
[cylc]
    [[parameters]]
        obs = ship, buoy, plane
        run = 1..5
    [[parameter templates]]
        run = -R%(run)s  # Make foo<run> expand to foo-R1 etc.
\end{lstlisting}
(See~\ref{RefParameterTemplates} for more on the string template syntax.)

Any number of parameters can be used at once. This parameterization:
\begin{lstlisting}
    model<run,obs>  # for run = 1..2 and obs = ship, buoy, plane
\end{lstlisting}
expands to these tasks names:
\begin{lstlisting}
    model_run1_ship, model_run1_buoy, model_run1_plane,
    model_run2_ship, model_run2_buoy, model_run2_plane
\end{lstlisting}

Here's a simple but complete example suite:
\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..2
[scheduling]
    [[dependencies]]
        graph = "prep => model<run>"
[runtime]
    [[model<run>]]
        # ...
\end{lstlisting}
The result, post parameter expansion, is this:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "prep => model_run1 & model_run2"
[runtime]
    [[model_run1]]
        # ...
    [[model_run2]]
        # ...
\end{lstlisting}

Here's a more complex graph using two parameters (\lstinline@[runtime]@ omitted):
\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..2
        mem = cat, dog
[scheduling]
    [[dependencies]]
        graph = """prep => init<run> => model<run,mem> =>
                      post<run,mem> => wrap<run> => done"""
\end{lstlisting}
%which expands to:
%\begin{lstlisting}
%[scheduling]
%    [[dependencies]]
%        graph = """
%            prep => init_run1 => model_run1_cat => post_run1_cat => wrap_run1 => done
%                init_run1 => model_run1_dog => post_run2_dog => wrap_run1
%            prep => init_run2 => model_run2_cat => post_run2_cat => wrap_run2 => done
%                init_run2 => model_run2_dog => post_run2_dog => wrap_run2"""
%\end{lstlisting}
Figure~\ref{fig-params-1} shows the result as visualized by \lstinline=cylc graph=.
\begin{figure}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/params1.png}
    \end{center}
    \caption[Parameter expansion example.]{\scriptsize
     Parameter expansion example.}
    \label{fig-params-1}
\end{figure}

\paragraph{Zero-Padded Integer Values}

Integer parameter values are zero-padded according to the size of their
largest value, so \lstinline@foo<p>@ for \lstinline@p = 9..10@ expands to
\lstinline@foo_p09, foo_p10@.

To get thicker padding, prepend extra zeroes to the upper range value: 
\lstinline@foo<p>@ for \lstinline@p = 9..010@ expands to
\lstinline@foo_p009, foo_p010@.


\subsubsection{Passing Parameter Values To Tasks}

Parameter values are passed as environment variables to tasks generated by
parameter expansion. For the example above, task \lstinline=model_run2_ship=
would get the following environment variables:
\begin{lstlisting}
    $CYLC_TASK_PARAM_run  # value "2" etc.
    $CYLC_TASK_PARAM_obs  # value "ship" etc.
\end{lstlisting}
These variables allow tasks to determine which member of a parameterized
group they are, and so to vary their behaviour accordingly.


\subsubsection{Selecting Specific Parameter Values}

Specific parameter values can be singled out in the graph and under
\lstinline=[runtime]= with the notation \lstinline@<p=5>@ (for example).
Here's how to make a special task trigger off just the first of a
set of model runs:
\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..5
[scheduling]
    [[dependencies]]
        graph = """model<run> => post_proc<run>  # general case
                   model<run=1> => check_first_run  # special case"""
[runtime]
    [[model<run>]]
        # config for all "model" runs...
    [[model<run=1>
        # special config (if any) for the first model run...
    #...
\end{lstlisting}

\subsubsection{Selecting Partial Parameter Ranges}

The parameter notation does not currently support partial range selection such
as \lstinline@foo<p=5..10>@, but you can achieve the same result by defining a
second parameter that covers the partial range and giving it the same expansion
template as the full-range parameter. For example:

\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..10 # 01, 02, ..., 10
        runx = 1..03  # 01, 02, 03 (note `03' to get correct padding)
    [[parameter templates]]
        run = _R%(run)s
        runx = _R%(runx)s
[scheduling]
    [[dependencies]]
        graph = """model<run> => post<run>
                   model<runx> => checkx<runx>"""
[runtime]
    [[model<run>]]
        # ...
    #...
\end{lstlisting}


\subsubsection{Parameter Offsets In The Graph}

A negative offset notation \lstinline@<NAME-1>@ is interpreted as the
previous value in the ordered list of parameter values. For example, to split
a model run into multiple steps with each step depending on the previous one,
this graph:
\begin{lstlisting}
    graph = "model<run-1> => model<run>"  # for run = 1, 2, 3
\end{lstlisting}
expands to:
\begin{lstlisting}
    graph = """model_run1 => model_run2
               model_run2 => model_run3"""
# or equivalently:
    graph = "model_run1 => model_run2 => model_run3"
\end{lstlisting}
And this graph:
\begin{lstlisting}
    graph = "proc<size-1> => proc<size>"  # for size = small, big, huge
\end{lstlisting}
expands to:
\begin{lstlisting}
    graph = """proc_small => proc_big
               proc_big => proc_huge"""
# or equivalently:
    graph = "proc_small => proc_big => proc_huge"
\end{lstlisting}

\subsubsection{Task Families And Parameterization}

Task family members can be generated by parameter expansion:
\begin{lstlisting}
[runtime]
    [[FAM]]
    [[member<r>]]
        inherit = FAM
# Result: family FAM contains member_r1, member_r2, etc.
\end{lstlisting}

Family names can be parameterized too, just like task names:
\begin{lstlisting}
[runtime]
    [[RUN<r>]]
    [[model<r>]]
        inherit = RUN<r>
    [[post_proc<r>]]
        inherit = RUN<r>
# Result: family RUN_r1 contains model_r1 and post_proc_r1,
#         family RUN_r2 contains model_r2 and post_proc_r1, etc.
\end{lstlisting}

As described in Section~\ref{FamilyTriggers} family names can be used to
trigger all members at once:
\begin{lstlisting}
    graph = "foo => FAMILY"
\end{lstlisting}
or to trigger off all members:
\begin{lstlisting}
    graph = "FAMILY:succeed-all => bar"
\end{lstlisting}
or to trigger off any members:
\begin{lstlisting}
    graph = "FAMILY:succeed-any => bar"
\end{lstlisting}

If the members of \lstinline=FAMILY= were generated with parameters, you can
also trigger them all at once with parameter notation:
\begin{lstlisting}
    graph = "foo => member<m>"
\end{lstlisting}
Similarly, to trigger off all members:
\begin{lstlisting}
    graph = "member<m> => bar"
    # (member<m>:fail etc., for other trigger types)
\end{lstlisting}

Family names are still needed in the graph, however, to succinctly express
``succeed-any'' triggering semantics, and all-to-all or any-to-all triggering:
\begin{lstlisting}
    graph = "FAM1:succeed-any => FAM2"
\end{lstlisting}
(Direct all-to-all and any-to-all family triggering is not recommended for
efficiency reasons though - see Section~\ref{EfficientInterFamilyTriggering}).

For family {\em member-to-member} triggering use parameterized members.
For example, if family \lstinline=OBS_GET= has members \lstinline=get<obs>= and
family \lstinline=OBS_PROC= has members \lstinline=proc<obs>= then this graph:
\begin{lstlisting}
    graph = "get<obs> => proc<obs>"  # for obs = ship, buoy, plane
\end{lstlisting}
expands to:
\begin{lstlisting}
    get_ship => proc_ship
    get_buoy => proc_buoy
    get_plane => proc_plane
\end{lstlisting}

\subsubsection{Parameterized Cycling}
\label{Parameterized Cycling}

Parameterized cycling is described and contrasted with iterative cycling in
Section~\ref{Ways Of Cycling}. In iterative cycling (which is recommended in
most cases) each instance of a cycled job is run by the same logical task at
different cycle points, and the scheduler extends the workflow to future cycle
points as the suite runs, potentially indefinitely.  In parameterized cycling,
parameter expansion (or Jinja2 loops) is used to generate the entire workflow
(every task instance from start to finish) at start-up. Each instance of a
cycled job is run by a different logical task with a different task name, and
the only ``cycling'' that occurs in the scheduler is during the workflow
definition phase at start-up.

Here's an iterative cycling workflow of two-monthly model runs for one year,
with previous-instance model dependence (e.g.\ for model restart files):
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020-01
    final cycle point = 2020-12
    [[dependencies]]
        [[[R1]]]  # Run once, at the initial point.
            graph = "prep => model"
        [[[P2M]]]  # Run at 2-month intervals between the initial and final points.
            graph = "model[-P2M] => model => post_proc & archive"
[runtime]
    [[model]]
        script = "run-model $CYLC_TASK_CYCLE_POINT"
\end{lstlisting}

And here's how to do the same thing with parameterized tasks:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    [[parameters]]
        chunk = 1..6
[scheduling]
    [[dependencies]]
        graph = """prep => model<chunk=1>
                     model<chunk-1> => model<chunk> =>
                       post_proc<chunk> & archive<chunk>"""
[runtime]
    [[model<chunk>]]
        script = """
# Compute start date from chunk index and interval, then run the model.
INITIAL_POINT=2020-01
INTERVAL_MONTHS=2
OFFSET_MONTHS=(( (CYLC_TASK_PARAM_chunk - 1)*INTERVAL_MONTHS ))
OFFSET=P${OFFSET_MONTHS}M  # e.g. P4M for chunk=3
run-model $(cylc cyclepoint --offset=$OFFSET $INITIAL_POINT)"""
\end{lstlisting}

The two workflows are shown together in Figure~\ref{fig-eg2}. They both achieve
the same result, and special tasks at the start or end, or anywhere in between,
can easily be added in both cases. But note that parameterized version has
several disadvantages: it must be finite in extent and not too large; the
date-time arithmetic has to be done manually; and the full extent of the workflow
will be visible at all times as the suite runs - because it does not have
``cycle points''.

\begin{figure}
    \begin{center}
        \includegraphics[width=16cm]{graphics/png/orig/eg2-static.png}
    \end{center}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/eg2-dynamic.png}
    \end{center}
    \caption[Parameterized (top) and iterative cycling (bottom) versions of the
        same workflow.]{\scriptsize parameterized and iterative cycling versions
            of the same workflow. The first three cycle points are shown in the
            iterative case (the parameterized case does not have ``cycle
        points'').}
    \label{fig-eg2}
\end{figure}

Here's a yearly-cycling suite with four parameterized chunks in each cycle
point:
\begin{lstlisting}
[cylc]
    [[parameters]]
        chunk = 1..4
[scheduling]
    initial cycle point = 2020-01
    [[dependencies]]
        [[[P1Y]]]
            graph = """model<chunk-1> => model<chunk>
                    model<chunk=4>[-P1Y] => model<chunk=1>"""
\end{lstlisting}
Note the inter-cycle trigger that connects the first chunk in each cycle point
to the last chunk in the previous cycle point. Of course it would be simpler
to just use 3-monthly iterative cycling:
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020-01
    [[dependencies]]
        [[[P3M]]]
            graph = "model[-P3M] => model"
\end{lstlisting}

A possible use-case for mixed iterative and parameterized cycling: consider a
portable date-time cycling workflow of model jobs that can each take too long
to run, or not, depending on the host platform.  This could be handled without
changing the (iterative) cycling structure of the suite, by splitting the run
(at each cycle point) into a variable number of shorter steps, using more steps
on less powerful hosts.

\paragraph{Cycle Point And Parameter Offsets At Start-Up}

In iterative cycling, cylc ignores anything earlier than the suite initial
cycle point. So this graph:
\begin{lstlisting}
    graph = "model[-P1D] => model"
\end{lstlisting}
simplifies at the initial cycle point to:
\begin{lstlisting}
    graph = "model"
\end{lstlisting}

Similarly, parameter offsets are ignored if they extend beyond the start of the
parameter value list. So this graph:
\begin{lstlisting}
    graph = "model<chunk-1> => model<chunk>"
\end{lstlisting}
simplifies for \lstinline@chunk=1@ to this:
\begin{lstlisting}
    graph = "model_chunk1"
\end{lstlisting}

Note however that the initial cut-off applies to every parameter list, but only
to cycle point sequences that start at the suite initial cycle point. Therefore
it may be somewhat easier to use parameterized cycling if you need multiple
date-time sequences {\em with different start points} in the same suite. We
plan to allow this sequence-start simplification for any date-time sequence in
the future, not just at the suite initial point, but it needs to be optional
because delayed-start cycling tasks sometimes need to trigger off earlier
cycling tasks.

\subsection{Jinja2}
\label{Jinja2}

{\em This section needs to be revised - the Parameterized Task feature
    introduced in cylc-6.11.0 (see Section~\ref{Parameterized Tasks}) provides
    a cleaner way to auto-generate tasks without coding messy Jinja2 loops.}

Cylc has built in support for the Jinja2 template processor in suite
definitions. Jinja2 variables, mathematical expressions, loop control
structures, conditional logic, etc., are automatically processed to
generate the final suite definition seen by cylc.

The need for Jinja2 processing must be declared with a hash-bang
comment as the first line of the suite.rc file:
\begin{lstlisting}
#!jinja2
# ...
\end{lstlisting}

Potential uses for this include automatic generation of repeated groups
of similar tasks and dependencies, and inclusion or exclusion of entire
suite sections according to the value of a single flag. Consider a
large complicated operational suite and several related parallel test
suites with slightly different task content and structure (the parallel
suites, for instance, might take certain large input files from the
operation or the archive rather than downloading them again) - these can
now be maintained as a single master suite definition that reconfigures
itself according to the value of a flag variable indicating the intended use.

Template processing is the first thing done on parsing a suite
definition so Jinja2 expressions can appear anywhere in the file (inside
strings and namespace headings, for example).

Jinja2 is well documented at \url{http://jinja.pocoo.org/docs}, so here
we just provide an example suite that uses it. The meaning of the
embedded Jinja2 code should be reasonably self-evident to anyone familiar
with standard programming techniques.

\begin{figure}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/jinja2-ensemble-graph.png}
    \end{center}
    \caption[The Jinja2 ensemble example suite graph.]{\scriptsize
    The Jinja2 ensemble example suite graph.}
    \label{fig-jinja2-ensemble}
\end{figure}

The \lstinline=jinja2.ensemble= example, graphed in
Figure~\ref{fig-jinja2-ensemble}, shows an ensemble of similar tasks
generated using Jinja2:
\lstset{language=suiterc}
\begin{lstlisting}
#!jinja2
{% set N_MEMBERS = 5 %}
[scheduling]
    [[dependencies]]
        graph = """{# generate ensemble dependencies #}
            {% for I in range( 0, N_MEMBERS ) %}
               foo => mem_{{ I }} => post_{{ I }} => bar
            {% endfor %}"""
\end{lstlisting}
Here is the generated suite definition, after Jinja2 processing:
\lstset{language=suiterc}
\begin{lstlisting}
#!jinja2
[scheduling]
    [[dependencies]]
        graph = """
          foo => mem_0 => post_0 => bar
          foo => mem_1 => post_1 => bar
          foo => mem_2 => post_2 => bar
          foo => mem_3 => post_3 => bar
          foo => mem_4 => post_4 => bar
                """
\end{lstlisting}

And finally, the \lstinline=jinja2.cities= example uses variables,
includes or excludes special cleanup tasks according to the value of a
logical flag, and it automatically generates all dependencies and family
relationships for a group of tasks that is repeated for each city in the
suite. To add a new city and associated tasks and dependencies simply
add the city name to list at the top of the file. The suite is graphed,
with the New York City task family expanded, in
Figure~\ref{fig-jinja2-cities}.

\lstset{language=suiterc}
\lstinputlisting{../examples/jinja2/cities/suite.rc}
\lstset{language=transcript}

\begin{figure}
    \begin{center}
        \includegraphics[width=16cm]{graphics/png/orig/jinja2-suite-graph.png}
    \end{center}
    \caption[Jinja2 cities example suite graph.]{\scriptsize
    The Jinja2 cities example suite graph, with the
    New York City task family expanded.}
    \label{fig-jinja2-cities}
\end{figure}

\subsubsection{Accessing Environment Variables With Jinja2}

This functionality is not provided by Jinja2 by default, but cylc
automatically imports the user environment to the template in a
dictionary structure called {\em environ}. A usage example:
\begin{lstlisting}
#!Jinja2
#...
[runtime]
    [[root]]
        [[[environment]]]
            SUITE_OWNER_HOME_DIR_ON_SUITE_HOST = {{environ['HOME']}}
\end{lstlisting}
This example is emphasizes that {\em the environment is read on the suite
host at the time the suite definition is parsed} - it is not, for
instance, read at task run time on the task host.

\subsubsection{Custom Jinja2 Filters}
\label{CustomJinja2Filters}

Jinja2 variable values can be modified by ``filters'', using pipe
notation. For example, the built-in \lstinline=trim= filter strips
leading and trailing white space from a string:
\lstset{language=suiterc}
\begin{lstlisting}
{% set MyString = "   dog   " %}
{{ MyString | trim() }}  # "dog"
\end{lstlisting}
(See official Jinja2 documentation for available built-in filters.)

Cylc also supports custom Jinja2 filters. A custom filter is a
single Python function in a source file with the same name as the
function (plus ``.py'' extension) and stored in one of the following
locations:
\begin{myitemize}
    \item \lstinline=$CYLC_DIR/lib/Jinja2Filters/=
    \item \lstinline=[suite definition directory]/Jinja2Filters/=
    \item \lstinline=$HOME/.cylc/Jinja2Filters/=
\end{myitemize}

In the filter function argument list, the first argument is the variable
value to be ``filtered'', and subsequent arguments can be whatever is
needed. Currently there is one custom filter called ``pad'' in the
central cylc Jinja2 filter directory, for padding string values to some
constant length with a fill character - useful for generating task names
and related values in ensemble suites:

\lstset{language=suiterc}
\begin{lstlisting}
{% for i in range(0,100) %}  # 0, 1, ..., 99
    {% set j = i | pad(2,'0') %}
    A_{{j}}          # A_00, A_01, ..., A_99
{% endfor %}
\end{lstlisting}

\subsubsection{Associative Arrays In Jinja2}

Associative arrays ({\em dicts} in Python) can be very useful.
Here's an example, from \\*
\lstinline=$CYLC_DIR/examples/jinja2/dict=:

\lstset{language=suiterc}
\begin{lstlisting}
#!Jinja2
{% set obs_types = ['airs', 'iasi'] %}
{% set resource = { 'airs':'ncpus=9', 'iasi':'ncpus=20' } %}

[scheduling]
    [[dependencies]]
        graph = OBS
[runtime]
    [[OBS]]
        [[[job]]]
            batch system = pbs
    {% for i in obs_types %}
    [[ {{i}} ]]
        inherit = OBS
        [[[directives]]]
             -I = {{ resource[i] }}
     {% endfor %}
 \end{lstlisting}

Here's the result:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc get-suite-config -i [runtime][airs]directives SUITE
-I = ncpus=9
\end{lstlisting}

\subsubsection{Jinja2 Default Values And Template Inputs}

The values of Jinja2 variables can be passed in from the cylc command
line rather than hardwired in the suite definition.
Here's an example, from \\*
\lstinline=$CYLC_DIR/examples/jinja2/defaults=:

\lstset{language=suiterc}
\begin{lstlisting}
#!Jinja2

title = "Jinja2 example: use of defaults and external input"

description = """
The template variable FIRST_TASK must be given on the cylc command line
using --set or --set-file=FILE; two other variables, LAST_TASK and
N_MEMBERS can be set similarly, but if not they have default values."""

{% set LAST_TASK = LAST_TASK | default( 'baz' ) %}
{% set N_MEMBERS = N_MEMBERS | default( 3 ) | int %}

{# input of FIRST_TASK is required - no default #}

[scheduling]
    initial cycle point = 20100808T00
    final cycle point   = 20100816T00
    [[dependencies]]
        [[[0]]]
            graph = """{{ FIRST_TASK }} => ENS
                 ENS:succeed-all => {{ LAST_TASK }}"""
[runtime]
    [[ENS]]
{% for I in range( 0, N_MEMBERS ) %}
    [[ mem_{{ I }} ]]
        inherit = ENS
{% endfor %}
\end{lstlisting}

Here's the result:

\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc list SUITE
Jinja2 Template Error
'FIRST_TASK' is undefined
cylc-list foo  failed:  1

shell$ cylc list --set FIRST_TASK=bob foo
bob
baz
mem_2
mem_1
mem_0

shell$ cylc list --set FIRST_TASK=bob --set LAST_TASK=alice foo
bob
alice
mem_2
mem_1
mem_0

shell$ cylc list --set FIRST_TASK=bob --set N_MEMBERS=10 foo
mem_9
mem_8
mem_7
mem_6
mem_5
mem_4
mem_3
mem_2
mem_1
mem_0
baz
bob
\end{lstlisting}

\lstset{language=suiterc}
Note also that
\lstinline@cylc view --set FIRST_TASK=bob --jinja2 SUITE@ will show the
suite with the Jinja2 variables as set.

{\em Note:} suites started with template variables set on the command
line will {\em restart} with the same settings. However, you can set
them again on the \lstinline=cylc restart= command line if they need to
be overridden.


\subsection{Omitting Tasks At Runtime}

It is sometimes convenient to omit certain tasks from the suite at
runtime without actually deleting their definitions from the suite.

Defining [runtime] properties for tasks that do not appear in the suite
graph results in verbose-mode validation warnings that the tasks are
disabled. They cannot be used because the suite graph is what defines
their dependencies and valid cycle points. Nevertheless, it is legal to
leave these orphaned runtime sections in the suite definition because it
allows you to temporarily remove tasks from the suite by simply
commenting them out of the graph.

To omit a task from the suite at runtime but still leave it fully
defined and available for use (by insertion or \lstinline=cylc submit=)
use one or both of [scheduling][[special task]] lists, {\em include at
start-up} or {\em exclude at start-up} (documented in~\ref{IASU}
and~\ref{EASU}). Then the graph still defines the
validity of the tasks and their dependencies, but they are not actually
loaded into the suite at start-up. Other tasks that depend on the
omitted ones, if any, will have to wait on their insertion at a later
time or otherwise be triggered manually.

Finally, with Jinja2 (\ref{Jinja2}) you can radically alter
suite structure by including or excluding tasks from the [scheduling]
and [runtime] sections according to the value of a single logical flag
defined at the top of the suite.
\subsection{Naked Dummy Tasks And Strict Validation}

A {\em naked dummy task} appears in the suite graph but has no
explicit runtime configuration section. Such tasks automatically
inherit the default ``dummy task'' configuration from the root
namespace. This is very useful because it allows functional suites to
be mocked up quickly for test and demonstration purposes by simply
defining the graph. It is somewhat dangerous, however, because there
is no way to distinguish an intentional naked dummy task from one
generated by typographic error: misspelling a task name in the graph
results in a new naked dummy task replacing the intended task in the
affected trigger expression; and misspelling a task name in a runtime
section heading results in the intended task becoming a dummy task
itself (by divorcing it from its intended runtime config section).

To avoid this problem any dummy task used in a real suite should not be
naked - i.e.\ it should have an explicit entry in under the runtime
section of the suite definition, even if the section is empty. This
results in exactly the same dummy task behaviour, via implicit
inheritance from root, but it allows use of
\lstinline=cylc validate --strict=
to catch errors in task names by failing the suite if any naked dummy
tasks are detected.

\section{Task Implementation}
\label{TaskImplementation}

Existing scripts or executables can be used as cylc tasks without any
modification so long as they return standard exit status on success or
failure, and do not internally spawn detaching processes
(see~\ref{DetachingJobs}).

\subsection{Inlined Tasks}

Simple tasks can be entirely implemented within the suite.rc file,
as the task {\em script} items can be multi-line strings.

\subsection{Returning Proper Error Status}

Tasks should abort with non-zero exit status if a fatal error occurs
(this is just good coding practice anyway). This allows cylc's task
job scripts to automatically trap errors and send
\lstinline=cylc task message "failed"= back to the suite. The shell
\lstinline=set -e= option can be used in lieu of explicit error
checks for every command:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
set -e  # abort on error
mkdir /illegal/dir  # this will abort the script with error status
\end{lstlisting}


\subsection{Other Task Messages}

General (non-output) messages can also be sent to report progress,
warnings, and so on, e.g.:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# a warning message (this will be logged by the suite):
cylc task message -p WARNING "oops, something's fishy here"
# information (this will also be logged by the suite):
cylc task message "Hello from task foo"
\end{lstlisting}

Explanatory messages can be sent before aborting on error:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
set -e  # abort on error
if ! mkdir /illegal/dir; then
    # (use inline error checking to avoid triggering the above 'set -e')
    cylc task message -p CRITICAL "Failed to create directory /illegal/dir"
    exit 1 # now abort non-zero exit status to trigger the task failed message
fi
\end{lstlisting}
Or equivalently, with different syntax:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
set -e
mkdir /illegal/dir || {  # inline error checking using OR operator
    cylc task message -p CRITICAL "Failed to create directory /illegal/dir"
    exit 1
}
\end{lstlisting}
But not this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
set -e
mkdir /illegal/dir  # aborted via 'set -e'
if [[ $? != 0 ]]; then  # so this will never be reached.
    cylc task message -p CRITICAL "Failed to create directory /illegal/dir"
    exit 1
fi
\end{lstlisting}
If critical errors are not reported in this way task failures will still
be detected and logged by the suite daemon, but you may have to examine task
logs to determine what the problem was.

\subsection{Avoid Detaching Processes}
\label{DetachingJobs}

\lstset{language=transcript}
If a task script starts background sub-processes and does not wait on them, or
internally submits jobs to a batch scheduler and then exits immediately, the
detached processes will not be visible to cylc and the task will appear to
finish when the top-level script finishes. You will need to modify scripts
like this to make them execute all sub-processes in the foreground (or use the
shell \lstinline=wait= command to wait on them before exiting) and to prevent
job submission commands from returning before the job completes (e.g.\
\lstinline=llsubmit -s= for Loadleveler,
\lstinline=qsub -sync yes= for Sun Grid Engine, and
\lstinline@qsub -W block=true@ for PBS).

If this is not possible - perhaps you don't have control over the script
or can't work out how to fix it - one alternative approach is to use another
task to repeatedly poll for the results of the detached processes:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "model => checker => post-proc"
[runtime]
    [[model]]
        # Uh-oh, this script does an internal job submission to run model.exe:
        script = "run-model.sh"
    [[checker]]
        # Fail and retry every minute (for 10 tries at the most) if the model's
        # job.done indicator file does not exist yet.
        retry delays = 10 * PT1M
        script = "[[ ! -f $RUN_DIR/job.done ]] && exit 1"
\end{lstlisting}

\section{Task Job Submission and Management}
\label{TaskJobSubmission}

For the requirements a command, script, or program, must fulfill in order
to function as a cylc task, see~\ref{TaskImplementation}.
This section explains how tasks are submitted by the suite daemon when they are
ready to run, and how to define new task job submission methods.

\subsection{Task Job Scripts}
\label{JobScripts}

When a task is ready cylc generates a {\em task job script} that encapsulates
the runtime settings (environment and scripting) defined for it in the suite.rc
file. The job script is submitted to run by the {\em job submission method}
chosen for the task. Different tasks can have different job submission methods.
Like other runtime properties, you can set a suite default job submission
method and override it for specific tasks or families:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
   [[root]] # suite defaults
        [[[job]]]
            batch system = loadleveler
   [[foo]] # just task foo
        [[[job]]]
            batch system = at
\end{lstlisting}

As shown in the Tutorial (\ref{RunningSuitesCLI}) job scripts are
saved to the suite run directory; the commands used to submit them are
printed to stdout by cylc in debug mode; and they can be printed with the
\lstinline=cylc cat-log= command or new ones generated and printed with the
\lstinline=cylc jobscript= command. Take a look at one to see exactly
how cylc wraps and runs your tasks.

\subsection{Supported Job Submission Methods}
\label{AvailableMethods}

Cylc supports a number of commonly used job submission methods.
See~\ref{CustomJobSubmissionMethods} for how to add new job
submission methods.

\subsubsection{background}

Runs tasks as Unix background processes.

If an execution time limit is specified for a task, its job will be wrapped
by the \lstinline=timeout= command.

\subsubsection{at}

Submits tasks to the rudimentary Unix \lstinline=at= scheduler. The
\lstinline=atd= daemon must be running.

If an execution time limit is specified for a task, its job will be wrapped
by the \lstinline=timeout= command.

\subsubsection{loadleveler}

Submits tasks to loadleveler by the \lstinline=llsubmit= command.
Loadleveler directives can be provided in the suite.rc file:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = loadleveler
            execution time limit = PT10M
        [[[directives]]]
            foo = bar
            baz = qux
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
# @ foo = bar
# @ baz = qux
# @ wall_clock_limit = 660,600
# @ queue
\end{lstlisting}

If restart=yes is specified as a directive for loadleveler, the job will
automatically trap SIGUSR1, which loadleveler may use to preempt the job. On
trapping SIGUSR1, the job will inform the suite that it has been vacated by
loadleveler. This will put it back to the submitted state, until it starts
running again.

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=wall_clock_limit= directive. The setting is assumed to be the soft
limit. The hard limit will be set by adding an extra minute to the soft limit.
Do not specify the \lstinline=wall_clock_limit= directive explicitly if
\lstinline=execution time limit= is specified. Otherwise, the execution time
limit known by the suite may be out of sync with what is submitted to the batch
system.

\subsubsection{lsf}

Submits tasks to IBM Platform LSF by the \lstinline=bsub= command.
LSF directives can be provided in the suite.rc file:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = lsf
            execution time limit = PT10M
        [[[directives]]]
            -q = foo
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#BSUB -q = foo
#BSUB -W = 10
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-W= directive. Do not specify the \lstinline=-W= directive
explicitly if \lstinline=execution time limit= is specified. Otherwise, the
execution time limit known by the suite may be out of sync with what is
submitted to the batch system.

\subsubsection{pbs}

Submits tasks to PBS (or Torque) by the \lstinline=qsub= command. PBS
directives can be provided in the suite.rc file:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = pbs
            execution time limit = PT1M
        [[[directives]]]
            -V =
            -q = foo
            -l nodes = 1
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#PBS -V
#PBS -q foo
#PBS -l nodes=1
#PBS -l walltime=60
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-l walltime= directive. Do not specify the \lstinline=-l walltime=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{moab}

Submits tasks to the Moab workload manager by the \lstinline=msub= command.
Moab directives can be provided in the suite.rc file; the syntax is very
similar to PBS:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = moab
            execution time limit = PT1M
        [[[directives]]]
            -V =
            -q = foo
            -l nodes = 1
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#PBS -V
#PBS -q foo
#PBS -l nodes=1
#PBS -l walltime=60
\end{lstlisting}
(Moab understands \lstinline=#PBS= directives).

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-l walltime= directive. Do not specify the \lstinline=-l walltime=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{sge}

Submits tasks to Sun/Oracle Grid Engine by the \lstinline=qsub= command.
SGE directives can be provided in the suite.rc file:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = sge
            execution time limit = P1D
        [[[directives]]]
            -cwd =
            -q = foo
            -l h_data = 1024M
            -l h_rt = 24:00:00
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#$ -cwd
#$ -q foo
#$ -l h_data=1024M
#$ -l h_rt=24:00:00
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-l h_rt= directive. Do not specify the \lstinline=-l h_rt=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{slurm}

Submits tasks to Simple Linux Utility for Resource Management by the
\lstinline=sbatch= command. SLURM directives can be provided in the
suite.rc file (note that since not all SLURM commands have a short form,
cylc requires the long form directives):
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = slurm
            execution time limit = PT1H
        [[[directives]]]
            --nodes = 5
            --account = QXZ5W2
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
#SBATCH --nodes=5
#SBATCH --time=60:00
#SBATCH --account=QXZ5W2
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=--time= directive. Do not specify the \lstinline=--time=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{Default Directives Provided}

For job submission methods that use job file directives (PBS, Loadleveler,
etc.) default directives are provided to set the job name, stdout and stderr
file paths, and the execution time limit (if specified).

Cylc constructs the job name string using a combination of the task ID and the
suite name. PBS fails a job submit if the job name in \lstinline=-N name= is
too long. For version 12 or below, this is 15 characters. For version 13, this
is 236 characters. The default setting will truncate the job name string to 15
characters. If you have PBS 13 at your site, you should modify your site's
global configuration file to allow the job name to be longer. (See also
Section~\ref{JobNameLengthMaximum}.) For example:

\begin{lstlisting}
[hosts]
    [[myhpc*]]
        [[[batch systems]]]
            [[[[pbs]]]]
                # PBS 13
                job name length maximum = 236
\end{lstlisting}

\subsubsection{Directives Section Quirks (PBS, SGE, ...) }

As shown in the example above, to get a naked option flag such as
\lstinline=-V= in PBS or \lstinline=-cwd= in SGE you must give a
null string as the directive value in the suite.rc file.

\subsection{Task stdout And stderr Logs}
\label{WhitherStdoutAndStderr}

When a task is ready to run cylc generates a filename root to be used
for the task job script and log files. The filename containing the task
name, cycle point, and a submit number that increments if the same task is
re-triggered multiple times:

\lstset{language=bash}
\begin{lstlisting}
# task job script:
~/cylc-run/tut.oneoff.basic/log/job/1/hello/01/job
# task stdout:
~/cylc-run/tut.oneoff.basic/log/job/1/hello/01/job.out
# task stderr:
~/cylc-run/tut.oneoff.basic/log/job/1/hello/01/job.err
\end{lstlisting}

How the stdout and stderr streams are directed into these files depends
on the job submission method. The \lstinline=background= method just uses
appropriate output redirection on the command line, as shown above. The
\lstinline=loadleveler= method writes appropriate directives to the job
script that is submitted to loadleveler.

Cylc obviously has no control over the stdout and stderr output from
tasks that do their own internal output management (e.g.\ tasks
that submit internal jobs and direct the associated output to other
files). For less internally complex tasks, however, the files referred
to here will be complete task job logs.

Some job submission methods, such as \lstinline=pbs=, redirect a job's stdout
and stderr streams to a separate cache area while the job is running. The
contents are only copied to the normal locations when the job completes. This
means that \lstinline=cylc cat-log= or the gcylc GUI will be unable to find the
job's stdout and stderr streams while the job is running. Some sites with these
job submission methods are known to provide commands for viewing and/or
tail-follow a job's stdout and stderr streams that are redirected to these
cache areas. If this is the case at your site, you can configure cylc to make
use of the provided commands by adding some settings to the global site/user
config. E.g.:

\begin{lstlisting}
[hosts]
    [[HOST]]  # <= replace this with a real host name
        [[[batch systems]]]
            [[[[pbs]]]]
                err tailer = qcat -f -e \%(job_id)s
                out tailer = qcat -f -o \%(job_id)s
                err viewer = qcat -e \%(job_id)s
                out viewer = qcat -o \%(job_id)s
\end{lstlisting}

\subsection{Overriding The Job Submission Command}
\label{CommandTemplate}

\lstset{language=suiterc}
To change the form of the actual command used to submit a job you do not
need to define a new job submission method; just override the
\lstinline=command template= in the relevant job submission sections of
your suite.rc file:
\begin{lstlisting}
[runtime]
    [[root]]
        [[[job]]]
            batch system = loadleveler
            # Use '-s' to stop llsubmit returning
            # until all job steps have completed:
            batch submit command template = llsubmit -s %(job)s
\end{lstlisting}
As explained in~\ref{SuiteRCReference}
the template's \%(job)s will be substituted by the job file path.

\subsection{Job Polling}

For supported job submission methods, one-way polling can be used to determine
actual job status: the suite daemon executes a process on the task host, by
non-interactive ssh, to interrogate the batch queueing system there, and to
read a {\em status file} that is automatically generated by the task job script
as it runs.

Polling may be required to update the suite state correctly after unusual
events such as a machine being rebooted with tasks running on it, or network
problems that prevent task messages from getting back to the suite host.

Tasks can be polled on demand by right-clicking on them in gcylc or using the
\lstinline=cylc poll= command.

Tasks are polled automatically, once, if they timeout while queueing in a
batch scheduler and submission timeout is set. (See~\ref{TaskEventHandling} for
how to configure timeouts).

Tasks are polled multiple times, where necessary, when they exceed their
execution time limits. These are normally set with some initial delays to allow
the batch systems to kill the jobs.
(See~\ref{ExecutionTimeLimitPollingIntervals} for how to configure the polling
intervals).

Any tasks recorded in the {\em submitted} or {\em running} states at suite
restart are automatically polled to determine what happened to them while the
suite was down.

Regular polling can also be configured as a health check on tasks submitted to
hosts that are known to be flaky, or as the sole method of determining task
status on hosts that do not allow task messages to be routed back to the suite
host.

To use polling instead of task-to-suite messaging set
\lstinline@task communication method = poll@
in cylc site and user global config (see~\ref{task_comms_method}).
The default polling intervals can be overridden for all suites there too
(see~\ref{submission_polling} and~\ref{execution_polling}), or in specific
suite definitions (in which case polling will be done regardless of the
task communication method configured for the host;
see~\ref{SubmissionPollingIntervals} and~\ref{ExecutionPollingIntervals}).

Note that regular polling is not as efficient as task messaging in updating
task status, and it should be used sparingly in large suites.

\subsection{Job Killing}

For supported job submission methods, the suite daemon can execute a process on
the task host, by non-interactive ssh, to kill a submitted or running job
according to its job submission method.

Tasks can be killed on demand by right-clicking on them in gcylc or using the
\lstinline=cylc kill= command.

\subsection{Execution Time Limit}

You can specify an \lstinline=execution time limit= for all supported job
submission methods. E.g.:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[task-x]]
        [[[job]]]
            execution time limit = PT1H
\end{lstlisting}

For tasks running with \lstinline=background= or \lstinline=at=, their jobs
will be wrapped using the \lstinline=timeout= command. For all other methods,
the relevant time limit directive will be added to their job files.

The \lstinline=execution time limit= setting will also inform the suite when a
a task job should complete by. If a task job has not reported completing within
the specified time, the suite will poll the task job. (The default
setting is PT1M, PT2M, PT7M. The accumulated times for these intervals will be
roughly 1 minute, 1 + 2 = 3 minutes and 1 + 2 + 7 = 10 minutes after a task job
exceeds its execution time limit.)

\subsubsection{Execution Time Limit and Execution Timeout}

If you specify an \lstinline=execution time limit= the
\lstinline=execution timeout event handler= will only be called if the job has
not completed after the final poll (by default, 10 min after the time limit).
This should only happen if the submission method you are using is not enforcing
wallclock limits (unlikely) or you are unable to contact the machine to confirm
the job status.

If you specify an \lstinline=execution timeout= and not an
\lstinline=execution time limit= then the
\lstinline=execution timeout event handler= will be called as soon as the
specified time is reached. The job will also be polled to check its latest
status (possibly resulting in an update in its status and the calling of the
relevant event handler). This behaviour is deprecated, which users should avoid
using.

If you specify an \lstinline=execution timeout= and an
\lstinline=execution time limit= then the execution timeout setting will be
ignored.

\subsection{Custom Job Submission Methods}
\label{CustomJobSubmissionMethods}

Defining a new job submission method requires a little Python programming. You
can use one of the built-in methods as an example, and read the documentation
in the header of the \lstinline=cylc.batch_sys_manager= module.

\lstset{language=Python}

\subsubsection{An Example}

The following user-defined job submission class, called {\em qsub},
overrides the built-in {\em pbs} class to change the directive
prefix from \lstinline=#PBS= to \lstinline=#QSUB=:

\begin{lstlisting}
#!/usr/bin/env python

from cylc.batch_sys_handlers.pbs import PBSHandler


class QSUBHandler(PBSHandler):
    """A user defined batch system handler."""
    DIRECTIVE_PREFIX = "#QSUB "


BATCH_SYSTEM_HANDLER = QSUBHandler()
\end{lstlisting}

To check that this works correctly save the new source file to
\lstinline=qsub.py= in one of the allowed locations (see just below),
use it in a suite definition:
\lstset{language=suiterc}
\begin{lstlisting}
# SUITE.rc
# $HOME/test/suite.rc
[scheduling]
    [[dependencies]]
        graph = "a"
[runtime]
    [[root]]
        [[[job]]]
            batch system = qsub
            execution time limit = PT1M
        [[[directives]]]
            -l nodes = 1
            -q = long
            -V =
\end{lstlisting}
and generate a job script to see the resulting directives:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc db reg test $HOME/test
shell$ cylc jobscript test a.1 | grep QSUB
#QSUB -e /home/hilary/cylc-run/my.suite/log/job/1/a/01/job.err
#QSUB -l nodes=1
#QSUB -l walltime=60
#QSUB -o /home/hilary/cylc-run/my.suite/log/job/1/a/01/job.out
#QSUB -N a.1
#QSUB -q long
#QSUB -V
\end{lstlisting}

\subsubsection{Where To Put New Job Submission Modules}

You new job submission class code should be saved to a file with
the same name as the class (plus ``.py'' extension). It can reside
in any of the following locations, depending on how generally useful
the new method is and whether or not you have write-access to the cylc
source tree:
\begin{myitemize}
    \item a \lstinline=lib/python= sub-directory of your suite definition
        directory.
    \item any directory in your \lstinline=$PYTHONPATH=.
    \item in the \lstinline=lib/cylc/batch_sys_handlers= directory of
        the cylc source tree.
\end{myitemize}

%\pagebreak


\section{Running Suites}
\label{RunningSuites}

This chapter currently features a diverse collection of topics related
to running suites. Please also see the Tutorial (\ref{Tutorial}) and
command documentation (\ref{CommandReference}), and experiment with
plenty of examples.

\subsection{Suite Start-up}
\label{SuiteStartUp}

There are three ways to start a suite running: {\em cold start} and {\em warm
start}, which start from scratch; and {\em restart}, which loads a prior suite
state. There is no difference between cold and warm start, except that the
latter starts from a point beyond the suite initial cycle point.

Once a suite is up and running it is typically a restart that is needed most
often (but see also \lstinline=cylc reload=). Be aware that cold and warm
starts wipe out any prior suite state, which prevents returning to a restart
if you decide that's what you really intended.

\subsubsection{Cold Start}

A cold start is the primary way to start a suite run from scratch:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc run SUITE [INITIAL_CYCLE_POINT]
\end{lstlisting}
The initial cycle point may be specified on the command line or in the suite.rc
file. The scheduler starts by loading the first instance of each task at the
suite initial cycle point, or at the next valid point for the task.

\subsubsection{Restart}

A restart starts a suite run from the state recorded at a checkpoint, which is
normally the end of a previous run. This allows restarting a suite that was
shut down or killed, without rerunning tasks that were already completed, or
which were already submitted or running when the suite went down.
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc restart SUITE
\end{lstlisting}
For a restart, the scheduler starts by loading each task in its recorded state.
Any tasks recorded as `submitted' or `running' will be polled automatically to
determine what happened to them while the suite was down.

See ~\ref{RestartingSuites} for more detail.

\subsubsection{Warm Start}

A warm start runs a suite from scratch like a cold start, but from a given
cycle point that is later than the suite's initial cycle point. All tasks from
the given cycle point will run. It can be considered an inferior alternative
to a restart because it may result in some tasks rerunning. A warm start may
be required if a restart is not possible because the suite run databases were
accidentally deleted (for instance). The warm start cycle point must be given
on the command line:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc run --warm SUITE [START_CYCLE_POINT]
\end{lstlisting}
The original suite initial cycle point is preserved, but all tasks and
dependencies before the given start cycle point are ignored.

The scheduler starts by loading a first instance of each task at the warm
start cycle point, or at the next valid point for the task.
\lstinline=R1=-type tasks behave exactly the same as other tasks - if their
cycle point is at or later than the given start cycle point, they will run; if
not, they will be ignored.

\subsection{How Tasks Interact With Running Suites}
\label{TaskComms}

Cylc has three ways of tracking the progress of tasks, configured per
task host in the site and user global config files
(\ref{SiteAndUserConfiguration}).
All three methods can be used on different task hosts within the same
suite if necessary.
\begin{myenumerate}
\item {\bf task-to-suite messaging:} cylc job scripts encapsulate task
scripting in a wrapper that automatically invokes messaging commands to
report progress back to the suite. The messaging commands can be
configured to work in two different ways:
    \begin{myenumerate}
        \item {\bf Pyro:} direct messaging via network sockets using
        Pyro (Python Remote Objects).
        \item {\bf ssh:} for tasks hosts that block access to the
        network ports required by Pyro, cylc can use non-interactive ssh to
        re-invoke task messaging commands on the suite host (where
        ultimately Pyro is still used to connect to the server process).
    \end{myenumerate}
\item {\bf polling:} for task hosts that do not allow return routing to
the suite host for Pyro or ssh, cylc can poll tasks at configurable
intervals, using non-interactive ssh.
\end{myenumerate}

The Pyro communication method is the default because it is the most
direct and efficient; the ssh method inserts an extra step in the
process (command re-invocation on the suite host); and task polling is
the least efficient because results are checked at predetermined
intervals, not when task events actually occur.

\subsubsection{Task Polling}

Be careful to avoid spamming task hosts with polling commands. Each poll
opens (and then closes) a new ssh connection.

Polling intervals are configurable under \lstinline=[runtime]= because
they should may depend on the expected execution time. For instance, a
task that typically takes an hour to run might be polled every 10
minutes initially, and then every minute toward the end of its run.
Interval values are used in turn until the last value, which is used
repeatedly
until finished:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        [[[job]]]
            # poll every minute in the 'submitted' state:
            submission polling intervals = PT1M
            # poll one minute after foo starts running, then every 10
            # minutes for 50 minutes, then every minute until finished:
            execution polling intervals = PT1M, 5*PT10M, PT1M
\end{lstlisting}
A list of intervals with optional multipliers can be used for both
submission and execution polling, although a single value is probably
sufficient for submission polling. If these items are not configured
default values from site and user global config will be used for the polling
task communication method; polling is not done by default under the
other task communications methods (but it can still be used if you
like).

Polling is also done automatically once on job submission timeout, and multiple
times on exceeding the execution time limit, to see if the timed-out task has
failed or not; and on suite restarts, to see what happened to any tasks that
were orphaned when the suite went down.

\subsection{Alternatives To Polling When Routing Is Blocked}

If Pyro and ssh ports are blocked but you don't want to use polling from
the suite host:
\begin{myitemize}
\item it has been suggested that network {\em port forwarding} may
provide a solution;
\item you may be able to persuade system administrators to provide
network routing to one or more dedicated cylc servers;
\item it is possible to run cylc itself on HPC login nodes, but
depending on what software is installed there this may preclude use
of the gcylc GUI and suite visualization tools.
\end{myitemize}

\subsection{Task Host Communications Configuration}

Here are the default site and user global config items relevant to task state
tracking (see these with \lstinline=cylc get-site-config=):

\lstset{language=suiterc}
\begin{lstlisting}
#SITE AND USER CONFIG

# Task messaging settings affect task-to-suite communications.
[task messaging]
    # If a message send fails, retry after this delay:
    retry interval in seconds = 5
    # If send fails after this many tries, give up trying:
    maximum number of tries = 7

    # This timeout is the same as --pyro-timeout for user commands. If
    # set to None (no timeout) message send to non-responsive suite
    # (e.g. suspended with Ctrl-Z) could hang indefinitely.
    connection timeout in seconds = 30

# Pyro is required for communications between cylc clients and servers
# (i.e. between suite-connecting commands and guis, and running suite
# server processes).
[pyro]

    # Each suite listens on a dedicated network port.
    # Servers bind on the first port available from the base port up:
# SITE ONLY
    base port = 7766

    # This sets the maximum number of suites that can run at once.
# SITE ONLY
    maximum number of ports = 100

    # Port numbers are recorded in this directory, by suite name.
    ports directory = "$HOME/.cylc/ports/"

[hosts]
    # The default task host is the suite host, i.e. localhost:
    # Add task host sections if local defaults are not sufficient.
    [[HOST]]
       # Method of communication of task progress back to the suite:
        #   1) pyro - direct client-server RPC via network ports
        #   2) ssh  - re-invoke pyro messaging commands on suite server
        #   3) poll - the suite polls for status of passive tasks
        # Pyro RPC is still required in all cases *on the suite host*
        # for cylc clients (commands etc.) to communicate with suites.
        task communication method = "pyro" # or "ssh" or "poll"
        # The "poll" method sets a default interval here to ensure no
        # tasks are accidentally left unpolled. You should override this
        # with run-length appropriate intervals under task [runtime] -
        # which will also result in routine polling to check task health
        # under the pyro or ssh communications methods.
        default polling interval in minutes = 1.0
\end{lstlisting}

\subsection{How Commands Interact With Running Suites}

User-invoked commands that connect to running suites can also choose
between direct communication across network sockets (Pyro) and
re-invocation of commands on the suite host using non-interactive ssh
(there is a \lstinline=--use-ssh= command option for this purpose).

The gcylc GUI requires direct Pyro connections to its target suite. If
that is not possible, run gcylc on the suite host.


\subsection{Client Authentication and Passphrases}
\label{ConnectionAuthentication}

Suite daemons listen on dedicated network ports for incoming client requests -
task messages and user-invoked commands (CLI or GUI). The \lstinline=cylc scan=
command reveals which suites are running on scanned hosts, and what ports they
are listening on.

Client programs have to authenticate with the target suite daemon before
issuing commands or requesting information. Cylc has two authentication
levels: full read and control via a suite-specific passphrase
(see~\ref{passphrases}); and configurable free ``public'' access
(see~\ref{PublicAccess}).

\subsubsection{Full Control - Suite Passphrases}
\label{passphrases}

A file called \lstinline=passphrase= is generated in the suite definition
directory at registration time, containing a single line of random text.
The passphrase is loaded by the suite daemon at start-up and used to
authenticate client connections. Suite passphrases are used in an encrypted
challenge-response scheme; they are never sent raw over the network.

Clients on the suite host account automatically pick up the passphrase from the
suite definition directory. On submission of the first task job on a host, the
suite host will attempt to install the passphrase to the run directory to
enable task jobs to connect to the suite. On other accounts that do not share
the same HOME directory with the suite host account and do not have
non-interactive SSH access to the suite host account, it is possible to install
the passphrase manually. Allowed passphrase locations are:
\begin{myenumerate}
    \item \lstinline=$HOME/.cylc/SUITE_OWNER@SUITE_HOST/SUITE_NAME/passphrase=
    \item \lstinline=$HOME/.cylc/SUITE_HOST/SUITE_OWNER/SUITE_NAME/passphrase=
    \item \lstinline=$HOME/.cylc/SUITE_HOST/SUITE_NAME/passphrase=
    \item \lstinline=$HOME/.cylc/SUITE_NAME/passphrase=
\end{myenumerate}

Note: A client will attempt to install the passphrase automatically via
non-interactive SSH to the first location listed above.

\subsubsection{Public Access - No Passphrase}
\label{PublicAccess}

Possession of a suite passphrase gives full read and control access to the
suite. Without the passphrase the amount of information revealed by a suite
daemon is determined by the public access privilege level set in global
site/user config (\ref{GlobalAuth}) and optionally overidden in suites
(\ref{SuiteAuth}):
\begin{myitemize}
    \item {\em identity} - only suite and owner names revealed
    \item {\em description} - identity plus suite title and description
    \item {\em state-totals} - identity, description, and task state totals
    \item {\em full-read} - full read-only access for monitor and GUI
    \item {\em shutdown} - full read access plus shutdown, but no other
        control.
\end{myitemize}
The default public access level is {\em state-totals}.
The \lstinline=cylc scan= command can print descriptions and task state totals
in addition to basic suite identity, if you have the right passphrases or if
the requested information is revealed publicly.


\subsection{How Tasks Get Access To Cylc}
\label{HowTasksGetAccessToCylc}

Running tasks need access to cylc via \lstinline=$PATH=, principally for
the task messaging commands. To allow this, the first thing a task job
script does is set \lstinline=$CYLC_VERSION= to the cylc version number of the
running suite. If you need to run several suites at once under different
incompatible versions of cylc, check that your site is using the cylc
version wrapper (see \lstinline=INSTALL= and \lstinline=admin/cylc-wrapper= in
a cylc installation) then set \lstinline=$CYLC_VERSION= to the desired
version. In the case of developers wishing to run their own copy
of cylc rather than a centrally installed one, set \lstinline=$CYLC_HOME=
to point to your cylc copy.

Access to the cylc executable for different hosts can be configured using
the site and user global configuration files.
If the environment for running the cylc executable is only set up correctly in
a login shell for a given host, you can set the
\lstinline=[hosts] \textrightarrow HOST \textrightarrow cylc executable=
setting for the relevant host to \lstinline=True=.
(This is the default behaviour.)
If the environment is already correct without the login shell, but the cylc
executable is not in \lstinline=$PATH=, then the
\lstinline=[hosts] \textrightarrow HOST \textrightarrow cylc executable=
setting can be used to specify the path to the cylc executable.

The site/user global config item \lstinline=global init-script= can also be
used if necessary (see~\ref{GlobalInitScript}).

\subsection{Restarting Suites}
\label{RestartingSuites}

A restarted suite (see \lstinline=cylc restart --help=) is initialized from a
previous recorded checkpoint, which is normally the end of a previous run, so
that it can carry on from wherever it got to before being shut down or killed.

\subsubsection{Restart Suite from Latest}

A normal restart is easy. Simply invoke the \lstinline=cylc restart= command
with the suite name:

\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc restart SUITE
\end{lstlisting}

It will restart the suite from the latest checkpoint.

\subsubsection{Restart Suite from a Check Point}

You can use the \lstinline=cylc ls-checkpoints= command to identify the
checkpoint to use to restart a suite. (See also
\lstinline=cylc ls-checkpoints --help=.)

The checkpoint ID 0 (zero) is always used for latest state of the suite, which
is updated continuously as the suite progresses. The checkpoint IDs of
non-latest states are positive integers starting from 1, and incremented each
time a new checkpoint is stored. Currently a suite will automatically stores
checkpoints before and after reloads, and on restarts (using the latest
checkpoints before the restarts).

Once you have identified the checkpoint to use, invoke the
\lstinline=cylc restart= command with the suite name and the
\lstinline@--checkpoint=CHECKPOINT@ option:

\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc restart --checkpoint=CHECKPOINT-ID SUITE
\end{lstlisting}

It will restart the suite from the specified checkpoint.

\subsubsection{Behaviour of Tasks on Suite Restart}

Tasks that were recorded in the submitted or running states are automatically
polled on restart, to see if they are still submitted (e.g. waiting in a PBS
batch queue or similar), still running, or if they finished (succeeded or
failed) while the suite was down.

Tasks recorded in the failed state at shutdown are not automatically
resubmitted on restarting the suite, in case the underlying problem has not
been addressed yet.

\subsection{Task States}

As a suite runs its task proxies may pass through the following states:

\begin{myitemize}
    \item {\bf waiting} - prerequisites not satisfied yet
    (note that clock-trigger tasks also wait on their trigger time).

    \item {\bf queued} - ready to run (prerequisites satisfied) but
    temporarily held back by an {\em internal cylc queue}
    (see~\ref{InternalQueues}).

    \item {\bf held} - will not be submitted even if ready to run.
    Tasks that spawn past the final cycle point are held automatically.

    \item {\bf ready} - ready to run (prerequisites satisfied) and
    handed to cylc's job submission sub-system.

    \item {\bf submitted} - submitted to run, but not executing yet
    (could be waiting in an external batch scheduler queue).

    \item {\bf submit-failed} - job submission failed {\em or}
    submitted job killed before commencing execution.

    \item {\bf submit-retrying} - job submission failed, but a submission retry
    was configured. Will only enter the {\em submit-failed} state if all
    configured submission retries are exhausted.

    \item {\bf running} - currently executing (a {\em task started}
    message was received, or the task polled as running).

    \item {\bf succeeded} - finished executing successfully (a {\em task
    succeeded} message was received, or the task polled as succeeded).

    \item {\bf failed} - aborted execution due to some error condition (a
    {\em task failed} message was received, or the task polled as failed).

    \item {\bf retrying} - job execution failed, but an execution retry
    was configured. Will only enter the {\em failed} state if all configured
    execution retries are exhausted.

\end{myitemize}

Note that greyed-out ``base graph nodes'' in the gcylc graph view do not
represent task states; they are displayed to fill out the graph
structure where corresponding task proxies do not currently exist
in the live task pool.

For manual task state reset purposes {\bf ready} is a pseudo-state that means
{\em waiting} with all prerequisites satisfied.


\subsection{Remote Control - Passphrases and Network Ports}
\label{RemoteControl}

Connecting to a running suite requires knowing the {\em network port} it
is listening on, and the {\em suite passphrase} to authenticate with once
a connection is made to the port.

Suites write their port number to \lstinline=$HOME/.cylc/ports/<SUITE>=
at start-up, and suite-connecting commands read this file to get the
number.\footnote{If you accidentally delete a port file while a suite
is running, use \lstinline=cylc scan= to determine the port number
then use it on the command line (\lstinline=--port=) or rewrite the port
file manually.} An exception to this is the messaging commands called by
tasks. Running tasks know the port number from the execution environment
provided by the suite (via the task job script).

So, to connect to a suite running on another account you must install
the suite passphrase (\ref{passphrases}), and configure
non-interactive ssh so that the port number can be retrieved from the
remote port file. Then use the \lstinline=--user= and
\lstinline=--host= command options to connect:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc monitor --user=USER --host=HOST SUITE
\end{lstlisting}

Alternatively, you can determine suite port numbers using \lstinline=cylc scan=,
and use them explicitly on the command line:
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc monitor --user=USER --host=HOST --port=PORT SUITE
\end{lstlisting}

Possession of a suite passphrase gives full control over the suite, and
ssh access to the port file also implies full access to the suite host
account, so it is recommended that this only be used to interact with
your own suites running on other hosts. We plan to implement
finer-grained authentication in the future to allow suite owners to
grant read-only access to others.



\subsection{Network Connection Timeouts}

A connection timeout can be set in site and user global config files
(see~\ref{SiteAndUserConfiguration}) so that messaging commands
cannot hang indefinitely if the suite is not responding (this can be
caused by suspending a suite with Ctrl-Z) thereby preventing the task
from completing. The same can be done on the command line for other
suite-connecting user commands, with the \lstinline=--pyro-timeout= option.

\subsection{Runahead Limiting}
\label{RunaheadLimit}

Runahead limiting prevents the fastest tasks in a suite from getting too far
ahead of the slowest ones. Newly spawned tasks are released to the task pool
only when they fall below the runahead limit. A low runhead limit can prevent
cylc from interleaving cycles, but it will not stall a suite unless it fails to
extend out past a future trigger (see~\ref{InterCyclePointTriggers}).
A high runahead limit may allow fast tasks that are not constrained by
dependencies or clock-triggers to spawn far ahead of the pack, which could have
performance implications for the suite daemon when running very large suites.
Succeeded and failed tasks are ignored when computing the runahead limit.

The preferred runahead limiting mechanism restricts the number of consecutive
active cycle points. The default value is three active cycle points;
see~\ref{max active cycle points}. Alternatively the interval between the
slowest and fastest tasks can be specified as hard limit;
see~\ref{runahead limit}.

\subsection{Limiting Active Tasks With Internal Queues}
\label{InternalQueues}

Large suites can potentially overwhelm task hosts by submitting too many
tasks at once. You can prevent this with {\em internal queues}, which
limit the number of tasks that can be active (submitted or running)
at the some time.

A queue is defined by a {\em name}; a {\em limit}, which is the maximum
number of active tasks allowed for the queue; and a list of {\em members},
assigned by task or family name.

Queue configuration is done under the [scheduling] section of the suite.rc file
(like dependencies, internal queues constrain {\em when} a task runs).

By default every task is assigned to the {\em default} queue, which by default
has a zero limit (interpreted by cylc as no limit). To use a single queue for
the whole suite just set the default queue limit:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[ queues]]
        # limit the entire suite to 5 active tasks at once
        [[[default]]]
            limit = 5
\end{lstlisting}
To use additional queues just name each one, set their limits, and assign
members:
\begin{lstlisting}
[scheduling]
    [[ queues]]
        [[[q_foo]]]
            limit = 5
            members = foo, bar, baz
\end{lstlisting}
Any tasks not assigned to a particular queue will remain in the default
queue. The {\em queues} example suite illustrates how queues work by
running two task trees side by side (as seen in the graph GUI) each
limited to 2 and 3 tasks respectively:
\lstset{language=suiterc}
\lstinputlisting{../examples/queues/suite.rc}

\subsection{Automatic Task Retry On Failure}
\label{TaskRetries}

See also~\ref{RefRetries} in the {\em Suite.rc Reference}.

Tasks can be configured with a list of ``retry delay'' periods, as
ISO 8601 durations, such that if a task fails it will go into a temporary
{\em retrying} state and then automatically resubmit itself after
the next specified delay period expires. A usage example is shown in the
suite listed below under~\ref{EventHandling}.

\subsection{Suite And Task Event Handling}
\label{EventHandling}

See also~\ref{SuiteEventHandling} and~\ref{TaskEventHandling} in the {\em
Suite.rc Reference}.

Cylc can call nominated event handlers when certain suite or task events occur.
This is intended to facilitate centralized alerting and automated handling of
critical events. Event handlers can be used to send a message, call a pager,
and so on; or intervene in the operation of their own suite using cylc
commands.

To send an email, you can use the built-in setting
\lstinline=[[[events]]]mail events= to specify a list of events for which
notifications should be sent. E.g. to send an email on (submission) failed and
retry:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        retry delays = PT0S, PT30S
        script = "test ${CYLC_TASK_TRY_NUMBER} -eq 3"
        [[[events]]]
            mail events = submission failed, submission retry, failed, retry
\end{lstlisting}

By default, the emails will be sent to the current user with:

\begin{myitemize}
    \item \lstinline=to:= set as \lstinline=$USER=
    \item \lstinline=from:= set as \lstinline=notifications@$(hostname)=
    \item SMTP server at \lstinline=localhost:25=
\end{myitemize}

These can be configured using the settings:
\begin{myitemize}
    \item \lstinline=[[[events]]]mail to= (list of email addresses),
    \item \lstinline=[[[events]]]mail from=
    \item \lstinline=[[[events]]]mail smtp=.
\end{myitemize}

Event handler commands can be located in the suite \lstinline=bin/= directory,
otherwise it is up to you to ensure their location is in \lstinline=$PATH= (in
the shell in which cylc runs, on the suite host). The commands should require
very little resource to run and should return quickly. (Each event
handler is invoked by a child process in a finite process pool that is also
used to submit, poll and kill jobs. The child process will wait for the event
handler to complete before moving on to the next item in the queue. If the
process pool is saturated with long running event handlers, the suite will
appear to hang.)

Task event handlers can be specified using the
\lstinline=[[[events]]]<event> handler= settings, where
\lstinline=<event>= is one of:
\begin{myitemize}
    \item `submitted' - the job submit command was successful
    \item `submission failed' - the job submit command failed
    \item `submission timeout' - task job submission timed out
    \item `submission retry' - task job submission failed, but will retry after a configured delay
    \item `started' - the task reported commencement of execution
    \item `succeeded' - the task reported successful completion
    \item `warning' - the task reported a warning message
    \item `failed' - the task failed
    \item `retry' - the task failed but will retry
    \item `execution timeout' - task execution timed out
\end{myitemize}

The value of each setting should be a list of command lines or command line
templates (see below).

Alternatively, task event handlers can be specified using the
\lstinline=[[[events]]]handlers= and the \lstinline=[[[events]]]handler events=
settings, where the former is a list of command lines or command line templates
(see below) and the latter is a list of events for which these commands should
be invoked.

A command line template may have any or all of these patterns which will be
substituted with actual values:
\begin{myitemize}
    \item \%(event)s: event name
    \item \%(suite)s: suite name
    \item \%(point)s: cycle point
    \item \%(name)s: task name
    \item \%(submit\_num)s: submit number
    \item \%(id)s: task ID (i.e. \%(name)s.\%(point)s)
    \item \%(message)s: event message, if any
\end{myitemize}

Otherwise, the command line will be called with the following command line
arguments:
\begin{lstlisting}
<task-event-handler> %(event)s %(suite)s %(id)s %(message)s
\end{lstlisting}

For an explanation of the substitution syntax, see
\href{https://docs.python.org/2/library/stdtypes.html#string-formatting}{String Formatting Operations}
in the Python documentation.

The retry event occurs if a task fails and has any remaining retries
configured (see~\ref{TaskRetries}).
The event handler will be called as soon as the task fails, not after
the retry delay period when it is resubmitted.

{\em Note that event handlers are called by cylc itself, not by the
running tasks} so if you wish to pass them additional information via
the environment you must use [cylc] \textrightarrow [[environment]],
not task runtime environments.

The following 2 \lstinline=suite.rc= snippets are examples on how to specify
event handlers using the alternate methods:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        retry delays = PT0S, PT30S
        script = "test ${CYLC_TASK_TRY_NUMBER} -eq 2"
        [[[events]]]
            retry handler = "echo '!!!!!EVENT!!!!!' "
            failed handler = "echo '!!!!!EVENT!!!!!' "
\end{lstlisting}

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        retry delays = PT0S, PT30S
        script = "test ${CYLC_TASK_TRY_NUMBER} -eq 2"
        [[[events]]]
            handlers = "echo '!!!!!EVENT!!!!!' "
            handler events = retry, failed
\end{lstlisting}

Note: The handler command is called like this:
\begin{lstlisting}
echo '!!!!!EVENT!!!!!' %(event)s %(suite)s %(id)s %(message)s
\end{lstlisting}

\subsection{Reloading The Suite Definition At Runtime}

The \lstinline=cylc reload= command reloads the suite definition at run
time. This allows:
 (a) changing task config items such as script or environment;
 (b) adding tasks to, or removing them from, the suite definition,
at run time - without shutting the suite down and restarting it. (It is
easy to shut down and restart cylc suites, but reloading may be useful
if you don't want to wait for long-running tasks to finish first).

Note that {\em defined tasks} can be already be added to or removed from
a running suite with the \lstinline=cylc insert= and \lstinline=cylc remove=
commands; the
reload command allows addition and removal of {\em task definitions}.
If a new task is definition is added (and used in the graph) you will
still need to manually insert an instance of it (with a particular cycle
point) into the running suite. If a task definition (and its use in the graph)
is deleted, existing task proxies of the of the deleted type will run their
course after the reload but new instances will not be spawned. Changes to a
task definition will only take effect when the next task instance is spawned
(existing instances will not be affected).


\subsection{Handling Job Preemption}
\label{PreemptionHPC}

Some HPC facilities allow job preemption: the resource manager can kill
or suspend running low priority jobs in order to make way for high
priority jobs. The preempted jobs may then be automatically restarted
by the resource manager, from the same point (if suspended) or requeued
to run again from the start (if killed). If a running cylc task gets
suspended or hard-killed
(\lstinline=kill -9 <PID>= is not a trappable signal so cylc cannot detect
task failure in this case) and then later restarted, it will just appear
to cylc as if it takes longer than normal to run. If the job is
soft-killed the signal will be trapped by the task job script and a
failure message sent, resulting in cylc putting the task into the failed
state. When the preempted task restarts and sends its started message
cylc would normally treat this as an error condition (a dead task is not
supposed to be sending messages) - a warning will be logged and the task
will remain in the failed state. However, if you know that preemption is
possible on your system you can tell cylc that affected tasks should be
resurrected from the dead, to carry on as normal if progress messages
start coming in again after a failure:

\lstset{language=suiterc}
\begin{lstlisting}
# ...
[runtime]
    [[HPC]]
        enable resurrection = True
    [[TaskFoo]]
        inherit = HPC
# ...
\end{lstlisting}

To test this in any suite, manually kill a running task then, after cylc
registers the task failed, resubmit the killed job manually by
cutting-and-pasting the original job submission command from the suite
stdout stream.

\subsection{Manual Task Triggering and Edit-Run}

Any task proxy currently present in the suite can be manually triggered at any
time using the \lstinline=cylc trigger= command, or from the right-click task
menu in gcylc. If the task belongs to a limited internal queue
(see~\ref{InternalQueues}), this will queue it; if not, or if it is already
queued, it will submit immediately.

With \lstinline=cylc trigger --edit= (also in the gcylc right-click task menu)
you can edit the generated task job script to make one-off changes before the
task submits.

\subsection{Runtime Settings Broadcast and Communication Between Tasks}
\label{cylc-broadcast}

The \lstinline=cylc broadcast= command overrides \lstinline=[runtime]=
settings in a running suite. This can
be used to communicate information to downstream tasks by broadcasting
environment variables (communication of information from one task to
another normally takes place via the filesystem, i.e.\ the input/output
file relationships embodied in inter-task dependencies). Variables (and
any other runtime settings) may be broadcast to all subsequent tasks,
or targeted specifically at a specific task, all subsequent tasks with a
given name, or all tasks with a given cycle point; see broadcast command help
for details.

Broadcast settings targeted at a specific task ID or cycle point expire and
are forgotten as the suite moves on. Un-targeted variables and those
targetted at a task name persist throughout the suite run, even across
restarts, unless manually cleared using the broadcast command - and so
should be used sparingly.

\subsection{The Meaning And Use Of Initial Cycle Point}

When a suite is started with the \lstinline=cylc run= command (cold or
warm start) the cycle point at which it starts can be given on the command
line or hardwired into the suite.rc file:
\begin{lstlisting}
cylc run foo 20120808T06Z
\end{lstlisting}
or:
\begin{lstlisting}
[scheduling]
    initial cycle point = 20100808T06Z
\end{lstlisting}
An initial cycle given on the command line will override one in the
suite.rc file.

\subsubsection[CYLC\_SUITE\_INITIAL\_CYCLE\_POINT]{The Environment Variable CYLC\_SUITE\_INITIAL\_CYCLE\_POINT}

In the case of a {\em cold start only} the initial cycle point is passed
through to task execution environments as
\lstinline=$CYLC_SUITE_INITIAL_CYCLE_POINT=. The value is then stored in
suite database files and persists across restarts, but it does get wiped out
(set to \lstinline=None=) after a warm start, because a warm start is really an
implicit restart in which all state information is lost (except that the
previous cycle is assumed to have completed).

The \lstinline=$CYLC_SUITE_INITIAL_CYCLE_POINT= variable allows tasks to
determine if they are running in the initial cold-start cycle point, when
different behaviour may be required, or in a normal mid-run cycle point.
Note however that an initial \lstinline=R1= graph section is now the preferred
way to get different behaviour at suite start-up.

\subsection{The Simulation And Dummy Run Modes}
\label{SimulationMode}

Since cylc-4.6.0 any cylc suite can run in {\em live}, {\em simulation},
or {\em dummy} mode. Prior to that release simulation mode was a
hybrid mode that replaced real tasks with local dummy tasks. This
allowed local simulation testing of any suite, to get the scheduling
right without running real tasks, but running dummy tasks locally does
not add much value over a pure simulation (in which no tasks are
submitted at all) because all job submission configuration has to be
ignored and most task job script sections have to be cut out to avoid
any code that could potentially be specific to the intended task host.
So at 4.6.0 we replaced this with a pure simulation mode (task proxies
go through the {\em running} state automatically within cylc, and no
dummy tasks are submitted to run) and a new dummy mode in which only the
real task scripting  is dummied out - each dummy task is
submitted exactly as the task it represents on the correct host and in
the same execution environment. A successful dummy run confirms not only
that the scheduling works correctly but also tests real job submission,
communication from remote task hosts, and the real task job scripts (in
which errors such as use of undefined variables will cause a task to
fail).

The run mode, which defaults to {\em live}, is set on the command line
(for run and restart):
\lstset{language=transcript}
\begin{lstlisting}
shell$ cylc run --mode=dummy SUITE
\end{lstlisting}
but you can configure the suite to force a particular run mode:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    force run mode = simulation
\end{lstlisting}
This can be used, for example, for demo suites that necessarily run out
of their original context; or to temporarily prevent accidental
execution of expensive real tasks during suite development.

Dummy mode task scripting just prints a message and sleeps for ten
seconds by default, but you can override this behaviour for particular
tasks or task groups if you like. Here's how to make a task sleep for
twenty seconds and then fail in dummy mode:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        script = "run-real-task.sh"
        [[[dummy mode]]]
            script = """
echo "hello from dummy task $CYLC_TASK_ID"
sleep 20
echo "ABORTING"
/bin/false"""
\end{lstlisting}

Finally, in simulation mode each task takes between 1 and 15 seconds to
``run'' by default, but you can also alter this for particular tasks or
groups of tasks:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
       [[[simulation mode]]]
        run time range = PT20S,PT31S # (between 20 and 30 seconds)
\end{lstlisting}
Note that to get a failed simulation or dummy mode task to succeed on
re-triggering, just change the suite.rc file appropriately and reload
the suite definition at run time with \lstinline=cylc reload SUITE=
before re-triggering the task.

Dummy mode is equivalent to removing all user-defined task scripting
to expose the default scripting.

\subsubsection{Restarting Suites With A Different Run Mode?}

The run mode is recorded in the suite run database files. Cylc will not let
you {\em restart} a non-live mode suite in live mode, or vice versa -
any attempt to do the former would certainly be a mistake (because the
simulation mode dummy tasks do not generate any of the real outputs
depended on by downstream live tasks), and the latter, while feasible,
would corrupt the task pool by turning it over to simulation mode.
The easiest way to test a live suite in simulation mode, if you don't want to
obliterate the current suite run database files by doing a cold or warm start
(as opposed to a restart from the previous state) is to take a quick copy of
the suite and run the copy in simulation mode. However, if you really want to
run a live suite forward in simulation mode without copying it, do this:
\begin{myenumerate}
    \item Back up the live mode suite database files.
    \item Using the \lstinline=sqlite3= command or otherwise,
          update the value of \lstinline=run_mode= in the
          \lstinline=suite_params= table in the
          \lstinline=cylc-suite-private.db= file.
    \item Later, restart the live suite from the restored back up.
\end{myenumerate}

\subsection{Automated Reference Test Suites}
\label{AutoRefTests}

Reference tests are finite-duration suite runs that abort with non-zero
exit status if any of the following conditions occur (by default):

\begin{myitemize}
    \item cylc fails
    \item any task fails
    \item the suite times out (e.g.\ a task dies without reporting failure)
    \item a nominated shutdown event handler exits with error status
\end{myitemize}

The default shutdown event handler for reference tests is
\lstinline=cylc hook check-triggering= which compares task triggering
information (what triggers off what at run time) in the test run suite
log to that from an earlier reference run, disregarding the timing and
order of events - which can vary according to the external queueing
conditions, runahead limit, and so on.

To prepare a reference log for a suite, run it with the
\lstinline=--reference-log= option, and manually verify the
correctness of the reference run.

To reference test a suite, just run it (in dummy mode for the most
comprehensive test without running real tasks) with the
\lstinline=--reference-test= option.

A battery of automated reference tests is used to test cylc before
posting a new release version. Reference tests can also be used to check that
a cylc upgrade will not break your own complex
suites - the triggering check will catch any bug that causes a task to
run when it shouldn't, for instance; even in a dummy mode reference
test the full task job script (sans \lstinline=script= items) executes on the
proper task host by the proper job submission method.

Reference tests can be configured with the following settings:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    [[reference test]]
        suite shutdown event handler = cylc check-triggering
        required run mode = dummy
        allow task failures = False
        live mode suite timeout = PT5M
        dummy mode suite timeout = PT2M
        simulation mode suite timeout = PT2M
\end{lstlisting}

\subsubsection{Roll-your-own Reference Tests}

If the default reference test is not sufficient for your needs, firstly
note that you can override the default shutdown event handler, and
secondly that the \lstinline=--reference-test= option is merely a short
cut to the following suite.rc settings which can also be set manually if
you wish:

\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    abort if any task fails = True
    [[events]]
        shutdown handler = cylc check-triggering
        timeout = PT5M
        abort if shutdown handler fails = True
        abort on timeout = True
\end{lstlisting}


\subsection{Inter-suite Dependence: Triggering Off Task States In Other Suites}
\label{SuiteStatePolling}

The \lstinline=cylc suite-state= command interrogates suite run databases. It
has a polling mode that waits for a given task in the target suite to achieve a
given state. This can be used to make task scripting wait for a remote task
to succeed (for example). The suite graph notation also provides a way to
define automatic suite-state polling tasks, which use the same polling command
under the hood. Note that cylc suite-state can only trigger off task
{\em states} in remote suites and does not support triggering off task
messages.

Here's how to trigger a task \lstinline=bar= off a task \lstinline=foo= in
a remote suite called \lstinline=other.suite=:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00, T12]]]
            graph = "my-foo<other.suite::foo> => bar"
\end{lstlisting}
Local task \lstinline=my-foo= will poll for the success of \lstinline=foo=
in suite \lstinline=other.suite=, at the same cycle point, succeeding only when
or if it succeeds. Other task states can also be polled:
\begin{lstlisting}
   graph = "my-foo<other.suite::foo:fail> => bar"
\end{lstlisting}

The default polling parameters (e.g.\ maximum number of polls and the interval
between them) are printed by \lstinline=cylc suite-state --help= and can be
configured if necessary under the local polling task runtime section:
\begin{lstlisting}
[scheduling]
    [[ dependencies]]
        [[[T00,T12]]]
            graph = "my-foo<other.suite::foo> => bar"
[runtime]
    [[my-foo]]
        [[[suite state polling]]]
            max-polls = 100
            interval = PT10S
\end{lstlisting}

For suites owned by others, or those with run databases in non-standard
locations, use the \lstinline=--run-dir= option, or in-suite:
\begin{lstlisting}
[runtime]
    [[my-foo]]
        [[[suite state polling]]]
            run-dir = /path/to/top/level/cylc/run-directory
\end{lstlisting}

If the remote task has a different cycling sequence, just arrange for the
local polling task to be on the same sequence as the remote task that it
represents. For instance, if local task \lstinline=cat= cycles 6-hourly at
\lstinline=0,6,12,18= but needs to trigger off a remote task \lstinline=dog=
at \lstinline=3,9,15,21=:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T03,T09,T15,T21]]]
            graph = "my-dog<other.suite::dog>"
        [[[T00,T06,T12,T18]]]
            graph = "my-dog[-PT3H] => cat"
\end{lstlisting}

For suite-state polling the cycle point of the target task is treated as a
literal string so the polling command has to be told if the remote suite has a
different cycle point format. Use the \lstinline=--template= option for this,
or in-suite:
\begin{lstlisting}
[runtime]
    [[my-foo]]
        [[[suite state polling]]]
            template = %Y-%m-%dT%H
\end{lstlisting}

Note that the remote suite does not have to be running when polling commences
because the command interrogates the suite run database, not the suite server
process.

\section{Other Topics In Brief}

The following topics have yet to be documented in detail.

\begin{myitemize}
    \item Intervening in suites, e.g.\ stopping, removing, inserting tasks:
        see \lstinline=cylc control help=.

    \item Interrogating suites and tasks:
        see \lstinline=cylc info help=, \lstinline=cylc show help=,
        and \lstinline=cylc discovery help=.

    \item Understanding suite evolution, particularly in delayed/catchup
        operation: the Tutorial helps here (\ref{Tutorial}),
        along with running the example suites.

    \item Task insertion (add a new task proxy instance to a running suite):
        see \lstinline=cylc insert --help=.

    \item Sub-suites: to run another suite inside a task, just invoke the
        sub-suite, with appropriate start and end cycle points (probably a
        single cycle point), via the host task's \lstinline=script= item:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        script = \
 "cylc run SUITE $CYLC_TASK_CYCLE_POINT --until=$CYLC_TASK_CYCLE_POINT"
\end{lstlisting}

\end{myitemize}
\lstset{language=transcript}

\section{Suite Storage, Discovery, Revision Control, and Deployment}
\label{SuiteStorageEtc}

Small groups of cylc users can of course share suites by manual copying,
and generic revision control tools can be used on cylc suites as for any
collection of files. Beyond this cylc does not have a built-in solution
for suite storage and discovery, revision control, and deployment, on a
network. That is not cylc's core purpose, and large sites may have
preferred revision control systems and suite meta-data requirements that
are difficult to anticipate. We can, however, recommend the use of {\em
Rose} to do all of this very easily and elegantly with cylc suites.

\subsection{Rose}
\label{Rose}

{\bf Rose} is {\em a framework for managing and running suites of
scientific applications}, developed at the UK Met Office for use with
cylc. It is available under the open source GPL license.

\begin{myitemize}
    \item Rose documentation: \url{http://metomi.github.io/rose/doc/rose.html}
    \item Rose source repository: \url{https://github.com/metomi/rose}
\end{myitemize}


\section{Suite Design Principles}
\label{SuiteDesignPrinciples}

%Simplicity, flexibility, efficiency, and portability of cylc suites.

\subsection{Make Fine-Grained Suites}
\label{Granularity}

A suite can contain a small number of large, internally complex tasks; a
large number of small, simple tasks; or anything in between. Cylc can
easily handle a large number of tasks, however, so there are definite
advantages to fine-graining:

\begin{myitemize}
    \item A more modular and transparent suite.

    \item More functional parallelism (multiple tasks running
        at once).

    \item Faster debugging and failure recovery: rerun just the tasks(s)
        that failed.

    \item More code reuse: similar tasks may be able to call the same
        underlying script or command with differing input parameters.

\end{myitemize}

\subsection{Make Tasks Re-runnable}

It should be possible to rerun a task by simply resubmitting it for the
same cycle point. In other words, failure at any point during execution
of a task should not render a rerun impossible by corrupting the state
of some internal-use file, or whatever. It is difficult to overstate the
usefulness of being able to rerun the same task multiple times,
either outside of the suite with \lstinline=cylc submit=, or by
re-triggering it within the running suite, when debugging a problem.

\subsection{Make Models Re-runnable}

If a warm-cycled model uses the exact same file names for its restart files
regardless of current cycle point, the only cycle point that can subsequently
run successfully is the next one. Instead, restart files should be labelled
with current cycle point and maintained in a simple rolling archive. Then you
can easily rerun the task for any cycle point still in the archive.

\subsection{Avoid False Dependence}
\label{LimitPID}

If a task does not depend on files generated by another task then generally
speaking it should not trigger off that task in the suite scheduling graph.
Unnecessary dependence between tasks restricts functional parallelism at run
time, and it makes the suite more difficult to understand. If you need to
restrict the number of tasks that are active at once, use runahead limiting
(\ref{RunaheadLimit}) and internal queues (\ref{InternalQueues}).


\subsection{Put Task Cycle Point In Output File Paths}
\label{PutCycleTimeinIO}

Putting task cycle point in output file or directory names makes archiving and
cleanup easier, and it facilitates re-runnability by ensuring that important
files do not get overwritten from one cycle to the next.

The \lstinline=cylc cycle-point= command computes offsets from a given or
current cycle point, and can insert the resulting computed date-time into
a filename template string.

\subsection{Managing Input/Output File Dependencies}
\label{HandlingDependencies}

Dependence between tasks usually, although not always, take the form of
files generated by one task and used by others. It is possible to manage these
files across a suite without compromising suite flexibility and portability with
hard wired I/O locations.

\subsubsection{Common I/O Workspaces}

You may be able to have all tasks, or groups of tasks that need to cooperate,
read and write from a common workspace, thereby avoiding the need to explicitly
move files around. The suite share directory
(\lstinline=$CYLC_SUITE_SHARE_DIR=) is provided for this purpose.
Similarly, task work directories are private to each task by default but they
can be shared to allow multiple tasks to simply read and write from their
current working directory. Even if you use other custom I/O directories, define
their locations in the suite.rc file rather than hard wiring them into task
implementation. Shared workspace locations can be passed to tasks as needed
without modifying the task implementation, like this:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
   [[SHARE]]
      [[[environment]]]
         WORKDIR = $CYLC_SUITE_SHARE_DIR/workspace1
   [[foo]]
      inherit = SHARE
      script = generate-data.exe
      [[[environment]]]
         MY_OUTPUT_DIR = $WORKDIR
   [[bar]]
      inherit = SHARE
      script = use-data.exe
      [[[environment]]]
         MY_INPUT_DIR = $WORKDIR
\end{lstlisting}

\subsubsection{Connector Tasks}

Special tasks can be used to move files around, from one task's output
directory to another's input directory. This should only be necessary across
host or filesystem boundaries, however; otherwise simply reference shared
locations as shown above.

\subsection{Reuse Task Implementation}

If your suite contains multiple logically distinct tasks that have similar
functionality (e.g.\ tasks that move files around or generate similar products
from different datasets) just have them all call the same underlying command,
script, or executable, but provide different input parameters as required.


\subsection{Make Suites Portable}

If all I/O is automatically done in suite-specific locations, such as the under
suite share and work directories (\lstinline=$CYLC_SUITE_SHARE_DIRECTORY=
and \lstinline=$CYLC_TASK_WORK_DIRECTORY=), you should be able to run multiple
copies of the same suite without interference between them, and other users
should be able to copy and run your suites with minimal modification.

\subsection{Make Tasks As Independent As Possible}

Where possible a task should not rely on the action of another task, except for
the inputs embodied in the suite dependency graph that it has no choice but to
depend on. This makes it as easy as possible to run single tasks alone during
suite development and debugging. For example, tasks should create their own
output directories if necessary rather than assuming their existence due to the
action of another task. Note that if the existing task implementation does not
handle output directory creation you can do it in suite
\lstinline=pre-script= or similar.

\subsection{Make Suites As Self-Contained As Possible}

Tasks can (of course) run external commands, scripts, and executables; and they
can read or otherwise make use of external files. In some cases this may be
necessary, but it does leave suites vulnerable to external breakages.
Alternatively, suites can be more or less completely self-contained (aside from
exposure to network, filesystem, and OS problems) if they have private copies
of every file they need at run time. Tasks can access files stored under their
suite definition directory via \lstinline=$CYLC_SUITE_DEF_PATH=, and the suite
bin directory is automatically added to \lstinline=$PATH= in the task execution
environment. If you have multiple suites there may be a tradeoff between
self-containment and duplication of files, but this does not particularly
matter if you can automatically extract, build, and install suite files from
external repositories prior to, or at the start of, a suite run.

\subsubsection{Distinguish Between Source and Installed Suites}

A suite definition and any files stored with it should be version controlled,
and a particular revision extracted before a run. The extracted source suite
will be a repository clone or working copy, depending on your choice of
revision control software, and can be used for further development. The source
files should then be {\em installed} to another location where the suite will
actually be executed (the cylc suite run directory is ideal for this).
External files may also be installed into the suite at this time, prior to the
run, or by special deployment tasks that run at suite start-up. This makes
self-containment easier to achieve, and the clean separation of source and
installed suite allows further development without breaking a running suite.
Rose (\ref{Rose}) supports this mode of working with cylc suites.


\subsection{Orderly Product Generation?}
\label{OrderlyProductGeneration}

Correct scheduling is not necessarily equivalent to orderly generation of
products in strict date-time order. Under cylc a product generation task will
trigger as soon as its private prerequisites are satisfied regardless of
whether other tasks at the same cycle point have finished or have yet to run.
If your product presentation system demands that all products are uploaded in
order, then be aware that this may be quite inefficient if your suite ever has
to catch up from a delay or run over historical data, but if necessary you can
force tasks to run in the right order even if their true dependencies do not
require that. One way to do this is to declare the product upload task to be
{\em sequential}, which is equivalent to making it depend on its own previous
instance (see~\ref{SequentialTasks}).

\subsection{Use Of Clock-Trigger Tasks}

Tasks that wait on external real time data should have a clock-trigger to delay
submission until roughly the expected time of data availability
(see~\ref{ClockTriggerTasks}), otherwise they could clutter up your batch
scheduler queue by submitting hours earlier. Similarly, suite polling tasks
(for inter-suite dependence in real time operation) should use a clock-trigger
to delay their submission until the expected time of the remote suite event.

\subsection{Tasks That Wait On Something}

Some tasks wait on external events and therefore need to repeatedly check and
wait for the event before reporting eventual success (or perhaps failure after
a timeout). For example, a task that waits for a file to appear on an ftp
server. Typically these should be clock-trigger tasks (see above), but once
triggered there are two ways to handle the repeated checking: the task itself
could implement a check-and-wait loop; or you could just configure multiple
retries for the task (see~\ref{TaskRetries}).

\subsection{Do Not Treat Real Time Operation As Special}

Cylc suites, without modification, can handle real time and delayed operation
equally well. In caught-up real time operation, clock-trigger tasks
constrain the behaviour of the whole suite, or at least of any tasks
downstream of them in the dependency graph. In delayed or historical operation
clock-trigger tasks will not constrain the suite at all, and cylc's cycle
point interleaving abilities come to the fore, because the clock-trigger times
have already passed. But if a clock-trigger task catches up to the wall
clock, it will automatically wait again. In this way cylc suites naturally
transition between delayed and real time operation as required.

\subsection{Factor Out Common Configuration}

To help avoid suite maintenance errors in the future, properties shared by
multiple tasks (job submission settings, environment variables, scripting,
etc.) should be defined only once, using runtime inheritance
(\ref{NIORP}) or Jinja2 variables (\ref{Jinja2}).

Multiple inheritance is efficient when tasks share many properties, but Jinja2
variables may be preferred when a small number of properties are shared by
tasks that don't have anything else in common (e.g.\ a single environment
variable for the location of a shared file).

For environment variables in particular it may be tempting to define all
variables for all tasks once under \lstinline=[root]=, but this is analagous to
overuse of global variables in programming and it can make it difficult to
determine which variables matter to which tasks. Environment filters
(\ref{EnvironmentFilter}) can be used to make this safer, but generally
it is best to provide each task with only the variables that it needs. It is
difficult to be sure if a task really needs a variable that is passed to it,
but you can be sure that it does not use a variable that is not passed to it.

Finally, Jinja2 can also be used to avoid polluting task environments with
variables used for the sole purpose of deriving other variables at task
run time. Instead of this:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[root]]
        [[[environment]]]
            OUTPUT_DIR=/my/top/outputdir
    [[foo]]
        [[[environment]]]
            FOO_OUTPUT_DIR=$OUTPUT_DIR/foo
            BAR_OUTPUT_DIR=$OUTPUT_DIR/bar
\end{lstlisting}
        do this:
\lstset{language=suiterc}
\begin{lstlisting}
{% set OUTPUT_DIR = "/my/top/outputdir" %}
[runtime]
    [[foo]]
        [[[environment]]]
            FOO_OUTPUT_DIR={{ OUTPUT_DIR }}/foo
            BAR_OUTPUT_DIR={{ OUTPUT_DIR }}/bar
\end{lstlisting}

If the values of these Jinja2 variables are needed in external
scripts, just translate them directly in environment sections:
\lstset{language=suiterc}
\begin{lstlisting}
    [[[environment]]]
        OUTPUT_DIR = {{ OUTPUT_DIR }}
\end{lstlisting}


\subsection{Use The Graph For Scheduling}

If you find yourself writing runtime scripting to change a task's behaviour
in some cycle points, consider that the graph is usually the proper place to
express this sort of thing. Use different task names, but have them inherit
common properties to avoid duplication. Instead of this:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            graph = "foo => shout => baz"
[runtime]
    [[shout]]
        script = """
if [[ $( cylc cycle-point --print-hour ) == 06 || \
      $( cylc cycle-point --print-hour ) == 18 ]]; then
    SENTENCE="the quick brown fox"
else
    SENTENCE="the lazy dog"
fi
echo $SENTENCE"""
        # (...other config...)
\end{lstlisting}
        do this:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "foo => shout_dog => baz"
        [[[T06,T18]]]
            graph = "foo => shout_fox => baz"
[runtime]
    [[SHOUT]]
        # (... other config...)
        script = "echo $SENTENCE"
    [[shout_fox]]
        inherit = SHOUT
        [[[environment]]]
            SENTENCE = "the quick brown fox"
    [[shout_dog]]
        inherit = SHOUT
        [[[environment]]]
            SENTENCE = "the lazy dog"
\end{lstlisting}

Similarly, if your task has a different behaviour at the initial or final
cycle point, consider using an \lstinline=R1= syntax to separate out the
functionality.

\subsection{Use Suite Visualization}

Effective visualization can make complex suites easier to understand.
Collapsible task families for visualization are defined by the {\em first
parents} in the runtime namespace hierarchy. Tasks should generally be grouped
into visualization families that reflect their purpose within the structure of
the suite rather than technical detail such as common job submission method or
task host. This often coincides nicely with common configuration inheritance
requirements, but if it doesn't you can use an empty namespace as a first
parent for visualization:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[OBSPROC]]
    [[obs1, obs2, obs3]]
        inherit = OBSPROC
\end{lstlisting}
    and you can demote parents from primary to secondary:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[HOSTX]]
        # common settings for tasks on host HOSTX
    [[foo]]
        inherit = None, HOSTX
\end{lstlisting}

\section{Style Guide}

Good style is to some extent a matter of taste. That said, for collaborative
development of complex systems it is important to settle on a clear and
consistent style, and you may find the following suggestions useful. Note that
the boundary between this section (style) and the previous (design) is somewhat
arbitrary.

\subsection{Indentation}

The suite.rc file format consists of \lstinline@item = value@ pairs
under nested section headings. Clear indentation is the best way to show
local nesting level inside large blocks.

\begin{myitemize}
    \item Indent suite.rc syntax four spaces per nesting level.

\lstset{language=suiterc}
\begin{lstlisting}
[SECTION]
    title = the quick brown fox
    [[SUBSECTION]]
        a short item = value1
        a very very long item = value2
\end{lstlisting}
Don't align \lstinline@item = value@ pairs on the \lstinline@=@ character -
this does not show nesting level clearly and it pushes everything off to
the right:
\lstset{language=suiterc}
\begin{lstlisting}
[SECTION]
             a short item = value1
    a very very long item = value2
\end{lstlisting}

The following layout does preserve proper indentation on the left,
but the whole block may need reformatting after changing one line, which
pollutes your revision history with spurious changes:
\lstset{language=suiterc}
\begin{lstlisting}
[SECTION]
    a short item          = value1
    a very very long item = value2
\end{lstlisting}

    \item Set your text editor to convert {\em TAB characters} to spaces -
        tabs may be displayed differently in different editors, so a
        mixture of space and tab indentations can render to a mess.

    \item {\em Line comments} should be indented to the same level as the
        section or item they refer to. Consistent local indentation makes
        block re-indentation operations easier in text editors.

    \item  {\em script strings} are interpreted by the
        associated task job script, not by cylc, so strictly speaking
        their internal lines should not be indented as if part of the
        suite.rc syntax. This, for example:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        script = \
"""echo Hello World!
echo Goodbye World!"""
\end{lstlisting}
is preferred over this (or similar):
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        script ="""
            echo Hello World!
            echo Goodbye World!
                           """
\end{lstlisting}
        The extra whitespace here translates directly to spurious
        indentation in the task job script. As it happens this is just
        an aesthetic problem in bash scripts, but for Python job scripts
        (which cylc may support in the future) it would be a technical error.

    \item The positioning of string-delimiting triple quotes is of no
        practical consequence either, but the following forms are
        suggested for the same reason - to avoid including spurious
        whitespace in the string:
\begin{lstlisting}
[runtime]
    [[foo]]
        # best:
        script = \
"""echo Hello World!
echo Goodbye World!"""
        # or (short first line):
        script ="""echo Hello World!
echo Goodbye World!"""
        # or (adds a single extra newline character):
        script ="""
echo Hello World!
echo Goodbye World!"""
\end{lstlisting}

    \item Multiline dependency \lstinline@graph@ strings have no meaning
        outside of the suite definition, so they can be free-form in
        order to most clearly present the structure of the suite:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
   foo => bar => baz => qux
     # failure recovery:
     qux:fail => recover
                """
\end{lstlisting}

   \item Embedded {\em Jinja2} code is not part of the suite.rc syntax, so
       it should be indented from the left margin on its own terms.
\lstset{language=suiterc}
\begin{lstlisting}
    [[OPS]]
{% for T in OPS_TASKS %}
    [[ops_{{T}}]]
        inherit = OPS
        # ...
{% endfor %}
\end{lstlisting}
   \end{myitemize}

\subsection{Comments}

Comments should be {\em minimal}, but not too minimal. If context and clear
item names will do, leave it at that. Extremely verbose comments tend to be
neglected and eventually get out of sync with the code, a result that may be
worse than having no comments at all.

    \begin{myitemize}
        \item {\em Indent line comments} to section or item level, as
            described above.

        \item Avoid {\em numbered comments} - future changes can create a
            renumbering nightmare.

        \item Avoid {\em full page width ``section divider'' comments} -
            these assume a particular line width, which can be a problem
            for text editors that auto line break on a smaller line width.

        \item Use the \lstinline=title= and \lstinline=description= items
            instead of comments to describe tasks and families under
            \lstinline=[runtime]= - these get displayed by mouse hover in
            gcylc.

    \end{myitemize}

\subsection{Line Length}

Keep to the standard maximum line length of 79 characters where possible. Very
long lines affect readability, may pose a problem for auto-line-breaking in
text editors, and make side-by-side diff display less effective.

\begin{myitemize}
    \item Line continuation markers can be used anywhere to break up long
        lines:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "prep => one => two => three \
            => four => five six => seven => eight"
[runtime]
    [[MY_TASKS]]
    [[one, two, three, four, five, \
        six, seven, eight ]]
        inherit = MY_TASKS
\end{lstlisting}
Graph lines can also be split up without line breaks, like this:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """prep => one => two => three => four
                   four => five six => seven => eight"""
\end{lstlisting}

\end{myitemize}

\subsection{Task Naming Convention}

Use \lstinline=UPPERCASE_NAMES= for families and \lstinline=lowercase_names=
for tasks, so that you can tell which is which at a glance.

\begin{myitemize}
    \item Put the most general components of task names first, for natural
        grouping in the GUI (under alphanumeric sorting) and in listings, e.g.\
        \lstinline=obsproc_sonde=, \lstinline=obsproc_radar=.
\end{myitemize}

\subsection{Inlined Task Scripting}

Trivial task scripting may be inlined in the suite definition but anything
more should be written to a script file. This keeps the suite definition tidy,
it allows proper shell-mode text editing, and it allows separate command line
testing of the script during development or debugging.

\pagebreak

\appendix

\input{suiterc.tex}
\pagebreak

\input{siterc.tex}

\pagebreak

\input{gcylcrc.tex}

\pagebreak

\input{gscanrc.tex}

\pagebreak


\section{Command Reference}
\label{CommandReference}

%This section is auto-generated from the self-documenting command set.

\lstset{language=usage}
\input{commands.tex}
\lstset{language=transcript}

\section{The gcylc Graph View}
\label{TheGraphBasedcontrolGUI}

The graph view in the gcylc GUI shows the structure of the suite as it
evolves. It can work well even for large suites, but be aware that the
graphviz layout engine has to do a new global layout every time a task
proxy appears in or disappears from the task pool. The following may help
mitigate any jumping layout problems:

\begin{myitemize}
    \item The disconnect button can be used to temporarily prevent the
        graph from changing as the suite evolves.
    \item The greyed-out base nodes, which are only present to fill out
        the graph structure, can be toggled off (but this will split the
        graph into disconnected sub-trees).
    \item Right-click on a task and choose the ``Focus'' option to restrict
        the graph display to that task's cycle point. Anything interesting
        happening in other cycle points will show up as disconnected
        rectangular nodes to the right of the graph (and you can click on
        those to instantly refocus to their cycle points).
    \item Task filtering is the ultimate quick route to focusing on just
        the tasks you're interested in, but this will destroy the graph
        structure.
\end{myitemize}

\section{Cylc README File}

\lstinputlisting{../README.md}

\section{Cylc INSTALL File}
\label{INSTALL}

\lstinputlisting{../INSTALL.md}

\section{Cylc Development History - Major Changes}

\begin{myitemize}

    \item {\bf pre-cylc-3} - early versions focused on the new
    scheduling algorithm. A suite was a collection of ``task definition
    files'' that encoded the prerequisites and outputs of each task,
    exposing cylc's self-organising nature. Tasks could be transferred
    from one suite to another by simply copying their taskdef files over
    and checking prerequisite and output consistency. Global suite
    structure was not easy to discern until run time (although cylc-2
    could generate resolved run time dependency graphs).

    \item {\bf cylc-3} - a new suite design interface: dependency graph
    and task runtime properties defined in a single structured,
    validated, configuration file - the suite.rc file; graphical user
    interface; suite graphing.

    \item {\bf cylc-4} - refined and organized the suite.rc file
    structure; task runtime properties defined by an efficient
    inheritance hierarchy; support for the Jinja2 template processor in
    suite definitions.

    \item {\bf cylc-5} - multi-threading for continuous network request
    handling and job submission; more task states to distinguish job
    submission from execution; dependence between suites via new suite
    run databases; polling and killing of real task jobs; polling as
    task communications option.

    \item {\bf cylc-6} - specification of all date-times and cycling
    workflows via the ISO8601 date-times, durations, and recurrence
    expressions; integer cycling; a multi-process pool to execute job
    submissions, event handlers, and poll and kill commands.

\end{myitemize}

\section{Pyro}
\label{Pyro}

Pyro (Python Remote Objects) is a widely used open source objected
oriented Remote Procedure Call technology developed by Irmen de Jong.

Earlier versions of cylc used the Pyro Nameserver to marshal
communication between client programs (tasks, commands, viewers, etc.)
and their target suites. This worked well, but in principle it
provided a route for one suite or user on the network to bring down
all running suites by killing the nameserver. Consequently cylc now
uses Pyro simply as a lightweight object oriented wrapper for
direct network socket communication between client programs and their
target suites - all suites are thus entirely isolated from one another.

\section{Cylc 6 Migration Reference}
\label{cylc-6-migration}

Cylc 6 introduced new date-time-related syntax for the suite.rc file. In
some places, this is quite radically different from the earlier syntax.

\subsection{Timeouts and Delays}
\label{cylc-6-migration-timeout-delays}

Timeouts and delays such as \lstinline=[cylc][[events]]timeout= or
\lstinline=[runtime][[my_task]]retry delays= were written in a purely
numeric form before cylc 6, in seconds, minutes (most common), or hours,
depending on the setting.

They are now written in an ISO 8601 duration form, which has the benefit
that the units are user-selectable (use 1 day instead of 1440 minutes)
and explicit.

Nearly all timeouts and delays in cylc were in minutes, except for:\\*
\lstinline=[runtime][[my_task]][[[suite state polling]]]interval= \\*
\lstinline=[runtime][[my_task]][[[simulation mode]]]run time range= \\*
which were in seconds, and\\*
\lstinline=[scheduling]runahead limit=\\*
which was in hours (this is a special case discussed below
in~\ref{cylc-6-migration-runahead-limit}).

See Table \ref{cylc-6-migration-timeout-delays-table}.

\begin{table}[ht]
\caption{Timeout/Delay Syntax Change Examples}
\centering
\begin{tabular}{ l c c }
Setting & Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[cylc][[events]]timeout= & 180 & PT3H \\
\lstinline=[runtime][[my_task]]retry delays= & 2*30, 360, & 2*PT30M, PT6H, \\
 & 1440 & P1D \\
\lstinline=[runtime][[my_task]][[[suite state polling]]]interval= & 2 & PT2S \\
\end{tabular}
\label{cylc-6-migration-timeout-delays-table}
\end{table}

\subsection{Runahead Limit}
\label{cylc-6-migration-runahead-limit}

See~\ref{runahead limit}.

The \lstinline=[scheduling]runahead limit= setting was written as a number of
hours in pre-cylc-6 suites. This is now in ISO 8601 format for date-time
cycling suites, so \lstinline@[scheduling]runahead limit=36@ would be written
\lstinline@[scheduling]runahead limit=PT36H@.

There is a new preferred alternative to \lstinline=runahead limit=,
\lstinline=[scheduling]max active cycle points=. This allows the user to
configure how many cycle points can run at once (default \lstinline=3=). See
\ref{max active cycle points}.

\subsection{Cycle Time/Cycle Point}
\label{cylc-6-migration-cycle-point}

See~\ref{initial cycle point}.

The following suite.rc settings have changed name (Table
\ref{cylc-6-migration-cycle-point-time-table}):

\begin{table}[ht]
\caption{Cycle Point Renaming}
\centering
\begin{tabular}{ l l }
Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[scheduling]initial cycle time= & \lstinline=[scheduling]initial cycle point= \\
\lstinline=[scheduling]final cycle time= & \lstinline=[scheduling]final cycle point= \\
\lstinline=[visualization]initial cycle time= & \lstinline=[visualization]initial cycle point= \\
\lstinline=[visualization]final cycle time= & \lstinline=[visualization]final cycle point= \\
\end{tabular}
\label{cylc-6-migration-cycle-point-time-table}
\end{table}

This change is to reflect the fact that cycling in cylc 6+ can now be over
e.g. integers instead of being purely based on date-time.

Date-times written in \lstinline=initial cycle time= and
\lstinline=final cycle time= were in a cylc-specific 10-digit (or less)
\lstinline=CCYYMMDDhh= format, such as \lstinline=2014021400= for 00:00 on
the 14th of February 2014.

Date-times are now required to be ISO 8601 compatible. This can be achieved
easily enough by inserting a \lstinline=T= between the day and the hour
digits.

\begin{table}[ht]
\caption{Cycle Point Syntax Example}
\centering
\begin{tabular}{ l c c }
Setting & Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[scheduling]initial cycle time= & 2014021400 & 20140214T00 \\
\end{tabular}
\label{cylc-6-migration-cycle-point-syntax-table}
\end{table}

\subsection{Cycling}
\label{cylc-6-migration-cycling}

Special {\em start-up} and {\em cold-start} tasks have been removed from cylc
6. Instead, use the initial/run-once notation as detailed
in~\ref{initial-non-repeating-r1-tasks} and~\ref{AdvancedStartingUp}.

{\em Repeating asynchronous tasks} have also been removed because non date-time
workflows can now be handled more easily with integer cycling. See for instance
the satellite data processing example documented in~\ref{IntegerCycling}.

For repeating tasks with hour-based cycling the syntax has only minor changes:

Pre-cylc-6:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    ...
    [[dependencies]]
        [[[0,12]]]
            graph = foo[T-12] => foo & bar => baz
\end{lstlisting}
\lstset{language=transcript}

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    ...
    [[dependencies]]
        [[[T00,T12]]]
            graph = foo[-PT12H] => foo & bar => baz
\end{lstlisting}
\lstset{language=transcript}

Hour-based cycling section names are easy enough to convert, as seen in Table
\ref{cylc-6-migration-cycling-hours-table}.

\begin{table}[ht]
\caption{Hourly Cycling Sections}
\centering
\begin{tabular}{ l l }
Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[scheduling][[dependencies]][[[0]]]= & \lstinline=[scheduling][[dependencies]][[[T00]]]= \\
\lstinline=[scheduling][[dependencies]][[[6]]]= & \lstinline=[scheduling][[dependencies]][[[T06]]]= \\
\lstinline=[scheduling][[dependencies]][[[12]]]= & \lstinline=[scheduling][[dependencies]][[[T12]]]= \\
\lstinline=[scheduling][[dependencies]][[[18]]]= & \lstinline=[scheduling][[dependencies]][[[T18]]]= \\
\end{tabular}
\label{cylc-6-migration-cycling-hours-table}
\end{table}

The graph text in hour-based cycling is also easy to convert, as seen in
Table \ref{cylc-6-migration-cycling-hours-offset-table}.

\begin{table}[ht]
\caption{Hourly Cycling Offsets}
\centering
\begin{tabular}{ l l }
Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=my_task[T-6]= & \lstinline=my_task[-PT6H]= \\
\lstinline=my_task[T-12]= & \lstinline=my_task[-PT12H]= \\
\lstinline=my_task[T-24]= & \lstinline=my_task[-PT24H]= or even \lstinline=my_task[-P1D]= \\
\end{tabular}
\label{cylc-6-migration-cycling-hours-offset-table}
\end{table}

\subsection{No Implicit Creation of Tasks by Offset Triggers}
\label{cylc-6-migration-implicit-cycling}

Prior to cylc-6 inter-cycle triggers implicitly created task instances at
the offset cycle points. For example, this pre cylc-6 suite automatically
creates instances of task \lstinline=foo= at the offset hours
\lstinline=3,9,15,21= each day, for task \lstinline=bar= to trigger off at
\lstinline=0,6,12,18=:
\lstset{language=suiterc}
\begin{lstlisting}
# Pre cylc-6 implicit cycling.
[scheduling]
   initial cycle time = 2014080800
   [[dependencies]]
      [[[00,06,12,18]]]
         # This creates foo instances at 03,09,15,21:
         graph = foo[T-3] => bar
\end{lstlisting}
You can run this suite to see how it works (cylc-6+ is backward-compatible: it
generates the old-style behaviour when it detects an old-style suite
definition). Here's the direct translation of this suite to cylc-6+ format:
\lstset{language=suiterc}
\begin{lstlisting}
# In cylc-6+ this suite will stall.
[scheduling]
   initial cycle point = 20140808T00
   [[dependencies]]
      [[[T00,T06,T12,T18]]]
         # This does NOT create foo instances at 03,09,15,21:
         graph = foo[-PT3H] => bar
\end{lstlisting}
If you run this suite, \lstinline=bar.20140808T00= will execute at start-up
because dependence on tasks prior to the initial cycle point is ignored, but
the suite will then stall with \lstinline=bar.20140808T06= waiting on
\lstinline=foo.20140808T03= (right-click on the task in the GUI to see its
prerequisites), which does not exist because the offset \lstinline=foo=
instances have not been created. Note that if you graph the suite the offset
\lstinline=foo= instances do appear. That's because the graph expresses
dependence, and \lstinline=bar= really does depend on these instances of
\lstinline=foo=. The problem is just that the tasks don't get created at run
time, which is why the suite stalls. Here's the correct way to get the desired
behaviour in cylc-6+:
\lstset{language=suiterc}
\begin{lstlisting}
# Cylc-6+ requires explicit task instance creation.
[scheduling]
   initial cycle point = 20140808T00
   [[dependencies]]
      [[[T03,T09,T15,T21]]]
         graph = foo
      [[[T00,T06,T12,T18]]]
         graph = foo[-PT3H] => bar
\end{lstlisting}
Implicit creation of task instances by offset triggers has been disabled
because it is error prone: if a trigger offset is wrong it should cause a
triggering failure rather than create task instances at incorrect cycle points.

\section{Known Issues}
\label{KnownIssues}

\subsection{Current Known Issues}
\label{CurrentKnownIssues}

The best place to find current known issues is on Github:
\url{https://github.com/cylc/cylc/issues}.

\subsection{Notable Known Issues}
\label{NotableKnownIssues}

\subsubsection{Use of pipes in job scripts}
\label{PipeInJobScripts}

In bash, the return status of a pipeline is normally the exit status of the
last command. This is unsafe, because if any command in the pipeline fails, the
script will continue nevertheless.

For safety, a cylc task job script running in bash will have the
\lstinline=set -o pipefail= option turned on automatically. If a pipeline
exists in a task's \lstinline=script=, etc section, the failure of any part of
a pipeline will cause the command to return a non-zero code at the end, which
will be reported as a task job failure. Due to the unique nature of a pipeline,
the job file will trap the failure of the individual commands, as well as the
whole pipeline, and will attempt to report a failure back to the suite twice.
The second message is ignored by the suite, and so the behaviour can be safely
ignored. (You should probably still investigate the failure, however!)

\section{GNU GENERAL PUBLIC LICENSE v3.0}
\input{gpl-3.0}
