Keywords: deal dealii finite elements fem triangulation
Date: $Date: 2013-12-08 10:28:38 -0500 (Sun, 08 Dec 2013) $

# The deal.II Testsuite

The deal.II testsuite consists of two parts: build tests and the regression testsuite. While the build tests are used to check that the library can be compiled on different systems and with different (versions of) compilers, the regression tests are actually run and their output compared with previously stored output files to verify that what worked yesterday still works today. These two testsuites are described below.

deal.II has a testsuite that has, at the time this article is written (mid-2013), some 2,900 small programs (growing by roughly one per day) that we run every time we make a change to make sure that no existing functionality is broken. The expected output for every test is stored in our subversion archive, and when you run a test you are notified if a test produces different output. These days, every time we add a significant piece of functionality, we add at least one new test to the testsuite, and we also do so if we fix a bug, in both cases to make sure that future changes do not break what we have just checked in. Machines running tests send results back home and these are then converted into a webpage showing the status of our regression tests.

## Quick instructions

If you're impatient, use the following commands:

    $ mkdir new_directory
    $ cd new_directory
    $ svn checkout https://svn.dealii.org/trunk .
    $ mkdir build
    $ cd build
    $ cmake ../deal.II
    $ make -j16
    $ make -j16 setup_tests
    $ ctest -j16

The exact meaning of all of these commands will be explained in much greater detail below.

## Setting up the testsuite

In order to run it, you need to download and set up the testsuite first. The following paragraphs detail how to do that.

### Downloading the testsuite

To download the testsuite, check it out from the subversion repository, along with deal.II. To this end, go to an empty directory where you want to test deal.II and do this:

    $ svn checkout https://svn.dealii.org/trunk .

(The period at the end puts everything from under `trunk/` into the current directory, rather than creating a new `trunk/` directory.) You will then have two folders:

    ./deal.II
    ./tests

Note: CMake will pick up any testsuite that is located in a `tests` folder next to the source directory (`../tests`). If your test directory is at a different location you have to hint during configuration by specifying `TEST_DIR`:

    $ cmake -DTEST_DIR=<...>

### Preparing the testsuite

To enable the testsuite, configure and build deal.II in a build directory as normal (installation is not necessary). After that you can setup the testsuite via the "setup_tests" target:s

    $ make setup_tests

This will set up all tests supported by the current configuration. The testsuite can now be run in the current build directory as described below.

Setup can be fine-tuned using the following commands:

    $ make regen_tests - reruns configure stage in every testsuite subproject

    $ make clean_tests - runs the 'clean' target in every testsuite subproject

    $ make prune_tests - removes all testsuite subprojects

In addition, when setting up the testsuite, the following environment variables can be used to override default behavior when calling `make setup_tests`:

    TEST_DIFF
      - The diff tool and command line to use for comparison. If numdiff is
        available it defaults to "numdiff -a 1e-6 -q", otherwise plain diff
        is used.

    TEST_TIME_LIMIT
      - The time limit (in seconds) a single test is allowed to take. Defaults
        to 180 seconds

    TEST_PICKUP_REGEX
      - A regular expression to select only a subset of tests during setup.
        An empty string is interpreted as a catchall (this is the default).

## Running the testsuite

The testsuite can now be run in the build directory via

    $ ctest [-j N]

Here, `N` is the number of concurrent tests that should be run, in the same way as you can say `make -jN`. The testsuite is huge and will need around 12h on current computers running single threaded.

If you only want to run a subset of tests matching a regular expression, or if you want to exclude tests matching a regular expression, you can use

    $ ctest [-j N] -R '<positive regular expression>'
    $ ctest [-j N] -E '<negative regular expression>'

Note: Not all tests succeed on every machine even if all computations are correct, because your machine generates slightly different floating point outputs. To increase the number of tests that work correctly, install the numdiff tool that compares stored and newly created output files based on floating point tolerances. To use it, simply export where the `numdiff` executable can be found via the `PATH` environment variable so that it can be found during `make setup_tests`.

### How to interpret the output

A typical output of a `ctest` invocation looks like:

    $ ctest -j4 -R "base/thread_validity"
    Test project /tmp/trunk/build
          Start 747: base/thread_validity_01.debug
          Start 748: base/thread_validity_01.release
          Start 775: base/thread_validity_05.debug
          Start 776: base/thread_validity_05.release
     1/24 Test #776: base/thread_validity_05.release ...   Passed    1.89 sec
     2/24 Test #748: base/thread_validity_01.release ...   Passed    1.89 sec
          Start 839: base/thread_validity_03.debug
          Start 840: base/thread_validity_03.release
     3/24 Test #747: base/thread_validity_01.debug .....   Passed    2.68 sec
    [...]
          Start 1077: base/thread_validity_08.debug
          Start 1078: base/thread_validity_08.release
    16/24 Test #1078: base/thread_validity_08.release ...***Failed    2.86 sec
    18/24 Test #1077: base/thread_validity_08.debug .....***Failed    3.97 sec
    [...]

    92% tests passed, 2 tests failed out of 24

    Total Test time (real) =  20.43 sec

    The following tests FAILED:
            1077 - base/thread_validity_08.debug (Failed)
            1078 - base/thread_validity_08.release (Failed)
    Errors while running CTest

If a test failed (like `base/thread_validity_08.debug` in above example output), you might want to find out what exactly went wrong. To this end, you can search through `Testing/Temporary/LastTest.log` for the exact output of the test, or you can rerun this one test, specifying `-V` to select verbose output of tests:

    $ ctest -V -R "base/thread_validity_08.debug"
    [...]
    test 1077
        Start 1077: base/thread_validity_08.debug

    1077: Test command: [...]
    1077: Test timeout computed to be: 600
    1077: Test base/thread_validity_08.debug: RUN
    1077: ===============================   OUTPUT BEGIN  ===============================
    1077: Built target thread_validity_08.debug
    1077: Generating thread_validity_08.debug/output
    1077: terminate called without an active exception
    1077: /bin/sh: line 1: 18030 Aborted [...]/thread_validity_08.debug
    1077: base/thread_validity_08.debug: BUILD successful.
    1077: base/thread_validity_08.debug: RUN failed. Output:
    1077: DEAL::OK.
    1077: gmake[3]: *** [thread_validity_08.debug/output] Error 1
    1077: gmake[2]: *** [CMakeFiles/thread_validity_08.debug.diff.dir/all] Error 2
    1077: gmake[1]: *** [CMakeFiles/thread_validity_08.debug.diff.dir/rule] Error 2
    1077: gmake: *** [thread_validity_08.debug.diff] Error 2
    1077:
    1077:
    1077: base/thread_validity_08.debug: ******    RUN failed    *******
    1077:
    1077: ===============================    OUTPUT END   ===============================

So this specific test aborted in the `RUN` stage.

The general output for a successful test `<test>` in category `<category>` for build type `<build>` is

    xx: Test <category>/<test>.<build>: PASSED
    xx: ===============================   OUTPUT BEGIN  ===============================
    xx: [...]
    xx: <category>/<test>.<build>: PASSED.
    xx: ===============================    OUTPUT END   ===============================

And for a test that fails in stage `<stage>`:

    xx: Test <category>/<test>.<build>: <stage>
    xx: ===============================   OUTPUT BEGIN  ===============================
    xx: [...]
    xx: <category>/<test>.<build>: <stage> failed. [...]
    xx:
    xx: <category>/<test>.<build>: ******    <stage> failed    *******
    xx: ===============================    OUTPUT END   ===============================

Hereby, `<stage>` indicates the stage in which the test failed:

- `CONFIGURE`: only for test in the "build_tests" category: The test project failed in the configuration stage
- `BUILD`: compilation error occured
- `RUN`: the test executable could not be run / aborted
- `DIFF`: the test output differs from the reference output
- `PASSED`: the test run successful

Typically, tests fail because the output has changed, and you will see this in the DIFF phase of the test.

### Parallelize testsuite configuration

If you have set up a good portion of the testsuite you'll notice that a reconfiguration of the build directory takes a noticeable time (in the order of minutes instead of seconds) with `Reconfiguring testsuite subprojects`. By default (with the "Unix Makefiles" generator) this is due to the fact that the reconfiguration processes the subprojects sequentally. If you want to increase the parallelism, specify e.g. `MAKEOPTS="-j8"` as environment variable or as CMake variable.

Alternatively, you can use `Ninja`.

## Testsuite development

The following outlines what you need to know if you want to understand how the testsuite actually works, for example because you may want to add tests along with the functionality you are currently developing.

### General layout

A test usually consists of a source file and an output file for comparison (under the testsuite directory `tests`):

    category/test.cc
    category/test.output

`category` will be one of the existing subdirectory under `tests/`, e.g., `lac/`, `base/`, or `mpi/`. Historically, we have grouped tests into the directories `base/, lac/, deal.II/` depending on their functionality, and `bits/` if they were small unit tests, but in practice we have not always followed this rigidly. There are also more specialized directories `trilinos/, petsc/, serialization/, mpi/` etc, whose meaning is more obvious. `test.cc` must be a regular executable (i.e. having an `int main()` routine). It will be compiled, linked and run. The executable should not output anything to `cout` (at least under normal circumstances, i.e. no error condition), instead the executable should output to a file `output` in the current working directory. In practice, we rarely write the source files completely from scratch, but we find an existing test that already does something similar and copy/modify it to fit our needs.

For a normal test, `ctest` will typically run the following 3 stages:

- `BUILD`: The build stage generates an executable in `BUILD_DIR/tests/<category>/<test>`.
- `RUN`: The run stage then invokes the executable in the directory where it is located. By convention, each test puts its output into a file simply called `output`, which will then be located in `BUILD_DIR/tests/<category>/<test>/output`. If the run fails (e.g. because the program aborts with an error code) the file `output` is renamed to `failing_output`.
- `DIFF`: As a last stage the generated output file will be compared to `SOURCE_DIR/tests/<category>/<test>[...].output`. and stored in `BUILD_DIR/tests/<category>/<test>/diff`. If the diff fails the file `diff` is renamed to `failing_diff`.

### Restricting tests to build configurations

Comparison file can actually be named in a more complex way than just `category/test.output`:

    category/test.[compiler=<regex>=<yes|no>.]*[with_<feature>=<on|off>.]*[mpirun=<x>.][expect=<y>.][binary.][<debug|release>.]output

Normally, a test will be set up so that it runs twice, once in debug and once in release configuration. If a specific test can only be run in debug or release configurations but not in both it is possible to restrict the setup by prepeding `.debug` or `.release` directly before `.output`, e.g.:

    category/test.debug.output

This way, the test will only be set up to build and run against the debug library. If a test should run in both configurations but, for some reason, produces different output (e.g., because it triggers an assertion in debug mode), then you can just provide two different output files:

    category/test.debug.output
    category/test.release.output

### Restricting tests to specific compiler and compiler versions

In a similar vain as for build configurations, it is possible to restrict tests to specific compiler versions, e.g.:

    category/test.compiler=GNU=yes.output, or
    category/test.compiler=ICC-14=no.output

These tests will only be set up if the specified regular expression matches (in case of `=yes`), or doesn't match (`=no`) the string `${CMAKE_CXX_COMPILER_ID}-${CMAKE_CXX_COMPILER_VERSION}`. Common compiler names are `GNU`, `Clang` or `Intel`.

### Restricting tests to feature configurations

In a similar vain as for build configurations, it is possible to restrict tests to specific feature configurations, e.g.:

    category/test.with_umfpack=on.output, or
    category/test.with_zlib=off.output

These tests will only be set up if the specified feature was configured. It is possible to provide different output files for disabled/enabled features, e.g.

    category/test.with_64bit_indices=on.output
    category/test.with_64bit_indices=off.output

It is also possible to declare multiple constraints subsequently, e.g.

    category/test.with_umfpack=on.with_zlib=on.output

Note: The tests in some subdirectories of `tests/` are automatically run only if some feature is enabled. In this case a feature constraint encoded in the output file name is redundant and should be avoided. In particular, this holds for subdirectories `distributed_grids`, `lapack`, `metis`, `petsc`, `slepc`, `trilinos`, `umfpack`, `gla`, and `mpi`

### Running tests with MPI

If a test should be run with MPI in parallel, the number of MPI processes `N` with which a program needs to be run for comparison with a given output file is specified as follows:

    category/test.mpirun=N.output

It is quite typical for an MPI-enabled test to have multiple output files for different numbers of MPI processes.

### Tests with binary output

If a test produces binary output add `binary` to the output file to indicate this:

    category/test.binary.output

The testsuite ensures that a diff tool suitable for comparing binary output files is used instead of the default diff tool, which (as in the case of `numdiff`) might be unable to compare binary files.

### Changing condition for success

Normally a test is considered to be successful if all test stages could be run and the test reached the `PASSED` stage (see the output description section for details). If (for some reason) the test should succeed ending at a specific test stage different than `PASSED` you can specify it via `expect=<stage>`, e.g.:

    category/test.expect=run.output

### Adding new tests

We typically add one or more new tests every time we add new functionality to the library or fix a bug. If you want to contribute code to the library, you should do this as well. Here's how: you need a testcase and a file with the expected output.

#### The testcase

For the testcase, we usually start from one of the existing tests, copy and modify it to where it does what we'd like to test. Alternatively, you can also start from a template like this:

    // ---------------------------------------------------------------------
    // $Id: testsuite.html 31939 2013-12-08 15:28:38Z heister $
    //
    // Copyright (C) 2013 by the deal.II Authors
    //
    // This file is part of the deal.II library.
    //
    // The deal.II library is free software; you can use it, redistribute
    // it, and/or modify it under the terms of the GNU Lesser General
    // Public License as published by the Free Software Foundation; either
    // version 2.1 of the License, or (at your option) any later version.
    // The full text of the license can be found in the file LICENSE at
    // the top level of the deal.II distribution.
    //
    // ---------------------------------------------------------------------

    // a short (a few lines) description of what the program does

    #include "../tests.h"
    #include <iostream>
    #include <fstream>

    // all include files you need here

    int main ()
    {
      std::ofstream logfile("output");
      deallog.attach(logfile);
      deallog.depth_console(0);

      // your testcode here:
      int i=0;
      deallog << i << std::endl;

      return 0;
    }

This code opens an output file `output` in the current working directory and then writes all output you generate to it, through the `deallog` stream. The `deallog` stream works like any other `std::ostream` except that it does a few more things behind the scenes that are helpful in this context. In above case, we only write a zero to the output file. Most tests of course write computed data to the output file to make sure that whatever we compute is what we got when the test was first written.

There are a number of directories where you can put a new test. Extensive tests of individual classes or groups of classes have traditionally been into the `base/`, `lac/`, `deal.II/`, `fe/`, `hp/`, or `multigrid/` directories, depending on where the classes that are tested are located. More atomic tests often go into `bits/`. There are also directories for PETSc and Trilinos wrapper functionality.

#### An expected output

In order to run your new test, copy it to an appropriate category and create an empty comparison file for it:

    category/my_new_test.cc
    category/my_new_test.output

Now, rerun

    $ make setup_tests

so that your new test is picked up. After that it is possible to invoke it with

    $ ctest -V -R "category/my_new_test"

If you run your new test executable this way, the test should compile and run successfully but fail in the diff stage (because of the empty comparison file). You will get an output file `BUILD_DIR/category/my_new_test/output`. Take a look at it to make sure that the output is what you had expected. (For complex tests, it may sometimes be impossible to say whether the output is correct, and in this case we sometimes just take it to make sure that future invokations of the test yield the same results.)

The next step is to copy and rename this output file to the source directory and replace the original comparison file with it:

    category/my_new_test.output

At this point running the test again should be successful:

    $ ctest -V -R "category/my_new_test"

#### Checking in

Tests are a way to make sure everything keeps working. If they aren't automated, they are no good. We are therefore very interested in getting new tests. If you have subversion write access already, you can add the new test and the expected output file:

    svn add category/my_new_test.cc
    svn add category/my_new_test.output
    svn commit -m "New test"

If you don't have subversion write access, talk to us in the discussion group; writing testcases is a worthy and laudable task, and we would like to encourage it by giving people the opportunity to contribute!

## Adding new categories

If you want to create a new category in the testsuite, create an new folder under ./tests that is named accordingly and put a `CMakeLists.txt` file into it containing

    CMAKE_MINIMUM_REQUIRED(VERSION 2.8.8)
    INCLUDE(${DEAL_II_SOURCE_DIR}/cmake/setup_testsubproject.cmake)
    PROJECT(testsuite CXX)
    INCLUDE(${DEAL_II_TARGET_CONFIG})
    DEAL_II_PICKUP_TESTS()

## Submitting test results

To submit test results to our CDash instance just invoke ctest within a build directory (or designated build directory) with the `-S` option pointing to the

    $ ctest [...] -V -S ../tests/run_testsuite.cmake

The script will run configure, build and ctest and submit the results to the CDash server. It does not matter whether the configure, build or ctest stages were run before that. Also in script mode, you can specify the same options for `ctest` as explained above.

Note: The default output in script mode is very minimal. Therefore, it is recommended to specify `-V` which will give the same level of verbosity as the non-script mode.

Note: The following variables can be set to via

    ctest -D<variable>=<value> [...]

to control the behaviour of the `run_testsuite.cmake` script:

    CTEST_SOURCE_DIRECTORY
      - The source directory of deal.II (usually ending in "[...]/deal.II"
        (equivalent to https://svn.dealii.org/trunk/deal.II)
        Note: This is _not_ the test directory ending in "[...]/tests"
      - If unspecified, "../deal.II" and "../../" relative to the location
        of this script is used. If this is not a source directory, an error
        thrown.

    CTEST_BINARY_DIRECTORY
      - The designated build directory (already configured, empty, or non
        existent - see the information about TRACKs what will happen)
      - If unspecified the current directory is used. If the current
        directory is equal to CTEST_SOURCE_DIRECTORY or the "tests"
        directory, an error is thrown.

    CTEST_CMAKE_GENERATOR
      - The CMake Generator to use (e.g. "Unix Makefiles", or "Ninja", see
        $ man cmake)
      - If unspecified the current generator of a configured build directory
        will be used, otherwise "Unix Makefiles".

    TRACK
      - The track the test should be submitted to. Defaults to "Experimental".
        Possible values are:

        "Experimental"     - all tests that are not specifically "build" or
                             "regression" tests should go into this track

        "Build Tests"      - Build tests that configure and build in a
                             clean directory and run the build tests
                             "build_tests/*"

        "Nightly"          - Reserved for nightly regression tests for
                             build bots on various architectures

        "Regression Tests" - Reserved for the regression tester

    CONFIG_FILE
      - A configuration file (see docs/development/Config.sample)
        that will be used during the configuration stage (invokes
        $ cmake -C ${CONFIG_FILE}). This only has an effect if
        CTEST_BINARY_DIRECTORY is empty.

    MAKEOPTS
      - Additional options that will be passed directly to make (or ninja).

Furthermore, the variables described above can also be set and will be handed automatically down to `cmake`.

## Build tests

Build tests are used to check that deal.II can be compiled on different systems and with different compilers as well as different configuration options. Results are collected in the "Build Tests" track in CDash.

Running the build test suite is simple and we encourage deal.II users with configurations not found on the CDash page to participate. Assuming you checked out deal.II into the directory `deal.II`, running it is as simple as:

    cd deal.II
    mkdir build
    cd build
    ctest -j4 -S ../cmake/scripts/run_buildtest.cmake

What this does is to compile and build deal.II in the directory `build`, try to configure, build (and run a subset) of all tutorial programs supported by the current configuration and send the results to the CDash instance.

Note: Build tests require the designated build directory to be completely empty and the source directory to be under subversion version control. If you want to specify a build configuration for cmake use a configuration file to preseed the cache as explained above:

    $ ctest -DCONFIG_FILE="[...]/Config.sample" [...]

### Dedicated build tests

Build tests work best if they run automatically and periodically. There is a detailed example for such dedicated build tests on the wiki.