# Maintaining GMT

Below are instructions for developers and advanced users.

## Contents

- [Building the source code](#building-the-source-code)
- [Building the documentation](#building-the-documentation)
- [Testing GMT](#testing-gmt)
- [Creating source packages](#creating-source-packages)
- [Packaging](#packaging)
- [Updating the development source codes](#updating-the-development-source-codes)
- [Debugging GMT](#debugging-gmt)
- [Using build and test aliases](#using-build-and-test-aliases)
- [Continuous Integration](#continuous-integration)

## Building the source code

Refer to the [building guide](BUILDING.md) for instructions on building GMT from source. Several sections include a
`Note for developers` that provides helpful or necessary information for running tests and building the documentation.

## Building the documentation

To build the GMT documentation you need to install the [Sphinx](http://www.sphinx-doc.org/) documentation builder.
After configuring and building GMT (see the [building guide](BUILDING.md)), you can build GMT documentation using
the following commands within the build directory:

```
cmake --build . --target docs_depends     # Generate images included in the documentation
cmake --build . --target optimize_images  # Optimize PNG images for documentation [optional]
cmake --build . --target animation        # Generate animations included in the documentation [optional]
cmake --build . --target docs_man         # UNIX manual pages
cmake --build . --target docs_html        # HTML manual, tutorial, cookbook, and API reference
```

> Note: Refer to the file `admin/bashrc_for_gmt` for useful aliases for building the documentation.

> Note: [pngquant](https://pngquant.org/) is needed for optimizing images.

## Testing GMT

GMT ships with more than 1000 tests to make sure that any changes won't break its functionality. In addition to the
tests located in the [test/](/test) directory, GMT tests all the plots included in its documentation. The
documentation tests are located in the [doc/scripts/](/doc/scripts) directory. The majority of GMT tests are plot-based,
with each test requiring a bash script for generating the plot and a reference PostScript file. These tests pass if the
difference between a new plot generated using the test script and the reference PostScript file is less than a defined
threshold. Other tests compute grids, tables, or other output, with the test passing if a suitable comparison is made
against a reference case.

Tests that are known to fail are excluded by adding `# GMT_KNOWN_FAILURE` anywhere in the test script.

### Configuring CMake for testing GMT

To enable testing, you need to *uncomment* the following lines in your `ConfigUserAdvanced.cmake` when configuring GMT:

```
enable_testing()
set (DO_EXAMPLES TRUE)
set (DO_TESTS TRUE)
set (DO_ANIMATIONS TRUE)

set (SUPPORT_EXEC_IN_BINARY_DIR TRUE)

set (DO_API_TESTS ON)
```

Optionally, uncomment the following line and change `4` to the number of ctest jobs to run simultaneously.
```
set (N_TEST_JOBS 4)
```

### Running tests

After configuring CMake and building GMT, you can run all the tests by running this command in the build directory:

    cmake --build . --target check

You can also run `ctest` commands in the build directory. Below are some common used ctest commands.

-  Run all tests in 4 parallel jobs:

        ctest -j 4

-  Re-run all failing tests in previous run in 4 parallel jobs:

        ctest -j 4 --rerun-failed

-  Select individual tests using regexp with ctest:

        ctest --output-on-failure -R ex2[3-6]

> Note: Refer to the file `admin/bashrc_for_gmt` for useful aliases for running the tests.

### Reviewing test failures

There are several tests that are "known to fail" for GMT. Unless the `GMT_ENABLE_KNOWN2FAIL` variable is set when
configuring CMake or setting up `ConfigUserAdvanced.cmake`, these tests are excluded when running ctest using the
instructions provided in the [Running tests](#running-tests) section. Therefore, you should expect all tests to pass
unless something new is broken.

Information about failing tests is produced in `test/fail_count.txt` inside the build directory. For plot-based tests,
the subdirectories `test/` and `doc/scripts/` inside the build directory contain folders for each failing test. For
plot-based tests, the directory associated with each failing tests contains a `gmtest.sh` script, a `gmt.conf` file, an
alias to the test script, a PostScript file and PDF document generated by the test script, and a PNG image that shows
differences between the reference plot and new plot in magenta. In addition to these files, running the failing tests
with verbose output can be helpful for evaluating failures:

        ctest --rerun-failed --verbose

### Updating reference plots for tests

Pull requests should avoid needing to change PostScript files in the [test/](/test) and [doc/scripts/](/doc/scripts)
directories. However, if this is unavoidable, new PostScript reference files can be generated by running
`ctest -R <test-script-name>` in the build directory after following the [building guide](BUILDING.md) and the
[Configuring CMake for testing GMT](MAINTENANCE.md#configuring-cmake-for-testing-gmt) instructions. The new PostScript
file can then be copied from the appropriate subdirectory within `build/test/` or `build/doc/scripts/`
to [test/](/test) or [doc/scripts/](/doc/scripts) respectively.

## Creating source packages

Edit `cmake/ConfigDefault.cmake` and set `GMT_PACKAGE_VERSION_MAJOR`, `GMT_PACKAGE_VERSION_MINOR`, and
`GMT_PACKAGE_VERSION_PATCH`. Also set `GMT_PUBLIC_RELEASE` to `TRUE.` Then create source packages with:

```
cmake --build . --target gmt_release      # export the source tree and documentation
cmake --build . --target gmt_release_tar  # create tarballs (in tar.gz and tar.xz formats)
```

## Packaging

Currently, packaging with CPack works on macOS (Bundle, TGZ, TBZ2),
Windows (ZIP, NSIS), and UNIX (TGZ, TBZ2). On Windows you need to install
[NSIS](http://nsis.sourceforge.net/). After building GMT and the documentation,
build and place the executables, including the supplements, with

```
cmake --build . --target install
```

and then create the package with either one of these:

```
cmake --build . --target package
cpack -G TGZ|TBZ2|Bundle|ZIP|NSIS
```

## Updating the development source codes

Assuming you did not delete the build directory, this is just as simple as

```
cd <path-to-gmt>
git pull
cd build
cmake --build .
cmake --build . --target install
```

CMake will detect any changes to the source files and will automatically reconfigure. If you deleted all files inside
the build directory you have to run CMake again manually.

> Note: Refer to the file `admin/bashrc_for_gmt` for useful aliases for updating the development source code

## Debugging GMT

Guides for debugging GMT are provided in the
[developer resources section](https://docs.generic-mapping-tools.org/dev/debug.html) of the GMT documentation.

## Using build and test aliases

The file [admin/bashrc_for_gmt](/admin/bashrc_for_gmt) contains useful aliases for building and testing GMT that some
developers chose to use. New pull requests with other aliases that you find helpful are welcome.
This file is version controlled, so you should copy the file to a different location in order to edit and use it.
For example, use these commands to copy it to your home directory:

```
cd <path-to-gmt>
cp admin/bashrc_for_gmt ~/.bashrc_for_gmt
```

Here are the steps for setting up [`bashrc_for_gmt`](/admin/bashrc_for_gmt) after copying it to a new location:

- If you do not have [ninja](https://ninja-build.org/) installed, you will need to change `builder=ninja` to
  `builder=make` and `Bname="Ninja"` to `Bname="Unix Makefiles"`. Ninja is recommended for speeding up build times.
- You may need to update `pngview=open` and `pngview=open` depending on your preferred program for viewing files.
- Optionally, change `ncores=4` to the number of cores to use for building and running tests.
- Change `MATLAB=/Applications/MATLAB_R2019a.app` to the path for your version of the MATLAB app.
- Set `REPO_DIR` to the path that contains the local `git clone` copy of the GMT repository.
- Set `DATA_DIR` to the path that contains the folders `dcw-gmt-2.0.0/` and `gshhg-gmt-2.3.7/` for the dcw and gshhg
  datasets respectively. If these folders are not located in the same path, you can instead delete the line
  (`DATA_DIR=<path to directory containing GSHHG and DCW>`) and set the individual paths to the GSHHG and DCW source
  by changing (`export GMT_GSHHG_SOURCE=${DATA_DIR}/gshhg-gmt-2.3.7`) and
  (`export GMT_DCW_SOURCE=${DATA_DIR}/dcw-gmt-2.0.0`).
- Edit the file `~/.bashrc` to include the line `source <path>/bashrc_for_gmt`. If you set up `bashrc_for_gmt` as a
  hidden file in your home directory, this line should be `source ~/.bashrc_for_gmt`.

Here are some of the shortcuts included in [`bashrc_for_gmt`](/admin/bashrc_for_gmt):

- `gmt6` and `gtop` can be used to quickly `cd` to the top of the GMT source directory and repository base respectively.
- `gmtfind` can be used to list all source, docs, scripts, and text files where a string appears in the file
  (e.g., `gmtfind "Grid increment is"` returns all files that contain the string 'Grid increment is'). This includes all
  files recursively from the current working directory; `gtop` or `gmt6` can be used prior to this command to get
  to the source directory or repository base.
- `cmakegmtd`, `cmakegmtr`, and `cmakegmtx` configures cmake for debug, release, and XCode debug respectively.
- `dlog` and `rlog` can be used to open the debug and release build check error logs respectively.
- There are several aliases with various combinations of pulling new changes, deleting the build directories,
  configuring cmake, and building the source code. Each of these are documented with comments in `bashrc_for_gmt`.
- `checkdbuild` and `checkrbuild` can be used to run the tests for the debug and release builds respectively.
- `vpngdbuild` and `vpdfdbuild` can be used to open the results from all failing image-based tests.
- `view_png_failures_r` and `view_pdf_failures_r` can be used for view failures of the release build with a lag between
  opening each file.

## Continuous Integration

We use GitHub Actions Continuous Integration (CI) services to build and test the project on Linux, macOS, and Windows.

There are 11 configuration files located in `.github/workflows/`:

1. `backport.yml` (Backports Pull Requests with specific labels into the matching branch)

   This workflow backports Pull Requests labelled by "backport xxx" into the "xxx" branch. For example, the workflow
   backports Pull Requests labelled "backport 6.1" into the "6.1" branch. This workflow does not apply to Pull Requests
   from forks ([issue #3827](https://github.com/GenericMappingTools/gmt/issues/3827)).

2. `build.yml` (Build GMT and run a few simple tests)

   This workflow is run on every commit to the *master* branch and Pull Request branches. The workflow configures
   and builds GMT on Linux, macOS, and Windows and runs some simple tests. The workflow uses the scripts in the `ci/`
   directory for downloading/installing dependencies (`ci/download-coastlines.sh` and `ci/install-dependencies-*.sh`),
   configuring GMT (`ci/config-gmt-*.sh`), building GMT, and running some simple
   tests (`ci/simple-gmt-tests.*`).

3. `check-links.yml` (Check links in the repository and website)

   This workflow is run every Sunday at 12:00 (UTC) to check all external links in plaintext and HTML files. It will
   create an issue if broken links are found.

4. `ci-caches.yml` (Cache GMT remote data files and Windows vcpkg libraries needed for GitHub Actions CI)

   This workflow is run every Sunday at 12:00 (UTC). If new remote files are needed urgently, maintainers can
   manually uncomment the 'pull_request:' line in the `ci-caches.yml` file to refresh the cache. The workflow uses the
   script `ci/install-dependencies-windows.sh` to cache the vcpkg libraries.

5. `code-validator.yml` (Validate code consistency)

   This workflow is run on every commit to the *master* branch and Pull Request branches. It runs the scripts
   `src/gmt_make_PSL_strings.sh`, `src/gmt_make_enum_dicts.sh`, and `src/gmt_make_module_purpose.sh` and exits with an
   error if any changes are detected by git. It also checks the GMT version year in `ConfigDefault.cmake` and the
   permissions of the bash scripts.

6. `docker.yml` (Build GMT on different Linux distros using dockers)

   This workflow is run when Pull Requests are merged into the *master* branch. It downloads/installs dependencies
   and configures and builds GMT on different Linux distributions using dockers.

7. `docs.yml`  (Build documentation on Linux/macOS/Windows)

   This workflow is run when Pull Requests are merged into the *master* branch, if the Pull Request involved changes to
   files in the `doc/`, `.github/workflows/`, or `ci/` directories.

   The workflow also handles the documentation deployment:

   * Updating the development documentation by pushing the built HTML pages from the *master* branch into the `dev`
     folder of the *gh-pages* branch.
   * Updating the `latest` documentation link to the new release.

8. `draft-release.yml` (Drafts the next release notes)

   This workflow is run to draft the next release notes when new tags are made. It downloads
   GMT tarballs, macOS bundle and Windows installers from the GMT FTP server, calculate their
   sha256sums and save into `gmt-X.Y.Z-checksums.txt`, draft the release notes, and uploads
   them as release assets. Maintainers still need to review the draft release notes and
   click the "publish" button.

9. `lint-checker.yml` (Run cppchecks)

   This workflow is run every day at 12:00 (UTC) to run `admin/run_cppcheck.sh` on the source code.

10. `scm-check.yml` (Check for new scientific color maps releases)

    This workflows is run every Sunday at 12:00 (UTC) to check whether there has been a new release of the
    [Scientific colour maps](http://www.fabiocrameri.ch/colourmaps.php). If a new release is found, it will open an
    issue automatically.

11. `tests.yml`  (Tests on Linux/macOS/Windows)

    This workflow is run when Pull Requests are merged into the *master* branch, if the Pull Request involved changes
    to the folders that contain source code, workflows, tests, or scripts for generating documentation figures. It runs
    the full GMT test suite on Linux, macOS, and Windows.
