The deal.II Readme on interfacing to MUMPS
Keywords: deal.II
Date: $Date: 2013-11-03 05:56:42 -0500 (Sun, 03 Nov 2013) $

# Interfacing deal.II to MUMPS

MUMPS should be readily packaged by most Linux distributions. Don't forget to install a develpment version of the library.

Below is a detailed instruction on how to compile and install MUMPS and some of its dependencies by hand (for the case you wish to do so).

## How to compile and install MUMPS by hand

In the following, we describe the compilation process of MUMPS and its dependencies starting with BLACS and following with SCALAPACK. The compilation of MUMPS and its dependencies is not as straightforward is we would like, however, it is not so difficult that we can not talk about it. That said, we cannot discuss every possible choice of parameters, because this is too involved (and we would probably much rather be doing something productive with the interface and deal.II). Thus we give only a short walk-through mentioning the most important parameters needed for successful compliation and linking.

The compilation scheme involves repeating the same three steps for each piece of software we want installed: (i) Selecting an appropriate makefile; (ii) Modifying symbols in this makefile to suit the environment; and (iii) running make to actually compile the software.

Note: If you are using a system, which provides Intel's MKL, you should only have to compile MUMPS, since BLACS and SCALPACK are already included in the MKL. Therefore you can jump directly to section 3, where the compilation of MUMPS is considered. Otherwise start right below with the installation of BLACS.

### 1. BLACS

The BLACS library provides basic linear algebra routines together with a communication model such that they can be used in parallel. The compilation of this library is highly dependent on the system and software used and we can only give a general guide through the configuration process.

The library can be found on the official BLACS webpage. We require the three files `mpiblacs.tgz`, `mpiblacs-patch03.tgz`, and `blacstester.tgz`. After extraction of these three files in the same directory, we find a subfolder named `BMAKES` in that directory. Inside this subfolder there are lots of different Bmake-files for different architectures. The naming of the files are conventional: `Bmake.XXX-YYY`, where `XXX` stands for the message passing software of your system (e.g. MPI, PVM) and `YYY` for the system architecture (e.g. LINUX, SUN4, SUN4SOL). Pick the corresponding file for your combination of message passing software and system architecture, copy it to the root directory of BLACS, and rename it `Bmake.inc`. Now that we have the correct makefile, we have to adapt it to the system we are using.

We start by opening Bamke.inc in our favourite text editor and taking a look at the following three sections:

- In `SECTION 1: PATHS AND LIBRARIES` set the MPI environment correctly by changing the variable `MPIDIR` to the directory, where your MPI installation is located in. Furthermore you will probably have to adjust the variables `MPILIBdir`, `MPIINCdir` and `MPILIB`.
- In the section `SECTION 2: BLACS INTERNALS` the most important variables are `INTFACE` and `TRANSCOMM`. If you are using OpenMPI, then set these as follows: `INTFACE = -DAdd_` and `TRANSCOMM = -DUseMpi2`. Otherwise, consult the BLACS webpage for details on how to set these values correctly.
- The last section is named `SECTION 3: Compilers`. Here we set the variables `F77` and `CC` to correspond to the MPI Fortran and CC compiler respectively, e.g. `mpif77` and `mpicc`. An important issue here are the variables `F77FLAGS` and `CCFLAGS`. It is absolutely vital to add the flag `-fPIC` to both of them, otherwise it will not be possible to link deal.II with the BLACS library correctly when shared libraries are enabled.

Assuming we have done all of that correctly, we go back to the BLACS root directory and compile the library with the command `make mpi`.

### 2. SCALAPACK

The SCALAPACK library provides high-performance linear algebra routines for distributed-memory message-passing MIMD computers. The library can be found on the official SCALAPACK webpage. We require the file `scalapack.tgz` version <= 1.8.0 only.

After extracting the file and changing to the directory which was created, we find a file called `SLmake.inc.example` and rename it to `SLmake.inc`. In this file we have to set a few parameters as we did before with BLACS' own `Bmake.inc`.

- The first important variable is the variable `home`. Here you have to take care, that this points to the SCALAPACK directory (i.e. where you are right now). Next, the variable `BLACSdir` points to the directory, where your BLACS libraries are located. Then set the variable `SMPLIB` to the MPI library.
- Now we have to get SCALAPACK to know about the location of the BLACS library, which we installed first. This is done by editing the following bunch of variables `BLACSFINIT`, `BLACSCINIT`, and `BLACSLIB`.
  - `BLACSFINIT` has to point to your F77init-library of BLACS. Check the name of the BLACS library file containing `F77init` in its name and put in this name here;
  - `BLACSCINIT` has to point to the BLACS library file containing `Cinit` in its name;
  - Finally, `BLACSLIB` points to the remaining BLACS library file.
- Next we set the compilers correctly again. `F77` and `CC` should point to your MPI Fortran and CC compiler respectively. Also at this place we have to add the flag `-fPIC` to the `F77FLAGS` and to the `CCFLAGS`. To the variable `CDEFS` you have to add the value of the variable `INTFACE` in the configuration file `Bmake.inc` in BLACS.
- Finally, we should check that the variables named `BLAS` and `LAPACK` point to your BLAS and LAPACK libraries compiled on your system.

If all variables are set correctly we can go ahead and compile the SCALAPACK library with the command `make`.

### 3. MUMPS

Now we come to the main program for which we are doing all this. We want to install the parallel sparse direct solver MUMPS.

After downloading the solver from its official webpage, extracting it, and changing to the directory which was created, we find a subfolder named `Make.inc` where there are bunch of different Makefiles for different machine architecture. Here it is important to realize that for each architecture there are two different files — one with the ending `.SEQ` and one without. The makefiles with the extension `.SEQ` are the ones required for the sequential version of MUMPS. Since we want to use MUMPS in parallel, we take the file without this extension which looks suitable for the architecture we are building on and copy this into the MUMPS root directory and rename it as `Makefile.inc`.

As before, we open the makefile `Makefile.inc` in our favourite text editor and start modifying variables:

- The first variables we have to edit are the compilers again. As usual the variable `CC` should point to your MPI CC compiler, e.g. `mpicc`. The variables `FC` and `FL` should both point to your MPI Fortran90 compiler, e.g. `mpif90`.
- Now let us consider the variable `SCALAP`. This one should point to your SCALAPACK0 and BLACS installations as follows: `-L/XXX -lscalapack -L/YYY -lblacs -lblacsCinit -lblacsF77init -lblacs -lblacsCinit`; where `XXX` stands for the directory containing the SCALAPACK library and `YYY` for the directory containing the BLACS library. The variable `INCPAR` should point to the MPI include directory.
- Now we turn to the variable `LIBPAR` which should contain the same value as the variable `SCALAP` plus a pointer to the MPI library. Therefore we propose to set it to the following value: `$(SCALAP) -L/XXX -lmpi`, where `XXX` stands for the directory containing your MPI libraries. Please check, if the variable `LIBBLAS` points to your BLAS installation. The variable `CDEFS` has to contain the same value as the variable `INTFACE` of BLACS' own `Bmake.inc`. If you are using OpenMPI this should be the value `-DAdd_`.
- Finally we add the flag `-fPIC` to the three compiler flag variables `OPTF`, `OPTC`, and `OPTL` which follow immediately after the above. Without adding this last flag it will not be possible to link deal.II with MUMPS as a shared library.

Now we are done with configuration and can compile MUMPS with the command `make`.

## Linking with deal.II

Support for MUMPS will be enabled automatically if a system wide installation of MUMPS can be found. To use a self compiled version, specify

        -DMUMPS_DIR=/path/to/mumps

on the command line when invoking `cmake`.

For SCALAPACK and BLACS a hint - if necessary - may be given by

        -DSCALAPACK_DIR=/path/to/scalapack
        -DBLACS_DIR=/path/to/blacs

You can override the autodetection by manually specifying

        -DDEAL_II_WITH_MUMPS=OFF|ON

.

Note: Throughout this description of the compilation process of MUMPS we have emphasised adding the compiler flag `-fPIC`. This is a definite requirement if we are compiling deal.II with shared libraries (which is the default). If we had preferred to be compiling deal.II without shared libraries, that's ok too; in that case we would do exactly the same thing as described above, but this time omitting the `-fPIC` flag from the scheme.