
<i>
This program was contributed by Katharina Kormann and Martin
Kronbichler.

The algorithm for the matrix-vector product is based on the article <a
href="http://dx.doi.org/10.1016/j.compfluid.2012.04.012">A generic
interface for parallel cell-based finite element operator
application</a> by Martin Kronbichler and Katharina Kormann, Computers
and Fluids 63:135&ndash;147, 2012, and the paper &quot;Parallel finite element operator
application: Graph partitioning and coloring&quot; by Katharina
Kormann and Martin Kronbichler in: Proceedings of the 7th IEEE
International Conference on e-Science, 2011.  </i>

<a name="Intro"></a>
<h1>Introduction</h1>

This program demonstrates how to use the cell-based implementation of finite
element operators with the MatrixFree class, first introduced in step-37, to
solve nonlinear partial differential equations. Moreover, we have another look
at the handling of constraints within the matrix-free framework.
Finally, we will use an explicit time-stepping
method to solve the problem and introduce Gauss-Lobatto finite elements that
are very convenient in this case since their @ref GlossMassMatrix "mass matrix" can be accurately
approximated by a diagonal, and thus trivially invertible, matrix. The two
ingredients to this property are firstly a distribution of the nodal points of
Lagrange polynomials according to the point distribution of the Gauss-Lobatto
quadrature rule. Secondly, the quadrature is done with the same Gauss-Lobatto
quadrature rule. In this formula, the integrals $\int_K \varphi_i \varphi_j
dx\approx \sum_q \varphi_i \varphi_j \mathrm{det}(J) \big |_{x_q}$ become
zero whenever $i\neq j$, because exactly one function $\varphi_j$ is one and
all others zero in the points defining the Lagrange polynomials.
Moreover, the Gauss-Lobatto distribution of nodes of Lagrange
polynomials clusters the nodes towards the element boundaries. This results in
a well-conditioned polynomial basis for high-order discretization
methods. Indeed, the condition number of an FE_Q elements with equidistant
nodes grows exponentially with the degree, which destroys any benefit for
orders of about five and higher. For this reason, Gauss-Lobatto points are the
default distribution for the FE_Q element (but at degrees one and two, those
are equivalent to the equidistant points).

<h3> Problem statement and discretization </h3>

As an example, we choose to solve the sine-Gordon soliton equation
\f{eqnarray*}
u_{tt} &=& \Delta u -\sin(u) \quad\mbox{for}\quad (x,t) \in
\Omega \times (t_0,t_f],\\
{\mathbf n} \cdot \nabla u &=& 0
\quad\mbox{for}\quad (x,t) \in \partial\Omega \times (t_0,t_f],\\
u(x,t_0) &=& u_0(x).
\f}

that was already introduced in step-25. As a simple explicit time
integration method, we choose leap frog scheme using the second-order
formulation of the equation. With this time stepping, the scheme reads in
weak form

\f{eqnarray*}
(v,u^{n+1}) = (v,2 u^n-u^{n-1} -
(\Delta t)^2 \sin(u^n)) - (\nabla v, (\Delta t)^2 \nabla u^n),
\f}
where <i> v</i> denotes a test function and the index <i>n</i> stands for
the time step number.

For the spatial discretization, we choose FE_Q elements
with basis functions defined to interpolate the support points of the
Gauss-Lobatto quadrature rule. Moreover, when we compute the integrals
over the basis functions to form the mass matrix and the operator on
the right hand side of the equation above, we use the
Gauss-Lobatto quadrature rule with the same support points as the
node points of the finite element to evaluate the integrals. Since the
finite element is Lagrangian, this will yield a diagonal mass matrix
on the left hand side of the equation, making the solution of the
linear system in each time step trivial.

Using this quadrature rule, for a <i>p</i>th order finite element, we use a
<i>(2p-1)</i>th order accurate formula to evaluate the integrals. Since the
product of two <i>p</i>th order basis functions when computing a mass matrix
gives a function with polynomial degree <i>2p</i> in each direction, the
integrals are not computed exactly.  However, the overall convergence
properties are not disturbed by the quadrature error on meshes with affine
element shapes with L2 errors proportional to <i>h<sup>p+1</sup></i>. Note
however that order reduction with sub-optimal convergence rates of the L2
error of <i>O(h<sup>p</sup>)</i> or even <i>O(h<sup>p-1</sup>)</i> for some 3D
setups has been reported <a href="https://dx.doi.org/10.1002/num.20353">in
literature</a> on deformed (non-affine) element shapes for wave equations
when the integrand is not a polynomial any more.

Apart from the fact that we avoid solving linear systems with this
type of elements when using explicit time-stepping, they come with two
other advantages. When we are using the sum-factorization approach to
evaluate the finite element operator (cf. step-37), we have to
evaluate the function at the quadrature points. In the case of
Gauss-Lobatto elements, where quadrature points and node points of the
finite element coincide, this operation is trivial since the value
of the function at the quadrature points is given by its one-dimensional
coefficients. In this way, the arithmetic work for the finite element operator
evaluation is reduced by approximately a factor of two compared to the generic
Gaussian quadrature.

To sum up the discussion, by using the right finite element and
quadrature rule combination, we end up with a scheme where we
only need to compute the right hand side vector corresponding
to the formulation above and then multiply it by the inverse of the
diagonal mass matrix in each time step. In practice, of course, we extract
the diagonal elements and invert them only once at the beginning of the
program.

<h3>Implementation of constraints</h3>

The usual way to handle constraints in <code>deal.II</code> is to use
the AffineConstraints class that builds a sparse matrix storing
information about which degrees of freedom (DoF) are constrained and
how they are constrained. This format uses an unnecessarily large
amount of memory since there are not so many different types of
constraints: for example, in the case of hanging nodes when using
linear finite element on every cell, most constraints have the form
$x_k = \frac 12 x_i + \frac 12 x_j$ where the coefficients $\frac 12$
are always the same and only $i,j,k$ are different. While storing this
redundant information is not a problem in general because it is only
needed once during matrix and right hand side assembly, it becomes a
bottleneck in the matrix-free approach since there this
information has to be accessed every time we apply the operator, and the
remaining components of the operator evaluation are so fast. Thus,
instead of an AffineConstraints object, MatrixFree uses a variable that
we call <code>constraint_pool</code> that collects the weights of the
different constraints. Then, only an identifier of each constraint in the
mesh instead of all the weights have to be stored. Moreover,
the constraints are not applied in a pre- and postprocessing step
but rather as we evaluate the finite element
operator. Therefore, the constraint information is embedded into the
variable <code>indices_local_to_global</code> that is used to extract
the cell information from the global vector. If a DoF is constrained,
the <code>indices_local_to_global</code> variable contains the global
indices of the DoFs that it is constrained to. Then, we have another
variable <code>constraint_indicator</code> at hand that holds, for
each cell, the local indices of DoFs that are constrained as well as
the identifier of the type of constraint. Fortunately, you will not see
these data structures in the example program since the class
<code>FEEvaluation</code> takes care of the constraints without user
interaction.

In the presence of hanging nodes, the diagonal mass matrix obtained on the
element level via the Gauss-Lobatto quadrature/node point procedure does not
directly translate to a diagonal global mass matrix, as following the
constraints on rows and columns would also add off-diagonal entries. As
explained in <a href="https://dx.doi.org/10.4208/cicp.101214.021015a">Kormann
(2016)</a>, interpolating constraints on a vector, which maintains the
diagonal shape of the mass matrix, is consistent with the equations up to an
error of the same magnitude as the quadrature error. In the program below, we
will simply assemble the diagonal of the mass matrix as if it were a vector to
enable this approximation.


<h3> Parallelization </h3>

The MatrixFree class comes with the option to be parallelized on three levels:
MPI parallelization on clusters of distributed nodes, thread parallelization
scheduled by the Threading Building Blocks library, and finally with a
vectorization by working on a batch of two (or more) cells via SIMD data type
(sometimes called cross-element or external vectorization).
As we have already discussed in step-37, you will
get best performance by using an instruction set specific to your system,
e.g. with the cmake variable <tt>-DCMAKE_CXX_FLAGS="-march=native"</tt>. The
MPI parallelization was already exploited in step-37. Here, we additionally
consider thread parallelization with TBB. This is fairly simple, as all we
need to do is to tell the initialization of the MatrixFree object about the
fact that we want to use a thread parallel scheme through the variable
MatrixFree::AdditionalData::thread_parallel_scheme. During setup, a dependency
graph is set up similar to the one described in the @ref workstream_paper ,
which allows to schedule the work of the @p local_apply function on chunks of
cells without several threads accessing the same vector indices. As opposed to
the WorkStream loops, some additional clever tricks to avoid global
synchronizations as described in <a
href="https://dx.doi.org/10.1109/eScience.2011.53">Kormann and Kronbichler
(2011)</a> are also applied.

Note that this program is designed to be run with a distributed triangulation
(parallel::distributed::Triangulation), which requires deal.II to be
configured with <a href="http://www.p4est.org/">p4est</a> as described
in the <a href="../../readme.html">deal.II ReadMe</a> file. However, a
non-distributed triangulation is also supported, in which case the
computation will be run in serial.

<h3> The test case </h3>

In our example, we choose the initial value to be \f{eqnarray*} u(x,t) =
\prod_{i=1}^{d} -4 \arctan \left(
\frac{m}{\sqrt{1-m^2}}\frac{\sin\left(\sqrt{1-m^2} t +c_2\right)}{\cosh(mx_i+c_1)}\right)
\f} and solve the equation over the time interval [-10,10]. The
constants are chosen to be $c_1=c_1=0$ and <i> m=0.5</i>. As mentioned
in step-25, in one dimension <i>u</i> as a function of <i>t</i> is the exact
solution of the sine-Gordon equation. For higher dimension, this is however
not the case.
