<h1>Results</h1>

<h3>Program output</h3>

Since this example solves the same problem as step-5 (except for
a different coefficient), there is little to say about the
solution. We show a picture anyway, illustrating the size of the
solution through both isocontours and volume rendering:

<img src="http://www.dealii.org/images/steps/developer/step-37.solution.png" alt="">

Of more interest is to evaluate some aspects of the multigrid solver.
When we run this program in 2D for quadratic ($Q_2$) elements, we get the
following output:
@code
Cycle 0
Number of degrees of freedom: 81
System matrix memory consumption:     0.008982 MB.
Multigrid objects memory consumption: 0.02617 MB.
Total setup time               (wall) 0.001811s
Time solve (5 iterations)  (CPU/wall) 0s/0.0002651s

Cycle 1
Number of degrees of freedom: 289
System matrix memory consumption:     0.01817 MB.
Multigrid objects memory consumption: 0.05779 MB.
Total setup time               (wall) 0.001223s
Time solve (5 iterations)  (CPU/wall) 0s/0.000926s

Cycle 2
Number of degrees of freedom: 1089
System matrix memory consumption:     0.05286 MB.
Multigrid objects memory consumption: 0.1581 MB.
Total setup time               (wall) 0.003045s
Time solve (6 iterations)  (CPU/wall) 0.012s/0.003393s

Cycle 3
Number of degrees of freedom: 4225
System matrix memory consumption:     0.1957 MB.
Multigrid objects memory consumption: 0.5228 MB.
Total setup time               (wall) 0.008561s
Time solve (6 iterations)  (CPU/wall) 0.02s/0.01133s

Cycle 4
Number of degrees of freedom: 16641
System matrix memory consumption:     0.7343 MB.
Multigrid objects memory consumption: 1.925 MB.
Total setup time               (wall) 0.02938s
Time solve (6 iterations)  (CPU/wall) 0.068s/0.03312s

Cycle 5
Number of degrees of freedom: 66049
System matrix memory consumption:     2.856 MB.
Multigrid objects memory consumption: 7.435 MB.
Total setup time               (wall) 0.1128s
Time solve (6 iterations)  (CPU/wall) 0.228s/0.09577s

Cycle 6
Number of degrees of freedom: 263169
System matrix memory consumption:     11.28 MB.
Multigrid objects memory consumption: 29.3 MB.
Total setup time               (wall) 0.4553s
Time solve (6 iterations)  (CPU/wall) 1.272s/0.3955s
@endcode

As in step-16, we see that the number of CG iterations remains constant with
increasing number of degrees of freedom. We can also see that the various
objects we have to store for the multigrid method on the individual levels of
our mesh together make up more than twice as much as the matrix on the finest
level. For the present example, about half the memory consumption of the
multigrid objects are the level transfer matrices, and the other half is
consumed by the matrix-free objects (and there, mainly the indices and the
variable coefficient).

Not much changes if we run the program in three spatial dimensions, with the
exception that the multilevel objects now take up some more memory (because
the level transfer matrices are denser) and the computing times are somewhat
larger:

@code
Cycle 0
Number of degrees of freedom: 125
System matrix memory consumption:     0.01093 MB.
Multigrid objects memory consumption: 0.03094 MB.
Total setup time               (wall) 0.002481s
Time solve (5 iterations)  (CPU/wall) 0s/0.000334s

Cycle 1
Number of degrees of freedom: 729
System matrix memory consumption:     0.04105 MB.
Multigrid objects memory consumption: 0.1274 MB.
Total setup time               (wall) 0.004471s
Time solve (5 iterations)  (CPU/wall) 0.004s/0.001979s

Cycle 2
Number of degrees of freedom: 4913
System matrix memory consumption:     0.2821 MB.
Multigrid objects memory consumption: 0.8048 MB.
Total setup time               (wall) 0.01651s
Time solve (4 iterations)  (CPU/wall) 0.036s/0.01295s

Cycle 3
Number of degrees of freedom: 35937
System matrix memory consumption:     1.948 MB.
Multigrid objects memory consumption: 5.734 MB.
Total setup time               (wall) 0.1072s
Time solve (5 iterations)  (CPU/wall) 0.16s/0.0709s

Cycle 4
Number of degrees of freedom: 274625
System matrix memory consumption:     14.49 MB.
Multigrid objects memory consumption: 44.41 MB.
Total setup time               (wall) 0.8173s
Time solve (5 iterations)  (CPU/wall) 1.52s/0.5093s

Cycle 5
Number of degrees of freedom: 2146689
System matrix memory consumption:     115.9 MB.
Multigrid objects memory consumption: 342.6 MB.
Total setup time               (wall) 6.387s
Time solve (5 iterations)  (CPU/wall) 12.45s/3.767s
@endcode


<h3>Comparison with a sparse matrix</h3>

In order to understand the capabilities of the matrix-free implementation, we
compare the performance on the 3d example above with a SparseMatrix
implementation and we measure the computation times for both initialization of
the problem (distribute DoFs, setup and assemble matrices, setup multigrid
structures) and the actual solution for the matrix-free variant and the
variant based on sparse matrices. We base the preconditioner on float
numbers and the actual matrix and vectors on double numbers, as shown
above. Tests are run on an Intel Core i7-2620M notebook processor (two cores
and <a href="http://en.wikipedia.org/wiki/Advanced_Vector_Extensions">AVX</a>
support, i.e., four operations on doubles can be done with one CPU
instruction, which is heavily used in FEEvaluation) and optimized mode. The
example makes use of multithreading, so both cores are actually used.

<table align="center" border="1">
  <tr>
    <th>&nbsp;</th>
    <th colspan="2">Sparse matrix</th>
    <th colspan="2">Matrix-free implementation</th>
  </tr>
  <tr>
    <th>n_dofs</th>
    <th>Setup + assemble</th>
    <th>&nbsp;Solve&nbsp;</th>
    <th>Setup + assemble</th>
    <th>&nbsp;Solve&nbsp;</th>
  </tr>
  <tr>
    <td align="right">125</td>
    <td align="center">0.0048s</td>
    <td align="center">0.00075s</td>
    <td align="center">0.0025s</td>
    <td align="center">0.00033s</td>
  </tr>
  <tr>
    <td align="right">729</td>
    <td align="center">0.014s</td>
    <td align="center">0.0022s</td>
    <td align="center">0.0026s</td>
    <td align="center">0.0018s</td>
  </tr>
  <tr>
    <td align="right">4,913</td>
    <td align="center">0.10s</td>
    <td align="center">0.012s</td>
    <td align="center">0.017s</td>
    <td align="center">0.013s</td>
  </tr>
  <tr>
    <td align="right">35,937</td>
    <td align="center">0.80s</td>
    <td align="center">0.14s</td>
    <td align="center">0.11s</td>
    <td align="center">0.071s</td>
  </tr>
  <tr>
    <td align="right">274,625</td>
    <td align="center">5.93s</td>
    <td align="center">1.05s</td>
    <td align="center">0.82s</td>
    <td align="center">0.51s</td>
  </tr>
  <tr>
    <td align="right">2,146,689</td>
    <td align="center">46.7s</td>
    <td align="center">8.44s</td>
    <td align="center">6.39s</td>
    <td align="center">3.77s</td>
  </tr>
</table>

The table clearly shows that the matrix-free implementation is twice as fast
for the solver, and more than six times as fast when it comes to
initialization costs. As the problem size is made a factor 8 larger, we note
that the times usually go up by a factor eight, too (as the solver iterations
are constant at 5). There are two deviations. The first is in the sparse
matrix between 5k and 36k degrees of freedom, where the time increases by a
factor 12. This is the threshold when the cache in the processor can no longer
hold all data necessary for the matrix-vector products and all matrix elements
must be fetched from main memory. The second deviation is the times for the
matrix-free solve which increase by less than a factor 8. This is because of
more parallelism from more cells, exploited by the (involved) dynamic task
scheduling approach taken in the cell loop of the MatrixFree class. Note
that about 30% of the time in the
matrix-free solver is spent on restriction and prolongation, which use sparse
matrices. So the speedup could be even better if all parts were done
efficiently.

Of course, this picture does not necessarily translate to all cases, as there
are problems where knowledge of matrix entries enables much better solvers (as
happens when the coefficient is varying more strongly than in the above
example). Moreover, it also depends on the computer system. The present system
has good memory performance, so sparse matrices perform comparably
well. Nonetheless, the matrix-free implementation gives a nice speedup already
for the <i>Q</i><sub>2</sub> elements used in this example. This becomes
particularly apparent for time-dependent or nonlinear problems where sparse
matrices would need to be reassembled over and over again, which becomes much
easier with this class. And of course, thanks to the better complexity of the
products, the method gains increasingly larger advantages when the order of the
elements increases (the matrix-free implementation has costs
4<i>d</i><sup>2</sup><i>p</i> per degree of freedom, compared to
2<i>p<sup>d</sup></i> for the sparse matrix, so it will win anyway for order 4
and higher in 3d).

<h3>Possibilities for extensions</h3>

Above, we have shown figures for second-order finite elements. Our
implementation gains more compared to sparse matrices if higher order elements
are used. However, FE_Q elements with equidistant nodes are badly conditioned
if the order increases. In this case, the smoother and the multigrid solver
break down. Node clustering close to the element boundaries resolves this
problem (and the multigrid solver converges in 5 or 6 iterations also for very
high order). Elements with this properties are the Gauss-Lobatto FE_Q
elements, which are presented in step-48.
