.. _classtracker_tutorial:

============================================
Tutorial - Tracking Class Instances in SCons
============================================

This tutorial demonstrates the class tracking facility to profile and optimize a
non-trivial program. SCons_ is a next-generation build system with a quite
elaborate architecture and thus an interesting candidate for profiling attempts.

Before we begin, it should be identified what shall be tracked, i.e. what
classes we want to connect to and whose instances are to be sized and profiled.
In this tutorial, the effect of a patch_ is analyzed that tries to size-optimize the very
heart of SCons - the ``Node`` class. Naturally, we will connect to the ``Node`` base
class and its sub-classes. It makes sense to put the profiling data
in context and track additional classes that are believed to contribute
significantly to the total memory consumption.

Installing hooks into SCons
---------------------------

The first step is to find the proper spots for connecting to the classes that
shall be tracked, taking snapshots, and printing the gathered profile data.
SCons has a simple memory profiling tool that we will override. The SCons
MemStats class provides all we need::
  
    from pympler.classtracker import ClassTracker

    class MemStats(Stats):
        def __init__(self):
            Stats.__init__(self)
            classes = [SCons.Node.Node, SCons.Node.FS.Base, SCons.Node.FS.File,
                       SCons.Node.FS.Dir, SCons.Executor.Executor]
            self.tracker = ClassTracker()
            for c in classes:
                self.tracker.track_class(c)
        def do_append(self, label):
            self.tracker.create_snapshot(label)
        def do_print(self):
            stats = self.tracker.stats
            stats.print_summary()
            stats.dump_stats('pympler.stats')

When SCons starts, ``MemStats`` is instantiated and the `ClassTracker` is
connected to a number of classes. SCons has predefined spots where it invokes
its statistics facilities with ``do_append`` being called. This is where
snapshosts will be taken of all objects tracked so far.

Because of the large number of instances, only a summary is printed to the
console via ``stats.print_summary()`` and the profile data is dumped to a file
in case per-instance profile information is needed later.

Test run
--------

Time for a test. In the following examples, SCons builds a non-trivial program
with a fair number of nodes. Running SCons via ``scons --debug=memory`` will
print the gathered data to the console::
    
    scons: Reading SConscript files ...
    scons: done reading SConscript files.
    scons: Building targets ...
    scons: `.' is up to date.
    scons: done building targets.
    ---- SUMMARY ------------------------------------------------------------------
    before reading SConscript files:         active      4.17 MB      average   pct
      SCons.Executor.Executor                     7      7.53 KB      1.08 KB    0%
      SCons.Node.FS.Base                          1      9.30 KB      9.30 KB    0%
      SCons.Node.FS.Dir                           6     17.77 KB      2.96 KB    0%
      SCons.Node.FS.File                          1      2.91 KB      2.91 KB    0%
      SCons.Node.Node                             0      0     B      0     B    0%
    after reading SConscript files:          active     13.06 MB      average   pct
      [...]
    before building targets:                 active     13.41 MB      average   pct
      [...]
    after building targets:                  active     34.77 MB      average   pct
      SCons.Executor.Executor                  1311      3.57 MB      2.79 KB   10%
      SCons.Node.FS.Base                       1102      4.84 MB      4.50 KB   13%
      SCons.Node.FS.Dir                         108      5.67 MB     53.72 KB   16%
      SCons.Node.FS.File                       2302     10.45 MB      4.65 KB   30%
      SCons.Node.Node                             1     84.93 KB     84.93 KB    0%
    -------------------------------------------------------------------------------

Making sense of the data
------------------------

The console output may give a brief overview how much memory is allocated by
instances of the individual tracked classes. A more appealing and well arranged
representation of the data can be generated with the ``HtmlStats`` class. The
dump generated previously can be loaded and a set of HTML pages can be emitted::

    from pympler.classtracker_stats import HtmlStats

    stats = HtmlStats()
    stats.load_stats('pympler.stats')
    stats.create_html('pympler.html')

If ``matplotlib`` is installed, charts will be embedded in the HTML output:

.. image:: ../images/classtracker_timespace.png

At first sight it might seem suspicious that the tracked classes appear to be
the sole contributors to the total memory footprint of the application. Because
the tracked objects are sized recursively, referenced objects which are not
tracked themselves are added to the referrers account. Thus, a root object's
size will include the size of every leaf unless the leaf is also tracked by the
`ClassTracker`.

Optimization attempt
--------------------

After applying the patch_ by Jean Brouwers, SCons is rerun under the supervision
of the `ClassTracker`. The differences in the last snapshot show that the
changes indeed reduce the memory footprint of ``Node`` instances::

    $ scons --debug=memory
    scons: Reading SConscript files ...
    scons: done reading SConscript files.
    scons: Building targets ...
    scons: `.' is up to date.
    scons: done building targets.
    ---- SUMMARY ------------------------------------------------------------------
    [...]
    after building targets:                  active     32.41 MB      average   pct
      SCons.Executor.Executor                  1311      3.50 MB      2.73 KB   10%
      SCons.Node.FS.Base                       1102      4.29 MB      3.98 KB   13%
      SCons.Node.FS.Dir                         108      5.52 MB     52.30 KB   17%
      SCons.Node.FS.File                       2302      8.82 MB      3.92 KB   27%
      SCons.Node.Node                             1     84.32 KB     84.32 KB    0%
    -------------------------------------------------------------------------------

The total measured memory footprint dropped from 34.8MB to 32.4MB, ``File``
nodes' average size from 4.6KB to 3.9KB.

Summary
-------

This tutorial illustrated how applications can be profiled with the
`ClassTracker` facility. It has been shown how the memory impact of changes
can be quantified.

.. _SCons: http://www.scons.org
.. _patch: http://scons.tigris.org/issues/show_bug.cgi?id=2198
