WiredTiger: Statistics

| Version 3.1.0

Fork me on GitHub Join my user group
Statistics

WiredTiger can be configured to maintain a variety of run-time and data-source statistics. As maintaining statistics may involve updating shared-memory data structures and traversing disk-based data structures, configuring statistics may decrease application performance.

The wiredtiger_open `statistics` configuration must be set in order for statistics to be maintained. When configured to `none` (the default), no statistics are maintained and attempting to read the statistics will result in an error. Alternatively, the `fast` configuration maintains a subset of the statistics which are relatively inexpensive, and the `all` configuration maintains all statistics regardless of cost.

The following example configures WiredTiger to maintain all statistics, regardless of cost:

error_check(wiredtiger_open(
home, NULL, "create,statistics=(all)", &conn));

Statistics are gathered and returned to the application using a statistics cursor, which returns key/value pairs to the application. See Statistics Data for information about accessing the gathered statistics.

When the WT_SESSION::open_cursor method is called to open a statistics cursor, the statistics configuration specifies the statistics to be gathered, similarly to the wiredtiger_open function. The `fast` configuration gathers the subset of the statistics that are relatively inexpensive, and the `all` configuration gathers all statistics, regardless of cost. If no configuration is specified, the current database statistics configuration is assumed.

The configuration of the WT_SESSION::open_cursor method must agree with the database configuration: if the database is configured to maintain `fast` statistics, attempts to open a statistics cursor in `all` mode will fail. If the database has been configured to maintain `all` statistics, the statistics cursor can be configured in either `fast` or `all` modes. For example, an application might configure the database to maintain all available statistics, but the application might gather expensive statistics less frequently than inexpensive ones.

The following example opens a statistics cursor on the database:

error_check(session->open_cursor(
session, "statistics:", NULL, NULL, &cursor));

The following example opens a statistics cursor on a table:

error_check(session->open_cursor(
session, "statistics:table:mytable", NULL, NULL, &cursor));

The following example opens a statistics cursor on a table, but gathering only the relatively inexpensive statistics:

error_check(session->open_cursor(session,
"statistics:table:mytable", NULL, "statistics=(fast)", &cursor));

The WT_SESSION::open_cursor method and the wiredtiger_open function also support the statistics configuration value `clear`.

When `clear` is specified to the WT_SESSION::open_cursor method, gathered statistics will be reset, where appropriate, after they are gathered. For example, a cache size statistic is not cleared because it's not expected to change rapidly over time, while the count of cursor insert operations will be cleared. This allows applications to easily monitor changes in the system over time.

The following example gathers all statistics for a table, regardless of cost, and then clears them:

error_check(session->open_cursor(session,
"statistics:table:mytable",
NULL, "statistics=(all,clear)", &cursor));

When `clear` is specified to the wiredtiger_open function, gathered statistics will be reset, where appropriate, after they are gathered. This applies to all statistics cursors (as if `clear` was configured when the cursor was opened), as well as statistics logging, when it is configured.

The following example configures WiredTiger to maintain only relatively inexpensive statistics, and to clears statistics after they are gathered or logged:

error_check(session->open_cursor(session,
"statistics:", NULL, "statistics=(fast,clear)", &cursor));

The following example opens a statistics cursor on an open join cursor:

error_check(session->open_cursor(session,
"statistics:join",
join_cursor, NULL, &stat_cursor));

The statistics gathered will be organized by reference cursors participating in the join (see WT_SESSION::join); the uri of each reference cursor appears as a prefix in the description field returned as a value by the statistics cursor.

# Statistics logging

WiredTiger will optionally log database statistics into files when the the wiredtiger_open `statistics_log` configuration is set.

The log files are named `WiredTiger`.%d.%H, where `%d` is replaced with the day of the month as a decimal number (01-31), and `%H` is replaced by the hour (24-hour clock) as a decimal number (00-23). Each log file contains the statistics for the hour specified in its name.

The location of the log files may be changed with the `statistics_log.path` configuration string.

The following example logs statistics every 30 seconds:

error_check(wiredtiger_open(
home, NULL, "create,statistics_log=(wait=30)", &conn));

Each record is formatted as a space-separated timestamp, unsigned 64-bit value and a variable length string which describes the statistic.

The timestamp format may be changed with the `statistics_log.timestamp` configuration string. The `timestamp` value may contain ISO C90 standard strftime conversion specifications.

The statistics the database is configured to maintain are logged, that is, configuring the database to maintain `fast` or `all` statistics will modify the statistics that will be logged.

Statistics for specific underlying data sources may be included by adding a list of data source URIs to the `statistics_log` configuration string:

error_check(wiredtiger_open(home, NULL,
"create, statistics_log=("
"sources=(\"table:table1\",\"table:table2\"), wait=5)", &conn));

Statistics for all underlying data sources of a particular type may be included by adding a partial data source URI to the `statistics_log` configuration string:

error_check(wiredtiger_open(home, NULL,
"create, statistics_log=(sources=(\"index:\"), wait=5)", &conn));

When database statistics are logged, the database home will be the first space-separated entry for each record in the log file. For example:

Mar 08 11:38:23 463 /database/home pthread mutex condition wait calls
Mar 08 11:38:23 0 /database/home files currently open
Mar 08 11:38:23 1855437 /database/home total heap memory allocations
Mar 08 11:38:23 1856622 /database/home total heap memory frees
Mar 08 11:38:23 1 /database/home total heap memory re-allocations
Mar 08 11:38:23 472 /database/home total read I/Os

When data source statistics are logged, the data source's URI will be the first space-separated entry for each record in the log file. For example:

Mar 20 10:42:36 21 table:mytable compressed pages written
Mar 20 10:42:36 0 table:mytable page written failed to compress
Mar 20 10:42:36 5 table:mytable page written was too small to compress
Mar 20 10:42:36 586 table:mytable cursor insert calls
Mar 20 10:42:36 0 table:mytable bulk-loaded cursor-insert calls

No statistics are logged for any data source for which a handle is not currently open in the database, nor will any statistics requiring the traversal of a tree (as if the `statistics_fast` configuration string were set).

A Python script that parses the default logging output and uses the gnuplot, utility to generate Portable Network Graphics (PNG) format graphs is included in the WiredTiger distribution in the file `tools/statlog.py`.