Postfix OpenLDAP LMDB Howto

-------------------------------------------------------------------------------

Introduction

Postfix uses databases of various kinds to store and look up information.
Postfix databases are specified as "type:name". OpenLDAP LMDB (called "LMDB"
from here on) implements the Postfix database type "lmdb". The name of a
Postfix LMDB database is the name of the database file without the ".lmdb"
suffix.

This document describes:

  * Building Postfix with LMDB support.

  * Configuring LMDB settings.

  * Using LMDB maps with non-Postfix programs.

  * Required minimum LMDB patchlevel.

  * Credits.

Building Postfix with LMDB support

Postfix normally does not enable LMDB support. To build Postfix with LMDB
support, use something like:

    % make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
        AUXLIBS="-L/usr/local/lib -llmdb"
    % make

Solaris may need this:

    % make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
        AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
    % make

The exact pathnames depend on how LMDB was installed.

When building Postfix fails with:

    undefined reference to `pthread_mutexattr_destroy'
    undefined reference to `pthread_mutexattr_init'
    undefined reference to `pthread_mutex_lock'

Add the "-lpthread" library to the "make makefiles" command.

    % make makefiles .... AUXLIBS="... -lpthread"

Configuring LMDB settings

Postfix provides one configuration parameter that controls LMDB database
behavior.

  * lmdb_map_size (default: 16777216). This setting specifies the initial LMDB
    database size limit in bytes. Each time a database becomes "full", its size
    limit is doubled. The maximum size is the largest signed integer value of
    "long".

Using LMDB maps with non-Postfix programs

Programs that use LMDB's built-in locking protocol will corrupt a Postfix LMDB
database or will read garbage.

Postfix does not use LMDB's built-in locking protocol, because that would
require world-writable lockfiles, and would violate Postfix security policy.
Instead, Postfix uses external locks based on fcntl(2) to prevent writers from
corrupting the database, and to prevent readers from receiving garbage.

See lmdb_table(5) for a detailed description of the locking protocol that all
programs must use when they access a Postfix LMDB database.

Required minimum LMDB patchlevel

Currently, Postfix requires LMDB 0.9.11 or later. The required minimum LMDB
patchlevel has evolved over time, as the result of Postfix deployment
experience:

  * LMDB 0.9.11 allows Postfix daemons to log an LMDB error message, instead of
    falling out of the sky without any notification.

  * LMDB 0.9.10 closes an information leak where LMDB was writing up to 4-kbyte
    chunks of uninitialized heap memory to the database. This would persist
    information that was not meant to be persisted, or share information that
    was not meant to be shared.

  * LMDB 0.9.9 allows Postfix to use external (fcntl()-based) locks, instead of
    having to use world-writable LMDB lock files, violating the Postfix
    security model in multiple ways.

  * LMDB 0.9.8 allows Postfix to recover from a "database full" error without
    having to close the database. This version adds support to update the
    database size limit on-the-fly. This is necessary because Postfix database
    sizes vary with mail server load.

  * LMDB 0.9.7 allows the postmap(1) and postalias(1) commands to use a bulk-
    mode transaction larger than the amount of physical memory. This is
    necessary because LMDB supports databases larger than physical memory.

Credits

  * Howard Chu contributed the initial Postfix dict_lmdb driver.

  * Wietse Venema wrote an abstraction layer (slmdb) that behaves more like
    Berkeley DB, NDBM, etc. This layer automatically retries an LMDB request
    when a database needs to be resized, or after a database was resized by a
    different process.

  * Howard and Wietse went through many iterations with changes to both LMDB
    and Postfix, with input from Viktor Dukhovni.

