Author: Mehdi Dogguy and Stéphane Glondu

# Ben’s reference manual

Mehdi Dogguy and Stéphane Glondu
version 0.9.1

Manual’s licence

This manual is free software; you may redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version.

This is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.

A copy of the GNU General Public License is available as /usr/share/common-licenses/GPL-2 in the Debian GNU/Linux distribution or on the World Wide Web at the GNU web site. You can also obtain it by writing to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.

| This document is still work in progress!

## Introduction

Ben is the name of a set of utilities written in OCaml and available through the `ben` command. The major feature of Ben is to provide a full featured transition tracker to follow the evolution of a set of packages in the Debian archive.

The aim of this document is to describe the features of Ben and its configuration.

## 1. Getting the source code

Ben is maintained in a Git repository available online at https://salsa.debian.org/debian/ben. Getting the sources is as simple as:

    git clone https://salsa.debian.org/debian/ben

The source tree has a `debian/` directory. It is a native Debian package, so you can use the standard building tools in order to build it.

## 2. Query language

The main feature of Ben is a transition tracker. The transition tracker selects a set of affected packages and computes a state for each of them. There exist already various ways to do that. One solution would be to use the dctrl-tools. We chose to make our own language. This allows us to perform more fine-grained queries, optimize queries and extend the query language easily.

A ben query is described by the following BNF formulae:

    <query> ::= true
              | false
              | ! <query>            /* Boolean negation     */
              | <query> '|' <query>  /* Boolean OR operator  */
              | <query> & <query>    /* Boolean AND operator */
              | ( <query> )
    /*set 2*/ | <field> ~ /<regex>/
              | <field> ~ "<list>"
              | source
              | <comparison> "<string>"
              | <field> ~ "<string>" <comparison> "<string>"

    <field> ::= .<string>

    <list>  ::= <string>
              | <string> '|' <list>

    <comparison> ::= '<<' | '<' | '>' | '>>' | '=' | '<=' | '>='

The first rules are the usual boolean constants and operators. The sixth rule is useful to group queries and override boolean operators' priotity.

The second set are the various type of queries implemented to match against different criteria. It matches against a package field (in lowercase).

We believe the syntax of queries is easy to grasp. Nonetheless, here are some exemples:

- `.build-depends ~ /lib.*ocaml-dev/`
  This query matches all packages that build-depend on a package whose name matches the Perl regular expression `lib.*ocaml-dev`.

- `.depends ~ "libapt-pkg4.12"`
  This one is simpler than the first one in the sense that `libapt-pkg4.12` is considered as a plain string and not a regular expression.

| Note the symbols around the selectors: /…​/ for regular expressions and "…​" for plain strings.

| Packages fields may contain a list of values comma-separated. Ben splits the list before looking with "…​" for a match.

- `>= "3.0"`
  The query `>= "3.0"` matches all packages with a version number equal to or higher than "3.0".

- `source`
  This matches all source packages.

Now that the concept of query is explained, we will focus on other parts of Ben. We will see how we can use the queries for each frontend.

## 3. Frontends

Ben is a set of utilities available through the `ben` command. Each utility is called a frontend and there are four:

- `download`

- `query`

- `monitor`

- `tracker`

- `rebuild`

All frontends share a set of options and each one defines its own specific set. The shared set of command-line options is:

Option | Action
--- | ---
--no-benrc | Do not read .benrc file at startup
--dry-run|-n | Dry run
--parallel|-P | Set parallelism level
--quiet|-q | Quiet mode
--verbose | Verbose mode
--mirror uri | Package mirror to use
--mirror-binaries uri | Package mirror to use for binaries
--mirror-sources uri | Package mirror to use for sources
--areas a,…​ | Areas to consider (comma separated)
--archs a,…​ | Architectures to consider (comma separated)
--suite a | Suite
--cache-dir d | Path to cache dir
--cache|-C f | Specify the name of the cache file
--use-cache | Use cache whenever possible
--config|-c c | Config file
--more-binary-keys l | Comma separated list of further relevant binary keys
--more-source-keys l | Comma separated list of further relevant source keys
--preferred-compression-format|-z format | Preferred compression format (Default: Gzip)
--version|-V | Display version number (and build date) and exit.

| The order of command-line flags is significant. More specifically, --config specifies a configuration file which may set other configuration keys (e.g.: areas, archs, suite, etc…​).

Ben knows about the following compression formats: Gzip, Bz2, Xz and Plain (for no compression).

By default, Ben keeps the following defined subset of field names. In fact, there are two subsets. The first one is for binary packages (Package, Source, Version, Maintainer, Architecture, Provides, Depends, Pre-Depends, Conflicts, Breaks) and the second one for source packages (Package, Source, Version, Maintainer, Binary, Build-depends, Build-depends-indep, Build-depends-arch). If your analysis requires more fields, you may use --more-binary-keys and --more-source-keys command-line flags to specify your needs.

The rest of this section describes each frontend.

### 3.1. download

This frontend is the simplest one as it has no command-line options, except the shared ones. It downloads all Sources.gz files and all Packages.gz files for selected architectures and areas. You may invoke it by running:

    $ ben download [options]

The requested files will be downloaded in $BEN_CACHE_DIR, if set, or in the current directory.

| `ben download` doesn’t read `ben.cache` files yet.

### 3.2. query

This utility is pretty much like `grep-dctrl(1)`. Given a list of Packages or Sources files, it performs a query and outputs the result.

Using it is as simple as shown below:

    $ ben query ".package ~ /gentoo/" Packages

Other valid uses:

    $ zcat Packages.gz | ben query ".package ~ /gentoo/" -
    [...]
    $ ben query ".package ~ /gentoo/" Packages_foo.gz Packages_bar.bz2 Packages_baz.xz
    [...]
    $ ben query ".package ~ /gentoo/" monitor.cache
    [...]

Just like `grep-dctrl(1)`, `ben query` has a simple mechanism to filter its output. Using the command-line option `-s`, one can specify a comma-separated list of fields that will be shown for matching paragraphs.

    $ ben query ".package ~ /gentoo/" -s Package,Version ...

| ben query considers files matching `[sS]ources.*` to be Sources files, to be able to make the distinction with Packages files. This is useful especially when you use the `source` predicate.

While other frontends can take advantage from multi-core architectures, ben query is still sequential. This may be fixed in future versions.

### 3.3. monitor

The `monitor` fontend builds a monitoring page for a transition. A transition is described by three queries:

- `is_affected`: matches source packages that are part of this transition; this query is evaluated against all source and binary packages and, for binary packages, their source package is picked;

- `is_good`: matches binary packages that are considered to be ready (fixed) for this transition;

- `is_bad`: matches binary packages that are considered to be broken (not fixed) for this transition.

Note that some packages can be neither good nor bad. For example, there are many packages that build-depend on some library but do not link against it. Thus, they do not need any dependency on the library at runtime.

Following previously defined criteria, packages will be divided into 4 categories:

- Good (✔) packages, matching `is_good` criterion

- Bad (✘) packages, matching `is_bad` criterion

- Partial (⁈) packages, matching both `is_good` and `is_bad` criteria

- Unknown state, mathing neither `is_good` nor `is_bad`

An example of a complete description of a transition is:

    is_affected = .build-depends ~ /libicu-dev/;
    is_good = .depends ~ /libicu44/;
    is_bad = .depends ~ /libicu42/;

There are other optional fields:

- `title`
  This puts a nice title in the HTML page.

- `notes`
  This can be used to put remarks about the transition or notes about the status of some packages or the whole transition.
  + Ben recognizes some special texts and turns them into links. The following formats are recognized:

Original text | Result
--- | ---
pts:ocaml | pts:ocaml
buildd:ocaml | buildd:ocaml
#123456 | #123456
<msg-id> | https://lists.debian.org/msg-id
http://example.com | http://example.com

`monitor` understands the following list of command-line options:

Option | Action
--- | ---
--run-debcheck | Run dose-debcheck and add virtual `.uninstallable` field
--use-projectb | Get package lists from Projectb database
--output|-o file | Select output file
--output-format|-f fmt | Select output format
--stdin | Use stdin to read the input file
--template template | Select an HTML template

Output can be formatted in simple text (`text`), colored text (`color`), Xhtml (`xhtml`), Json (`json`) or dependency levels (`levels`). `html` is an alias for `xhtml`.

### 3.4. tracker

This frontend uses `monitor` to generate a summary page about all known transitions. An example of such summary can be found at https://release.debian.org/transitions/.

`tracker` has a notion of `profiles` that gives a hint on the actual state of the transition. The default profiles are:

- `planned` for known planned transitions or requested but not processed or acknowledged yet;

- `permanent` is a special profile made to keep an eye on some set of packages, not necessarily a transition;

- `ongoing` for the actual list of (known) on-going transitions;

- `finished` has a list of some transitions that are almost finished (e.g. new binary packages migrated but old binary packages left in testing to not decrease installability, or a finished transition with a number of still broken packages in testing).

Technically, each `profile` is a sub-directory of the global config directory[1]. The latter can be specified by a command-line option.

By default, `tracker` looks for a directory named `config` which should contain a file named `global.conf`, unless otherwise told. It reads data (`.ben` files) from the `config` directory and outputs the result in the `base` directory.

`tracker` has the following command-line options:

Option | Action
--- | ---
--base|-b [dir] | Specifies the "base" directory
--config-dir|-cd [dir] | Location of ben trackers
--global-conf|-g | Specify tracker’s global configuration file
--transition|-t [profile/transition] | Generate only that tracker page
--update|-u | Updates cache files
--use-projectb | Get package lists from Projectb database
--template template | Select an HTML template
--no-clean | Leave unknown generated HTML files

#### 3.4.1. HTML Templates

Ben has a simple templating mechanism to generate customized HTML pages. Templates are loaded dynamically when `monitor` or `tracker` frontends are used. When none specified, Ben chooses to load the Debian template.

You can write your own HTML template for Ben. All you have to do is to install the package `libben-ocaml-dev` which provides the Ben library. We recommend users to start from an existing template to ease the task. You can compile a template (e.g. named `foo`) by running the following commands:

    $ ocamlbuild -pkg ben foo.cmx
    $ ocamlopt -shared -o foo.cmxs _build/foo.cmx

| .cmxs files are native dynamically loadable shared modules. If you’re building on a bytecode-only architecture, you should build a `foo.cma` instead.

If your template needs a CSS (Cascading Style Sheets) or some images, you should install them under `/usr/share/ben/media`.

### 3.5. rebuild

This frontend recompiles a set of packages matching a query, respecting the order of build-dependencies inside the set. It takes as command-line argument a configuration file with the following keys:

- `is_affected`: the query used to select packages;

- `rebuild_command`: the command used to rebuild a given package.

It outputs in Makefile syntax the dependency graph of packages that could not be built.

### 3.6. migrate

This frontend simulates a migration of some packages from unstable to testing. It is useful to investigate Britney’s behaviour. It outputs information related to the migration of the given packages, the resulting Sources and Packages_* files, and an easy hint.

Note that the generated hint is not guaranteed to succeed. Moreover, this command does not take into account ages, RC bugs, tests…​ as Britney does.

A typical workflow is:

    $ cd /an/empty/working/dir
    $ source /usr/share/doc/ben/examples/migrate/functions.sh
    $ update
    $ migrate package1 package2 package3
    $ debcheck

If the `debcheck` function above does not report significant differences, there are good chances that the generated hint will succeed. However, in this situation, Britney usually manages to migrate the packages on its own.

## 4. Reporting issues

Please report bugs against Ben through the Debian BTS. Reported bugs can be seen on https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=ben

1. https://release.debian.org/transitions/config/