WRITING A TEST
==============

This test framework is intended to test low-level Lintian checks
and behavior with very malformed packages.  It provides a mechanism
to manually construct pathological deb files and run lintian on
them.  It should only be used when this degree of low-level control
is necessary.  For other, more conventional cases, use the tests
framework.

A test in this framework is a directory containing (at least) a tags
file and a Makefile.  The directory may contain other files as needed
for that test.

The tags file
-------------

The expected output of Lintian when run on the package, including info
and experimental tags.  The Lintian output will be lexicographically
sorted before comparing it with tags.  This file may be empty if the
test case should produce no Lintian output.

The Makefile
------------

Makefile must be a GNU makefile which, when invoked via make (using
the default target), creates a <testname>.deb file in the current
directory.  <testname> must match the directory name.  This makefile
may assume that the current working directory when it is invoked is
the directory in which it's located (or a copy of it).  It can take
any action needed to generate the deb file, such as running dpkg-deb
or ar.

The Makefile does not have to clean up after itself.  It can assume
that the directory is deleted and recreated between test runs.

The packages are constructed using fakeroot, so the Makefile doesn't
need to worry about file ownership when creating the files as long
as all files included in the archive are newly created or chowned
before creating the deb.

Note that t/helper/bin/create-deb can handle most of the common deb
creation cases;  the Makefile can assume this script is in PATH.

The desc file
-------------

Please refer to t/tests/README for the full description of the desc
file.

Skipping tests
--------------

To mark a test to be skipped, create a file named "skip" in the test
directory containing the reason for why this test should be skipped.
This should be used for tests for work that hasn't been finished or
tests that must be temporarily skipped due to bugs in other packages.


TEST NAMING CONVENTIONS
=======================

Each test name should begin with the name of the part tested, e.g.

<checkname>-...
<unpackname>-...
lintian-...
lintian-info-...

Use generic- as a prefix for test cases that don't cover a specific
portion of Lintian but instead test Lintian's behavior on a useful
special case of package (such as a minimal package).


RUNNING THE TEST SUITE
======================

The complete test suite will be run with debian/rules runtests, but
this can take quite a lot of time.  Normally this is only necessary
after significant structural changes or before a release as a final
check.

To run a specific test case, run:

    debian/rules runtests onlyrun=<test-name>

It's often more useful to run every test that is relevant to a
particular tag.  To do that, run:

    debian/rules check-tag tag=<tag>

This will run all tests that list that tag in the tags file.


TEST WRITING TIPS
=================

Please keep each test case focused.  One of the problems that
developed with the old test suite is that each test was serving many
separate purposes and testing large swaths of Lintian, which made it
difficult to know what could be changed and what would destroy some
other useful test.  Test cases should only test a set of closely
related tags and new tests should be added for new issues that aren't
part of that closely-related set.

Test cases should be as Lintian-clean as possible except for the tags
that they're testing for.  The template is intended to help with this.
It generates a Lintian-clean basic package for you to start with.  You
should override only the minimal required to trigger your test, and
try to fix any unrelated problems.  Sometimes this won't be possible
and the only way to trigger a tag is to also trigger another tag, and
that's fine, but it shouldn't be the normal case.
