=head1 NAME

expireover - Expire entries from the news overview database

=head1 SYNOPSIS

B<expireover> [B<-ekNpqs>] [B<-f> I<file>] [B<-w> I<offset>]
[B<-z> I<rmfile>] [B<-Z> I<lowmarkfile>]

=head1 DESCRIPTION

B<expireover> expires old entries from the news overview database.  It
reads in a list of newsgroups (by default from I<pathdb>/active, but a
different file can be specified with the B<-f> option) and then removes
from the overview database mentions of any articles that no longer exist
in the news spool.

If I<groupbaseexpiry> in F<inn.conf> is true, B<expireover> also removes
old articles from the news spool according to the expiration rules in
F<expire.ctl>.  Otherwise it only removes overview entries for articles
that have already been removed by some other process, and B<-e>, B<-k>,
B<-N>, B<-p>, B<-q>, B<-w>, and B<-z> are all ignored.

When I<expiretombstone> and I<groupbaseexpiry> are enabled in F<inn.conf>,
B<expireover> appends each cancelled token to F<expireover.tombstone.NEW>
in I<pathdb> under an exclusive POSIX lock, and atomically renames the file
into place on a clean run.  In delayrm mode (B<-z>), the rename is performed
by B<expirerm> after B<fastrm> succeeds.  The next B<expire> run consumes
this log to skip per-article storage existence checks.  See inn.conf(5) under
I<expiretombstone>.

When I<groupbaseexpiry> is set, the default behavior of B<expireover> is
to remove the article from the spool once it expires out of all of the
newsgroups to which it was crossposted.  The article is, however, removed
from the overview database of each newsgroup as soon as it expires out of
that individual newsgroup.  The effect is that an article crossposted to
several groups will be removed from the overview database from each group
one-by-one as its age passes the expiration threshold for that group as
set in F<expire.ctl>, and then when it expires out of the last newsgroup,
it will be deleted from the news spool.

Articles that are stored in self-expiring storage backends such as CNFS
are normally treated differently and not expired until they expire out of
the backend regardless of F<expire.ctl>.  See B<-N>, however.

By default, B<expireover> purges all overview information for newsgroups
that have been removed from the server; this behavior is suppressed if
B<-f> is given.

To speed up the existence check for each article, B<expireover> builds a
Bloom filter from the history database at startup.  This replaces, for the
C<hisv6> history method, per-article random I/O into the F<history> file
with a single sequential read, which is critical for large spools.  It is
also a win for per-article SQL requests with the C<hissqlite> history method.
The Bloom filter is a positive-only cache: if it reports an article probably
exists, the slow history lookup is skipped.  If it reports the article is not
found, B<expireover> falls back to a direct history lookup for correctness.
False positives (the Bloom filter incorrectly reporting an article exists) are
benign; the orphaned overview entry will be cleaned up on the next expiration
run.

The false positive rate and memory usage of the Bloom filter are
controlled by the I<expirebloomfp> setting in F<inn.conf>.  Setting it
to C<0> disables the Bloom filter.  The Bloom filter is also disabled
when the B<-s> flag is used.

=head1 OPTIONS

=over 4

=item B<-e>

Remove articles from the news spool and all overview databases as soon as
they expire out of any newsgroup to which they are posted, rather than
retain them until they expire out of all newsgroups.  B<-e> and B<-k>
cannot be used at the same time.  This flag is ignored if
I<groupbaseexpiry> is false.

=item B<-f> I<file>

Use I<file> as the newsgroup list instead of I<pathdb>/active.  I<file>
can be C<-> to indicate standard input.  Using this flag suppresses the
normal purge of all overview information from newsgroups that have been
removed from the server.

=item B<-k>

Retain all overview information for an article, as well as the article
itself, until it expires out of all newsgroups to which it was posted.
This can cause articles to stick around in a newsgroup for longer than the
F<expire.ctl> rules indicate, when they're crossposted.  B<-e> and B<-k>
cannot be used at the same time.  This flag is ignored if
I<groupbaseexpiry> is false.

=item B<-N>

Apply F<expire.ctl> rules to expire articles even from storage methods
that have self-expire functionality.  This may remove articles from
self-expiring storage methods before the articles "naturally" expire.
This flag is ignored if I<groupbaseexpiry> is false.

=item B<-p>

By default, B<expireover> bases decisions on whether to remove an article
on the arrival time on the server.  This means that articles may be kept a
little longer than if the decision were based on the article's posting
date.  If this option is given, expiration decisions are based on the
article posting date instead.  This flag is ignored if I<groupbaseexpiry>
is false.

=item B<-q>

B<expireover> normally prints statistics at the end of the expiration
process.  B<-q> suppresses this report.  This flag is ignored if
I<groupbaseexpiry> is false.

=item B<-s>

B<expireover> normally only checks the existence of articles in the news
spool if querying the storage method for that article to see if it still
exists is considered "inexpensive".  To always check the existence of all
articles regardless of how resource-intensive this may be, use the B<-s>
flag.  See storage.conf(5) for more information about this metric.

=item B<-w> I<offset>

"Warps" time so that B<expireover> thinks that it's running at some time
other than the current time.  This is occasionally useful to force groups
to be expired or not expired without changing F<expire.ctl> for the expire
run.  I<offset> should be a signed floating point number specifying the
number of days difference from the current time to use as "now".  This
flag is ignored if I<groupbaseexpiry> is false.

=item B<-z> I<rmfile>

Don't remove articles immediately but instead write the path to the
article or the token of the article to I<rmfile>, which is suitable input
for fastrm(1).  This can substantially speed up deletion of expired
articles for those storage methods where each article is a single file
(such as tradspool and timehash).  See the description of the I<delayrm>
keyword in news.daily(8) for more details.  This flag is ignored if
I<groupbaseexpiry> is false.

=item B<-Z> I<lowmarkfile>

Write the lowest article numbers for each newsgroup as it's expired to the
specified file.  This file is then suitable for C<ctlinnd lowmark>.  See
ctlinnd(8) for more information.

=back

=head1 EXAMPLES

Normally B<expireover> is invoked from news.daily(8), which handles such
things as processing the I<rmfile> and I<lowmarkfile> if necessary.
Sometimes it's convenient to manually expire a particular newsgroup,
however.  This can be done with a command like:

    echo example.test | expireover -f - -Z <pathtmp in inn.conf>/lowmark
    ctlinnd lowmark <pathtmp>/lowmark

This can be particularly useful if a lot of articles in a particular group
have expired but the overview information is still present, causing some
clients to see a lot of "this article may have been cancelled" messages
when they first enter the newsgroup.

=head1 HISTORY

Written by Rob Robertson <rob@violet.berkeley.edu> and Rich $alz
<rsalz@uunet.uu.net> (with help from Dave Lawrence <tale@uunet.uu.net>)
for InterNetNews.

=head1 SEE ALSO

active(5), ctlinnd(8), expire(8), expire.ctl(5), inn.conf(5),
news.daily(8).

=cut
