Subject: Re: History DB Performance

From: James Brister <[email-masked]>
Date: 20 Nov 1996 20:03:33 +0100

[email-masked] (Matthew Dwyer) writes:

> As with 1.4, We are still having trouble with the history database once it
> exceeds about 35megs. Our responses to incoming articles become too large to
> accept a full feed.

What do you mean by 'too large'?

> There seems to be little information on the relationship between the size of 
> the history DB and performance.

The performance gets worse as the size gets bigger (obviously). It also
gets worse if the database is not rebuilt nightly (which it is by default
with expire). There's a bug in 'makehistory' when using the '-i' and '-s'
flags, which results in poor performance when your history file is over
16MB (untested patch below). Using '-i' to expire will also result in poor
performance (but that's not a bug).

A good indicator of your performance characteristics would be how much
smaller is the number generated by this,

   head -1 /var/news/etc/history.dir | perl -ane 'print 2 ** $F[7], "\n";'

than the size of your history text file. If it's bigger you're OK. if it's
smaller, then lookups for the message ids at the tail of the history text
file (past the byte indexed by the number just generated) will be much
slower than for those at the front.
 
> One idea we are considering is to keep the "live" database small during the 
> day and appending it to the large database at expire time.. then producing a 
> new "live" database for the next day.

What exactly is the 'live' database, and what's the 'large' database.

> We just don't need to check every 
> incoming article against 14 days worth. We only need about 1 day of history 
> for incoming news and the rest for expiration.

Actually, you have that backwards. You need 14 days history so that when
someone upstream starts re-feeding you week old articles, you can reject
them properly.

James
-- 
James Brister                                            [email-masked]
Internet Software Consortium       http://www.isc.org    [email-masked] 
---------------------------------------------------------------------------
*** makehistory.c	1996/11/07 00:50:51	1.20
--- makehistory.c	1996/11/13 22:15:01
***************
*** 125,131 ****
      /* Open the new database, using the old file if desired and possible. */
      (void)dbzincore(1);
      if (IgnoreOld) {
! 	if (dbzfresh(p, dbzsize(size), HIS_FIELDSEP, 'C', dbztagmask(size)) < 0) {
  	    (void)fprintf(stderr, "Can't do dbzfresh, %s\n",
  		    strerror(errno));
  	    if (temp[0])
--- 125,132 ----
      /* Open the new database, using the old file if desired and possible. */
      (void)dbzincore(1);
      if (IgnoreOld) {
!         /* 70 == average history file line length */
! 	if (dbzfresh(p, dbzsize(size), HIS_FIELDSEP, 'C', dbztagmask(size*70)) < 0) {
  	    (void)fprintf(stderr, "Can't do dbzfresh, %s\n",
  		    strerror(errno));
  	    if (temp[0])