Subject: re: Berkeley v Tenex mailbox formats for the INBOX (fwd)

From: Mark Crispin <[email-masked]>
Date: Thu, 9 Mar 1995 18:47:39 -0800 (PST)

Hello Mike -

I think that your test was fundamentally flawed, because you measured resident
memory instead of virtual memory.  The amount of resident memory is more an
artifact of how much the scheduler loves you than of any direct performance
measurement.

Bezerk/mbox format keeps every message in memory, in separate malloc() buffers
for each message.  In many implementations of malloc(), the size of a malloc()
buffer is a power of 2, thus you have to round up each message size to the
nearest power of two and sum that to get the total virtual memory consumed.
This is needed, because expunge and checkpoint operations require a copy of
the mailbox.  Many times, user's don't have the disk quota to do a disk copy,
and a disk copy is very slow anyway because the mailbox's inode needs to be
preserved (thus a disk copy would triple the amount of disk traffic!).

Tenex format uses a size count and reserved space for flags.  This permits
updates to be done in place (effectively, no checkpoints) and the source can
be used as a target for expunge (since you're guaranteed to be blatting data
down).

As a result, Tenex format does not keep message in memory.  It may read in a
single message for processing, but it'll never keep more than one in memory at
a time.

In general, Tenex format will be less memory hungry and faster than bezerk
format.  There is one exception; free text searches will be slower in tenex
format, since it actually has to read messages from the disk.  In bezerk
format, it'll do searches from memory, so that is faster.  Envelope/MIME
parses are potentially faster in bezerk format because of all the data being
in memory as well.  This is rarely much of an issue in actuality though.

---

From: Mike Brudenell <[email-masked]>
Date: Fri, 10 Mar 1995 08:51:15 +0000 (GMT)

Hi, Mark -

> artifact of how much the scheduler loves you than of any direct performance

That's what I had initially thought.  However I tried both Berkeley and 
Tenex tests 3 times each, and each time got *exactly* the same SZ (total 
virtual memory usage) and RSS (resident set size) for each type of 
instantiation.  As this seemed so constant I started wondering if the 
test might be valid.

Interestingly, if you run with what still may prove to be a flawed test 
for a while, the SZ value for Berkeley format was 782 pages (1 page = 
4Kb), and 820 pages for Tenex.  This is an even bigger difference than 
the 299 v 327 RSS values! :-}

> preserved (thus a disk copy would triple the amount of disk traffic!).

This is very useful information too.  The implication is that the point I 
was measuring memory usage at is not the maximum, certainly for Berkeley 
(*WHY* can I never spell that right first time!!) format.  Hence apparent 
initial differences (if my test was valid, which I'm now doubting) are 
probably misleading for general use of the program.

> in memory as well.  This is rarely much of an issue in actuality though.

On the whole I am being very tempted to go with the Tenex format for the 
INBOX.  I'm not overly worried about the free text searching, etc, which 
are used relatively infrequently.

I'm glad you said it is generally faster... I had thought there was a 
slight speed increase and wondered if I was hallucinating it.

Mike Brudenell                                               <[email-masked]>
------------------------------------------------------------------------------
The Computing Service, University of York, Heslington, York, YO1 5DD, UK
Phone: +44-(0)1904-433811  FAX: +44-(0)1904-432767

---

From: Mark Crispin <[email-masked]>
Date: Fri, 10 Mar 1995 14:10:01 -0800 (PST)

Hi Mike -

I did a similar comparison test between two identical 89 message mailboxes,
differing only in format.  The tenex version is 616,679 bytes, the bezerk
version is 619968 bytes (the difference being in the internal data headers).

On a DECSYSTEM-5000 running Ultrix, the tenex version was 1772 SZ, 400 RSS;
whereas the bezerk version was 2608 SZ, 1260 RSS.

It is theoretically possible that the malloc() on SGI may be doing something
bizarre, such as free() being a no-op... ;-)

-- Mark --