Subject: ipop3d and $HOME/Mailbox

From: Matt Hamilton <[email-masked]>
Date: Tue, 31 Dec 1996 13:44:00 -0500

Dear All,
  I want to store users mail in their home dirs and now I want to use
ipop3d to retrieve mail from there.  Is this possible?  I don't want to
have to use qpopper as I don't like it very much and don't trust it very
much.  Plus I like the IMAP<->POP3 features of ipop3d.

-Matt

-------------------------------------------------------------------------------
  Matt Hamilton                                         Clintondale Aviation
  [email-masked]                            http://www.clintondale.com
-------------------------------------------------------------------------------

---

From: Mike Brudenell <[email-masked]>
Date: Fri, 3 Jan 1997 09:46:20 +0000 (GMT)

Yes, it is possible, although you may have to recompile the C-client
library to include an extra mailbox driver, then relink Pine and imapd
against the library.  If this doesn't fill you with fear read on...

There are certainly at least two routes open to you: one uses the
"standard" ("Berkeley") mailbox format to store the INBOX folder (this is 
called the "mbox" driver); whilst the other uses a different format called
Tenex (the "Tenex" driver).

Tenex is included by default in the C-client library, so if you decide to
use this one you won't need to recompile anything.  The mbox driver isn't
included as standard.  You can add it by setting the EXTRADRIVERS variable
in the "build" script (or is it in a Makefile? I forget!) to "mbox".  I
_believe_ you can also do it when you give the "build" command simply by
changing this to:

	build xxx EXTRADRIVERS=mbox

where "xxx" is your usual three character platform code.

So which to choose - mbox or Tenex?  (The following thoughts are purely
personal observations; they may be added to/reversed by Others Who Know
More Than I.)

If you are already using the Berkeley mailbox format for your folders and
are happy with its performance then you should probably use the mbox
driver as this will keep things simple for you.

If you are wanting a bit more "oomph" from your mail server you may like
to consider the Tenex format.  This is much more efficient in its memory
use because Tenex messages are a fixed size (whereas Berkeley ones can
change size as their status flags are added/altered).  This fixed size
allows the Tenex driver to build an in-memory index into where the
messages are in the folder and access them from there; the Berkeley driver
has to read the whole folder into memory and keep it there.

There are a couple of Gotchas with the Tenex driver though:

*  For locking to work properly your Tenex folder(s) should be on a disk
local to the machine running the IMAP server software (or your platform
should have reliable locking across NFS/whatever ... something Mark
Crispin really doubts exists (and so do I after some empirical
observation)).  This implies that your users' home directories must be
local to the IMAP server system.

*  You cannot save messages from a mail folder in one format to one in
another.  This means you cannot use a Tenex INBOX and save messages from
it to a Berkeley folder.  If you already have Berkeley folders you will
either need to convert them to Tenex as well, or put up with not being
able to add more messages to them from Tenex folders.  The recommendation
from the Pine Team is to pick one folder format and stick with it for all
your folders.

So how do you use either driver?

To use the mbox driver (having compiled and linked it in) you just need to
create an empty file called "mbox" in your home directory.  Pine and the
imapd will spot this and rip all the mail in your delivery file (eg,
/var/mail/username) and move it across to the mbox file.

Newly arriving mail continues to be delivered to the delivery file, and
gets moved across to the mbox file whenever Pine/imapd starts up, and
periodically thereafter (whenever the check for new mail is performed).

The Tenex driver is used similarly, only this time the empty file you
create in your home directory must be called "mail.txt".  It works in the
same way as the mbox driver.

[See below for a note about mail delivery and "tmail".]

Gotchas?

At one time we used the mbox driver but backed out from using it after
finding a couple of problems...

*  Users were prone to try editing this mysterious "mbox" file with an
editor or (worse) a word processor.  I had to do any amount of patching up
of these to edit out the header junk Word Perfect sprinkles at the start
of the file.  Until the file was fixed Pine/imapd refused to touch it
(rightly) because it was corrupt.  (Note that mail continued to be
delivered to the delivery file in /var/mail/username for the user;
Pine/imapd just couldn't transfer it to the mbox file for them to see it
in/read it from.)

*  If a user ran short of disk quota and had a large message in their
delivery file it couldn't get transferred across to the mbox file
(insufficient quota left).  This prevented the user from being able to
read/delete the big message and all following ones until they had deleted
enough files from their home directory to allow the big message to
transfer across to the mbox file.  (Needless to say this is a Real Fun Way
for students to mess each other up! :-(

What we did here...

After backing out from using the mbox driver to store files in people's
home directories (because of the above problems) we introduced disk quotas
on the delivery directory (/var/mail).  We give people a reasonable soft
disk quota (0.5Mb for students, 4Mb for staff) with a hard quota which is
double their soft one.

This prevented people from letting their INBOX files grow indefinitely
(some people seem remarkably loathe to delete messages from or save them
out of their INBOX!).  It also prevents accidental/deliberate mail loops
filling up the entire disk.

But then...

We were having real performance problems after the latest intake of
students.  We traced this to now having around 9,000+ INBOX/delivery files
stored in the /var/mail directory.

So we decided to split users up into groups (we decided to go for a
structure of /var/mail/NN/username, where NN is the last two digits of the
user's UID).

This *vastly* improved performance, and may be a possible solution to
other sites having such problems who have staff confident of hacking the
mailbox driver code to implement it (fairly straightforward changes, but
just pray that you'll be able to keep the patches working against new
releases of the Pine/imapd software!:-)

Basically we fool the code into checking this /var/mail/NN/username area
instead of the home directory when looking for the delivery file and/or
mail.txt files (we haven't bothered about the mbox file as we no longer
use these).  The advtange is that this area is local to the IMAP server
machine (hence avoids file locking problems over NFS), and is mounted
anywhere else (so users can't edit their mail files with Word Perfect!).

Whilst investigating all this we discovered the "tmail" program included
with the other IMAP utilities (including something to convert between
mailbox formats, eg from Berkeley to Tenex (or back!:-) ) 

This can be used as a pretty much drop-in replacement for /bin/mail, which
most sendmail's use to actually write an arriving message into the INBOX
folder's file.

An advantage of tmail is that it built on top of the C-client library, and
hence fully co-operates with Pine and imapd over mail file locking.

It also knows about the mbox (if lined in) and Tenex drivers.  This allows
it to spot the existence of an mbox or mail.txt file in the user's home
directory and deliver mail stright to it.  This bypasses the normal
delivery file, and hence the overhead of Pine/imapd having to move its
contents across to the mbox/mail.txt file.

Warning: I don't know how "supported" tmail is by the Pine Team.  It seems
to be developing (I have spotted a number of changes go by), but certainly
don't assume it is fully operational and supported without hearing this
from the Pine Team themselves.

I hope this helps a bit...

Cheers,

-- 
Mike Brudenell                                         <[email-masked]>
------------------------------------------------------------------------
The Computing Service, University of York, Heslington, York, YO1 5DD, UK
Tel: +44-1904-433811  FAX: +44-1904-433740  http://www.york.ac.uk/~pmb1/