Subject: UW IMAPd cluster, traditional UNIX spool format, options?

From: Jim Lawson <[email-masked]>
Date: Mon, 29 Dec 2003 18:08:40 -0500

Hi c-client readers -

We're attempting to replace our existing mail server here with a cluster
of Linux nodes running sendmail/procmail/UW IMAPd.  The Linux nodes use
are attached to shared storage (SAN) and using Sistina's GFS to manage
the locking.

The locking appears to be working just fine - no mailboxes are getting
stepped on, no mail is getting lost.  However, we are running into a
problem where an old stale imapd is running, and new ones can't start
up.  (e.g. someone left their MUA running at the office, now they're at
home trying to check their mail with another MUA.)  The new IMAP client
is able to log in and check their mail, but they're stuck read-only -
they can't get read-write mode like we could when all the imapds were on
the same host.

In searching through the code, I can see that when imapd can't get a RW
lock on the mailbox, it looks for the other imapd's PID and sends the
"kiss-of-death" (SIGUSR2).  The other imapd is supposed to give up the
lock, then the first imapd can get it.  Of course, this doesn't work
when the imapds are on separate hosts.  :-)

So, we see that we have at least 2 options:

- Move to a different mailbox format, one that supports multiple access
(mbx).  Frankly, this scares us, since we have around 40,000 users -
this is going to be a complex enough conversion as it is.

- Alter unix.c and other files to send the kiss-of-death to the actual
host the imapd is running on.  (Anyone done this already?)

Does anyone have any other suggestions about these, or other options we
should consider?  We realize we could do some type of proxy-based or
DNS-based routing of users to specific mail hosts, but we'd like to
avoid that if possible...

-- 
Jim Lawson
University of Vermont

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 30 Dec 2003 01:10:08 -0800 (PST)

> are attached to shared storage (SAN) and using Sistina's GFS to manage

I have never seen any sort of network filesystem clustering work out well
for IMAP servers.  Nor have I seen network locking work out well.

mbx format is probably not your answer; it does shared write which is
almost certainly guaranteed to lose on a network filesystem.  Let's put it
this way; if your vendor says that it will work, make them put that claim
in writing before you do it.  You might get some free hardware out of them
before they throw in the towel.

In general, it works better to split the mail store across a set of
machines, each of which has direct access to its own piece of the mail
store.  At UW, we use the DNS to direct users to the right server;
mrc.deskmail.washington.edu is the IMAP server which has mrc's mail, the
machine may change as mrc gets moved about, but the name that mrc uses to
get to his mail is always the same.

Remember that IMAP servers are generally not CPU bound; they are I/O
bound.  Thus, fast access to the filesystem is more important than CPU
load balancing.  Our IMAP servers tend to have low load averages but very
high disk/network utilization.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.