Subject: c-client "unix" driver and AFS

From: Robert Banz <[email-masked]>
Date: Tue, 9 Feb 1999 17:24:57 -0500

I've done some preliminary work on getting this to work right, but I'm
wondering if anyone out there has done this "right" :)

If you're not too familiar with AFS, AFS uses last-closer-wins filesystem
semantics, that tend to cause trouble with the "unix" mailbox driver,
since it keeps the file "open for writing" even if, at the time, it's
not actually writing out the mail spool.

The problem becomes:

o User is using "pine" on a machine, has mail file open for writing.
o User receives mail (appended to mail spool on the mail delivery server)
o Pine never "sees" changes to file, because of last-closer-wins
  semantics.  (if the file is open for writing on that machine, it assumes
  that it's cached copy is correct and ignores all server callbacks to
  update it's cache, since it's changes will "win" when it closes the
  file) 
o In worst case, user exists pine, mailbox is saved out, erasing newly
  appended mail.

For obvious reasons, this is "not acceptable" :)  The work-around I've
been using is to "close and re-open" the mailbox at times before the
driver goes to write to the mailbox, and also in-between that operation,
call the AFS ioctl to invalidate the cached blocks of that file.  This
works, however, a more viable solution to the problem is to have the
unix driver only have the mailbox open for writing when it's actually
going to be writing to the mailbox.

Is there a preferred method to obtain this behaviour from the unix driver?

Robert Banz ([email-masked])
UMBC University Computing
(410) 455-3933  fax: (410) 455-1065

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 9 Feb 1999 16:30:50 -0800 (PST)

I strongly recommend *against* using any network filesystem such as NFS or
AFS.  This is what IMAP is supposed to accomplish.  It isn't just a matter of
dealing with poor or incomplete filesystem semantics; network filesystems also
do not scale well.  It should be noted that CMU, where AFS originally came
from, punted AFS based mail in favor of IMAP.

I suspect that your workaround is about the best that you can do.

I know that this isn't what you were looking for, but sometimes the correct
answer to "Doctor, it hurts when I do this!" is "So don't do it".

---

From: Mark Plaksin <[email-masked]>
Date: 10 Feb 1999 13:35:12 -0500

Mark Crispin <[email-masked]> writes:

> AFS.  This is what IMAP is supposed to accomplish.

Can you give more details about how IMAP accomplishes this?  I don't want
to argue about whether you're right or wrong, just to understand what
you're saying.

Are you saying that folks should just run one big machine that can handle
all of their mail and IMAP load?

If you have multiple machines serving one domain (e.g.,
cac.washington.edu), how does that work?  A meta-sendmail determines to
which machine to send mail destined for mrc@cac, etc?

Or...?

Thanks!

Oh, and for what it's worth, we've been running UW imapd with DCE/DFS for
over two years and haven't had any horrible problems.  It works great.  We
deliver mail into home directories in DFS.

-- 
Mark Plaksin                                http://www.arches.uga.edu/~happy/

---

From: Robert Banz <[email-masked]>
Date: Thu, 11 Feb 1999 13:03:28 -0500

> which machine to send mail destined for mrc@cac, etc?

I know we're probably at the risk of starting a great religious war here,
but I have to also agree with Mark here -- I *wish* sincerely that a
monolithic mail server with a non-distributed filesystem were the way to
go -- but with the insane rate in which system usage (in "space", "hits",
and required uptime) grows, many sites, such as mine, have found it
necessary to to further embrace the distributed services mentality -- we
have multiple fileservers holding user files (their home directories, with
their mailspools in their home volumes), with multiple relatively
low-powered mail delivery machines (making for a quite redundant mail
delivery environment -- I really "don't care" if one of those machines
goes down), and right now one server handling IMAP/POP access into the
home directories.  Our users choose to use MANY methods to access their
mail (mutt, elm, pine, netscape, eudora, *eech* BSD Mail, etc.), running a
variety of auto-filter mechanisms (procmail, etc) -- with these factors,
one monolithing way of dealing with all mail (imap) is bit of an insane
dream for us.  I think most sites out there are in the same situation as
we are.

Having c-client's unix driver not deal well with most popular network
filesystems, is well, not really acceptable...  Guess it's back to hacking
it up again, oh well.  I'll post my changes to the list if anyone is
interested...

> deliver mail into home directories in DFS.

DFS's file system semantics, are well, a little better than AFSs :)

-rob

---

From: Terry Gray <[email-masked]>
Date: Thu, 11 Feb 1999 10:16:44 -0800 (Pacific Standard Time)

At UW we support a community of 70,000 with an almost purely IMAP
infrastructure.  We have a small handful of folks who "are not with the
program" and there is one Unix machine for them to play in, but has worked
fine to treat them as exceptions.

We have been doing this for many years; the concept is proven.  It works
fine, you just have to decide that it is worth going there.  For us, there
is no doubt in my mind.

Our IMAP-centric email architecture is robust and scalable.  It is easy to
add capacity, and users are insulated from changes in the backroom.  The
key to this is the service location strategy.  We use a modified DNS
server that exports a virtual email hostname for every user, e.g.
gray.deskmail.washington.edu is the name of my email host, which then gets
mapped to the actual mail server with my data on it.  The incoming mail
forwarders consult the same database for final mail delivery.  By using
DNS in this way, we have backward compatibility with all known email
clients.  It works great...

-teg

---

From: Mark Crispin <[email-masked]>
Date: Thu, 11 Feb 1999 16:54:07 -0800 (Pacific Standard Time)

On Thu, 11 Feb 1999, Robert Banz wrote:
> Our users choose to use MANY methods to access their
> dream for us.

IMAP is a "monolith" only if you configure your system architecture as a
monolith.  If you insist upon having all mail at one place, you're going
to have problems no matter what you do.

Of the clients you mentioned above, only BSD Mail and perhaps Elm stand
out as requiring access to a mail file as opposed to using a protocol
specifically designed for mail (such as IMAP or POP).

> filesystems, is well, not really acceptable...

It is impossible to "deal well" with NFS.  Some formats are impossible to
use with NFS at all, since NFS fails to provide critical atomic filesystem
semantics.  With the unix format, it's possible to keep things from
breaking too badly, as long as you don't try to access the same mail file
from two or more places at a time (or from the same place multiple times
simultaneously).

Second hand reports state that AFS has additional problems; it is reported
that AFS deliberately discards updates.  If these reports are true, then
AFS can not be considered to be a suitable filesystem for mail files.

c-client goes to considerable effort to write lock files and to get the
disk copies updated via fsync() before removing the lock files.  Even NFS
does this right.  If AFS can't do this, then don't use AFS.

---

From: Robert Banz <[email-masked]>
Date: Sun, 14 Feb 1999 10:58:06 -0500

> does this right.  If AFS can't do this, then don't use AFS.

Yes, it does discard updates to files -- but the situations in which it
does are well known.  The "trick", I guess, to making it deal safely
with AFS is to only have the mail file open RW when you have it in
a locked state, so that no other processes will try to make an update to
the file, and keep it open RO all the other times.  I'm currently in the
process of fire-testing a c-client unix driver (from the imap-4.5 dist)
modified to have this behaviour, and currently feel comfortable enough
to read my mail with it :)

The problem is, AFS cache callbacks telling a client a file is updated
are "ignored" if that client has the file open in a RW state (to enforce
it's last-closer-wins semantics), so like you mentioned, AFS does
deliberatly ignore updates, but only in "known" situations that can be
avoided.  Luckily, I haven't seen any problems on the locking end of it,
since AFS supports flock() with acceptable reliability. ( I say
acceptable, since I haven't seen any problems with it, but just because
I haven't seen it doesn't prove reliablity, in an absolute sense )

-rob