Subject: [Imap-uw] imap and NFS

From: Roy Giles <[email-masked]>
Date: Tue, 29 Nov 2005 16:45:03 +1100

Hello,

Our mail system resides on a fairly old Sparc machine running Solaris 2.6
which is grinding to a halt; as a start in eventually moving off this machine,
we are thinking of moving our unix style mailbox format imap server :-

IMAP4rev1 2004.357

on to a Linux box with mail still being delivered by smtp to the Solaris box.

Could anyone with experience of running imap over NFS please comment on their
experiences pro and con, especially in the realm of locking.

Thanks

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: scott hollatz <[email-masked]>
Date: Tue, 29 Nov 2005 09:54:30 -0600 (CST)

> experiences pro and con, especially in the realm of locking.

Many years ago we experienced similar problems with NFS.

The use of NFS made sense under a simpler email services architecture but
begain to fail (i.e., poor IMAP/POP services) as load and user base grew;
adding to the problem was increasing the number of IMAP/POP systems and
using mbox files (we still do, but that's another issue to be resolved).

The solution was to scrap the architecture used for at leat 10 years,
and redesign without the need for any NFS anywhere email is involved.

I understand NFS 4 mitigates or eliminates the locking issues, but we're
not interested anymore...

-- 
scott hollatz                                        net [email-masked]
information technology systems and services          tel +1 218 726 8851
university of minnesota duluth mn usa                fax +1 218 726 7674
                                                                         --
                                            "gabba gabba hey" - the ramones

---

From: Mark Crispin <[email-masked]>
Date: Tue, 29 Nov 2005 08:24:11 -0800 (PST)

> and redesign without the need for any NFS anywhere email is involved.

I agree.

NFS-exported mail spools were a bad idea then, and are a worse idea now; 
it *never* worked well even though in the old days you could hold it 
together with duct tape.

> not interested anymore...

I doubt very much that NFS 4 fixes the problems, which are far deeper than 
locking issues.  You also have to have synchronization of inode and data 
state across all clients, which requires a token-passing network 
filesystem.

To my knowledge, TOPS-20 and VAX/VMS were the only operating systems which 
ever implemented a token-passing network filesystem.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Dan Pritts <[email-masked]>
Date: Tue, 29 Nov 2005 12:00:06 -0500

One thing you might do on the existing system that should improve
performance is switch to mbx-format mailboxes from standard mbox.

The downside is that anything other than UW IMAP, pine (or other c-client
applications maybe) will not be able to directly access the mailboxes.
If mailbox access is IMAP-only then this will be fine.  If needed,
you can convert them back to mbox.

the short summary is that you need to use mailutil to create the new
mailboxes and dmail to deliver into them.  Beware that if you do not
specify the correct flags to mailutil, it will default to creating a
plain old mbox format file.  

> experiences pro and con, especially in the realm of locking.

danno
--
dan pritts - systems administrator - internet2
734/352-4953 office        734/834-7224 mobile
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: "Chr. v. Stuckrad" <[email-masked]>
Date: Tue, 29 Nov 2005 18:19:39 +0100

>begain to fail (i.e., poor IMAP/POP services) as load and user base grew;
...

Well, same here, nearly.  The problem of 'locking' is not so
much depending on UW-IMAP as on which *combination* of tools
runs on which side of the NFS!

You have to make absoutely sure that every tool working on
an mbox file is using the same locking (or at least is
locking 'somehow' against every other tool).

As there are three typical 'locks'
- network-locking for NFS
- kernel locking for files
- dotlocks (which are th 'lockfiles' named '<box>.lock')
you'll habe to find out which way is the safe way.

We here have working (but will move away from soon ...)

A Solaris 9 (SunOS 5.9) Server with local spool files
which are exported via NFS to the Workstations, and
'Home-Servers' which export User-Homes to the Mailer.
This results in:
- UW-IMAP (and pop) for remote access on MTA-local-Spool
  and on Remote(NFS)-Home-Files of users.,
- The /var/mail/mailbox-files exported via NFS for local old programs
- 'qmail' for the MTA delivering via 'procmail'
  because 'qmail' ***NEVER*** works directly on NFS!!!
- Mail delivered via procmail directly over NFS into Homes.

'procmail' and 'UW-Imap' and 'pine' work absolutely correct together.
'mutt' and 'SUN's [dt]mail' seem to be secure too, others we did not
check completely as they mostly allow access via IMAP.

We seldom (but not 'never'!) see mailbox corruptions.
More often we see (network/NFS-)locks hanging 'forever'
which mostly means a dead machine holding a nfs-lock.
Even more often we see procmail-sorted deliveries into
'remote and hanging NFS-Homes', which then block all
available delivery-'slots' of qmail, which brings the
Mailer to a full stop(!).

The latter scenario causes us to move away from
the old system to 'only IMAP' via 'Cyrus' (cyrus
does allow 'folders with mail AND folders in them'
and thus is more Win*compatible) and away from
qmail because we need some newer MTA features.
(programmable retry times, authentication schemes, ...)

Stucki

-- 
Christoph von Stuckrad      * * |nickname |<[email-masked]>  \
Freie Universitaet Berlin   |/_*|'stucki' |Tel(days):+49 30 838-75 459|
Mathematik & Informatik EDV |\ *|if online|Tel(else):+49 30 77 39 6600|
Arnimallee 2-6/14195 Berlin * * |on IRCnet|Fax(alle):+49 30 838-75454/
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: David B Funk <[email-masked]>
Date: Tue, 29 Nov 2005 17:08:00 -0600 (CST)

> ever implemented a token-passing network filesystem.

DCE-DFS also has a token-passing network filesystem. It handles concurrent
reads/writers to a filesystem object and is implemented in a vender
independent fasion (there is even an OpenDCE project ;).

Dave

-- 
Dave Funk                                  University of Iowa
<dbfunk (at) engineering.uiowa.edu>        College of Engineering
319/335-5751   FAX: 319/384-0549           1256 Seamans Center
Sys_admin/Postmaster/cell_admin            Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Roy Giles <[email-masked]>
Date: Fri, 02 Dec 2005 15:14:51 +1100

> experiences pro and con, especially in the realm of locking.

Thanks to all who replied on this, it seems to have provoked the debate ...
that we probably had to have !!!

---

From: Erik Kangas <[email-masked]>
Date: Fri, 02 Dec 2005 08:28:03 -0500

Hello?

As a next step to the discussion of UW Imap on NFS, has anyone played
with UW IMAP on GFS (RedHat Global File System -
http://www.redhat.com/en_us/USA/home/solutions/gfs/) in an environment
where multiple IMAP machines are sharing the files.

>From what I have read, GFS should take care of all the locking and
synchronization across a large number of attached machines without any
problem. Specifically, the file system appears and behaves like a local
file system. However, GFS is relatively new and not specifically
supported by most SAN vendors.

If anyone has any experience setting up a SAN with GFS with load
balanced UW IMAP servers using a folder format that depends on the
NFS-averse file system operations (i.e. mbx) please let us all know what
you have seen.

Thank you,

-Erik Kangas
http://LuxSci.com

---

From: Mark Crispin <[email-masked]>
Date: Fri, 2 Dec 2005 10:09:34 -0800 (PST)

> supported by most SAN vendors.

I've heard many such claims in the past.  They've always been proven 
false.  It often is not easy to prove the claims false either; all too 
often it appears to be right, and only after user data has been corrupted 
is it recognized that the claims are false.

I also see little or no point to using a network filesystem for IMAP. 
IMAP, by its nature, is heavily I/O bound.  A network filesystem adds more 
I/O load as well as synchronization problems which do not exist with a 
direct filesystem.

Many people perceive a network filesystem as a means of obtaining 
reliability.  What is missed is that ultimately, there is a single point 
of failure for the network filesystem: the server for the network 
filesystem.  There is no particular reason to believe that a filesystem 
server is more reliable than a dedicated IMAP server.

Timesharing systems, with an unpredictable mix of applications and users, 
are intrinsically less reliable; and that's where file servers are useful. 
IMAP server systems are a different animal.

-- Mark --

http://panda.com/mrc

---

From: Jim Lawson <[email-masked]>
Date: Fri, 02 Dec 2005 13:34:04 -0500

We have been running UW-IMAP on top of GFS for almost 2 years now at 
UVM, running both mbox and mbx formatted mailboxes.  It is a 6-node 
cluster, with 3 lock managers running in failover (not distributed) 
mode.  The 6 mail nodes are attached to a switched SAN, and the storage 
is provided by an IBM DS4800.

This is not a recommended config, obviously (and MRC has reminded us of 
this :-)   However, when we still had a single-system running UW-IMAP, 
we decided to give the cluster a try, which would hopefully provide 
better scalability while still providing a single-system image - 
preventing us from having to touch 15,000+ desktop systems spread across 
the state.

To make this work, we had to make a few small changes to the UW-IMAP 
code base.  The INBOX directory (/var/spool/mail) and c-client lockfile 
location (/tmp) had to be moved to a GFS-shared filesystem, and into a 
directory hash.    Spool files were hashed by username (mine is 
/var/spool/mail/j/t/jtl), c-client lockfiles were hashed by the LSB of 
the inode number (so lockfile .fe06.7bcdf3 gets put in 
/gfs/tmp/f/3/.fe06.7bcdf3).  The hashing reduces directory lock 
contention, which can make GFS perform worse than a single system with 
local filesystems.

We also had to write a simple daemon to listen for the "kiss of death" 
used to break locks on older connected imapds.  Basically, the c-client 
lockfile also includes the node ID, and if c-client wants to KOD someone 
on another node, a UDP packet gets sent to that node, received on that 
node by a daemon, who sends the KOD locally.  The daemon is 
affectionately named "dkod".

We're still running it, so we've had some level of success, but it is 
not problem-free.  Occasionally, we see imapds that have deadlocked, for 
reasons that we can't explain.  The problem exhibits itself to the user 
as a mailbox that can't be opened read-write.  However, we've *never* 
seen corruption of mailboxes under GFS (and we DID see that occasionally 
on the old single-system.)

In the beginning, there were bugs in GFS that we tickled with our 
particular config, but Sistina (now RedHat) has fixed the problems that 
we've run into.  A good piece of advice is: don't create very large 
(1TB) filesystems, because they can take days to fsck.  We haven't had 
to fsck in over a year, but in the first months of running it when we 
had some bad hardware and a filesystem with minor corruption, we had to 
take some filesystems down for hours while it was fsck'd.  Sometimes we 
were better off moving all the data we could off the filesystem and 
creating a new filesystem in its place.  Again, that hasn't happened in 
over a year.

Overall, we're mostly happy with it at this point, although it would 
have been nice to avoid the first few months of pain that we had due to 
bugs in the filesystem.

We'd be happy to share code if anyone's interested.

Jim