Subject: AFS (instead of NFS) and mbox

From: [email-masked]
Date: Sun, 26 Aug 2001 14:11:42 +0100

Hi,

I'm running into problems with many of my users - whose mail boxes are getting very large in 
size (100's of megabytes). I employ NFS to deal with home area - which is where mail lives
and as such I'm using trad. unix mail format for the mail boxes as everything I've read 
indicates that mbx format will 'kill' NFS.

I'm not really in a position to move away from the mail deliveries to home areas so was 
wondering if using a file systems such as AFS instead of NFS would ewnable me to use mbx
reliably?

Does anyone have any experience in this? I'm not sure at what point (file size) the unix 
format  becomes inefficient - is there a recommended max size and I could then try and 
get users to try and stick to these limits - I know they wont :-(

Anybody have any other suggestions on scaling this type of setup - considering that I'm 
not in position to move to one large mail spool somewhere - certainly not for a long time
and without plenty of planning. 

Thanks
Surinder

PS. I'm currently on uw-imap 2000-c - and will shortly be upgrading to the latest 2001 snapshot 
- maybe this might help somewhat?
-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Sebastian Hagedorn <[email-masked]>
Date: Sun, 26 Aug 2001 18:31:21 +0200

-- [email-masked] is rumored to have mumbled on Sonntag, 26. August 2001 
14:11 Uhr +0100 regarding AFS (instead of NFS) and mbox:

> I'm not really in a position to move away from the mail deliveries to
> home areas so was  wondering if using a file systems such as AFS instead
> of NFS would ewnable me to use mbx reliably?

Unfortunately not, AFAIK. I'm in the same position as you and don't know 
what to do about it either. Actually we do deliver to a large spool, but 
the spool file has to be in Unix format anyway if you want to stick with 
mail or mail.local for local delivery. For mbx you generally need a file 
named INBOX in the home directory that's in mbx format. But all the home 
directories are either NFS or AFS....

Cheers, Sebastian
--
Sebastian Hagedorn
Ehrenfeldgürtel 156, 50823 Köln, Germany
http://www.spinfo.uni-koeln.de/~hgd/
This mail was sent using Mac OS X

---

From: David Carter <[email-masked]>
Date: Tue, 28 Aug 2001 09:50:29 +0100 (BST)

On Sun, 26 Aug 2001 [email-masked] wrote:

> I'm running into problems with many of my users - whose mail boxes are
> getting very large in size (100's of megabytes).

I'm amazed that mailboxes this size work at all over NFS. Are you using
Gigabit ethernet as a backbone?

> I employ NFS to deal with home area - which is where mail lives and as
> such I'm using trad. unix mail format for the mail boxes as everything
> I've read indicates that mbx format will 'kill' NFS.

Yep. I suspect that AFS would have a hard time as well. MBX uses file
locking really hard, and this is the thing that tends to be most fragile
on distributed filesystems. ".lock" files are much more reliable on NFS.

> Anybody have any other suggestions on scaling this type of setup -
> considering that I'm not in position to move to one large mail spool
> somewhere - certainly not for a long time and without plenty of
> planning.

An brief aside, before MRC comes down on me like a ton of bricks:

I inherited a mail infrastructure based on NFS, and came to the conclusion
quite rapidly that this is not the correct way to do things. Modern NFS
appliances may be able to cope with the heavy workload, but its still a
desperately inefficient way to ship data compared to an IMAP proxy or
better still telling users to connect directly to the relevant server.

Aside other: We introduced index files on unix format mail folders about
18 months back, a local hack. Pine and the POP and IMAP daemons load a
short index file (typically a few Kbytes in size) rather than the entire
mailfolder. A separate file provides an ENVELOPE cache for folder listing
display and clients which like to thread messages. There's a 3 x write
penalty for certain update operations, but the same is true for the Cyrus
server. This seems to have made a huge difference for large mailfolders.

I wouldn't recommend this as a long term solution: we plan to move to a
cluster of systems without NFS (and probably using MBX) as the next
generation of our mail infrastructure. However if you are in bind, I would
be willing to donate my code as a stop-gap measure.

-- 
David Carter                             Email: [email-masked]
University Computing Service,            Phone: (01223) 334502
New Museums Site, Pembroke Street,       Fax:   (01223) 334679
Cambridge UK. CB2 3QG.

---

From: "Surinder S. Dio" <[email-masked]>
Date: Tue, 28 Aug 2001 11:49:58 +0100

Hi,

Firstly thanks for the reply - I got a number of other replies via
private email and it seems that there's quite a few people in a
similar situation.

[Apologies for all the details that follow - I thought it might help
in explaining my issues].

> Gigabit ethernet as a backbone?

All the hosts are currently switched 100Mb ethernet, I'm planning a
move to gigabit over the next year for all the servers at least.

It does work - and actually not too badly as it currently goes - it's
just the larger (and these are users who are sending and receiving
huge powerpoint *arrgh* type files - thus filling up mailboxes very
rapidly) mailboxes that are suffering.

Maybe I could just take a moment to explain my setup -I know that some
of this might make eyebrow's rise - the following is a result of
history, lack of funding and making the best of it to the
best of ones capability :-)

The MTA in use is exim and this is setup to deliver mail to users
homeareas to ~/.Mailbox. This is considered to be the users InBox,
mail folders live under ~/Mail directory. The homeareas are served via
NFS from a variety of servers - but most of them are currently sun
ultra 5's with multi disk packs hanging of them - a few are sparc 20
systems - all disks are ultrawide scsi. In total we're talking approx
500Gb of homeareas - distributed amongst NFS servers and multi disk
packs designated for staff, student and research use. This is
departmental use only and not for whole institution.

I will be migrating to A1000 raid on some of these and upgrading the
spec of the NFS servers as well (netra's maybe)- but on a piecemeal
basis as and when I can scrape resources together from various parties
- such is academia.

As well as receiving mail, these homeareas are receiving a battering
from normal use - esp on the researcher side - where we can have files
of many Gbytes being read/written to.

There is a dedicated imap (uw-imap) server - which I've configured to
pick up mail from the above specified locations. Additionally there is
a separate webmail (IMP) server. This has proved to be very popular
with staff and is now planned to roll out to students as well. These
will be dual SunBlade 600 type systems - with plenty of memory to
support IMP and the student imap server. There is a distinct
separation between staff (inc research)and student mail services.

Users are free to use an imap client (netscape, outlook,etc) or
webmail. Additionally there is a whole set of "power" email users,
like myself dedicated to the one true mailer - mutt. They will run
this on variety of desktops, servers etc and it will pick up their
mail via NFS/automount. 

It actually holds together pretty well - but the real problems are on
the imap server - it hits high loads, esp when large mail boxes
are being loaded. Also members of staff will use multiple mailers from
multiple locations - ie leave themselves logged into Outlool/Imap at
work and use Webmail from home - I've advised against this; though I
havent had any corruptions - I'm assuming that uw-imap is dealing with
all the necessary locking issues (as it exim trying to deliver to
their .Mailbox - when an imap write occurs).

> on distributed filesystems. ".lock" files are much more reliable on

Thanks - thats what I suspected but I wasnt sure.

> better still telling users to connect directly to the relevant

I'm not sure what an IMAP proxy is - is this a piece of
software/hardware sitting in front of the imap server and "caching"
on imap? I've never come across this before.

> server. This seems to have made a huge difference for large

This sounds quite interesting - anything that can help on the large
mailboxes would be most welcome. Is there any selective way to apply
this - ie if a mailbox reaches a certain size or is it a all or
nothing scenario *actually thinking about that - that could be tricky*.
 
> be willing to donate my code as a stop-gap measure.

I'd appreciate a look - if only to see if it I could buy myself
sometime.

Would you mind expanding on your proposed cluster system (if you can)
as I'm starting to look at alternatives. Is the cluster a dedicated
mail spool and all other homearea type functions dealt with in the
normal distributed filesystem way?

I would like to go for a few dedicated large servers (sun e220/440
etc) with hardware raid 5. These would hold the homeareas and be
configured with a large quota'd mail spool. Then imap/exim can run
locally on the servers - avoiding the NFS issues. However I do like
the way disk storage is currently distributed around several servers -
it helps with load issues and makes managing the disk space quite
easy. It also wont be painless to migrate away from the current setup.

However for those users that want to use mutt etc then this will mean
us having to ssh into these servers and running our mail client there,
this then opens up the whole server to users running other tasks - and
they will try and find compute resource wherever they can. Something
else that would need dealing with :-(

Ideally I guess what is required for my situation is a NFS compatible,
high performance mail format - and for it to be supported by uw-imap,
exim/postfix etc. It will be also love large mail folders :-)

Apologies for the length of this post. I'm fishing for ideas on a good
way forward for us.

Rgds
Surinder

---

From: David Carter <[email-masked]>
Date: Tue, 28 Aug 2001 12:30:22 +0100 (BST)

On Tue, 28 Aug 2001, Surinder S. Dio wrote:

> a separate webmail (IMP) server.

IMP has the particularly amusing habit of reconnecting to the IMAP server
several times for each HTTP request. I'm sure that this isn't helping you.

> software/hardware sitting in front of the imap server and "caching" on
> imap? I've never come across this before.

A piece of software that allows you to advertise an existing DNS service
name to users, but which transparently redirects the IMAP session to a
backend server (using IMAP referrals as an optimisation if the client
supports them). Something like: "http://vergenet.net/linux/perdition/".

Telling your users to connect directly to the correct IMAP server is a
better solution if you can arrange it. However you may have political
complications if your users are used to an existing scheme and it appears
to all "work just fine" from their perspective. There are a few advantages
in having a mail system that is clearly creaking at the seams :).

I understand that Washington use and recommend "username.servicename" DNS
mapping to hide the details. This seems an entirely sensible approach.

> I'd appreciate a look - if only to see if it I could buy myself sometime.
>

I'll have a separate chat with you about this rather than boring the rest
of the list with details.

---

From: Chris Newman <[email-masked]>
Date: Tue, 28 Aug 2001 14:23:48 -0700

AFS gets its scalability boost over NFS by having a cache optimized for a 
small working set of files.  As a result, the cache will actually result in 
a significant slowdown for email delivery since there's no small working 
set.  Expect to add more and more delivery servers which will put more and 
more load on your AFS servers (which have to break callbacks to the 
unusually large working set of files).  If you get too many users, the 
entire filesystem will collapse under the email load (yes, I've seen this 
happen, admittedly with AMS rather than mbx).

Putting email on distributed filesystems is a bad idea.  The data models 
between file access and email access are simply too different.  The system 
as a whole ends up sufficiently complex that diagnostics are difficult and 
you'll end up being on your own if a problem occurs.

Horizontal scalability is best achieved with a DNS hack (like UW does) or a 
good IMAP proxy (like CMU does).

		- Chris

--On Sunday, August 26, 2001 14:11 +0100 [email-masked] wrote:
> getting very large in  size (100's of megabytes). I employ NFS to deal
> with home area - which is where mail lives and as such I'm using trad.
> unix mail format for the mail boxes as everything I've read  indicates
> that mbx format will 'kill' NFS.
>
> Does anyone have any experience in this? I'm not sure at what point (file
> size) the unix  format  becomes inefficient - is there a recommended max
> size and I could then try and  get users to try and stick to these limits
> - I know they wont :-(
>
> somewhere - certainly not for a long time and without plenty of planning.
>
> Thanks
> Surinder
>
> PS. I'm currently on uw-imap 2000-c - and will shortly be upgrading to
> the latest 2001 snapshot  - maybe this might help somewhat?