Subject: Spreading IMAP loading on one mailhub...

From: [email-masked] (Adam Bentley)
Date: 27 Oct 1995 12:36:39 +0100

Hi,
	I've got a Sun Sparc 20 (160 MB ram, Sol 2.4) which is acting as a 
	primary mailhub. We provide mail access to the whole site via IMAP 
	and run sendmail 8.6.12 on the hub for mail....

	My problem is I really need to spread the load of my IMAP users as
	they can eat a fair bit of memory between them (when I get more than 
	about 125-130 the machine loading really starts to climb). I could
	throw more memory and CPU at it, but as I've got a couple of other
	less powerful but underused Sun SS10's I'd like them to help. Ideally,
	I'd NFS mount the mailspool accross to the other two suns....

	The problem as always is file locking. If my 8.6.12 sendmail is writing
	to a user mailbox as an IMAPD on another host is doing something, the
	mailbox is gonna get trashed. So, can anyone suggest a more elegant
	solution or a RELIABLE method of file locking? I really don't want
	to start splitting our domain internally into two internal subdomains
	which are then munged into one for outgoing mail, but at the moment 
	this is the only sensible method I can see...

	any suggestions appreciated.

	Otherwise, is there anyway to make my IMAP processes more machine 
	friendly? 

	cheers.
-- 
                                                _
                                    	       /-\dam
-------------------------------------------------------------------------
FLESH: Adam Bentley, Systems/Networking, Coventry University. UK
INET : [email-masked]
-------------------------------------------------------------------------
			#include <std/disclaimer.h>				
-------------------------------------------------------------------------

---

From: Terry Gray <[email-masked]>
Date: Fri, 27 Oct 1995 23:30:49 -0700

On 27 Oct 1995, Adam Bentley wrote:

> 	My problem is I really need to spread the load of my IMAP users...

Adam,
The way I would approach this problem is as follows...

Goals:
 o Want to use IMAP and avoid NFS for performance and locking reasons.
 o Need multiple IMAP servers for scalability.
 o Want to preserve single email domain view.

Need:
 o An algorithm *OR* a database for mapping usernames
   into the name or cname of a mail server.
 o An MX mail forwarder for the email domain that uses the above algorithm
   or database.  (We use sendmail as the MTA).
 o A DNS server for the email domain that uses the above algorithm or
   database.

Configuration:
 o Clients are configured (using Pine terminology) to have an
   inbox-path=<username>.email.coventry.ac.uk
   e.g.
   inbox-path=ccx009.email.coventry.ac.uk

The customized DNS server for email.coventry.ac.uk would then return the
IP address of the mail server for user ccx009.

Because MX routing allows load sharing across multiple mail forwarders and
the the modified sendmail on the MX hosts and DNS servers allow use of
multiple mail servers, I believe this model could be scaled to support
very large numbers of users, on the order of hundreds of thousands,
though I/O (esp. seek) bandwidth probably limits each server to around 200
concurrent sessions.

-teg

---

From: Mark Crispin <[email-masked]>
Date: Fri, 27 Oct 1995 21:22:35 -0700

> 	I'd NFS mount the mailspool accross to the other two suns....

There are several things which you can do.

If memory use is the concern, you can use one of the alternate mail
formats which do not require reading the entire folder into memory, such
as tenex, albeit at the cost of making text searches slower.  Refer to the
Pine technical notes for more defaults.

You must not use NFS with tenex format, though.  NFS is not a true UNIX
filesystem; it is a subset and lacks several important characteristics
(robust locking and atomic file/directory operations).  Tenex format uses
random access update-mode I/O which depends upon these.  So if you must
use NFS, you're pretty much stuck with UNIX mbox format.

> 	mailbox is gonna get trashed.

Why do you think that this would be the case?  Sendmail should be using
the .lock file locking which works even over NFS.  [Not that I recommend
using NFS -- I don't.  It is always worse than local file access.]

> 	I really don't want

This isn't necssarily a bad idea, since at some time you may want to split
users by workgroup or other category.

-- Mark --

DoD #0105, R90/6 pilot, FAX: (206) 685-4045  ICBM: N 47 39'35" W 122 18'39"
Science does not emerge from voting, party politics, or public debate.

---

From: Mike Brudenell <[email-masked]>
Date: Mon, 30 Oct 1995 10:28:57 +0000 (GMT)

Mark's question about Sendmail's ".lock" file locking begs me to ask a 
question which has been troubling me vaguely for some time now...

We are using Silicon Graphics kit, running IRIX.  This offers three types
of file locking calls: flock; lockf; fcntl.  Pine/imapd uses some subset
of these to take out locks on the mail folders (either directly, or on an
intermediate temporary file, I forget which). 

As far as I can tell from our sednmail man page sendmail *only* uses 
.lock file locking (creation of a file called username.lock in the 
/var/mail directory to lock the username file).

Pine/imapd will try and create such a file, but only if it has write 
access to the /var/mail directory.

We access mail via an IMAP server running on a machine with a locally 
mounted mail store disk (ie, no NFS involved).  However we cannot give 
everyone write access to the /var/mail directory, otherwise 
Undergraduates, being Undergraduates, will start misusing the area.

Am I right in saying, therefore, that we potentially have a window where 
Pine/imapd *can* conflict with sendmail's delivery?

And if so, how can it be resolved?  Other mailers (eg, Elm) run set-group 
to mail so that the software can create the necessary .lock files.  
However I have never seen anything indicating the Pine or imapd CAN be 
run like this safely, and various notes from users and the authors saying 
that it SHOULDN'T be run like this.

So what's the lowdown, please?

Mike Brudenell                                         <[email-masked]>
------------------------------------------------------------------------
The Computing Service, University of York, Heslington, York, YO1 5DD, UK
Tel: +44-1904-433811  FAX: +44-1904-433740  http://www.york.ac.uk/~pmb1/

---

From: Mark Crispin <[email-masked]>
Date: Mon, 30 Oct 1995 11:55:31 -0800 (PST)

On Mon, 30 Oct 1995 10:28:57 +0000 (GMT), Mike Brudenell wrote:
> of file locking calls: flock; lockf; fcntl.

I am certain that upon further examination, you will find that lockf() is not
a system call, but rather a subroutine that invokes fcntl().  Furthermore, I'm
willing to bet that you will find that flock() is not implemented as a system
call, but rather is emulated by BSD compatibility and actually does fcntl().

lockf() is so simple just about any locking system call can implement it.  For
the most part, fcntl() can be used to emulate flock() except for two very
serious bugs in fcntl():
	1) If you do an open() on a file that you have already open and locked
	   the lock gets blown away.
	2) fcntl() tries to work over NFS via the rpc.lockd/rpc.statd daemons,
	   but this mechanism doesn't work worth a damn.

> Pine/imapd uses some subset

Both lock files and flock() are used in the Pine code, with the necessary
action taken to emulate flock() through fcntl() for those poor unfortunates
who were suckered into buying SysV ("consider it standard") systems.

> /var/mail directory to lock the username file).

Actually, it's done by /bin/mail (which is what sendmail usually invokes to
deliver mail -- look at the Mlocal line in your sendmail.cf).  But yes, it is
true that on almost all implementations of /bin/mail, only .lock file locking
is done.

This is alright; .lock file locking is exclusive which is what mail delivery
needs.  The only problem with .lock file locking is if the locker neglects to
remove the file, you'll have a stuck lock (hence you need timeouts when seeing
that a file is .lock locked against you).

> access to the /var/mail directory.

This is true.

> Undergraduates, being Undergraduates, will start misusing the area.

You should look into the proper setting of the "sticky" bit on /var/mail.  If
you have this set properly, you should not have the abuse problems.

You aren't the only site that has ill-advisedly turned off write access to the
mail spool.  You shouldn't do this unless you change /bin/mail to use flock()
style locking.  But you can't do that over NFS.

> Pine/imapd *can* conflict with sendmail's delivery?

Absolutely so.  If you give the software no way to lock, you can't expect it
to lock!

> And if so, how can it be resolved?

Basically, you have a bad configuration, and you need to change something to
correct it.

One possible solution -- assuming you can get away with it -- is to remove NFS
access to /var/mail, and force all access to be via IMAP.  Tell the partisans
of Elm, etc., to fix their software to support IMAP.

Other, less brutal, solutions include setting the sticky bit correctly (which
fixes the mail security problems).  If undergrads use /var/mail as a scratch
file area, perhaps your real problem is that you don't give them enough disk
quota and/or enough scratch file area on /tmp (or wherever) and you should
resolve that.

> Other mailers (eg, Elm) run set-group

And from time to time there are various security problems which pop up with
mailers that are given privileges.  Sometimes it is disasterous, such as the
infamous movemail security bug (granted, that guy was setuid root).  It was a
design feature from the very beginning that Pine/imapd would *not* require
privileges.

> that it SHOULDN'T be run like this.

That's right.  The UNIX world is filled with people who think that the
solution to any access problem is to make software run privileged.  Without a
doubt, the setuid and setgid bits on files was one of the worst design
decisions made in UNIX from a security standpoint.

This is what makes UNIX so delightful to every net.brat on the planet (and is
why "secure" versions of UNIX disable setuid and setgid files), since any
vulnerability in any piece of software can be used to break in.  Sometimes you
need a cascade of vulnerability, such as subverting one program to create data
in a secure place that will be read by another program with higher privileges
but trusts that lower-level data.

You need to think in a different paradigm, which is "how can users do what
they want to do without requiring privileges?"  Or another way, suppose the
only software you supply is the shell and C compiler, how can users do what
they want to do if they have to write all the software?  By creating a "mail"
user group, you essentially make the writing of MUAs be a privileged
operation.

If you don't want to provide access through the normal filesystem calls, then
you have to provide access through other means.  One such means is POP and
IMAP.  Another such means is to implement a new system call that will let
unprivileged users do what they need to do.

Or, you need to separate the real threat (shanghaied mail files) from what is
essentially a non-threat (anyone can make a file in /var/mail).  The sticky
bit does most of what is needed.  You can also make /bin/mail (or whatever
delivers mail on your system) be much more cautious about what it will deliver
mail to (this is done in most newer implementations) to catch a shanghai after
the fact.  The remaining problems are then in the noise.  Yes, a brat could
create a false .lock file, but he'll need to do that every 5 minutes in order
for his harassment to have effect, and it's easy to catch the responsible
party.  Once caught, you have a social, not a technical problem.

---

From: [email-masked] (Alan Thew)
Date: Mon, 30 Oct 1995 19:49:45 GMT

Has anyone done this with say > 15,000 users? 

Thanks

--
Alan Thew
[email-masked]   ...!uknet!liv!alan.thew   Tel: +44 151 794-4497
University of Liverpool, Computing Services     Fax: +44 151 794-4442

On Fri, 27 Oct 1995 23:30:49 -0700 , Terry Gray ([email-masked]) wrote:

: On 27 Oct 1995, Adam Bentley wrote:

: > 	My problem is I really need to spread the load of my IMAP users...

: Adam,
: The way I would approach this problem is as follows...

: Goals:
:  o Want to use IMAP and avoid NFS for performance and locking reasons.
:  o Need multiple IMAP servers for scalability.
:  o Want to preserve single email domain view.

: Need:
:  o An algorithm *OR* a database for mapping usernames
:    into the name or cname of a mail server.
:  o An MX mail forwarder for the email domain that uses the above algorithm
:    or database.  (We use sendmail as the MTA).
:  o A DNS server for the email domain that uses the above algorithm or
:    database.

: Configuration:
:  o Clients are configured (using Pine terminology) to have an
:    inbox-path=<username>.email.coventry.ac.uk
:    e.g.
:    inbox-path=ccx009.email.coventry.ac.uk

: The customized DNS server for email.coventry.ac.uk would then return the
: IP address of the mail server for user ccx009.

: Because MX routing allows load sharing across multiple mail forwarders and
: the the modified sendmail on the MX hosts and DNS servers allow use of
: multiple mail servers, I believe this model could be scaled to support
: very large numbers of users, on the order of hundreds of thousands,
: though I/O (esp. seek) bandwidth probably limits each server to around 200
: concurrent sessions.

: -teg

---

From: [email-masked] (Mark A. Stevens)
Date: 30 Oct 1995 14:31:53 -0600

In article <[email-masked]>,
Terry Gray  <[email-masked]> wrote:
>...

We spread the load of all of our e-mail users on (now) 6 UNIX boxes.
Users log on or telnet, etc. to these machines, but the mail server is
a machine they do not log on to. We have a script that tests the load
of the 'ecom' boxes to allow a user to log on to the one with the
lightest load and their files are available regardless of the machine,
via NFS mounting. The user's files live on one of two machines that
only (sort of) serve files via NFS.

All mail goes to and comes from our uxa box, the user logs on to ecom?
and files are on fsa or fsb. News is on another box.

IHTH
-- 
Mark A. Stevens                           Phone:    708-235-2204
Systems Programmer                        Internet: [email-masked]
Educational Computing Network             BITNET:   [email-masked]
Board of Governors Universities           VMSHARE:  ECE/MARK

---

From: Terry Gray <[email-masked]>
Date: Mon, 30 Oct 1995 21:16:53 -0800

Alan,
We now have about 40,000 users on our primary email cluster,
with 2-3,000 concurrent sessions.

(Caveat: we are just now implementing the DNS-based client binding
mentioned below; we're using a homebrew hack right now, but I don't forsee
any scaling problems with the DNS binding model. )

-teg