Subject: imapd process count

From: John Kelly <[email-masked]>
Date: Wed, 16 May 2001 13:11:56 -0400

Hi,

I read elsewhere that Linux scheduler performance starts degrading
badly above 1,000 processes or so.  There are some unofficial patches
to fix this, but who knows if they will ever appear in the official
kernel.

That severely limits the number of users that could be served from a
single Linux server, since imapd will keep the connection alive for up
to 30 minutes before timing out, and many processes could pile up.

I suppose you might think of using some kind of hybrid process/threads
model like Apache 2.0 to deal with this problem; but OTOH, I also read
that the Linux scheduler treats a thread like a process for purposes
of scheduling, and thus a threads model would not solve the scheduler
performance problem.

What, if any, are the design alternatives for reducing the number of
processes required to serve a large population of IMAP users?

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Wed, 16 May 2001 10:28:41 -0700 (PDT)

> to fix this, but who knows if they will ever appear in the official

If this is true, this is a serious bug in Linux.  1000 processes is not a
particularly large number for a large system.  I can't imagine this not
getting fixed.

However, this shouldn't affect IMAP.  An IMAP server runs out of disk I/O
bandwidth long before it runs out of CPU.

> processes required to serve a large population of IMAP users?

Don't put all your eggs in one basket.  Instead of having a single expensive
system, have many cheap systems.  Just because the user sees it as "one
system" doesn't mean that it's one CPU.

The deskmail cluster at the University of Washington consists of a few dozen
dedicated IMAP server boxes.  We can add new server boxes any shuffle users to
them at any time, and use the DNS as a user locator to the currect systems.
User "mrc" uses mrc.deskmail.washington.edu, which today happens to be the
mailer31.u.washington.edu machine (I know because I just looked).

---

From: John Kelly <[email-masked]>
Date: Wed, 16 May 2001 14:19:23 -0400

>system" doesn't mean that it's one CPU.

Fair enough.

However, I meant software design instead of system environment design.

---

From: John Kelly <[email-masked]>
Date: Wed, 16 May 2001 15:57:02 -0400

On Wed, 16 May 2001 13:18:41 -0500 (CDT), "Kevin W. Mullet"
<[email-masked]> wrote:

>I believe the 2.4 kernel should practically support however many
>your hardware can sustain.

They eliminated the former 2.2 restrction which stopped you at about
4000, but that still does not mean the scheduler can cope with higher
numbers:

  http://www.cs.helsinki.fi/linux/linux-kernel/2001-13/0294.html

---

From: Malcolm Beattie <[email-masked]>
Date: Thu, 17 May 2001 15:39:09 +0100

>   http://www.cs.helsinki.fi/linux/linux-kernel/2001-13/0294.html

There's an important point you both seem to have missed: it's not the
high number of processes that causes a problem; it's a high number of
*runnable* processes. So, for example, we currently have 236 imapd
processes on one of our IMAP server nodes but the load average (roughly
the number of runnable processes at any instant) is 1.59. IMAP serving
is I/O bound so almost all of the imapd processes at any time will be
TASK_{UN,}INTERRUPTIBLE and not TASK_RUNNING and hence not affect the
scheduler at all.

--Malcolm

-- 
Malcolm Beattie <[email-masked]>
Unix Systems Programmer
Oxford University Computing Services

---

From: Mark Crispin <[email-masked]>
Date: Thu, 17 May 2001 11:30:09 -0700 (PDT)

On Thu, 17 May 2001 15:39:09 +0100, Malcolm Beattie wrote:
> *runnable* processes.

Oh, so you're saying that Linux gets unhappy when the load average reaches
1024?  I guess it has a right to be cranky at that!  :-)

[Please don't flame me; I know that "load average" and "number of runnable
processes" is not the same thing.  But as a very rough first-order
approximation, you can pretend that they are.]

> IMAP serving

That's our experience as well.  We run out of disk bandwidth long before we
run out of CPU.

---

From: "Mark Elvers" <[email-masked]>
Date: Tue, 22 May 2001 11:00:05 +0100

> mailer31.u.washington.edu machine (I know because I just looked).

So, in this scenario each of the mailerXX.u.washington.edu boxes has its own
local storage which is monitored for space.  As space becomes tight users
are moved off to other servers.  Just for interest how is the move done?
NFS copy while the mailbox is not in use?

As NFS seems to be out, is it possible to have a large central disk solution
which can be accessed by multiple IMAP servers?  Or is this simply not worth
doing as you say that the disk usage is higher than CPU usage so such a
central disk solution would be crippled by all the concurrent accesses from
the servers?

And sorry if this is all slightly off the topic are you saying that all
users have a different IMAP server name - doesn't that mean for an
organisation the size of UW you have tens of thousands of DNS entries?  Is
there an alternative method of doing this kind of thing?

--mte

---

From: "Mark Elvers" <[email-masked]>
Date: Tue, 22 May 2001 11:44:06 +0100

> This is a controversial point, but I believe the latter.  That is, I
believe
> that a central disk solution is not the way to distribute something as
I/O-
> intensive as mail.
>
> Of course, a central disk solution is ideal for files, which have very
> different usage/access patterns than mail.
>

But here I have some confusion, if we have a central disk solution for say
users home directories export via NFS.  We have a number of IMAP boxes with
their own storage for the email.  Then where do users folders sit in this
environment?

Do each of the users have a home directory on 'their' IMAP server which is
used exclusively for email?  They would probably be barred from logging into
server directly.  I'm thinking that if they call mail_create(s,
"{server}SentBox") the mailbox defaults to their home directory.  Is their
perhaps a way of allowing folders to be created in /var/spool/mail ?

--mte

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 May 2001 02:58:40 -0700 (PDT)

On Tue, 22 May 2001 11:00:05 +0100, Mark Elvers wrote:
> are moved off to other servers.

Actually, as load becomes more than trivial.  Disk space is cheap enough to
add as needed.  It's much more leisurely to move users when you only have a
modest load than when a crisis is in progress.

> Just for interest how is the move done?

I'm not sure, but it's not NFS.

> central disk solution would be crippled by all the concurrent accesses from

This is a controversial point, but I believe the latter.  That is, I believe
that a central disk solution is not the way to distribute something as I/O-
intensive as mail.

> there an alternative method of doing this kind of thing?

Yes, there is.  The DNS server for the *.deskmail.washington.edu does not use
DNS zone files.  Instead, it is a special version which interacts with the
accounting database which (surprise!) already has tens of thousands of
entries.  "Which IMAP server" is just another field in the per-user entry.

Needless to say, a zone transfer isn't a meaningful concept with that domain!

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 May 2001 13:36:24 -0700 (PDT)

> their own storage for the email.  Then where do users folders sit in this

Users' mailboxes sit on the IMAP server, and not on the NFS-exported
filesystem.

Of course, users can also put mailboxes on their NFS-exported home directories
on the central disk filesystem, if they configure their clients that way.
This may be better for UNIX shell users, but not for PC users.

> used exclusively for email?

Yes, and this is separate from the NFS-exported central disk home directory.

> They would probably be barred from logging into
> server directly.

Generally, yes.

> I'm thinking that if they call mail_create(s,
> "{server}SentBox") the mailbox defaults to their home directory.

Yes.

> Is their

There's a way of allowing anything, but trust me, this isn't something that
you want to do.