Subject: Re: A note on multiprocess IMAP server

From: "Alex Shvedov" <[email-masked]>
Date: Fri, 5 Mar 1999 10:45:00 -0500

>On Thu, 4 Mar 1999 19:44:16 -0500, Alex Shvedov wrote:
>> Normally one wants just to copy imapd.c, mail_link some new driver there
and
>> live happily afterwards; that's what I did and that's why I decided to
send
>> this comment.
>>
>> Since You provided these mm_(no)critical as wrappers for
>> critical=TRUE/FALSE, forked variant is not a problem at all.
>

>This also makes no sense.
>
>As distributed, imapd handles a single IMAP session, and for the most part
>does it in a single process.  IMAP's I/O for the session is via stdio.  All
>drivers run in the same process.
The way all - repeat, all - connections work on UNIX is different. Its inetd
who accepts the connection and (sic!) spawns a separate process of imapd (or
telnetd or ftpd etc.).

Therefore you always - repeat, always - have a separate process for each
login (=successful connection).

It is true that inetd retranslates what it hears from outside world to
imapd's stdin and listens (and retranslates to the outside world) what imapd
has to say on its stdout. But inetd does it SEPARATELY for each login. In
other words, if two clients simultaneously login to the same IMAP account,
there will be TWO imapd processes that DO NOT share any memory.

This is a standard picture which has nothing to do with IMAP protocol
whatsoever, it's the way inetd handles requests (=connections).

My problem was:
- suppose we have two simultaneous identical logins (and, as we know now,
two independent imapd processes), so that name, pwd, mailbox and message(s)
that has(ve) been fetched are the same;
- suppose one guy deletes a message and expunges;
What will happen with the slower guy, who drew second?

After your explanation I see that mm_(no)critical has nothing to do with
solution.

What's the solution then? Or, for starter, is there a problem?

Actually, I tested our server using that same MS Outlook Express v.5 (beta
2) I'm using right now; I myself was surprised to see two or more sessions
(!) while going through my IMAP account - I was the only one (!) who was
logged in, and I didn't expect my client to do it twice during one session.
It's MS OE (+inetd scheme) that creates multiple logins and hence
multiprocesses imapd.

>The only thing that mm_critical() is used for is to allow a driver to tell
the
>main program that it is in dangerous code that should be allowed to run to
>completion.  That, in turn, is used by imapd's signal handlers to determine
>whether or not it should exit on a signal.
>
>You seem to be saying that you have taken the original imapd and modified
it
>to run in an entirely different way.  If you do that, then it has become a
>different program, and you're responsible for doing whatever has to be done
to
>make that program work.
I don't think that our code is really different, since we just made one more
layer of retranslation: inetd starts some special thing called imap_link
which does nothing except piping each byte to/from a privileged port 143 to
some nonprivileged port any non-root can bind. So we do things in exactly
the same - alas, spawning - manner inetd does, but deal with port number >
1024.

>That may include changing how it does command I/O, or how it handles
signals, or any number of other >details.
OK. We - hopefully - don't and will not need that.

>More importantly, imapd is not c-client.
A rose is a rose is a rose.
I tried to compile and link imapd without c-client.a and failed. After an
hour of deliberations I came to a conclusion that I have every reason to
suspect some sort of connection between the two.

---

From: Mark Crispin <[email-masked]>
Date: Fri, 5 Mar 1999 13:25:55 -0800 (PST)

Thank you for your clarification.  I believe that you are very confused about
how the software works.  I will try to clear things up for you.

Let me start by assuring you that I understand quite well how inetd works; and
that imapd is designed to work in precisely that environment.  In general, it
is a good assumption that an implementor knows what he's doing, particularly
when it comes down to such fundamental details as this.

mm_critical() has nothing to do with mailbox interlocking between multiple
imapds accessing the same mailbox.  mm_critical() deals only with blocking
interrupt-termination of an imapd that is engaged in an operation which must
run to completion to avoid mailbox damage.

There is entirely different code to handle multiple imapds accessing the same
mailbox, and the interlocks necessary to support this.  You can find out more
about these issues by reading the file "docs/locking.txt" in the IMAP toolkit
source directory.

Each c-client driver is responsible for managing its own locking discipline.
In some drivers, such as the ones to handle the standard UNIX format of mail
and the MMDF format of mail, it is not permitted for multiple imapds to access
the same mailbox.  Consequently, the locking algorithms for these formats deal
with the question of which imapd is allowed read/write access, and of how to
seize the lock from another imapd.

In certain other formats, most notably mbx, the driver permits multiple imapds
to access the same mailbox; and there is a rather elaborate mechanism to
handle the contention issues.  To the best of my knowledge, nobody had done
anything like this on UNIX prior to my doing it about 10 years ago; I was told
by all the UNIX Gurus that it wasn't possible.

In other words, this is a problem that was solved many years ago.  What may
have confused you is that, in the standard UNIX format of mail, you are not
permitted to have multiple imapds accessing the same mailbox.  However, imapd
properly detects this condition, and prevents it from happening.  Any attempt
to make it happen results in one of the imapds failing.