Subject: RE: mail_uid_sequence: a performance hog

From: "Rubinstein, Dmitry" <[email-masked]>
Date: Mon, 23 Aug 1999 22:45:27 +0300

> From: Mark Crispin [mailto:[email-masked]]
> 
> I don't understand your comment about the IMAP driver.  If 
> you're using IMAP,
> then mail_uid_sequence() is generally never called.  If you 
> give it a UID
> sequence, that sequence as text is passed to the IMAP server.

I can't see how it can be. The function mail_uid_sequence() is being called
from imapd.c every time a UID FETCH (and not only it) is issued. Once this
function is called, and once the sequence has '*' in it, a call to
mail_uid() is issued, which, in case the UID is not already there, causes a
fetch of UID (over network, in case of IMAP driver), which might have been
spared.

> I don't see any situation in which you can have the purported 
> "unbearable"
> problem in unmodified c-client code.  

Consider a client of a 'middleware' server sending the following command:

1 UID FETCH 1:* FLAGS

We'd like this command to go to the 'real' IMAP server unchanged. At the
same time we'd like to be able to start sending the data to the client as
soon as it (the data) arrives, without waiting for the whole response from
the real server. Therefore we're facing the necessity of prepending this
command by another one:

2 UID FETCH * UID

It's not a _very_ good thing, but it's OK. However, in our application (not
in the c-client part, which goes in virtually with no changes at all, at
least not the IMAP stuff), the mailbox is constructed on the fly (and there
are additional complications), which means that fetching the UID of the last
message will result in fetch of all messages. Now it gets ugly. You have to
wait for a result of a pretty heavy command in order to learn something you
don't really need to know.

> Even in the one case 
> (overviews, which
> aren't part of IMAP) where the IMAP driver can call 
> mail_uid_sequence(), the
> worst that'll have is a fetch of the UID for the message with sequence
> matching stream->nsgs.  Once the UID is fetched for that 
> message, it's cached
> and isn't fetched again.

Sorry, perhaps I haven't phrased my previous letter clearly enough. The
calls to mail_uid_sequence are performed by the 'application' in terms of
c-client (in my case - by imapd.c).

> If you're modifying the code to do IMAP UID sequence parsing 
> locally, then you
> can just as easily also make the type of modification that 
> you suggest to
> accomodate your other modification.

Actually I try to stick to c-client as much as possible, including the
sequences parsing.

> In any case, a better programming technique is to use sequence numbers
> whenever possible.  UIDs are never the preferred way to get 
> at messages if you
> have the sequence number available.  Applications which try 
> to think that "all
> the world is UID" will never be able to use c-client (or 
> IMAP) as effectively
> as those which use both UIDs and sequence numbers, choosing 
> the one that is
> appropriate for the task at hand.

The client I'm using for testing is mainly Netscape Messenger. It uses UIDs
_only_. I believe Outlook Express does the same. What can I do but optimize
for UIDs?

--
Dmitry Rubinstein

---

From: Mark Crispin <[email-masked]>
Date: Mon, 23 Aug 1999 12:49:13 -0700 (PDT)

> fetch of UID (over network, in case of IMAP driver), which might have been

This isn't a c-client issue at all.  It's an imapd issue.  Imapd can be used
as a proxy server to another IMAP server, but it was not designed for that
purpose.  Imapd was designed for access to local mailboxes first and foremost,
then for NNTP proxy.

IMAP and POP3 proxy in imapd are more or less afterthoughts.  Neither will
work at all well unless you use plaintext authentication, since imapd can't
log in to the target server without the password.  There's no support for
passing on Kerberos, etc.

If you want a high-performance IMAP proxy server, you are better off building
a server designed for that exclusive purpose.  I would not even think of using
a server which does anything other than pass through the IMAP protocol from
the target server.  When you use imapd as an IMAP proxy server, you
essentially parse and store all the target server's IMAP into imapd's local
cache, only to regenerate it to sent to the client.

> We'd like this command to go to the 'real' IMAP server unchanged.

You should not use imapd for this purpose.  What you want is actually much
simpler than imapd: a pass-through server.  It shouldn't use c-client at all,
since all of c-client's caching is nothing but a waste of memory and CPU.

The simplest pass-through proxy server is probably no more than about 50 lines
of C code.  You'd want more than that to provide filtering or firewall
services, but then again, you'd have to add the same additional code to imapd.