Subject: RE: mail_uid_sequence: a performance hog

From: "Rubinstein, Dmitry" <[email-masked]>
Date: Mon, 23 Aug 1999 23:25:15 +0300

> From: Mark Crispin [mailto:[email-masked]]
> On Mon, 23 Aug 1999 22:45:27 +0300, Rubinstein, Dmitry wrote:
> > I can't see how it can be. The function mail_uid_sequence() 
> is being called
> > from imapd.c every time a UID FETCH (and not only it) is 
> issued. Once this
> > function is called, and once the sequence has '*' in it, a call to
> > mail_uid() is issued, which, in case the UID is not already 
> there, causes a
> > fetch of UID (over network, in case of IMAP driver), which 
> might have been
> > spared.
> 
> This isn't a c-client issue at all.  It's an imapd issue.  
> Imapd can be used
> as a proxy server to another IMAP server, but it was not 
> designed for that
> purpose.  Imapd was designed for access to local mailboxes 
> first and foremost,
> then for NNTP proxy.

Doesn't pine (based on c-client) works with remote IMAP servers? Anyway, if
no massive use of c-client with network drivers is intended, then I remove
my (original) question (and will have to make the modification in my copy of
the code).

> IMAP and POP3 proxy in imapd are more or less afterthoughts.  
> Neither will
> work at all well unless you use plaintext authentication, 
> since imapd can't
> log in to the target server without the password.  There's no 
> support for
> passing on Kerberos, etc.

I'm aware of that, and had to perform the appropriate modifications.

> If you want a high-performance IMAP proxy server, you are 
> better off building
> a server designed for that exclusive purpose.  I would not 
> even think of using
> a server which does anything other than pass through the IMAP 
> protocol from
> the target server.  When you use imapd as an IMAP proxy server, you
> essentially parse and store all the target server's IMAP into 
> imapd's local
> cache, only to regenerate it to sent to the client.

Our product is not just a proxy server. It has some added value, which needs
a framework to reside upon. However, I aggree that imapd is not as effective
as needed in order to be used with network drivers (that was exactly my
original remark).

> The simplest pass-through proxy server is probably no more 
> than about 50 lines
> of C code.  

;-))) That's an exaggeration of the year, I'd dare say. Unless you use very,
very, VE-E-E-RY long lines.

--
Dmitry Rubinstein

---

From: Mark Crispin <[email-masked]>
Date: Mon, 23 Aug 1999 13:29:02 -0700 (PDT)

On Mon, 23 Aug 1999 23:25:15 +0300, Rubinstein, Dmitry wrote:
> Doesn't pine (based on c-client) works with remote IMAP servers?

What does this have to do with anything?  Pine uses message sequence numbers
for most things.  When Pine uses UIDs, the UID is sent directly to the server.
Pine handles IMAP better than almost any other IMAP client that I've seen
(I've seen some truly wretched IMAP clients from Big Vendors).

Even with a c-client application that only uses UIDs, the UID text is passed
as given to the IMAP server to interpret.  A lot of people do this, but I
don't recommend it since there's a lot of neat things which you can do with
IMAP if you use both UIDs and sequences.

Your problem comes up *only* when you use IMAP to connect to imapd and that
imapd then subsequently uses IMAP to connect to a third-party.

> as needed in order to be used with network drivers (that was exactly my

I am at a loss to think of what sort of framework that you need from c-client.
You shouldn't do much IMAP parsing or generation at all; just pass through the
IMAP from the target server.  It's absurd to go to all the trouble of parsing
IMAP into MESSAGECACHE/ENVELOPE/BODY structures and cached texts if all you're
going to do is spit it out the other end.  That's *much* more of a performance
issue that whether or not an RTT is taken up for a single UID lookup on "*"
(and it *is* a single lookup unless there's new mail -- c-client will cache
this).

You can't effectively use the SASL technology from c-client at all, since
there's nothing for pass-through there.  You need to build a different SASL
technology, plus deal with the security issues (e.g. the stuff that prevents
man-in-the-middle attacks since you are effectively a man-in-the-middle).  So
what you need is a capability to do credentials forwarding for those protocols
which support it (e.g. Kerberos) and just pass-through on those which do not
(e.g. CRAM-MD5).  In other cases, you may need a special mechanism to do
authentication from the proxy IMAP server to the target IMAP server.

c-client has none of this.  The code will show you how you go about
implementing some of these things, but you need completely different code.

> > than about 50 lines of C code.
>

Not at all.  The simplest pass-through simply transfers data between stdio and
an open network socket.  Most of the work is opening the socket.  50 lines may
be too much.

This doesn't include any of your value-added features, but you need to write
that code anyway, whether you use imapd as your base or your own code.

Now, if you wish to have local mailbox capability in your server, then it
makes sense to use imapd/c-client as a base.  However, I would still recommend
treating proxy as a special case.  Either have a separate application for
proxy, or if the same machine needs to do both proxy and local, then have a
way to switch to the proxy application once you realize which mode to be in.

Imapd can do proxy, but it isn't recommended or suitable for any sort of heavy
duty use.  The lack of credentials forwarding is also a problem.