Subject: how to cache IMAP headers?

From: Vadim Zeitlin <[email-masked]>
Date: Thu, 5 Apr 2001 19:41:27 +0200 (CET)

Hello,
 
 I am looking for advice about implementing caching of the headers of the IMAP
folders. Currently, our program retrieves all messages from the folder each
time it is opened which is too slow for big folders. As a temporary work
around, we limit the number of messages retrieved to some optional number
(100) but it's not really a solution because sometimes you need more headers
than the configured limit and sometimes retrieveing even 100 of them is still
too slow.

 So the logical (I hope!) idea is to cache the headers locally. However there
is a problem of synchronisation - how can I know if the cached headers still
correspond to the ones on the server? For a local file, I'd just save the file
modification time (and/or size, and/or MD5 hash depending on paranoia level)
in the header cache file and check it when opening the folder but I don't see
any way to do something like this with a remote IMAP folder.

 I can test if the cached UIDs still correspond to the current msgnos which
should let me detect if any messages were deleted from server, but I don't
know at all how to detect changes to the message flags - i.e. if the message
was read from another client, how can I know when reopening the folder that
its \SEEN flag changed?

 I'd be very grateful for any ideas or pointers to documentaion/code/whatever!

 Thanks in advance,
VZ

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Thu, 5 Apr 2001 11:57:15 -0700 (PDT)

On Thu, 5 Apr 2001 19:41:27 +0200 (CET), Vadim Zeitlin wrote:
>  I am looking for advice about implementing caching of the headers of the
> IMAP folders. Currently, our program retrieves all messages from the folder
> each time it is opened which is too slow for big folders. As a temporary
> work around, we limit the number of messages retrieved to some optional
> number (100) but it's not really a solution because sometimes you need more
> headers than the configured limit and sometimes retrieveing even 100 of them
> is still too slow.

A review of RFC 1733 can be helpful in understanding this.

There's an old military slogan: "There's three ways to do something: the right
way, the wrong way, and the Army way.  Do it the Army way."

You've discovered the wrong way to get message header data at startup, and you
know why it is wrong.

There is a right way, involving caching and synchronization.  Synchronization
is done using UIDs (unique identifiers), not MD5 hashes.  IMAP UIDs have a
number of interesting and useful properties that are often overlooked by
programmers who are accustomed to using MD5 hashes and/or POP3 UIDs; fully
exploiting these capabilities requires careful study.

However, you should also consider the IMAP way.

The rest of this message will be about the IMAP way.  You will hear from other
folks how to do the right way.  All I will say is "many have tried; most have
failed."

Many programmers overlook the IMAP way.  The IMAP way is the only good way
when you have users who share client machines and/or use multiple clients,
because it does not require a cache.  Nor does it require large downloading.
But it's completely foreign to anyone who's used to POP.

Pine is the epitome of a client that does things the IMAP way.  This is why
Pine is so much faster/better than most other IMAP clients.

In the IMAP way, you retrieve data on demand, and anticipate limited
additional needs.

For example, at startup time, the need is to draw the initial message browser.
To do this, you only need to have the information for the messages which will
be visible.  You don't need information about messages which are scrolled out
of the current view.

Your first decision is "where do I position the initial view?"  By virtue of
opening the mailbox, you know how many messages there are and which is the
lowest-numbered message which does not have \Seen status.  So, given a browser
with n visible rows, you know enough to position the first n messages, the
last n messages, or n messages around the first unseen message.  If you want a
different position, you may have to do a SEARCH command.

So, assuming that you've decided upon a position.  You now need to draw the
browser view.  You can't do this without information for message n.  So, at
this point, you get it.  If you're used to POP, you may want to do a TOP
command to get the header.  Or, you might decide to get the IMAP "ENVELOPE",
which has the information you probably need already digested for you; it's the
equivalent of NNTP overviews.

You may find that having the flags and size are also convenient, so you may
decide to use "ALL" (which incorporates all of this) or "FULL" (which also
gives you the BODYSTRUCTURE).  I'd advise against using "FULL" unless you
really need the MIME structure at this point, e.g. to draw a paperclip icon by
messages which have attachments.

You now have what you need to draw the browser line for message n, so you draw
it.

Now, the problem with fetching just the data for that one message is that
doing a fetch per-message will whack you with so many round trips that your
client will be just as slow as one that does the Big Download.  So, you do
some caching and anticipating.

Here's how that works.

Caching is easy.  Once you get the data for a message, remember it in your
client's memory.  That way, if you need to draw that browser line again, you
have it at hand.

Now, for anticipation.  Given that you're drawing a browser line, you probably
will want to draw for the next several messages.  Pick a number, e.g. 20, of
messages which is not noticably more expensive to fetch than just 1.  Now,
when you fetch-on-demand for a message, fetch ahead 20 messages IF THEY ARE
NOT ALREADY IN THE CACHE.  So, you look ahead in your cache up to 20 message,
and if you see a hole, add that to the fetch request.

So, suppost you're at message 1 and this is just after startup.  You'd end up
doing
	tag FETCH 1:20 ALL

This means that when the browser goes to draw message 2, it'll find it already
in the cache so it doesn't do an IMAP command.

As the user scrolls through the browser, do subsequent fetching as necessary
whenever a cache fault happens.  But never do the Big Download.

Of course, the equivalent of the Big Download will happen if the user scrolls
through his entire mailbox, but it'll be broken up into smaller chunks so that
the user doesn't really notice that it happens.

NOW, here comes the really good news:

c-client does the caching and anticipation for you!  All you have to do is ask
c-client for message information on demand.

Of course, you have to use envelopes; that is, mail_fetch_structure() and/or
its aliases such as mail_fetchenvelope().  If you use mail_fetch_header(), you
won't get that benefit.