Subject: Re: How do I determine number of unseen and deleted message guickly

From: "Mark A. Keasling" <[email-masked]>
Date: Wed, 1 Jun 1994 17:28:40 +0900 (JST)

On Tue, 31 May 1994 22:31:50 -0700 (PDT), Mark Crispin <[email-masked]> wrote...
> On Wed, 1 Jun 1994 11:40:56 +0900 (JST), Mark A. Keasling wrote:
> > That sounds like a reasonable approach.  Before, I was using the
> > mail_fetchstructure() and then mail_elt and examining the MESSAGECACHE for
> > this information as documented in Internal.doc.  Since all of the envelopes
> > have been fetched already to build list items, would it be better in this
> > case to use mail_fetchstructure() and mail_elt()?
> 
> The only disadvantage to doing this is in the case of multiple shared
> read/write access to the mailbox; you won't get the flag updates.  However,
> the very latest c-client/imapd does quite a bit towards making sure that flag
> updates are communicated to the client in this case.
> 
> One general comment, though, is that doing all those mail_fetchstructure()
> calls is likely to be much slower than the search.  As I told you before, it's
> better to do a mail_fetchstructure() only when you actually have to display
> it, since that is one of the most expensive functions in c-client.
> 
Ok, then I should avoid unnecessary mail_fetchstructure calls.
All I want to do is examine 5 bits (the flags) in the message cache.
The preferred method to get the cache is in conjunction with a
mail_fetchstructure.  I call mail_fetchstructure for every message
in the mailbox to build the message list header items when it is
opened (makes opening a folder very slow).  But that being the case,
should I not be able to use mail_elt on the stream after that without
having to first call mail_fetchstructure every time?  Is there a way
to determine if mail_fetchstructure needs to be called for a particular
message?  Is there a way to sync the local message cache with the IMAP
server's cache?  What is the most efficient way to examine those 5 bits?

Regards,

-- Mark Keasling
   AIR Company LTD, Nishikawa Mitsui Bldg, 1-3-14 Kitahama, Chuo-ku, Osaka 541
   email: [email-masked]      phone: +1 81 6201 4307    fax: +1 81 6201 2107

---

From: Mark Crispin <[email-masked]>
Date: Sun, 5 Jun 1994 17:41:05 -0700 (PDT)

> Ok, then I should avoid unnecessary mail_fetchstructure calls.

By ``avoid unnecessary mail_fetchstructure() calls'' this does not mean 
that mail_fetchstructure() is any slower if you have earlier fetched the 
structure of that message.  c-client caching makes that a no-op.

My recommendations are as follows:

1) mail_fetchflags() is faster than mail_fetchstructure(), but although 
it loads the cache it does not consult the cache.  That means that it 
will happily refetch flags that are already cached.  Since modern 
versions of c-client update the cache dynamically with shared folders 
(including via IMAP), there is generally never any reason to do a 
mail_fetchflags() of all messages call more than once.

2) If you are going to be calling mail_fetchstructure() on a message 
shortly, it is better not to use mail_fetchflags() and instead to 
structure your code so that you can defer sniffing at the flags until you 
are ready to do a mail_fetchstructure().

3) If you are not going to need the MIME body structure, it is better to 
give a NIL argument to the returned body pointer to mail_fetchstructure() 
to avoid doing the MIME parse.  Later, when you need the MIME body 
structure, you can do a new mail_fetchstructure() with a non-NIL returned 
body pointer.

4) Operations on all messages are slow.  Avoid them.  Instead of fetching
all messages to build a browser list, build the list with dummy contents 
and replace the dummy contents with the real data only when you need to 
display it on the screen.  That way, you only do a mail_fetchstructure() 
when you really need it.

5) Although mail_fetchstructure() is slow, it is cached, so there is no 
cost in doing it again after the first time (modulo the question of the 
MIME parse data).

6) SEARCH for flags is faster than mail_fetchflags() and then running
through the flags in the cache.  Consider turning off prefetching (this 
is a mail_parameters() call in recent versions of c-client).

> I call mail_fetchstructure for every message
> having to first call mail_fetchstructure every time?

As I said above, you should avoid doing this when a mailbox is opened.
But, having done this, subsequent mail_fetchstructure() calls are going to
be no-ops; no IMAP will be generated.  So you will not gain any 
performance advantage from getting rid of mail_fetchstructure() calls, 
and getting rid of those calls runs the risk of your program breaking 
when you fix your program not to do mail_fetchstructure() when opening 
the folder.

So, the answer to your question is ``yes, but you won't gain anything''.

>  Is there a way
> message?

This is not a meaningful concept.  If it doesn't need to be called, then 
calling it is a no-op.

>  Is there a way to sync the local message cache with the IMAP
> server's cache?

This should be automatic in modern c-clients.

>  What is the most efficient way to examine those 5 bits?

If you have already fetched flags for all messages (either via 
mail_fetchflags() or by mail_fetchstructure()) then you might as well run 
through the cache via mail_elt() and look at the bits.

My own preference is to avoid fetching flags for all messages, and then 
call mail_fetchstructure() as needed.

---

From: John Gardiner Myers <[email-masked]>
Date: Sun, 5 Jun 1994 21:47:05 -0400 (EDT)

While I certainly agree with the bulk of Mark's advice (especially of
not calling mail_fetchstructure() on every message on open--mailboxes
can have thousands or tens of thousands of messages), I should point
out that some of the costs are slightly different when talking to a
Cyrus server.

The Cyrus server implements fetching the ENVELOPE and/or BODY by
sending back a precalculated string from a memory-mapped file.  The
cost of the first mail_fetchstructure() call is essentially the
network round-trip plus the network bandwith to transfer the data.
The strategy of calling mail_fetchstructure() with a NIL body pointer
followed by a later non-NIL call will cost you a round trip at the
benefit of potentially not having to use some network bandwith.

-- 
_.John G. Myers		Internet: [email-masked]
			LoseNet:  ...!seismo!ihnp4!wiscvm.wisc.edu!give!up