Subject: [Imap-uw] ideas on improving efficiency of calls to mail_sort?

From: Andrew Biggs <[email-masked]>
Date: Thu, 18 Aug 2005 22:01:58 -0600

Hello,

I'm using the c-client library as part of a client application, and was
hoping for some advice on how I might change the way I'm using it to
improve performance.  Specifically, I've run into some slowdowns when I
use the mail_sort function to retrieve a sorted list of message UID's
from a large mailbox (several thousand messages).  What appears to be
happening is that the mail_sort is dutifully executing an IMAP search,
but subsequently calling what appears to be the equivalent of a
mail_fetchfast on each message individually:

0000000e SEARCH UNDELETED UNSEEN NOT (HEADER X-Priority 1)

* SEARCH 26 27 28 29 30 31 32 <...snipped over 1000 UID's ...>
1205 1206 1207 1208

0000000e OK SEARCH completed.

0000000f FETCH 26 (UID INTERNALDATE RFC822.SIZE FLAGS)

* 26 FETCH (UID 46 INTERNALDATE "10-Aug-2005 21:33:47 -0600"
RFC822.SIZE 20687 FLAGS ())

0000000f OK FETCH completed.

00000010 FETCH 27 (UID INTERNALDATE RFC822.SIZE FLAGS)

* 27 FETCH (UID 47 INTERNALDATE "10-Aug-2005 21:33:47 -0600"
RFC822.SIZE 20687 FLAGS ())

00000010 OK FETCH completed.

00000011 FETCH 28 (UID INTERNALDATE RFC822.SIZE FLAGS)

* 28 FETCH (UID 48 INTERNALDATE "10-Aug-2005 21:33:48 -0600"
RFC822.SIZE 20685 FLAGS ())

00000011 OK FETCH completed.

00000012 FETCH 29 (UID INTERNALDATE RFC822.SIZE FLAGS)

* 29 FETCH (UID 49 INTERNALDATE "10-Aug-2005 21:33:48 -0600"
RFC822.SIZE 20686 FLAGS ())

00000012 OK FETCH completed.

00000013 FETCH 30 (UID INTERNALDATE RFC822.SIZE FLAGS)

* 30 FETCH (UID 50 INTERNALDATE "10-Aug-2005 21:33:48 -0600"
RFC822.SIZE 20687 FLAGS ())

00000013 OK FETCH completed.

00000014 FETCH 31 (UID INTERNALDATE RFC822.SIZE FLAGS)

* 31 FETCH (UID 51 INTERNALDATE "10-Aug-2005 21:33:48 -0600"
RFC822.SIZE 20687 FLAGS ())

00000014 OK FETCH completed.

00000015 FETCH 32 (UID INTERNALDATE RFC822.SIZE FLAGS)

* 32 FETCH (UID 52 INTERNALDATE "10-Aug-2005 21:33:48 -0600"
RFC822.SIZE 20687 FLAGS ())

00000015 OK FETCH completed.

<... etc etc ...>

Any suggestions or advice on other calls or perhaps different cache
settings I could try to speed this up a bit?  I figure there must be
some way to either pre-fetch the internaldate (which I'm sorting on) or
at least batch the fetch calls up somehow.

Thanks!

Andrew

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Thu, 18 Aug 2005 21:35:26 -0700 (PDT)

Hello -

The c-client mailing list has been renamed to the imap-uw mailing list.

The fundamental problem is that your server does not implement the IMAP 
SORT extension.  You would not have this problem with a server that offers 
SORT.

Next, what options are you giving to mail_sort()?  If you are setting 
SE_UID, then the code does not (can not) do any prefetching.  This is 
because c-client's cache is completely implemented by message sequence 
number, not by UID.  In general, options such as SE_UID and FT_UID should 
be avoided, especially when doing multiple message operations.

Instead, get the sequence numbers, and then afterwards translate them to 
UIDs (if that is what you want) via mail_uid().  This may sound backwards 
to someone who is thinking in terms of UIDs, but trust me, c-client cache 
will be happier and will reward you for your cooperation.

You may find the use of sequence numbers to be beneficial in other ways as 
well; use UIDs just for synchronization and do everything else with 
sequence numbers.  c-client will reward you for that as well.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Andrew Biggs <[email-masked]>
Date: Fri, 19 Aug 2005 14:59:38 -0600

Excellent suggestions, thanks Mark.  You're correct that I've been 
working mainly from UID's, so I'll see if I can move away from that a 
bit to try and improve the situation.  And yeah, Exchange2K3 doesn't 
seem to support SORT :-(.

Andrew

(and apologies for spamming the mailer with multiple copies of the 
original post)

Mark Crispin wrote:

> The fundamental problem is that your server does not implement the 
> IMAP SORT extension.  You would not have this problem with a server 
> that offers SORT.
>
> number, not by UID.  In general, options such as SE_UID and FT_UID 
> should be avoided, especially when doing multiple message operations.
>
> Instead, get the sequence numbers, and then afterwards translate them 
> to UIDs (if that is what you want) via mail_uid().  This may sound 
> backwards to someone who is thinking in terms of UIDs, but trust me, 
> c-client cache will be happier and will reward you for your cooperation.
>
> You may find the use of sequence numbers to be beneficial in other 
> ways as well; use UIDs just for synchronization and do everything else 
> with sequence numbers.  c-client will reward you for that as well.
>
> -- Mark --
>