Subject: What does CLIENT BUG in result of STATUS operation mean?

From: Mike Schmidt <[email-masked]>
Date: Sat, 16 Apr 2005 11:03:27 -0400

Hi,

Can  someone tell me what the CLIENT BUG references mean in the status 
report?

This is a transcript of the IMAP status operation. The client is based 
on c-client, running in Windows XP, and the server is uw-imapd running 
over a TLS connection (on Linux):

4/16/2005 10:55:05 AM: 00000005 STATUS INBOX (MESSAGES RECENT UNSEEN 
UIDNEXT UIDVALIDITY)
4/16/2005 10:55:12 AM: * NO CLIENT BUG DETECTED: STATUS on selected 
mailbox: INBOX
4/16/2005 10:55:21 AM: WARN - CLIENT BUG DETECTED: STATUS on selected 
mailbox: INBOX
4/16/2005 10:55:30 AM: * STATUS INBOX (MESSAGES 150 RECENT 0 UNSEEN 0 
UIDNEXT 247 UIDVALIDITY 1108742917)
4/16/2005 10:56:53 AM: 00000005 OK STATUS completed

Thanks very much. This certainly looks pretty confusing to me, but I' m 
just getting started writing a client using c-client.

Mike

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Sat, 16 Apr 2005 09:16:07 -0700 (PDT)

> Can  someone tell me what the CLIENT BUG references mean in the status 

It means that you did a STATUS command on the selected (opened) mailbox; a 
completely unnecessary and wasteful operation which could also have 
additional severe negative consequences.

When you have a mailbox selected, the IMAP state already has all the data 
that a STATUS command could return.  The STATUS command is to the be used 
only to get this data from a mailbox which is *NOT* selected.

This has been shown to be an area which many novices fail to understand; 
many seem to believe (incorrectly) that you have to do a STATUS to get 
updated mailbox state.  Learning how IMAP really works in this case is 
likely to help client authors understand IMAP in other cases, hence this 
is a mistake that I felt is important to correct.

> over a TLS connection (on Linux)

If the client was written using c-client, then you did a mail_status() 
call on a mailbox name when you already had a perfectly good MAILSTREAM 
with all the data you need from a mail_open() call.

The correct procedure is to get the data from the MAILSTREAM.  For 
example, in your case of getting:
 	(MESSAGES RECENT UNSEEN UIDNEXT UIDVALIDITY)
you should use
 	stream->nmsgs
 	stream->recent
 	either do a mail_search() for unseen and count the number of
 	 responses coming back, or if all message metadata is in
 	 c-client's cache just count the number of messages which have
 	 !mail_elt(stream,i)->seen, for i=1 to nmsgs.
 	stream->uid_last+1
 	stream->uid_validity

The simplest way to get the RECENT count is:
 	mail_fetch_fast (stream,"1:*",NIL);
 	for (i = 1, j = 0; i <= stream->nmsgs; ++i)
 	 if (!mail_elt (stream,i)->seen) ++j;
However that mail_fetch_fast() should only be done once in any session. 
If you've already gotten all the message metadata once you don't need to 
get it again.  IMAP automatically updates it for you.

A STATUS command forces the IMAP server to do its open mail_open() call 
get the data from the resulting MAILSTREAM, then close it.  But it already 
has done a mail_open() on that mailbox, as has your client.  Hence the 
waste.  The multiple opens can also cause other problems.

> Thanks very much. This certainly looks pretty confusing to me, but I' m just 
> getting started writing a client using c-client.

The message is obnoxious for a reason; it's to get the client author's 
attention and get him to ask "what's going on?" (and hopefully the 
explanation will convince him to do the right thing instead of abusing 
STATUS).  So, in this case, it worked as designed.

If there's anything unclear about any of the above, please ask.  I'm not 
sure if the above answer adequately explains "why" as opposed to "what".

Also, please don't feel bad; you have lots of company in having made this 
mistake.  You responded intelligently as well (ask and find out what's 
wrong); too many people just try to hide the message and never fix what 
their client is doing wrong...  :-(

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Mike Schmidt <[email-masked]>
Date: Sat, 16 Apr 2005 12:55:47 -0400

Thank you very much, Mark. This is exactly the information I was looking 
for. I am just learning to understand c-client, and, although I have 
read the various doc files many times, especially the internal.txt one, 
there are many aspects of the operational use of c-client that I do not 
yet understand. Your help is great, and I greatly appreciate your clear 
responses. Many of the commands I am issuing at this time are 
exploratory; since I don't yet understand the proper use of imap (I was 
only a user and server jock with respect to imapd), I am still 
exploring. I appreciate the easy callbacks that c-client provides for 
this kind of information because it means I can see (and display) all 
the imap command traffic in both directions. Hence the questions. I will 
continue asking questions because I am not capable of writing code I 
don't understand the why and wherefore of.  

I noticed there are a number of functions exposed in mail.h that are not 
documented in internal.txt. I don't want to bother you unnecessarily, 
but you may get questions about some of these functions from time to 
time, if they seem to be useful to my approach.

Now I have an additional quick question, more a design question than 
anything else: if I understand correctly, I should have 1 mailstream 
open to an imap server, not one per mailbox. I should reuse this 
mailstream when switching mailboxes instead of making a new one, is that 
correct? What happens to the message cache in this case?

Thanks for your quick response. You help is invaluable to me.

Mike

Mark Crispin wrote:

>> Can  someone tell me what the CLIENT BUG references mean in the 
>> status report?
>
>
> It means that you did a STATUS command on the selected (opened) 
> mailbox; a completely unnecessary and wasteful operation which could 
> also have additional severe negative consequences.
>
> When you have a mailbox selected, the IMAP state already has all the 
> data that a STATUS command could return.  The STATUS command is to the 
> be used only to get this data from a mailbox which is *NOT* selected.
>
> This has been shown to be an area which many novices fail to 
> understand; many seem to believe (incorrectly) that you have to do a 
> STATUS to get updated mailbox state.  Learning how IMAP really works 
> in this case is likely to help client authors understand IMAP in other 
> cases, hence this is a mistake that I felt is important to correct.
>
>> This is a transcript of the IMAP status operation. The client is 
>> based on c-client, running in Windows XP, and the server is uw-imapd 
>> running over a TLS connection (on Linux)
>
>
> call on a mailbox name when you already had a perfectly good 
> MAILSTREAM with all the data you need from a mail_open() call.
>
> However that mail_fetch_fast() should only be done once in any 
> session. If you've already gotten all the message metadata once you 
> don't need to get it again.  IMAP automatically updates it for you.
>
> A STATUS command forces the IMAP server to do its open mail_open() 
> call get the data from the resulting MAILSTREAM, then close it.  But 
> it already has done a mail_open() on that mailbox, as has your 
> client.  Hence the waste.  The multiple opens can also cause other 
> problems.
>
>> Thanks very much. This certainly looks pretty confusing to me, but I' 
>> m just getting started writing a client using c-client.
>
>
> If there's anything unclear about any of the above, please ask.  I'm 
> not sure if the above answer adequately explains "why" as opposed to 
> "what".
>
> Also, please don't feel bad; you have lots of company in having made 
> this mistake.  You responded intelligently as well (ask and find out 
> what's wrong); too many people just try to hide the message and never 
> fix what their client is doing wrong...  :-(
>
> -- Mark --
>

---

From: Mark Crispin <[email-masked]>
Date: Sat, 16 Apr 2005 10:21:28 -0700

On Sat, 16 Apr 2005, Mike Schmidt wrote:
> Now I have an additional quick question, more a design question than anything 
> else: if I understand correctly, I should have 1 mailstream open to an imap 
> server, not one per mailbox. I should reuse this mailstream when switching 
> mailboxes instead of making a new one, is that correct? What happens to the 
> message cache in this case?

It's a bit more complicated than that.

You should have one MAILSTREAM open for each mailbox which you want to be 
open *simultaneously*.

If you want to "switch" from one open mailbox to another, it is better to 
reuse the mailstream than to close the existing mailstream and open a new 
mailstream on the new name.

With any of these operations, there are certain costs:

Opening a new connection has the cost of TCP/IP session initiation, 
encryption negotiation, and authentication negotiation.

Selecting a mailbox has the cost of server overhead to load/process the 
mailbox, and the protocol overload of loading the client cache.

Closing a mailstream has the cost of discarding the message cache for the 
old name and closing the TCP/IP session.

Opening a new mailstream has both the "opening a new connection" and the 
"selecting a mailbox" cost.

Recycling a mailstream has the cost of discarding the message 
cache for the old name and "selecting a mailbox", but avoids the costs of 
closing the TCP/IP session and "opening a new connection".

Keeping multiple sessions open allows you to monitor multiple mailboxes 
simultaneously, but there is a practical limit of how many mailboxes that 
you should have open at a time, no more than a handful.  You want to do 
this if you want immediate real-time access to the mailbox and its 
messages, as opposed to passive monitoring.

-- Mark --

---

From: Mike Schmidt <[email-masked]>
Date: Sat, 16 Apr 2005 13:44:18 -0400

Thanks Mark.

When I look at the server logs for Thunderbird or Outlook, I never see 
more that one IMAP logon at a time. That also goes for all the users on 
our production server, although they are nearly all either Outlook or 
Thunderbird, with maybe an occasional Eudora thrown in for good measure. 
In my own case, on one account, I have maybe 30 or more mail folders, on 
another maybe 6 or 7. Yet I never see these  IMAP clients logging on 
more than once simultaneously.  Now, they don't use c-client  as far as 
I can tell.  Does this mean they are operating in a different  way ?  
Something roughly equivalent to 1 connection, multiple mailstreams? For 
the moment, I expect to use just an inbox and a sent folder, so I only 
have two mailboxes to monitor/operate, and that may not even be 
simultaneously. But of course I'm just getting started. I 'd rather not 
be obliged to re-design my system later, so I need to at least consider 
the implications. If this becomes an issue, is it 
possible/useful/interesting to separate the connection from the 
mailstream? Or is uw-imap effective in this environment? (BTW, I have no 
intention of changing my mail server under any circumstances) unless 
forced to, and that would be kicking and screaming.

Mark Crispin wrote:

>> mailstream when switching mailboxes instead of making a new one, is 
>> that correct? What happens to the message cache in this case?
>
>
> You should have one MAILSTREAM open for each mailbox which you want to 
> be open *simultaneously*.
>
> If you want to "switch" from one open mailbox to another, it is better 
> to reuse the mailstream than to close the existing mailstream and open 
> a new mailstream on the new name.
>
> Selecting a mailbox has the cost of server overhead to load/process 
> the mailbox, and the protocol overload of loading the client cache.
>
> Closing a mailstream has the cost of discarding the message cache for 
> the old name and closing the TCP/IP session.
>
> Opening a new mailstream has both the "opening a new connection" and 
> the "selecting a mailbox" cost.
>
> Recycling a mailstream has the cost of discarding the message cache 
> for the old name and "selecting a mailbox", but avoids the costs of 
> Keeping multiple sessions open allows you to monitor multiple 
> mailboxes simultaneously, but there is a practical limit of how many 
> mailboxes that you should have open at a time, no more than a 
> handful.  You want to do this if you want immediate real-time access 
> to the mailbox and its messages, as opposed to passive monitoring.
>
> -- Mark --
>

---

From: Pawel Salek <[email-masked]>
Date: Sat, 16 Apr 2005 18:08:35 +0000

On 04/16/2005 07:44:18 PM, Mike Schmidt wrote:
> Thanks Mark.
> 
> When I look at the server logs for Thunderbird or Outlook, I never  
> see more that one IMAP logon at a time. That also goes for all the  
> users on our production server, although they are nearly all either  
> Outlook or Thunderbird, with maybe an occasional Eudora thrown in for  
> good measure. In my own case, on one account, I have maybe 30 or more  
> mail folders, on another maybe 6 or 7. Yet I never see these  IMAP  
> clients logging on more than once simultaneously.  Now, they don't  
> use c-client  as far as I can tell.  Does this mean they are  
> operating in a different  way ?  Something roughly equivalent to 1  
> connection, multiple mailstreams?

There are many clients that open several concurrent connections.  
Mozilla/Thunderbird allows limiting number of concurrent IMAP  
connections (Server Settings/Advanced) and often uses just one by  
chosing to SELECT mailboxes frequently and doing extensive client-side  
caching. I guess the reason for this kind of model was that mozilla was  
designed to work efficiently in offline imap mode which obviously is  
not able to take advantage of many concurrent connections anyway. As  
Mark wrote, it's a tradoff. If you have relatively short-lived  
connections or your server's has some database-based backend with very  
fast SELECT operation, opening one/very few connections may be a good  
idea. If you need to interactively watch several mailboxes for  
modifications, keeping separate connections open may be better. (BTW,  
some imap servers allegedly limit number of concurrent connections for  
the same user).

> For the moment, I expect to use
> just an inbox and a sent folder, so I only have two mailboxes to  
> monitor/operate, and that may not even be simultaneously. But of  
> course I'm just getting started. I 'd rather not be obliged to  
> re-design my system later, so I need to at least consider the  
> implications. If this becomes an issue, is it  
> mailstream?

IIRC, each c-client mailstream opens a separate connection.

Pawel

> Or is uw-
> imap effective in this environment? (BTW, I have no intention of  
> changing my mail server under any circumstances) unless forced to,  
> and that would be kicking and screaming.
> 
> Mark Crispin wrote:
> 
>>> Now I have an additional quick question, more a design question  
>>> than anything else: if I understand correctly, I should have 1  
>>> mailstream open to an imap server, not one per mailbox. I should  
>>> reuse this mailstream when switching mailboxes instead of making a  
>>> new one, is that correct? What happens to the message cache in this  
>>> case?
>> 
>> 
>> You should have one MAILSTREAM open for each mailbox which you want  
>> to be open *simultaneously*.
>> 
>> If you want to "switch" from one open mailbox to another, it is  
>> better to reuse the mailstream than to close the existing mailstream  
>> and open a new mailstream on the new name.
>> 
>> Closing a mailstream has the cost of discarding the message cache  
>> for the old name and closing the TCP/IP session.
>>