Subject: HELP: How can a IMAP4 server inform a client of a changing message in a Mobile environment

From: Surendar Chandra <[email-masked]>
Date: Thu, 29 Aug 1996 02:56:36 -0400

Hi, 
 I am implementing a IMAP4 server in a mobile environment, where the
server can drop certain messages that are locally cached and served on
the fly. I would appreciate your help in a problem I'm facing of the
server notifying this correctly to clients.

When the server drops messages, the server notifies the clients using
unilateral messages. For example, suppose the client transaction starts
of with 15 message and this later drops to 4 messages. The server would
behave as follows:
============<>==============
* 15 EXISTS
* OK [UIDVALIDITY 839563907] UID validity status
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS ()] Permanent flags
* OK [READ-ONLY] Mailbox status
* 0 RECENT
* 15 EXISTS
A01 NOOP
* OK [UIDVALIDITY 553076] UID validity status
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS ()] Permanent flags
* OK [UNSEEN 1] 1 is first unseen
* 0 RECENT
* 4 EXISTS
* 0 RECENT
* OK NOOP completed
==============<>===============
I expect the clients, once they see the new EXISTS message to dump
whatever they have cached and refetch new headers from the server.
However, most of the clients I've tested abort as a FATAL condition.
Here is what I notice:
* Pine
I was in the header menu and after the server had sent the above
message, I tried to use the uparrow key and pine noted.
	Problem detected: "Bad msgno in mail_fetchstructure".
	Pine Exiting.

* MMAIL: 
  Similar response. Quietly quits after closing all windows.

* Solstice:
 	?FATAL ERROR IN C_CLIENT, Bad msgno in mail_fetchstructure_full
	Abort
To avoid the abort, I could choose "Check for new mail", then "View
All", at which time it updates the screen to display just the four
headers insteam of 15 so that I cannot access those unavailable headers.

My question is, how do I do this correctly, ie, how can the server tell
the clients that there has been a catastrophic change in the message
counts and that the client must refetch whatever cached messages
structures that they have. I would prefer a portable solution if
possible as I would prefer not to munge with pine or other clients to
acheive this. BTW, is my usage a proper usage of the IMAP4 protocol?

Thanks much for any help,
-S

---

From: Mark Crispin <[email-masked]>
Date: Thu, 29 Aug 1996 00:01:40 -0700 (PDT)

The only way that the number of messages in the mailbox can decrease is if you
send untagged EXPUNGE responses for each message that is removed.  An untagged
EXISTS response can only increase the number of messages in the mailbox.

Thus, your scenario could be:

A00 SELECT FOO
* 15 EXISTS
A00 OK Select done
A01 NOOP
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 4 EXISTS
* OK NOOP completed

It would, however, be better to figure out which messages were not dropped and
refrain from sending an untagged EXPUNGE for those messages.

---

From: [email-masked]
Date: Fri, 30 Aug 1996 03:24:19 -0400 (EDT)

I tried doing untagged EXPUNGE, both counting up and counting down and
the clients (Sun Solstice/Pine) seem to ignore it. Even if they do
accept an EXPUNGE, seems to be that EXPUNGE cannot be sent when no
command is in progressl nor while responding to a FETCH, STORE, or
SEARCH command. This restricts the window of opportunity for the
server.

Imagine that the server decides to drop some message. As long as the
client is not doing a CHECK, the server cannot notify the client. When
the client does a request, the server has to wait for a CHECK or NOOP
or some such operation to force EXPUNGE. When the client requests the
body of this expunged message and if the client is not notified and the 
if server responds with a NIL for body[1], [* 5 FETCH (BODY[1] NIL)]
the clients dump core . Please note that I am sending unsolicited
'EXISTS' messages whenever the server drops/adds messages.

What would be a good protocol for the server to continously update the
clients on the current message availability and force the clients to
always ask the server rather than make local cached decisions(or, how
can I always keep the client cache uptodate)

Thanks much for all your help,
-S

: The only way that the number of messages in the mailbox can decrease is if you
: send untagged EXPUNGE responses for each message that is removed.  An untagged
: EXISTS response can only increase the number of messages in the mailbox.
: 
: Thus, your scenario could be:
: 
: A00 SELECT FOO
: * 15 EXISTS
: * OK [UIDVALIDITY 839563907] UID validity status
: * FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
: * OK [PERMANENTFLAGS ()] Permanent flags
: * OK [READ-ONLY] Mailbox status
: * 0 RECENT
: A00 OK Select done
: A01 NOOP
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 1 EXPUNGE
: * 4 EXISTS
: * OK NOOP completed
: 
: It would, however, be better to figure out which messages were not dropped and
: refrain from sending an untagged EXPUNGE for those messages.

---

From: Mark Crispin <[email-masked]>
Date: Fri, 30 Aug 1996 00:27:18 -0700 (PDT)

I'm not sure what version of Pine you tried.  The current version of Pine is
3.95, and it is supposed to handle untagged EXPUNGE even when an EXPUNGE
command was not done.

Ideally, the server should not allow the message data to be deleted until
after it has issued the untagged EXPUNGE.  If you really must return empty
message data, try returning "" instead of NIL.

---

From: [email-masked]
Date: Sat, 31 Aug 1996 02:54:13 -0400 (EDT)

: I'm not sure what version of Pine you tried.  The current version of Pine is
: 3.95, and it is supposed to handle untagged EXPUNGE even when an EXPUNGE
: command was not done.
I was using 3.93. Now, I have downloaded 3.95 and it does the same thing.
Here are excerts from the debug file. I could mail you the whole debug
file if you want. Basically, I am in this mail folder, and there are 4
messages in it. Then the number of messages increases to 16 at which
time the server sends 4 EXPUNGES. Pine works fine. Now the number of
messages drops to 4 again. Server sends 16 EXPUNGES. I am still using 
down keys in the header window when pine quits with 
	Pine Panic: Bad msgno in mail_fetchstructure

============DEBUG LOG==========
IMAP DEBUG 02:42:35 8/31: * 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 1 EXPUNGE
* 4 EXISTS
mm_expunged called {xos.cs.duke.edu}#jail/INBOX 1
* OK [UIDVALIDITY 553076] UID validity status
IMAP DEBUG 02:42:35 8/31: * FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
IMAP DEBUG 02:42:35 8/31: * OK [PERMANENTFLAGS ()] Permanent flags
IMAP DEBUG 02:42:35 8/31: * OK [READ-WRITE] Mailbox status
IMAP DEBUG 02:42:35 8/31: * OK [UNSEEN 1] 1 is first unseen
IMAP DEBUG 02:42:35 8/31: * 0 RECENT
IMAP DEBUG 02:42:35 8/31: * 4 EXISTS
IMAP DEBUG 02:42:35 8/31: * 0 RECENT
IMAP DEBUG 02:42:35 8/31: * 0 RECENT
IMAP DEBUG 02:42:35 8/31: A00008 OK NOOP completed
Ping complete: Sat Aug 31 02:42:35 1996

New mail, inbox,  new_mail_count:0  expunge count:2,  max_msgno:16
******** new mail returning 2  ********
set_titlebar - style: 1  current message cnt:1  current_pl: 0
total_pl: 0
end_screen called
about to end_tty_driver
end_signals(1)
Pine Panic: Bad msgno in mail_fetchstructure
============DEBUG LOG==========

: Ideally, the server should not allow the message data to be deleted until
: after it has issued the untagged EXPUNGE.  If you really must return empty
: message data, try returning "" instead of NIL.
True, the server does not expunge the messages till it had sent the
EXPUNGE. But, even if it did send an EXPUNGE and the client asks for
an EXPUNGED message, I had modified the server to send a NIL, but the
client never sends any message to the server, it just PANIC's by
itself.

Thanks much for all your help, I really appreciate it.
-S

---

From: [email-masked]
Date: Mon, 2 Sep 1996 03:34:23 -0400 (EDT)

Oops, please ignore my previous email. Pine does work correctly,
except I was printf'g "* 1 EXPUNGE\n", instead of "* 1 EXPUNGE\015\012", 

Sorry for the trouble. It works fine now.

Thanks much,
-S