Subject: [Imap-uw] Instantaneous Notification of Received Mail

From: "Aaron W. LaFramboise" <[email-masked]>
Date: Sun, 21 Aug 2005 21:13:15 -0500

I am seeking advice on how to cause my IMAP client to give me 
instantaneous (or as fast as possible) notification of newly received email.

Firstly, let me sidestep the philosophical issue.  I understand that 
instant email delivery can be a major time-waster in the modern 
workflow.  However, for a select few of my correspondents, such as 
family and close friends, I would prefer that I receive communications 
from them immediately, as the email may contain time-critical 
information, and they may be unable to reach me otherwise.  I hope this 
rationale legitimizes my request.

I receive about a thousand emails a day, which is mostly mailing list 
traffic, spam, and other bulk email.  Of this, I need to read about a 
dozen messages, and perhaps briefly scan a few dozen more.  I am able to 
do this without wasting huge amounts of time due to extensive mail 
filters and fast threading features in my mail reader.

Due to the nature of IMAP, filtering needs to be performed 
server-side—otherwise a heterogeneous and error-prone system of 
client-side filters would need to be deployed at each location I read 
mail for each client to benefit from filtering.

Unfortunately, I have almost 100 different folders, many which have 
upwards of 30K messages in them.  As I understand it, the ordinary IMAP 
notification mechanism is only able to receive notifications from a 
single selected folder.

So, as far as I know, this rules out automatic notification from the 
IMAP server.  Polling is also unacceptable due to the large size of many 
of the mailboxes, the largest of which takes UW IMAP more than a minute 
to select, even in MBX format.

Is this a common problem?  If so, how is it handled?  If not, why not?

The best hypothetical solution I have thought of so far would involve a 
hackish alteration to the mailbox driver.  When a mailbox was selected, 
a cache file containing vital information about the mailbox could be 
created.  When the mailbox was selected again, the information could be 
quickly regurgitated from the cache without even opening the mailbox. 
When the mailbox was written to, the cache file would be deleted.  This 
would allow very rapid polling, since dispatching each poll request 
would take a negligible amount of time.

Has anything like this been tried?  In fact, I am puzzled as to why the 
current preferred formats do not maintain some sort of rapid-lookup 
index that would be a superset of such caching functionality.

My setup is presently composed of a Mozilla Thunderbird 0.7.3 client 
running on Windows XP SP2 on a Pentium 4, connecting via IMAP to UW IMAP 
2004e running on Linux 2.6 on a Pentium.  The mailboxes are in MBX 
format, on a local ReiserFS filesystem.  sendmail delivers email which 
is filtered by procmail and sent to mailboxes with dmail.

Aaron W. LaFramboise

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Sun, 21 Aug 2005 19:42:55 -0700 (PDT)

Aaron -

Your needs are common ones.  Unfortunately, for the most part, they remain 
unfufilled needs; but help is on the way (slowly...).

One realization that took a long time coming was that the proper place to 
perform notification is not in message *access* (which is what IMAP does), 
but rather in message *delivery*.

On many systems, the mail delivery software has no relationship to the 
POP or IMAP server software.  Often, the software is from entirely 
different vendors.  Since the IMAP server software is often not part of 
the message delivery process, it has no good way of knowing that new 
messages have been delivered other than by polling.

As you have noted, client-initiated polling is not cheap.  It is not 
substantially cheaper by specifying that IMAP should notify, and giving it 
no other way than to do the work than by polling (albeit under the covers 
where the client doesn't see it happening).  Not surprisingly, IMAP server 
developers who face this problem are not enthusiastic about IMAP being 
given that mandate.

This does not necessary mean that there could not be some facility in IMAP 
to request a notification, and perhaps even to receive the notification 
and pass it on to the client.

What it means is that there needs to be a new service, which the 
developers of mail delivery software will feel encouraged to implement. 
Roughly, this new service would work as follows.  When a message is 
delivered, the facts of the delivery are passed to the notification 
service which determines if there are any requests to be notified of such 
a message (you don't want to spam someone with notification if he hasn't 
asked for them!).  If there is such a request, the notification service 
transmits a notification to the requesting entity.

An IMAP client could be the "requesting entity".  Then again, the 
middleman could be eliminated by having the user's application request the 
notification itself (maybe you want notifications but don't want to keep 
your IMAP client running all the time).  Personally, I feel that it is 
best to have IMAP out of the loop, but I'm not adamant on that point.

Of course, in your environment, you're using dmail so all the code is 
under my control.  However, that would only provide for notifications for 
people who use both dmail and UW imapd.  Ideally, we would have a standard 
that would not require a single vendor's (even if it is UW) solution.

As you noticed, none of the current mailbox formats used by UW imapd are 
particular ameniable to doing poll-type notification.  At least in part 
this is due to legacy.  However, with delivery-initiated notifications, 
there is no need to do polling or to have special format considerations 
for notification.

I actually developed a prototype version of a protocol and software that 
did this, but that project got shelved.  More recently, the IETF Lemonade 
Working Group has considered the question of notification, and a hack (RFC 
4146) to do this has actually been published as an RFC.  They are actually 
chartered to do this work, see item 4 of:
 	http://www.ietf.org/html.charters/lemonade-charter.html

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: "Aaron W. LaFramboise" <[email-masked]>
Date: Mon, 22 Aug 2005 00:22:52 -0500

> there is no need to do polling or to have special format considerations 

I still think it would be very useful to have support for a state where 
results are returned very quickly (near constant time with respect to 
number of messages or size of mailbox) in the case that 'nothing has 
changed since last time.'  Even with a notification mechanism, 
wholescale synchronization must happen every now and then, such as 
whenever the mail client is started.

For users like myself who have many mail folders that may not have 
changed since the last access, it would be nice if this initial 
synchronization could happen quickly, rather than the 3-4 minutes it 
takes me.  (In my case, this time is solely spent reading MBX files; 
there is no appreciable network latency or bandwidth limitations.)

> did this, but that project got shelved.  More recently, the IETF 
> Lemonade Working Group has considered the question of notification, and 
> a hack (RFC 4146) to do this has actually been published as an RFC.  
> They are actually chartered to do this work, see item 4 of:

I suppose there is also the classical biff/comsat mechanism.  I guess my 
next step is to see what it takes to get Mozilla to accept new mail 
notifications from external sources (other than polling).  It seems like 
it would be easy enough to have procmail, or whatever, generate a 
notification of some sort when it delivers mail, before passing off to 
dmail.

Aaron W. LaFramboise

---

From: Mark Crispin <[email-masked]>
Date: Mon, 22 Aug 2005 09:05:39 -0700 (Pacific Daylight Time)

On Mon, 22 Aug 2005, Aaron W. LaFramboise wrote:
> results are returned very quickly (near constant time with respect to number 
> of messages or size of mailbox) in the case that 'nothing has changed since 
> last time.'  Even with a notification mechanism, wholescale synchronization 
> must happen every now and then, such as whenever the mail client is started.

This is already being addressed by other portions of the IMAP protocol: 
the \Marked and \UnMarked mailbox flags, the STATUS command, and the 
proposed CONDSTORE extension.

It's important to recognize that there is no disagreement over the 
desirability of such capabilities.  It's a question of what is feasible to 
accomplish, given a number of technical and non-technical constraints 
(some of which are not always obvious).

> For users like myself who have many mail folders that may not have changed 
> since the last access, it would be nice if this initial synchronization could 
> happen quickly, rather than the 3-4 minutes it takes me.  (In my case, this 
> time is solely spent reading MBX files; there is no appreciable network 
> latency or bandwidth limitations.)

The mbx format, while a substantial improvement over traditional Unix 
format, has likely reached the limits of its scalability in today's world. 
You are correct that a new format is needed, one which has open time that 
is not dependent upon the size of the file (as traditional Unix format is) 
or the number of messages (as mbx format it).

> I suppose there is also the classical biff/comsat mechanism.  I guess my next 
> step is to see what it takes to get Mozilla to accept new mail notifications 
> from external sources (other than polling).  It seems like it would be easy 
> enough to have procmail, or whatever, generate a notification of some sort 
> when it delivers mail, before passing off to dmail.

This is indeed the right direction.  Your message has encouraged me to 
dust off my work from three years ago and call it to the attention of the 
IETF Lemonade Working Group which is working on this very issue (but is 
still in discussion stage on it).

Thank you for getting the ball rolling again, and please feel free to send 
reminders to nudge it along periodically.

-- Mark --