Subject: proposed IMAP protocol enhancement

From: Mark Crispin <[email-masked]>
Date: Sun, 18 Apr 1993 19:18:29 -0700 (PDT)

I, perhaps more than anyone else, want to put a lid on further IMAP additions
in the name of stability and getting a Proposed Standard out of this.
However, something has come out that has survived even my harsh filters, and
I'm throwing it out to the list to be blessed (or damned, as the case may be).

The proposed change is a new form of FETCH which allows the selecting fetching
of message headers.  The problem to be solved is that RFC-822 certain header
lines are not represented in the IMAP envelope structure, nor is there any
reasonable way to extend the envelope structure to accomodate them.  It is
considered mandatory that any extension be upwards/downwards compatible and
not require revisiting the next time we need to be able to snarf another
header.

The most obvious example is the ReSent-Date/ReSent-From/ReSent-To header
lines.  There is an additional problem with these particular header lines in
that they can not be arbitrarily reordered and retain the same meaning.

Other headers, such as Newsgroups:, are also perceived as interesting.

The proposed two new functions are a ``fetch all header lines of this message
whose field names match any in this list'' and ``fetch all header lines of
this message whose names do not match any in this list''.  They take an
argument which consists of a list of the field names.  The result of these
functions is a text string similar to that from RFC822.HEADER.  All header
lines are returned in the original order of the message (note this is a
requirement for useful ReSent information).

For example (note that line breaks are here only for clarity):

A001 FETCH 23:30 (ENVELOPE RFC822.HEADER.LINES ("Resent-Date" "Resent-From"
		 "Resent-To")

would fetch the envelopes and remail information for messages 23 to 30.

A001 FETCH 23 RFC822.HEADER.LINES.NOT ("Return-Path" "Received")

would fetch the header of message 23 without the ``Return-Path'' or
``Received'' header lines.

This would become a fundamental API call in c-client, and c-client would
simulate its results if it finds itself talking to an IMAP server that does
not support it.

Comments please.

---

From: Laurence Lundblade <[email-masked]>
Date: Sun, 18 Apr 1993 23:10:39 -0400 (EDT)

I'd welcome the addition of these to IMAP. I've got a few questions:

Would the functions just return plain unparsed strings? If so, then the
Resent-xxxx: cases would require the client to do address parsing. This
isn't a huge problem since the routines are there in the client anyway.

It sounds like you have a choice of requesting the header lines that you
want, one at a time, or parsing a big string that comes back. The problem
with requesting the lines one at a time would be an RTT for each one,
right? 

I'm also having trouble coming up with a use for the LINES.NOT function.
Did you have something in mind? That's not to say it should be take out. 

I'm interested in the References: field for threading news groups and other
mail folders. I think you need it because the full tree of In-Reply-To:'s
might not be in the mailbox being threaded.

Laurence Lundblade
   [email-masked] or [email-masked] (forwarded to same place)
      Blacksburg, Virginia or Seattle, Washington

---

From: Mark Crispin <[email-masked]>
Date: Sun, 18 Apr 1993 22:00:17 -0700 (PDT)

> isn't a huge problem since the routines are there in the client anyway.

Yes, just plain unparsed strings would be returned.  I don't particularly
understand why you would want to address-parse the ReSent-* strings (other
than perhaps to canonicalize their format), but as you point out the routines
you need are in c-client anyway.

> with requesting the lines one at a time would be an RTT for each one,

Yes, that's correct.  The real intent is to be able to gobble down the useful
header lines in addition to what the envelope gives to you and possibly just
blat them to the screen without any processing.

> Did you have something in mind? That's not to say it should be take out.

It's in there primarily for symmetry.  I don't think Pine will need it, but
I'm fairly confident that if I don't put it in now, someone will be nagging me
for it later!  :-)  Also, it was a basic function in MM's header filters, so
there is some precedent for its use.

> might not be in the mailbox being threaded.

I would be delighted if Pine solved the threading problem this way!  ;-)  Yes,
enabling this sort of thing without having to go and change IMAP again was one
of the motivations.  It doesn't rescue me from someday having to deal with
cross-post suppression in the .newsrc on the server case though.  :-(

-- Mark --

---

From: John Gardiner Myers <[email-masked]>
Date: Mon, 19 Apr 1993 13:46:00 -0400 (EDT)

I strongly support the FETCH RFC822.HEADER.LINES parameter.

I don't see the point of FETCH RFC822.HEADER.LINES.NOT.  It could be
helpful to a client that wants to avoid presenting "uninteresting"
headers like "Received:" to a user, but any client that wants to
provide this feature is going to have to deal with a server's not
providing that parameter anyway.  I suppose it comes down to a
question of whether the bandwidth saved would justify the cost of the
additional parameter.

Would it be possible to consider adding an analogous "HEADER name
string" SEARCH criteria without opening the entire Pandora's box that
is associated with SEARCH?  This extension would help greatly in
implementing kill files.

-- 
_.John G. Myers		Internet: [email-masked]
			LoseNet:  ...!seismo!ihnp4!wiscvm.wisc.edu!give!up

---

From: Mark Crispin <[email-masked]>
Date: Mon, 19 Apr 1993 13:53:26 -0700 (PDT)

On Mon, 19 Apr 1993 13:46:00 -0400 (EDT), John Gardiner Myers wrote:
> I don't see the point of FETCH RFC822.HEADER.LINES.NOT.

My main concern is to avoid regretting having *not* put it in.  I think it is
rather trivial to do at the same time.  The same arguments against it could
also be applied against the RFC822.HEADER.LINES functionality as well, but
there is a definite groundswell of support for it (and especially for support
in c-client).

> is associated with SEARCH?  This extension would help greatly in

If you can come up with a way of doing this without opening Pandora's box on
searching, please feel free to suggest it.  I'm totally clueless!

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 20 Apr 1993 23:52:45 -0400 (EDT)

> you need are in c-client anyway.

OK, The reason I had in mind was canonicalization of their format for
presentation to the user -- it's a nice thing to keep all the addresses
looking similar.

> blat them to the screen without any processing.

I'm a little concerned about RTT's in a mailer that regularly fetches the
Resent-XXX:, References:  and other fields (presumably many if not most
good IMAP clients will do this). You're probably one to think about this
more than I, but wouldn't it at least double the number of RTT's for a lot
of the normal operations? Is doubling the RTT's a problem? I know there's
probably not much else that can be done without breaking existing IMAP
clients. 

Laurence Lundblade
  [email-masked] or [email-masked] (both forward to same place)
     Blacksburg, Virginia or  Seattle, Washington

---

From: Mark Crispin <[email-masked]>
Date: Fri, 23 Apr 1993 02:56:50 -0700 (PDT)

> probably not much else that can be done without breaking existing IMAP

That's a good point.  I think that when the API is done for this there should
be some sort of means to all for it to be fetched it along with something else
to avoid the extra RTT.  How about fetching it along with section 1 of the
body?

It'd be a new API function, no matter what.

---

From: Laurence Lundblade <[email-masked]>
Date: Fri, 23 Apr 1993 09:02:07 -0400 (EDT)

Yes, creating a new call in the API that fetches the extended header along
with the 1st body part seems fine. 

I was going to mention this last time, but I was also thinking about the
case when you're fetching envelopes.  If we do threading based on the
References: field (I'm still investigating whether or not that is the way
to go) then you'll want to fetch this extended header information with all
the envelopes.  Since you can fetch them all with one RTT it's still only
doubling, so I guess it's OK. 

I am a little worried about performance for threading. To do threading one
has to fetch the envelopes of all the messages in the mailbox, as well as
the References fields. 

While you're redesigning API's maybe an extended mail_fetchstructure could
get the Resent-xxx, References and Newsgroups fields. Those are the ones
for which we've got a clear need that I can think of.

LL

---

From: Jim Conklin <[email-masked]>
Date: Fri, 23 Apr 1993 09:47:35 -0400 (EDT)

Until a Mail Requirements group defines a real standard for list mail,
CREN's RFP for Internet list-management software proposes to use the
Resent- headers, so capturing those headers is of considerable importance to
us.  (It's available from info.cren.net as the files ip-listserv.txt,
ip-listserv.ps, or ip-listserv.rtf in the directory /cren-rfp, if you
haven't seen it and are interested.)
  Thanks,

Jim Conklin