Subject: Re: IMAP bug: incoming mail with "Status: D" or "X-Status: D" is lost forever!

From: Jonathan Kamens <[email-masked]>
Date: Thu, 9 Aug 2001 23:20:14 -0400

(I have been informed that I started this discussion on the wrong
mailing list -- I thought that this list was for discussion of the
wu-imap package, when in fact that's the "c-client" list.  Therefore,
this will be my last message to this list about this topic; any
further responses I send will be directed only to the "c-client"
list.  I apologize for bothering those of you who don't care about the
uw-imap package.)

>  Date: Fri, 10 Aug 2001 00:04:54 +0100 (GMT Daylight Time)
>  From: Mark Crispin <[email-masked]>
>  
>  > Why would someone examine
>  > the deleted messages in his mailbox to determine if someone else just
>  > happened to accidentally forward a message to him with "Status: D" or
>  > "X-Status: D" in its header?
>  
>  If a user finds a deleted message in his mailbox when he opens it, it is
>  generally because he made the decision not to expunge it in the previous
>  session.  One would assume that a user would make a note of such messages,
>  particularly ones that aren't remembered.

Um, that may be how you use your E-mail client, but there are many,
many people in the world who do not expunge after every session --
they only expunge manually.  They would never notice messages marked
"deleted" accidentally by the sender.

>  > While "Status" is arguable, "X-Status" is not, because "X-Status" is a
>  > perfectly valid RFC 822 header.  Software which does not handle
>  > compliant messages is by definition broken.
>  
>  You got it backward;  the use of X-Status is absolutely
>  standards-compliant, and the use of Status is questionable.  You are
>  welcome to launch a campaign to get every MUA on every UNIX system in the
>  world changed so that they do not use Status:.

No, YOU'VE got it backwards.

The question is whether it is legal for an SMTP message on the wire to
contain a "Status" or "X-Status" field.  "Status" is illegal since it
isn't specified anywhere and applications aren't supposed to add
unspecified fields without "X-" (nevertheless, MTAs and MUAs are
supposed to try to cope well with such nonstandard headers, and most
of them do).  "X-Status", on the other hand, is perfectly legal
because all headers starting with "X-" are by definition legal.

The use of "Status" or "X-Status" in the header of a message in a Unix
mailbox file is completely orthogonal to the question of what the
header of the message is allowed to contain before it arrives at the
site.  You can't take the standard which says that "X-Status" is legal
in SMTP messages and claim that that proves that it's legal for
wu-imap to use "X-Status" in Unix mailbox headers.  In fact, it proves
exactly the opposite -- since "X-Status" is legal in SMTP messages,
wu-imap needs to do something special to ensure that it doesn't break
such legal SMTP headers.

>  > Finally, I'm curious if you can justify your assertion that "Status"
>  > and "X-Status" "have long use" in the Unix mailbox format.
>  
>  The use of Status: on UNIX definitely preceeds RFC 822, and probably also
>  RFC 733.  X-Status: was in use in the late 1980s.

Any proof?

>  RFC [2]822 (not SMTP) says nothing whatsoever about local interpretation
>  of messages.  It refers only to transmission of messages on the wire.
>  [2]822 is quite explicit that the interpretation of X- headers is
>  application specific.

You're trying to use RFC 2822 for a purpose for which it specifically
is not intended.  Some excerpts (emphasis mine):

   This standard specifies a syntax for text messages that are sent
   BETWEEN COMPUTER USERS, within the framework of "electronic mail"
   messages. ...

   This specification is intended as a definition of what message
   content format is to be passed between systems.  Though some message
   systems locally store messages in this format (which eliminates the
   need for translation between formats) and others use formats that
   differ from the one specified in this standard, LOCAL STORAGE IS
   OUTSIDE OF THE SCOPE OF THIS STANDARD.

   Note: THIS STANDARD IS NOT INTENDED TO DICTATE THE INTERNAL FORMATS
   USED BY SITES, the specific message system features that they are
   expected to support, or any of the characteristics of user interface
   programs that create or read messages.  In addition, this standard
   does not specify an encoding of the characters for either transport
   or storage; that is, it does not specify the number of bits used or
   how those bits are specifically transferred over the wire or stored
   on disk.

You're claiming that it's legal for wu-imap to use X-Status as magic
because RFC 2822 says X- headers are legal.  That's wrong, because
RFC 2822 says nothing about internal storage formats, which is what
wu-imap's usage of Status and X-Status is.  What RFC 2822 does say is
that X-Status is a legal header in the "Internet Message Format" and
that therefore, software which manipulates "Internet Messages" has to
deal properly with it.

>  In circumstances where it is plain that somebody is playing around, yes.

I lost three E-mail messages since July 27 because of this bug.  I
have no idea how many E-mail messages I've lost in the past because of
it.  The people who sent me those E-mail messages were not "playing
around"; they were simply unaware that using a header called
"X-Status" would cause problems.

This problem could be tickled simply by someone who doesn't know
better writing software that uses "Status" or "X-Status" for its own
purposes; are you claiming that everyone in the world knows as much
about the standards as you (think you) do?

This problem could be tickled simply by someone who believes that it's
legitimate to use a mail header called "X-Status" in their software,
since after all, the standards stay it's legal and it's a pretty
logical name to use.

And this problem could be tickled simply by someone resending a
message which has in it a "Status" or "X-Status" header introduced by
wu-imap.

Just for kicks, I went through my E-mail archives (over 89,000
messages extending back over 12 years) and tried to analyze them to
see if there were *other* messages I would have missed over the years
if I'd been reading my mail through ipop3d.  This was a difficult
exercise because I have no way of telling for how much of that time my
*own* software was inserting the "Status:" header into the messages I
put into my archive.  However, using various heuristics to eliminate
those, I found at least 34 messages which almost certainly had this
problem, and there are certainly more than that which I had to omit
due to uncertainty.  In short, this problem *does* happen.

>  The only messages that are dropped are messages which contain the
>  indicator of "deleted" used by several MUAs.  There is no legitimate
>  reason to send a message with such an indicator.

Hogwash.  "I didn't know it would cause trouble and it was useful to
me," "The standards say it shouldn't cause trouble, so why shouldn't I
use it?" and, "I did it by accident," are all "legitimate reasons" for
a message to contain such an indicator.

>  If you want to scream, rave, and pull hair at various MUAs,

I don't.  I want wu-imap to (a) accept RFC-compliant E-mail messages
without dropping them on the floor and (b) operate on the principle of
"be generous in what you accept and conservative in what you
generate," which is a far more accepted standard for Internet software
than your "GIGO".

Even if you're right that binmail, procmail, etc. should escape this
field and the other fields that are magic to wu-imap, wu-imap itself
should still cope well when they don't.  There's that "be generous in
what you accept" thing again.

>  It is the only practical solution if you choose to stay with the
>  traditional UNIX mailbox format.

Here are some other "practical solutions":

1) Store this information somewhere other than in the mailbox.  This
is not impossible.  Other MTA's (e.g., www.courier-mta.org) do it.

2) Store this information in the magic first message in the mailbox,
rather than in the header of individual messages.

3) Store a hash on the X-UID (using a key that is stored in the
mailbox header), and ignore magic fields (Status, X-Status,
X-Keywords, etc.) on messages which do not contain an X-UID field with
a valid hash.

4) Modify ipop3d to allow the user to access deleted messages just
like undeleted messages, until they are actually deleted during that
POP3 session.  Even if the user set the flag (which is unlikely, since
users do not frequently switch between POP3 and IMAP), if he didn't
expunge the messages from the mailbox during the IMAP session, he
presumably didn't want to lose the message, so expunging it when the
POP3 session is closed is the wrong behavior.  Granted, this is only a
partial solution, since it doesn't deal with IMAP users who get sent
deleted messages and never notice them, but it's better than the
current behavior of the messages disappearing completely without the
user ever seeing them.

If I were to submit patches to implement one of these other solutions,
would you accept them?

  Jonathan Kamens