Subject: tmail/mbx/imapd truncate message body

From: Malcolm Beattie <[email-masked]>
Date: Tue, 15 Sep 1998 15:07:29 +0100 (BST)

I'm using cclient/imapd 4.1.FINAL and tmail (from an imap-utils.tar.Z
distribution which had a last mod time of 22 July 1998) on Solaris
2.6. I'm using exim as the MTA and delivering messages with
"tmail username" to the MBX format file INBOX in each user's home
directory. This seemed to work fine on the face of it. However I now
find that imapd is "losing" the body of some messages. The body text
is definitely in there and mbxcvt generates a Berkeley format
equivalent which looks and works fine. The body loss occurs with
both Pine 4.03 as client and also my own cclient-based client but not
Netscape's IMAP client. Tracing system calls shows that the failing
clients issue the following IMAP commands:

00000005 FETCH 5 (BODYSTRUCTURE FLAGS)
00000006 FETCH 5 BODY.PEEK[HEADER.FIELDS (From Subject To Date Message-Id)]
00000007 FETCH 5 BODY[TEXT]

To test what was going on I ran mbxcvt to make a Berkeley format copy
of INBOX called foo. Talking to the IMAP server with telnet, I get:

2 select inbox

-- * 9 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 905337305] UID validity status
* OK [UIDNEXT 15] Predicted next UID
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags
* OK [UNSEEN 1] 1 is first unseen message in /x0/xzug/validate/INBOX
2 OK [READ-WRITE] SELECT completed
5 FETCH 5 (BODYSTRUCTURE FLAGS)
* 5 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "7BIT" 0 0 NIL ("INLINE" NIL) NIL) FLAGS (\Seen))
5 OK FETCH completed
6 fetch 4 (BODYSTRUCTURE FLAGS)
* 4 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "7BIT" 19 2 NIL NIL NIL) FLAGS (\Seen))
6 OK FETCH completed

Here, 4 is a message that does appear: note the NIL NIL rather than
("INLINE" NIL): the failing messages have headers
    Content-Type: text/plain
    Content-Disposition: inline
although a manually delivered one (via direct tmail) with such headers
seemed to work OK.

a1 fetch 5 BODY[TEXT]
* 5 FETCH (BODY[TEXT] "")
a1 OK FETCH completed

Note the body text is empty (my client does a fetchtext() to produce
this IMAP command).

If I now select the foo mailbox (supposedly the same contents, just in
Berkeley format), I get:

a4 select foo
* 9 EXISTS
* 9 RECENT
* OK [UIDVALIDITY 905864403] UID validity status
* OK [UIDNEXT 10] Predicted next UID
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags
* OK [UNSEEN 1] 1 is first unseen message in /x0/xzug/validate/foo
a4 OK [READ-WRITE] SELECT completed
a5 FETCH 4 BODY[TEXT]
* 4 FETCH (BODY[TEXT] {19}
test from wing1

)
a5 OK FETCH completed
a5 FETCH 5 BODY[TEXT]
* 5 FETCH (BODY[TEXT] {25}
Herald compose test 3

)
a5 OK FETCH completed

In other words, the body text really is present.

Is this a problem with tmail or cclient? These IMAP servers are
shortly going live with 4000 users, increasing to 30000 over a year
or two. I obviously need a mailbox format which doesn't have this
truncation problem but equally I don't want to be stuck with Berkeley
format mailboxes unless there's no other choice. (There's no way I'd
want to do a mass Berkeley -> MBX conversion on large live systems).

--Malcolm

-- 
Malcolm Beattie <[email-masked]>
Unix Systems Programmer
Oxford University Computing Services

------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 15 Sep 1998 10:19:56 -0700 (PDT)

The best address for reporting c-client bugs is to send the report directly to
me and not to a mailing list.

This is the first time that I have ever heard of such a bug.  It is plainly
clear from the telemetry that the MIME parser considers the body of the
failing message to be empty, so there is some other problem.

Are you ABSOLUTELY sure that the failing message is correct?  9 times out of
10, a "missing body" bug report turns out to be that the message does not have
a blank line delimiting the header and body.  A line consisting of only a
space is a continuation line according to RFC 822.

Please send me (as a MIME attachment) a mailbox which demonstrates the problem
for analysis.

---

From: Malcolm Beattie <[email-masked]>
Date: Tue, 15 Sep 1998 18:35:32 +0100 (BST)

> me and not to a mailing list.

OK, I'll remember in future.

> space is a continuation line according to RFC 822.

I think I'm number 10. I'm aware of the importance of the blank line.
I'm using exim to deliver the message via tmail. Some messages
delivered that way work fine. mbxcvt converts the mailbox into a
Berkeley mailbox which can be seen to be correct (i.e. headers/body
separated OK and cclient can see the body when reading it).

> Please send me (as a MIME attachment) a mailbox which demonstrates the problem

This message is cc'ed to the list but I'll send example mailboxes to
your address alone to avoid clutter.

--Malcolm