Subject: Re: Content-length header val and embedded From lines

From: [email-masked] (Chris Koenigsberg)
Date: Fri, 15 Mar 1996 23:52:01 GMT

You are talking about the "mbox" or "Bezerkeley" file format. The
"mtext" or "tenex" file format encodes the length of a message, into
the flags on the first line.

Escaping embedded "From" lines is the responsibility of the local
delivery agent (/bin/mail, procmail, etc.) which is appending the
incoming message to a mail spool file. Sendmail hands the LDA a chunk
of ascii bytes as one message, and the LDA escapes all non-initial
"From" lines. If it won't do that, then either you are giving it the
wrong flags, or else it is broken :-) (are we starting an argument
yet, is Solaris /usr/lib/mail.local broken? :-)

What if your imapd (or popper, whatever) adds a new header to store
some status flags, after you've read a message; do you expect it to
compute a new Content-Length and edit the old Content-Length header?
(similarly how MH will delete the "Unix From" line and replace it with
a "Return-Path" line, also altering the content length)

If the mailbox is in mtext format, the new status flags go into the
same header line as the content length, which IS recomputed and
updated properly by the imapd.

But if the mailbox is in mbox Berzerkeley format, the only thing
guaranteed is that there will be a leading Unix From line.

Nay, my humble opinion is that Content-Length headers, in mbox format
files, are a curious abomination which should be stomped out :-)

Chris Koenigsberg
U. of Chicago Academic Computing Services

---------------------
In article <[email-masked]>,
Steven Rich - Knoxville TN SE <[email-masked]> wrote:
>O.k.  I just read the FAQ entry on file formats and it is pretty clear
>that it uses 'From blah blah blah' to separate mail messages (where
>'blah blah blah' is a real From line).
>
>The problem is that some MUAs and LDAs don't escape embedded From
>lines (such as during forwarding messages) and expect the
>'Content-length' field of the header to be accurate (which is always
>true here, since /usr/lib/mail.local is doing the counting).
>
>The consequence is that Pine and/or the imapd process return multiple
>messages when there aren't multiple messages.  Is there a way to tell
>Pine/imap to trust/use the Content-length field?
>
>If not, can someone give me pointers as to where I should make
>modifications to fix this?
>
>Thanks.
>Steve
>-- 
>Steve Rich
>Systems Engineer - Sun Microsystems Computer Company
>in the sunny Knoxville, TN office.

---

From: [email-masked] (Steven Rich - Knoxville TN SE)
Date: 15 Mar 1996 21:45:07 GMT

Thanks.
Steve

---

From: [email-masked] (Chris Koenigsberg)
Date: Mon, 18 Mar 1996 22:54:11 GMT

I thought about this over the weekend, I felt like maybe I was too
rude to the guy from Sun who initially asked how to make Pine/imapd
honor Content-Length: headers.

Anyway, I thought about how to "do it right" and here's the proposal,
for the only possible way to "do it right". 

The biggest problem with Content-Length as an Internet (rfc822) HEADER
is that it really doesn't belong there at all. Mail Transfer Agents
have no business computing and updating a content length HEADER,
although there IS an ESMTP extension to indicate message
length. Existing Mail User Agents freely muck with certain selected
headers (X-Status for example) and have no obligation to
respect/update a Content Length among the HEADERS.

Instead, your stored content length belongs as an attribute only of a
message SEPARATOR LINE as stored in a local mailbox file, to be
STRIPPED OFF before transmission.

(similar to how the initial Unix From line is only used as a separator
in the mailbox, and is stripped off before transmission)

You are trying to add a new feature to the existing, well established
mbox file format, a feature (fixed length which must be computed and
stored/updated in Content-Length header) which contradicts a basic
mbox assumption (free format of headers/text, can be added/changed
arbitrarily subject to rfc 822/1123 restrictions, after initial From
line).

SO, here's the proposal: you will have to define yet another mailbox
file format, call it "clmbox". Write a new driver for imapd/c-client,
based on the mbox driver. Change the local mail delivery agent and the
mail applications on your system (Solaris, presumably) so they all
understand the new format and all respect/preserve its main attribute:

Each message in the new clmbox mailbox format would START with a
single separator line, perhaps in this format: "Content-Length [n
bytes] From [usual From line stuff]".

Thus the imapd and c-client would still be able to determine the
mailbox format from reading the first line of the file, as they do
now.

Note that imapd/c-client already HAVE at least one fixed-length
format, namely the "mtext/tenex" format, in which a message begins
with a line that starts with a MM-DD-YY (or YYYY?) date, followed by
octal flag bytes, including the content length.

The mtx c-client/imapd driver respects this content length,
recomputing it as necessary when rewriting a message to a mailbox
file. It makes sense to have this computed number appear in the
initial out of band separator line, AT THE BEGINNING of the message it
refers to..... rather than a header stuck randomly somewhere in the
middle of the Internet rfc822 header set....

So what would you gain from introducing a new clmbox mailbox format,
instead of just using mtext/tenex? Good question. Maybe Solaris should
adopt the mtext/tenex format as its native mailbox format, if you
really really need to have a computed content length for a Solaris
mail user agent?

On the other hand, remember that mtext/tenex mailbox handling involves
actions which are not reliable when the mailbox is accessed via NFS...

Chris Koenigsberg
[email-masked], [email-masked]
http://www2.uchicago.edu/ns-acs/ckk/index.html
(also http://www.pobox.com/~ckk)
U. of Chicago Academic Computing Services postmaster