Subject: beginning of messages/parsing

From: [email-masked] (Patti Johnson)
Date: Wed, 2 Nov 1994 10:22:13 +0500

Hello all!
I've recently begun using my Sun/Solaris 2.3 - this may be an issue I need to
take up elsewhere but thought I'd start here.

On my old OS (SCO Xenix), I could process a mail file with one of my own 
apps - I knew that messages begin with "From ".  Knowing this, I could break
a mail file down to its individual messages and work with them from there.

Then, I moved to my Sun/Solaris 2.3 box and found that that is no longer true.
I thought the problem was mine until I received a couple of messages that
included several imbedded "From "'s - they were majordomo list logs.  Well,
mailx could tell that there was a total of 3 messages, where the messages
included many other messaages.  Pine, on the other hand could not - and parsed
them out as several smaller messages.  

What's the deal on mailbox parsing?  How can I tell where one message begins
and one ends?  Should I cross post this message to another list that discusses
Sun or are others of you finding the same "problem"?

patti johnson
[email-masked]

---

From: Ed Greshko <[email-masked]>
Date: Thu, 3 Nov 1994 10:08:55 -0800 (GMT)

> Sun or are others of you finding the same "problem"?

	The problem is Sun's twisted notion that the "Content-Length:"
field is a valid field.   You have several options:

	One is to adopt a recent suggestion from Mark Crispin:

     "Please get a copy of our sendit tool (available on our FTP server), or
some similar mail delivery tool that does not depend upon the highly bogus
Solaris ``Content-Length'' header to skip over message texts with embedded
mbox headers.  Use it in place of the mail delivery tool (usually /bin/mail)
as called by sendmail."

	I'm not familiar with SUN's sendmail implementation or the /bin/mail
program on their system....but you can also try adding the "E" flag to the
local mailer definition in the sendmail.cf file.  This flag is designed to
Escape lines beginning with "From " in the message with an ">" sign.

					Ed

Edward M. Greshko                       Technical Manager, Electronic Commerce
                                        Control Data Asia/Pacific Region
Voice: +886-2-715-2222 x287             6/F, 131 Nanking East Road, Section 3
FAX  : +886-2-712-9197                  Taipei, Taiwan R.O.C

---

From: Mark Crispin <[email-masked]>
Date: Wed, 2 Nov 1994 21:01:08 -0800

> Sun or are others of you finding the same "problem"?

This is likely to be a Frequently Asked Question.  So, without further 
ado...:

		 Content-Length Considered Harmful
				or
	  Fear and Loathing in the Unix Mail File Header

		Mark Crispin, UW-NDC 2 November 1995

INTRODUCTION:

The UNIX mbox format was delimited by lines of the form
	From <address> <date-time>
for example:
	From [email-masked] Wed Nov  2 20:01:52 1994

Any line in the message text in that format has to be escaped in some way,
usually by prefixing it with a ">" character:
	>From [email-masked] Wed Nov  2 20:01:52 1994

Some software quotes all lines that began with "From ":
	>From the data above, you can see the blurdybloop of the garply...
although this is not necessary since such lines are obviously not the 
start of a new message.

There are variations.  Not all systems include the seconds:
	From [email-masked] Wed Nov  2 20:01 1994
Some systems have a symbolic timezone (e.g. "PST") before the year:
	From [email-masked] Wed Nov  2 20:01 PST 1994
Other systems have a numeric timezone after the year:
	From [email-masked] Wed Nov  2 20:01:52 1994 -0800
Some systems have an additional field that says "remote from <UUCP-node>":
	From [email-masked] Wed Nov  2 20:01:52 1994 -0800 remote from foo
Pine supports 20 possible variations in all.

And all was more or less joy and light.  Or at least begrudged acceptance 
and enough to see your hand in front of your face.

LET'S FIX WHAT AIN'T BROKE:

Certain individuals at unnamed organizations [one used to be the US
telephone monopoly, the other a large workstation vendor whose name used
to mean Stanford University Network] decided to "improve" the format of
UNIX mail files. 

These individuals felt that it was "inefficient" to look for lines that 
begin with "From ".  They invented a Content-Length: header line, which 
contained a byte count of the number of text bytes after the header.  
This meant, in theory, after reaching the blank line that separates the 
message text from the message body, you could do an fseek() to move that 
many bytes forward.

WHY IT'S BROKE:

There were problems with this notion.

1) Stdio does not guarantee that the numbers used by fseek() are really
   bytes from the start of the file.  So the "efficient" algorithm was
   by definition non-portable.

2) This header line was written by the originator of the message.  If the
   message was modified in transit (e.g. through a gateway that converts
   spaces to tabs, deletes trailing spaces, etc.), the size count would
   be messed up.

3) Mischievious users could messages with bogus Content-Length values to
   confuse your mail reader.

4) There are differences between UNIX newline (a single byte \012) and
   Internet newline (a two byte \015\012), with resultant impact on message
   text lengths.

5) One of the benefits of the UNIX mailbox format was that you could run a
   text editor on the file without damaging its subsequent usability with
   your mail reader.  Size-count based mailbox formats break this ability.

6) One of the benefits of a size-count mailbox format is to be able to find
   the next message quickly, even if you can not edit it with an editor.
   But because the Content-Length: header line could only be used as an 
   estimate, you end up not only having to scan the header for this header
   line but then hunting around in the mailbox at about the appropriate
   point for the next message.  This requires a lot of clever code and
   ends up not saving much at all.

7) It is possible to do fast block searches that are much faster than
   character-at-a-time searches.  Pine does a very basic form of fast
   search (that is nonetheless faster than the character-at-a-time
   search).  But you can't use such searches effectively if you have to 
   stop and do header parsing, or have to backtrack.

8) It forces the mail reader to parse the message header just to find
   the next message.  So, mail readers that try to save time by deferring
   message header parsing lose the startup time advantage.

9) Even when things work right, we weren't talking big savings here:
   less than 1/1000 of a second of CPU time per typical message, somewhat 
   more for the Multi-Megabyte MIMEgram From Hell.  But, you have to
   subtract the cost of doing a message header parse for every message (to
   find the Content-Length) which previously was unnecessary.

Bottom line: Content-Length combines the disadvantages of the UNIX mailbox
format (slow to find the next message) and a size-count format (can't edit
reliably) while retaining the advantages of neither.  It can have some 
performance benefits, but also entails performance costs.  Software which 
tries to support it takes a performance hit on *all* systems, even those 
which don't use it.  Finally, it has reliability and robustness costs.

WHERE WE STAND IN THE RESULTING FETID BOG:

There are individuals who are fans of Content-Length format, and loudly
evangelize its cause.  People with more experience shudder at the thought.
We on the Pine team fall into the latter category; if we tried to support
Content-Length, we'd slow down mail file reading for *all* users, and it
would make Pine less reliable and more vulnerable to the antics of
mischievious individuals.  So, we decline to do so. 

Although the above-named Silicon Valley workstation vendor is chiefly
responsible for distributing software that inflicts Content-Length upon
the email world, many internal users at that vendor avoid it like the
plague.  They replace the offending software with traditional format
software ("do what we say, not what we do").  Many of this vendor's
customers do likewise ("we'll do what you do, not what you say").

We recommend this course of action as well. 

WHAT YOU CAN DO TO ESCAPE THE SWAMP:

If sendmail is your mailer, the sendmail.cf file usually has an entry that
identifies the program that actually writes mail files.  Typically, it is
/bin/mail.  You can get a copy of our tools sendit or tmail (sendit is
more traditional, tmail is mostly for wizards only) from our FTP server
(ftp.cac.washington.edu), install it, and have sendmail point to that tool
instead. 

Sendit is smart enough to recognize that it only needs to insert the nasty
">" prefix in front of lines that are 100% in start-of-message header
format, and otherwise ignores lines that start with "From ".  This cures
99% of user complaints, and helps preserve a fast and reliable Pine.

---

From: [email-masked] (Jim Davis)
Date: 2 Nov 1994 11:43:42 -0700

In article <[email-masked]>,
Patti Johnson <[email-masked]> wrote:
:
:Then, I moved to my Sun/Solaris 2.3 box and found that that is no longer true.
:I thought the problem was mine until I received a couple of messages that
:included several imbedded "From "'s - they were majordomo list logs.  Well,
:mailx could tell that there was a total of 3 messages, where the messages
:included many other messaages.  Pine, on the other hand could not - and parsed
:them out as several smaller messages.  

There's a sendmail flag that will turn 'From ' into '>From ' inside a
message body during local delivery; otherwise that 'From ' will look
like it's the start of a new message.  Mailx uses another method to
tell messages apart, which is why it didn't get confused.  Ask your
system administrator to check the flags used on sendmail's local
delivery agent.  I thought that /bin/mail, the default local delivery
agent, doesn't need that flag, but perhaps your administrator is using
some other program for local delivery.