Subject: Detecting headers in folders

From: [email-masked] (Graham Minchin)
Date: 1 Nov 1995 04:36:43 GMT

I have the following problem:
How do I look through a Pine folder (containing headers and messages 
saved as text) and know where the header for each message starts?
I know the end of each header is marked as a blank line, but you can't 
mark the start of a header like this because the message itself could 
contain blank lines.
I'm writing a program to go through these folders and remove most of the 
fields from the headers, but I don't know of a reliable way of detecting 
the start of each header.  Obviously Pine can do it, does anyone out 
there know how?

Thanks
Graham

---

From: [email-masked] (Sven Guckes)
Date: Wed, 1 Nov 1995 23:20:04 +0100 (MET)

Quoting Graham Minchin:
> the start of each header.
> Obviously Pine can do it, does anyone out there know how?

Mail headers are described by RFC0822.
If you just need a simple utility to weed away some header lines
where it does not matter whether the header lines are inside other mails
then you can take a look at the "weedout.sed" at
	http://www.math.fu-berlin.de/~guckes/elm/scripts

Sven

---

From: [email-masked] (Rick Troxel)
Date: Wed, 1 Nov 1995 17:45:42 GMT

In article <[email-masked]> [email-masked]
(Graham Minchin) writes:

A standard Berkeley style folder delineates messages by a "From " line
following a blank line.  Reading from the pine Tech Notes:

Berkeley Mail Format
     This format comes to us from the ancient UNIX
     mail  program,  /bin/mail.   (Note  that this
     doesn't have anything to  do  with  Berkeley,
     but  we call it the Berkeley mail file format
     anyway.) This program was  actually  used  to
     interactively  read  mail at one time, and is
     still used  on  many  systems  as  the  local
     delivery agent.  In the Berkeley mail format,
     a folder is a simple text file.  Each message
     (including  the  first)  must  start  with  a
     separator line which takes approximately  the
     form:

     From [email-masked]  Wed Aug 11 14:32:33 1993

     Each  message  ends  with  two  blank  lines.   *
     There  are  actually several different varia-
     tions in the date part of the string,  twenty
     at  last  count.

* I think this should read "two newlines" -- i.e. _one_ blank line.

Hope this helps,
--
Rick Troxel     [email-masked]     [email-masked]     301/496-4823
///////////////////////////////////////////////////////////////////////////
     All effort and exertion put forth by man from the fullness of his
     heart is worship, if it  is prompted  by the  highest motives and
     the will to do service to humanity.                 --Abdu'l-Baha

---

From: Graham Minchin <[email-masked]>
Date: Thu, 2 Nov 1995 11:32:59 +0800 (WST)

Hi!  Thanks for your little pointer, but the problem is more the 
detection of where each header starts.  If the header is included in 
other mail (ie if somebody mailed me a header) then I'd want to consider 
it as just part of the message.

Graham

---

From: [email-masked] (Rick Troxel)
Date: Thu, 2 Nov 1995 18:37:09 GMT

In article
<[email-masked]>
[email-masked] (Graham Minchin) writes:

I'm confident that your mail program won't!  That is, an unescaped "From
" line in the body of a message will cause your mailer to treat it as
two messages.  That's why many mail transport agents prepend '>' to
lines starting with "From ".
--
Rick Troxel     [email-masked]     [email-masked]     301/496-4823
///////////////////////////////////////////////////////////////////////////
     All effort and exertion put forth by man from the fullness of his
     heart is worship, if it  is prompted  by the  highest motives and
     the will to do service to humanity.                 --Abdu'l-Baha

---

From: [email-masked] (Sven Guckes)
Date: Fri, 3 Nov 1995 21:18:42 +0100 (MET)

Quoting Rick Troxel:
> > If the header is included in other mail (ie if somebody mailed me a header)
> > then I'd want to consider it as just part of the message.
> I'm confident that your mail program won't!  That is, an unescaped "From_"
> line in the body of a message will cause your mailer to treat it as two
> messages.

So - how doee Pine recognize mails within mails?
The FAQ says it does not regard the Content-Length header.
Will Pine stumble over mails in mails when the MTA does not escape "From_"
lines?

Sven  [too lazy to test]

---

From: [email-masked] (Graham Minchin)
Date: 5 Nov 1995 04:57:48 GMT

Rick Troxel ([email-masked]) wrote:
: A standard Berkeley style folder delineates messages by a "From " line
: following a blank line.  Reading from the pine Tech Notes:

This seems a bit simplistic when you consider that the message might well 
contain a line starting with the word "From ".  From some of the 
information I've received following this thread I've come across another 
way of doing it:
Whenever you come across a line starting with "From " you then say "This 
might be a new header".  You look at the following lines which (if it's a 
header) must all start with "Received from: ", "X-Status: " or a large 
number of other field names, unless they are part of a "To: " or "Cc: " 
field, in which case they probably start with a few spaces.  Then when 
you get a blank line (end of header) you count the number of fields you 
came across.  If this number is greater than a certain limit, you say 
"right I've found a header".  I guess about 9 or so fields would be a 
good sign of a header.

Any comments on this algorithm?  It will still think that a header within 
a message is a new header :(

Graham

---

From: [email-masked] (Sven Guckes)
Date: Mon, 6 Nov 1995 03:58:38 +0100 (MET)

Quoting Graham Minchin:
> field, in which case they probably start with a few spaces.

Any header might be broken across several lines, providing each following
line starts with white space.
Example:
	X-Example:  This header line
		is more than one line
		but it is still a valid
		header line!

> Then when you get a blank line (end of header) you count the number of
> fields you came across.  If this number is greater than a certain limit, you
> say "right I've found a header".  I guess about 9 or so fields would be a
> good sign of a header.  Any comments on this algorithm?

Nine lines may be a good sign, but the tenth line might be a non valid header
lines.  This would allow anyone to send you mails with pseudo headers which
only require nine lines and thus fool your script.

> It will still think that a header within a message is a new header :(

OK!

>From cac.washington.edu!owner-pine-info Mon Nov  6 12:34:56 1995
From: me to you
To: you
Date: Mon Nov  6 12:34:56 MET 1995
Subject: 
Organization: Free University of Berlin
X-URL: http://www.math.fu-berlin.de/~guckes/
X-Smiley: :-)
X-X-L:  SMILE!
X-X-X-X-X-X-X: This exxxxtra line for Graham Minchin.
	May his algorithm succeed!

Oops, looks like my signature got into the next mail.  ;-)

Sven

---

From: [email-masked] (Rick Troxel)
Date: Mon, 6 Nov 1995 17:43:22 GMT

In article <[email-masked]>
[email-masked] (Sven Guckes) writes:

   Sven  [too lazy to test]

If the line in question has a valid format (i.e. contains a properly
formatted date after the From_), pine will parse the mail as two
adjacent messages.  If not, pine will see it all as one message.

BTW, this message parsing convention is not at all unique to pine.  The
standard Unix mail utilities have always (?) used it.  Regarding
alternatives, I've seen lots of reports about Content-Length: breaking
frequently.
--
Rick Troxel     [email-masked]     [email-masked]     301/496-4823
///////////////////////////////////////////////////////////////////////////
     All effort and exertion put forth by man from the fullness of his
     heart is worship, if it  is prompted  by the  highest motives and
     the will to do service to humanity.                 --Abdu'l-Baha

---

From: [email-masked] (Sven Guckes)
Date: Tue, 7 Nov 1995 21:27:40 +0100 (MET)

Quoting Rick Troxel:
> > Will Pine stumble over mails in mails when the MTA does not
> > escape "From_" lines?

Is a valid "From_" line enough?
Or does Pine do a test on the following lines for a vaild header?

> BTW, this message parsing convention is not at all unique to pine.
> The standard Unix mail utilities have always (?) used it.
> Regarding alternatives, I've seen lots of reports about Content-Length:
> breaking frequently.

Which mailers honor the Content-Length: line?

Sven

---

From: [email-masked] (Rick Troxel)
Date: Wed, 1 Nov 1995 17:45:42 GMT

Berkeley Mail Format

Hope this helps,