Subject: Re: PINE, please don't generate headers

From: Mark Crispin <[email-masked]>
Date: Wed, 31 May 1995 16:36:41 -0700

On 31 May 1995, Stephen R. van den Berg wrote:
> In article <[email-masked]>, Rahul Dhesi  <[email-masked]> wrote:
> >What saves the user is that in almost all cases, syntactically-invalid
> >message-ids will be accepted by receiving software.
>
> As far as I know, the only two requirements for message id fields
> are that:
> 1. They are unique.
> 2. To the right of the @-sign, there must be some text that should *look*
>    like a domain name.

Van den Berg is correct and Dhesi is wrong.

According to RFC 822, the contents of the Message-ID header is a rule
called "msg-id".  Buried inside a Message-ID is something called a
"domain"; which, in spite of the name, is not necessarily an Internet
domain name.  Unfortunately, this tends to confuse certain sorcerer's
apprentices.

Nor is it required to have any embedded periods.  A domain consists of
a "sub-domain", followed by zero or more period/sub-domain pairs.  A
sub-domain, in turn, may be a "domain-ref", which is a single atom.

The bottom line is that:
	Message-ID: <a@b>
is syntactically valid according to RFC-822; and those who claim
otherwise have mis-read the specification.

So, by the way, is:
	Message-ID: <a@blurdy\@bloop.[1.2.buckle.my.shoe].3.4.shut.the.door>
Try and see if your software swallows this!  [Hint: Pine does just fine.]

The relevant elements of RFC-822 syntax follow:
     msg-id      =  "<" addr-spec ">"            ; Unique message id
     addr-spec   =  local-part "@" domain        ; global address
     local-part  =  word *("." word)             ; uninterpreted
                                                 ; case-preserved
     word        =  atom / quoted-string
     atom        =  1*<any CHAR except specials, SPACE and CTLs>
     CTL         =  <any ASCII control           ; (  0- 37,  0.- 31.)
                     character and DEL>          ; (    177,     127.)
     SPACE       =  <ASCII SP, space>            ; (     40,      32.)
     specials    =  "(" / ")" / "<" / ">" / "@"  ; Must be in quoted-
                 /  "," / ";" / ":" / "\" / <">  ;  string, to use
                 /  "." / "[" / "]"              ;  within a word.
     qtext       =  <any CHAR excepting <">,     ; => may be folded
                     "\" & CR, and including
                     linear-white-space>
     quoted-pair =  "\" CHAR                     ; may quote any char
     quoted-string = <"> *(qtext/quoted-pair) <">; Regular qtext or
                                                 ;   quoted chars.
     linear-white-space =  1*([CRLF] LWSP-char)  ; semantics = SPACE
                                                 ; CRLF => folding
                                                 ; case-preserved
     CRLF        =  CR LF
     CR          =  <ASCII CR, carriage return>  ; (     15,      13.)
     LF          =  <ASCII LF, linefeed>         ; (     12,      10.)
     LWSP-char   =  SPACE / HTAB                 ; semantics = SPACE
     HTAB        =  <ASCII HT, horizontal-tab>   ; (     11,       9.)
     domain      =  sub-domain *("." sub-domain)
     sub-domain  =  domain-ref / domain-literal
     domain-literal =  "[" *(dtext / quoted-pair) "]"
     domain-ref  =  atom                         ; symbolic reference
     dtext       =  <any CHAR excluding "[",     ; => may be folded
                     "]", "\" & CR, & including
                     linear-white-space>

-- Mark --

DoD #0105, R90/6 pilot, FAX: (206) 685-4045  ICBM: N 47 39'35" W 122 18'39"
Science does not emerge from voting, party politics, or public debate.

---

From: [email-masked] (Stephen R. van den Berg)
Date: 1 Jun 1995 02:44:47 +0200

John Stanley <[email-masked]> wrote:
>Stephen R. van den Berg <[email-masked]> wrote:
>>>dropped by inn sites, unless the gateway takes steps to correct the
>>>syntax in the message-id or generate a new one.  Fortunately most such
>>>gateways just replace the old message-id with a new valid one.

>As the manager of a gateway, I have had to include code to massage
>message id's received from mail into ones acceptable to news. Mostly 
>removing spaces, but also removing illegal characters.

Just as a side note, simply looking for `illegal' characters is not
enough.  You have to do full rfc822 parsing.
I.e. <"bla_BLA-"@some.domain.name> is a correct messageid.

>There is also a limitation on what characters they may contain.

Well, yes, if you must, there is indeed a third requirement:
   3. To the left of the @-sign, there must be some text that should *look*
      like the local part of an e-mail address.

>>If Pine chooses to append just the hostname and not the FQDN, then
>>there is nothing wrong with the generated msgid, as long as (in its
>>entirity) it is unique (in time and space).

>There are two parts of the message id for a reason. There is the local
>side, which a system CAN guarantee is unique locally but cannot
>guarantee globally, and the FQDN, which can be guaranteed globally
>unique but is not locally unique. Together, the whole is guaranteed
>globally unique.

>same news code), they can generate the same message id. Unlikely, but
>the RFC doesn't say "probably unique".

Right, indeed.  The RFC says unique in time and space.  But the rfc
does not say that the part to the left of the @ sign should be
locally unique and the part to the right of the @ sign should be
globally unique.

There is absolutely no reason why one couldn't switch the meaning of the
left and the right side (as long as one adheres to the formatting requirements
of the left and right sides).  You could easily put a string that encodes
the FQDN to the left of the @ sign, and put some locally generated
unique string to the right of the @ sign.

The only thing you have to ensure is that the *entire* message id is unique
in time and space.  How you accomplish this, is entirely your choice
(be it by magic, or whatever encoding scheme you have in mind).

E.g. if you put merely the hostname without the fully qualified domain
part to the right of the @ sign, then you could, for example, ensure
uniqueness by encoding the IP number of the machine in the left part
(or some other particularly unique info you have about your machine).

So, using the FQDN to the right of the @ sign sure is one way to get
to a unique and valid message id, but it is by no means the only way.
-- 
Sincerely,                                                          [email-masked]
           Stephen R. van den Berg (AKA BuGless).

This signature third word omitted, yet is comprehensible.

---

From: [email-masked] (Barry Margolin)
Date: 1 Jun 1995 18:26:25 -0400

>(be it by magic, or whatever encoding scheme you have in mind).

The designers of RFC 822 didn't choose to use the addr-spec format
arbitrarily.  It was clearly the *intent* of RFC 822 that this global
uniqueness be accomplished by creating a locally-unique local part and
using the globally-unique name or address of the sending host to the right
of the @.  Other than "by magic", there's simply no other way to ensure
that message ID's are globally unique (given existing protocols -- another
possibility is some central message-id registry, but there isn't one).
-- 
Barry Margolin
BBN Planet Corporation, Cambridge, MA
barmar@{bbnplanet.com,near.net,nic.near.net,netcom.com}
Phone (617) 873-3126 - Fax (617) 873-5124

---

From: Rahul Dhesi <[email-masked]>
Date: 1 Jun 1995 22:17:02 GMT

>A sub-domain, in turn, may be a "domain-ref", which is a single atom.

>     msg-id      =  "<" addr-spec ">"            ; Unique message id

Being right has has never stopped me from being wrong.  My philosophy
re 'right' vs 'wrong' may be extracted from the saved posting that is
appended below.

It would be interesting to know:

   What is pine's algorithm for guaranteeing uniquenss without ensuring
   that the right-hand-side is in the form of a dotted domain name?

Date:    25 May 95 20:26:25 GMT
From:    Rahul Dhesi <dhesi>
Newsgroups: comp.mail.misc,comp.mail.smail,comp.mail.headers
Subject: Re: is this a bad header? (gurus welcome)
Organization: a2i network

In <bjohnson.800692827@sibyl> [email-masked] (Bernard Johnson) writes:

>I have been told that the following header is bad because of a "blank"
>line.  The line is actually not blank (or null as defined in RFC-822),
>but a TAB and a SPACE character.

All my mail scripts written in perl test for the end of header with
something like this:

   if (/^\s*$) {
      ...
   }

A line with only blanks is treated as end of header.  I don't try too
hard to be RFC-conforming, but I do try to accomodate BITNET and other
IBM-based sites, which do not distinguish between empty lines and blank
lines.  I also try to accomodate people who manually generate messages
and might accidentally add some blanks in the separating empty line.

In your argument with the other party, you were probably talking at
cross-purposes.  Many wars begin when people talk about being 'correct'
or 'portable' when they really mean 'standard-conforming'.  Correctness
has to do with whether or not something works in its intended
environment.
-- 
Rahul Dhesi <[email-masked]>
"please ignore Dhesi" -- Mark Crispin <[email-masked]>

---

From: [email-masked] (John Stanley)
Date: 1 Jun 1995 22:16:49 GMT

In article <[email-masked]>,
>I.e. <"bla_BLA-"@some.domain.name> is a correct messageid.

No, I do not have to parse anything. I am not simply looking for
illegal characters, I am removing them. The result is a valid message
id. It may not be the same message id, but I have a copy of the
incoming one, and either a copy of the outgoing or the ability to
recreate it, so it doesn't matter.

>      like the local part of an e-mail address.

Then there is also a fourth requirement: 

    4. The message id cannot contain certain characters. For example,
	spaces are prohibited in the domain part. This is not a
	complete list.

>locally unique and the part to the right of the @ sign should be

Please read the RFC before continuing to comment. The RFC does, indeed,
specify that the local part is unique, but since it does not specify
how it is to be made unique, it cannot be guaranteed to be anything but
locally unique.

It also specifies "domain" for the right side, and because of DNS and
the way domain names are created, it will be globally unique.

>(be it by magic, or whatever encoding scheme you have in mind).

No, the RFC does limit you. 

>uniqueness by encoding the IP number of the machine in the left part

No, sorry. How the local part is created is up to each system. That
means, yes, you can encode what you want there, but that also means
others can encode the same things the same way and come up with the
same thing.

>to a unique and valid message id, but it is by no means the only way.

With the current RFC, it is the only way. Yes, you can come up with
other schemes, but those schemes are not RFC compliant.

---

From: [email-masked] (Trey Harris)
Date: 4 Jun 1995 03:34:32 GMT

I've found the discussion of Message-Id headers kind of interesting, 
because on all of our hosts running SunOS, AIX, ConvexOS, and HP-UX, the 
pice that Pine writes to the right of the @ is the fully-qualified domain 
name.  I should mention that on these hosts, hostname(1) also produces a 
FQDN.

However, on our Solaris hosts, Pine only places the machine name on the 
right side of the @, not a FQDN.  Solaris's hostname(1) also returnns 
just the machine name.

I am told that if we were to run NIS on the Solaris machines, the problem 
would fix itself (and have verified that that is the case) but that isn't 
an option.

Is there a fix to this problem, besides running NIS or NIS+?  We don't 
run NIS and likely never will (NIS just doesn't scale when you get into 
thirty thousand or more users) and I worry that we are producing 
Message-IDs on our Solaris machines that have the potential for being 
non-unique.
-- 
Trey Harris                              http://sunsite.unc.edu/harris
  System Administrator, Project Isis, Office of Information Technology
                       The University of North Carolina at Chapel Hill

---

From: Mark Crispin <[email-masked]>
Date: Mon, 5 Jun 1995 11:41:25 -0700 (PDT)

> right side of the @, not a FQDN.  Solaris's hostname(1) also returnns

Edit the /etc/hosts on the Solaris machine so that the FQDN for the local
machine appears first.

-- Mark --

DoD #0105, R90/6 pilot  FAX: (206) 842-0758  ICBM: N 47.36'24" W 122.34'08"
TOPS-20: A Great Improvement Over Its Successors

---

From: Trey Harris <[email-masked]>
Date: Mon, 5 Jun 1995 15:00:17 -0400 (EDT)

> Edit the /etc/hosts on the Solaris machine so that the FQDN for the local

At least a dozen people have emailed me with that.  But I've always put 
FQDNs first in /etc/hosts; I regard it a security hole to do otherwise.

What else could be causing this?

---

From: Mark Crispin <[email-masked]>
Date: Mon, 5 Jun 1995 13:07:31 -0700 (PDT)

> What else could be causing this?

Hmm, you've got me.  I think it may be time to ask SUN what could cause
Solaris to return anything other than the official host name in the h_name
member of the returned hostent struct from gethostbyname().  h_name *is*
documented to be the official name.

If /etc/hosts isn't lying, is there any chance that you might have NIS/YP or
some other such thing running without knowing about it?  e.g. a rump YP that
is "switched off" but still responds to the local host name?

---

From: [email-masked] (Robert A. Rosenberg)
Date: Mon, 5 Jun 95 06:21:55 GMT

In Article <[email-masked]>, [email-masked] (Trey
>Message-IDs on our Solaris machines that have the potential for being 

Off the top of my head, I'd expect that if there were a call that returned
the FQDN, using it [in lieu of the call to hostname(1)] in the code that
builds the right hand side would fix your problem if you made this
replacement in the Pine code prior to compiling it <g>.

---

From: [email-masked] (Phillip Vandry)
Date: 5 Jun 1995 18:22:24 -0400

>>right side of the @, not a FQDN.  Solaris's hostname(1) also returnns 

I'm the one who originally complained that FQDN's aren't generated under
Solaris, but I must retract my complaint after realizing it's not PINE's
fault. PINE uses a perfectly valid way of obtaining the local host's FQDN.

But the code doesn't work under Solaris 2.3 (library bugs). It does
under 2.4

>>Message-IDs on our Solaris machines that have the potential for being 

Upgrade to 2.4 :-)

-Phil

---

From: Mark Crispin <[email-masked]>
Date: Mon, 5 Jun 1995 15:29:42 -0700

> replacement in the Pine code prior to compiling it <g>.

There already is such a call in the Pine code.

There always has been such a call in the Pine code.

That call is gethostbyname(), which accepts a name (in this case, the name
returned by hostname()) and returns a hostent struct.  The h_name member
of the hostent struct is documented as being the FQDN (or, in old ARPANET
terms, the "official name").

The problem is that many systems are misconfigured; often by accident, but
on occasion by bad advice from sorcerer's apprentices.  The single most
common misconfiguration is an entry in /etc/hosts for the local system
which puts the non-FQDN first.  There have been reports this problem which
appear to have other causes.  Both Yellow Pages/NIS from SUN and NetInfo
from NeXT are perfectly capable of being misconfigured as well.

The problem with the proferred "solution" of manually appending the domain
name is that if/when the system's configuration is ever fixed, then you'll
see doubled domain names, e.g. foo.bar.com.bar.com   So, instead of adding
a bug to work around a bug, it's usually best to fix the underlying bug
(assuming you can find it).

-- Mark --

---

From: [email-masked] (John Stanley)
Date: 7 Jun 1995 19:56:19 GMT

In article <[email-masked]>,
James C Deikun <[email-masked]> wrote:
>In fact, you should be creating an entirely new one.  If one message
>is sent to two or more mail-news gateways and goes through each one
>with message-id preserved, but to different newsgroups or sets of
>newsgroups, the multiple copies will interfere with each others'
>propagation.

Just as if someone posted the same article to two different newsgroups
from two different servers with the same message id.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 7 Jun 1995 20:31:23 -0700

On 7 Jun 1995, Rahul Dhesi wrote:
> Assuming this is a misconfiguration, and recalling earlier discussion,
> we now have a list of two common misconfigurations:
>
> 1. gethostbyname() system call does not return fully-qualified domain
>    name.
> 2. Mail transport agent does not correctly insert From: line and/or
>    message id.
>
> Pine goes to some trouble to work around 2, but not 1.  I assume this
> was a design decision.  It would be interesting, useful, and
> informative, to know why.  I have always found it educational to
> understand the reasoning behind decisions made by software designers.

There is no good way to determine that (1) is a misconfiguration, as
opposed to being a private TCP/IP network that isn't connected to Internet
and does not use DNS.  I speculate that there are many more TCP/IP
machines that are not on Internet than those which are.  If you just check
to see if h_name doesn't have a dot, you'll get a false positive.  You can
try running down the h_alias list and seeing if any alias names have dots
in them.  But you'll get a false negative if the FQDN isn't in /etc/hosts.

3.92 is going to have some form of guess/warning, although I wouldn't be
at all surprised if someone on a private net with a perfectly valid
configuration gets bit and flames us.

(2) is not a misconfiguration.  It is perfectly valid according to the
standards, which make it clear that the MUA is responsible for generating
the From: and Message-ID: header.  If sendmail on certain systems applies
a From: and/or Message-ID:, it is acting as part of the MUA (not MTA).

And, of course, all the world is not necessarily sendmail.

-- Mark --

---

From: [email-masked] (James C Deikun)
Date: 7 Jun 1995 00:45:47 GMT

In article <[email-masked]>,
>recreate it, so it doesn't matter.

If you're running a mail-news gateway, it'd better NOT be the same ID.

--
James "[email-masked]" Deikun

---

From: Rahul Dhesi <[email-masked]>
Date: 7 Jun 1995 08:13:06 GMT

In <looo-o-ong-message-id> Mark Crispin <[email-masked]> writes:

>The problem is that many systems are misconfigured; ....
>The single most
>which puts the non-FQDN first.

A possible attempt to work around misconfiguration 1 would be:

   if (host name does not contain a dot) and (userdomain is defined) then
      append a dot and userdomain to host name
   end if

The code needed to work around 2 must be more complex than that.

---

From: [email-masked] (Michael Shields)
Date: 9 Jun 1995 03:09:02 -0000

In article <[email-masked]>,
Mark Crispin  <[email-masked]> wrote:
> the From: and Message-ID: header.

Where?
-- 
Shields.

---

From: Mark Crispin <[email-masked]>
Date: Thu, 8 Jun 1995 18:31:54 -0700

On 8 Jun 1995, John Stanley wrote:
> >and does not use DNS.
> It isn't really a question of using DNS or not. It is a question of
> whether the network uses a domain name or not.

This distinction is unimportant.  The only reason for using a domain
format name without using DNS is a future expectation of using DNS.

> And private networks are
> quite capable of having a domain name. It is a good policy to HAVE such
> a name, for the day when the network is no longer isolated from the net.

This is irrelevant.  Whether or not it is "good policy", it is perfectly
valid for a private network not to use domain names.  It is much more
typical for a private network to be domainless, no matter what you or I
may feel about "good policy".

> Since the problem is one of message id's that show up in news, then
> saying "the network isn't connected" is incorrect.

News has nothing to do with this.  Anyway, you don't use Pine for news, so
it shouldn't bother you.

Certain individuals claimed (incorrectly) that a message ID without a FQDN
was invalid, and (correctly) that such a message ID was more likely to
have false matches.

It is one thing to say that a system on a domain-using network is
misconfigured if the FQDN is not returned by gethostbyname().  It is quite
another to say that it is misconfigured if the FQDN isn't a meaningful
concept.

> And then you tell them to set their domain name.

In a domainless private network, that is about as meaningful as saying
"and then you tell them to blurdybloop the sarasoop".

-- Mark --

---

From: [email-masked] (Peter da Silva)
Date: Thu, 8 Jun 1995 21:06:58 GMT

In article <[email-masked]>,

> from two different servers with the same message id.

Which is something that takes a deliberate effort to do, as opposed to
sending a message to two mailing lists that have gateways which is very
easy to do accidentally.
-- 
Peter da Silva    (NIC: PJD2)                             `-_-'
Network Management Technology Incorporated                 'U`
1601 Industrial Blvd.     Sugar Land, TX  77478  USA
+1 713 274 5180                                "Har du kramat din varg idag?"

---

From: [email-masked] (John Stanley)
Date: 9 Jun 1995 20:09:22 GMT

In article <[email-masked]>,
Mark Crispin  <[email-masked]> wrote:
>This distinction is unimportant.  

I would think that the reason that Pine doesn't use FQDN in some
message id's is rather important in a discussion about why Pine
doesn't use FQDN in some message id's. It isn't an issue of DNS, it
is an issue of how the system is named. 

>The only reason for using a domain

Or so that software that needs it can have it.

>valid for a private network not to use domain names.  

Right. And just why is it we are caring if private networks are
generating bad message id's? It seems that the problem becomes a problem
when those networks aren't private anymore. 

>News has nothing to do with this.  Anyway, you don't use Pine for news, so

Right. I'll just start saying "screw anyone who hasn't a proper message
id and just drop their messages on the floor".

>was invalid, and (correctly) that such a message ID was more likely to

Since the RFC specifies "unique", "more likely to have false matches"
means that it doesn't meet the RFC specifications.

>another to say that it is misconfigured if the FQDN isn't a meaningful

1) FQDN is meaningfull for private networks. 2) I didn't say it was
misconfigured.

>"and then you tell them to blurdybloop the sarasoop".

Malarky. Private networks are just as capable of having domain names as
are "public" ones. "Not needed except for certain things" is not the
same as "meaningless".

---

From: [email-masked] (John Stanley)
Date: 9 Jun 1995 20:14:53 GMT

>sending a message to two mailing lists that have gateways which is very

Yes. So easy that it hasn't happened once here.

---

From: [email-masked] (John Stanley)
Date: 8 Jun 1995 19:00:19 GMT

In article <[email-masked]>,

It isn't really a question of using DNS or not. It is a question of
whether the network uses a domain name or not. And private networks are
quite capable of having a domain name. It is a good policy to HAVE such
a name, for the day when the network is no longer isolated from the net.

---

From: Mark Crispin <[email-masked]>
Date: Sat, 10 Jun 1995 09:46:01 -0700

There does not seem to be any point to continuing this discussion any
further.  I hope that John Stanley will attend the email-related meetings
of the IETF in the future and share his vast knowledge.

-- Mark --

---

From: [email-masked] (Michael Shields)
Date: 10 Jun 1995 22:22:59 -0000

In article <[email-masked]>,
> Yes. So easy that it hasn't happened once here.

Whether it is easy or not, the fact that it can be done accidentally
and easily defended against is reason to defend against it.
-- 
Shields.

---

From: [email-masked] (Phillip Vandry)
Date: 11 Jun 1995 15:32:48 -0400

>right side of the @, not a FQDN.  Solaris's hostname(1) also returnns 

/usr/ucb/hostname always returns the same as uname -n. After all, take a look
at it, it's a shell script!

But the Solaris 2.4 gethostbyname() does correctly return the FQDN. Only in
Solaris 2.3 (and possibly earlier) does the problem occur.

>would fix itself (and have verified that that is the case) but that isn't 

If it does not occur when using NIS, then it must depend on the access method,
which means the bug might be in nss_dns.so.1. On the other hand, It also
occurs when using files, so maybe it's in both nss_dns.so.1 and
nss_files.so.1 but not in nss_nis.so.1. Who knows?

>Message-IDs on our Solaris machines that have the potential for being 

Well, you can hack PINE, like I did, but that's not really a solution. You
can also upgrade to Solaris 2.4.

Follow ups to comp.unix.solaris.

-Phil

---

From: [email-masked] (Phillip Vandry)
Date: 26 May 1995 18:28:05 -0400

Quite a while ago, I noticed that the message IDs of the messages coming
from PINE do not use the FQDN of the sending host. The reason? It does
a simple uname() call (or something similar, I forget) to fetch the
node name, which is not the FQDN.

I left this alone because it's not a big deal. But IMHO it would be better
to not generate any such headers and leave the task to the MTS. That's how
it works with most/all other MUAs.

Recently, I ran into the same problem with From: lines. Not all of our
users live under the same domain name. We have some which are mapped to
other names (different domain name, possibly different username) using
sendmail 8.6.x's userdb facility. Our mail configuration accomodates
this well. But PINE insists on slapping a constant string on the end of
the username.

The solution I've used for both problems is the same: supress the
generation of the Message-ID: and From: headers in outgoing mail & news.

Why doesn't PINE work that way in the first place? It doesn't make sense
for the MUA to compute the correct values of those headers. It makes many
MUAs to configure when we could get away with only configuring the MDA.
Not to mention complex configurations supported by sendmail but not
some or all MUAs!

My 2 cents...

If anyone is interested in my patches to pine-3.91 to supress these
headers (including the implementation of posting via inews since it was
previously unimplemented), mail me.

-Phil