Subject: Address parse error on weird From: header

From: Axel Reinhold <[email-masked]>
Date: Wed, 9 Jan 2002 15:36:45 +0100 (MET)

Dear devellopers,

The c-client lib does not parse this (maybe) not standard
From-header correctly:

From: "Daniel Aimufua"[email-masked]

The ->mailbox field returned is: 'Daniel Aimufuadaniel'
The ->personal field returned is empty.

This results in an unusable return-address:
        <Daniel [email-masked]>

Can this be fixed? I looked in the source,
but that's much too high level for me to fix.

Regards
Axel Reinhold
-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Jan 2002 09:34:46 -0800 (PST)

> From: "Daniel Aimufua"[email-masked]

There is no "correct" way to parse that; it's invalid syntax and non-compliant
with the standards.  The rule of "garbage in, garbage out" applies.

c-client's guess of how to parse that garbage is as good as your guess.  In
some other cases, c-client's guess would be right.

---

From: Vadim Zeitlin <[email-masked]>
Date: Wed, 9 Jan 2002 18:50:33 +0100 (CET)

On Wed, 9 Jan 2002 09:34:46 -0800 (PST) Mark Crispin <[email-masked]> wrote:

MC> c-client's guess of how to parse that garbage is as good as your guess.  In
MC> some other cases, c-client's guess would be right.

 Yes, but wouldn't it be better to leave the address as is instead of
adding SYNTEX-ERROR and other ugliness to it? I have to undo this stuff
manually now as I have no intention of showing it to the user and it seems
pretty stupid.

 Besides, GIGO principle is really incorrect for the code like this. "Be
conservative in what you send and liberal with what you receive" is a much
better one IMO.

 Regards,
VZ

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Jan 2002 10:23:55 -0800 (PST)

On Wed, 9 Jan 2002 18:50:33 +0100 (CET), Vadim Zeitlin wrote:
> adding SYNTEX-ERROR and other ugliness to it?

All error indications (such as ".SYNTAX-ERROR") consist of a "host name" that
has a leading ".".  That indicates to the application that there was something
wrong with the string that it fed to c-client to parse.  This is how c-client
tells the application that something was wrong, and gives the application a
chance to say "whoa!  Maybe we don't want to act on this data."

> I have to undo this stuff

It "seems pretty stupid" to give the application, and possibly the end user, a
clue that there is something wrong with the data?  Most people would disagree.

Most people think that it's "pretty stupid" to act on corrupted data as if
nothing happened, with no warning to application or user.

> conservative in what you send and liberal with what you receive" is a much

"Be conservative in what you send and liberal with what you receive" is often
misquoted by people who fail to understand what Jon Postel meant when he said
it.  I knew Jon, and I was around when he said it.

It *NEVER* was intended to indicate a requirement to accept non-compliant
protocol, no matter how much the vendors of broken applications which transmit
non-compliant protocol claim to the contrary.

Instead, the correct interpretation is: "understand the full spectrum of what
the protocol permits to be sent, but send only the subset that the majority of
people actually use."

In other words, "don't send odd-ball stuff, even if the protocol says you can;
but be sure that you understand the odd-ball stuff if the protocol allows it."

---

From: "Peter Derr" <[email-masked]>
Date: Thu, 10 Jan 2002 14:17:37 -0500

I want to build and ship an IMAP server that can optionally be configured to
use SSL/TLS, but it seems that
starting in imap-2001, the c-client library's mail_open() function (and
Pine) always tries to use TLS if it's built with that capability, unless
explicity told not to with the hostname /notls option.

If the IMAP server was built with SSL capability, but isn't configured to
use it (e.g., no certificate), the connection fails.

Is there some way to configure and/or modify impad so that it will only
advertise the STARTTLS capability if it actually has a certificate?

Thanks,

Peter Derr
Compaq Tru64 UNIX Internet Engineering Group
[email-masked]

---

From: Mark Crispin <[email-masked]>
Date: Thu, 10 Jan 2002 11:36:38 -0800 (PST)

On Thu, 10 Jan 2002 14:17:37 -0500, Peter Derr wrote:
> it seems that

That is provably not true.

The correct statement is: if imapd is built with SSL capability, it always
advertises STARTTLS.

> advertise the STARTTLS capability if it actually has a certificate?

There's no test for all the ways in which a certificate could be missing or
invalid without doing certificate initialization unconditionally at all
startups.  That would introduce a performance problem; certificate
initialization is expensive!

I recommend instead that as part of your installation procedure, you generate
and install a self-signed certificate and leave appropriate instructions on
why it should be replaced with a real one.

---

From: "Peter Derr" <[email-masked]>
Date: Thu, 10 Jan 2002 15:35:41 -0500

> The correct statement is: if imapd is built with SSL capability, it always

...which then leads to connections failures if the imapd doesn't actually
have a certificate.

> initialization is expensive!

Something simple like testing whether the {SSLDIR}/imapd.pem exists was what
I had in mind.  A command option might be another possiblity.

> why it should be replaced with a real one.

But, until you got a real certificate connections would continue to fail
because of the default behavior of c-client that rejects self-signed
certificates.

A site that didn't want to enable SSL/TLS or spend the money for a real
certificate would be out of luck.

Peter

---

From: Mark Crispin <[email-masked]>
Date: Thu, 10 Jan 2002 12:36:55 -0800 (PST)

> ...which then leads to connections failures if the imapd doesn't actually

Yes, that is the case.

Tell me, isn't it true that Digital UNIX doesn't work too well if certain
files on /etc are erased?  I'm not trying to beat up on Digital UNIX, but
rather to point out that software can, and often does, have required files
which must be installed for proper installation.

I have been very aggressive on not making imapd have required files, but in
the case of SSL/TLS-enabled imapd, the certificate file is a required file.

> I had in mind.

That isn't as easy to do as it sounds, and it doesn't check for all forms of
certificate lossage.

> A command option might be another possiblity.

No good.  Blame certain security packages for stealing the usability of argv.

> because of the default behavior of c-client that rejects self-signed

People can use /novalidate-cert, and applications (such as modern versions of
Pine) can query on a self-signed certificate rather than reject it.

> certificate would be out of luck.

Such sites should install the non-SSL version.

---

From: "Peter Derr" <[email-masked]>
Date: Fri, 11 Jan 2002 13:23:37 -0500

Surely I'm not the only one who would like to distribute an easily
installed, pre-built imapd that can be configured to use SSL/TLS, or not.

> That isn't as easy to do as it sounds, and it doesn't check for all forms
of

I think it might be safe to assume that if an appropriately named
certificate file for imapd is installed, the expectation is that it should
work, and if it is not, it's not.  Therefore, a simple, cheap check as to
whether the certificate file exists before including STARTTLS in the
capability string is reasonable and desirable.  The complete initialization
of the certificate can wait until a client actually wants to use it.

It was pretty straightforward to separate out into a separate function the
code that finds the certificate name in ssl_server_init()  in
osdep/unix/ssl_unix.c.  I called the new function ssl_certificate_file() :

  /* Check whether a certificate file exists, and determine its name.
   * Accepts: server name, certificate name buffer
   * Returns: 0 for success, non-zero for failure
   */
    int ssl_certificate_file (char *server, char *cert_name)

Then I had ssl_server_init() call it and also, in imapd/imapd.c,
pcapability() can call it to see if there is any applicable certificate file
before including "STARTTLS" in the capability string (see below).  It works
great, checking for the address-specific or the general imapd.pem file name.

I've done this with imap-2001a (diffs are attached) but I'd be glad to merge
this into your latest imap-2002 snapshot and send you those diffs.   (I'd
also be glad to redo for imap-2002 the IPv6 diffs that I sent before.)

Peter

---

From: Vadim Zeitlin <[email-masked]>
Date: Fri, 11 Jan 2002 19:39:22 +0100 (CET)

On Wed, 9 Jan 2002 10:23:55 -0800 (PST) Mark Crispin <[email-masked]> wrote:

MC> On Wed, 9 Jan 2002 18:50:33 +0100 (CET), Vadim Zeitlin wrote:
MC> >  Yes, but wouldn't it be better to leave the address as is instead of
MC> > adding SYNTEX-ERROR and other ugliness to it?
MC> 
MC> All error indications (such as ".SYNTAX-ERROR") consist of a "host name" that
MC> has a leading ".".  That indicates to the application that there was something
MC> wrong with the string that it fed to c-client to parse.  This is how c-client
MC> tells the application that something was wrong, and gives the application a
MC> chance to say "whoa!  Maybe we don't want to act on this data."

 Yes, but why mangle the address? I surely like the fact that c-client
notifies me about the address parsing error but I don't understand why does
it have to do it in a so intrusive way. Why not just set the error flag in
the ADDRESS struct?

MC> > I have to undo this stuff
MC> > manually now as I have no intention of showing it to the user and it seems
MC> > pretty stupid.
MC> 
MC> It "seems pretty stupid" to give the application, and possibly the end user, a
MC> clue that there is something wrong with the data?  Most people would disagree.

 I meant that it was stupid to undo the work done by c-client in the
program (in particular I didn't mean to offend anyone).

MC> Most people think that it's "pretty stupid" to act on corrupted data as if
MC> nothing happened, with no warning to application or user.

 But surely I (the application) will do something about it. I just want to
be able to show to the user something readable instead of this
.SYNTAX-ERROR. thing.

MC> "Be conservative in what you send and liberal with what you receive" is often
MC> misquoted by people who fail to understand what Jon Postel meant when he said
MC> it.  I knew Jon, and I was around when he said it.

 You certainly do, but IMHO it still makes sense to try harder to parse the
addresses just because there are so many broken mailers around which
generate incorrect but obviously understandably by humans addresses.

 It's not much help to me that c-client conforms to the standards better
when a user sends me a mail asking why do Outlook and Eudora can parse the
address field in a message he received but Mahogany cannot (again, this, of
course, only applies to the incoming part - I am extremely glad that
c-client is standard-conforming with what it sends out).

 Anyhow, if you think there is no problem here I know that I have no chance
at all to change your opinion but I just wanted to let you know how this is
seen from the point of view of the library user.

 Regards,
VZ

---

From: Mark Crispin <[email-masked]>
Date: Fri, 11 Jan 2002 12:00:55 -0800 (PST)

On Fri, 11 Jan 2002 19:39:22 +0100 (CET), Vadim Zeitlin wrote:
>  Yes, but why mangle the address?

No address is "mangled".  If you see an address with a host field with a token
that starts and end with a ".", that address is an error block and should be
treated as such.

There are two types of error blocks.

An error block with .MISSING-HOST-NAME. as its token indicates just that; it
is what c-client uses as the "default host" when parsing a message in a
mailbox.  Normally a UI will use the local domain as the default host when
parsing input from a user's keyboard.  You can't do that when parsing a
message since there's no way to know for certain what the intended host is.
Decades of experience (and documents) demonstration why it is evil not to
write fully-qualified addresses.

An error block with .SYNTAX-ERROR. (this string is accessible via the global
errhst) can have one of the following texts in the mailbox field:
	UNEXPECTED_DATA_AFTER_ADDRESS
	INVALID_ADDRESS
	UNEXPECTED_DATA_AFTER_ADDRESS_IN_GROUP
	INVALID_ADDRESS_IN_GROUP
	MISSING_MAILBOX_TERMINATOR
Any other text in the mailbox field of a .SYNTAX-ERROR. error block indicates
an incompletely parsed address.  There should not be any other ADDRESS blocks
after a .SYNTAX-ERROR. block, even if there is additional text containing
possible good addresses in the original text.  So, once you see one of these,
you should assume that at best you have a truncated representation of the
address list and not the complete list.

> I surely like the fact that c-client

It is intrusive because applications MUST be aware of the underlying condition
and MUST NOT blithely ignore it.

The "error" field is writeable by the SMTP code, and used solely for SMTP
errors.

The "error" field only contains one unit of data.  As you can see, what is
needed are two units.

Most damning of all, the "error" field is not transmitted via IMAP.  That,
alone, is enough to preclude its use.

> program

If all you do is "undo" the work done by c-client, your application has a bug.
If you get one of these, the address list MUST be considered to be corrupt and
untrustworthy, and your application MUST do remedial action over and above
"undoing" it.

> be able to show to the user something readable instead of this

When Pine sees a .SYNTAX-ERROR. block, it refetches the address list in
question as text for display as literal text instead of using the c-client
parsed representation.  This conveys the maximum amount of information to the
user, as opposed to c-client's truncated understanding of the message.

You may want to consider something similar.

> generate incorrect but obviously understandably by humans addresses.

The situation was much much worse in the 1970s and 1980s.  I can assure you
from FIRST-HAND experience that the practice of "try harder to parse bogus
syntax" (and worse, "fix bogus syntax") leads inevitably to worse horrors.  I
am not eager to repeat those battles.  I am especially not eager to put myself
on the wrong side of those battles.

c-client's parser does not check for all possible errors.  Rather, c-client
strives to parse all possible valid input correctly (this is its "be liberal
in what you accept").  In the case of error input, either that error input
runs afoul of a c-client check or it does not.  A c-client check occurs when
it must see one of a particular set of tokens to know what to do next, and it
got something that it didn't expect; at that point c-client has lost parse
synchronization and is obliged to give up (otherwise it'll probably spew forth
tons of error messages).

There are a (very) few cases of error input which c-client does not check.  In
such cases, c-client still has parse synchronization, and by continuing the
result will be a reasonable guess at how it should be interpreted.  For
example, c-client does not enforce the requirement that there be a period
between a quoted-string and an atom in a local-part.  This is the other part
of "be literal in what you accept"; c-client doesn't reject error input if it
can continue in some reasonable manner.

c-client always generates strictly compliant headers.  This is its "be
conservative in what you generate."

If there are any changes made to c-client's error input handling, it will be
in the direction of strict enforcement and consequent refusal to accept any
addresses which do not strictly comply.

> address field in a message he received but Mahogany cannot

Tell such naive users that they feel this way, they should use Outlook and
Eudora, and enjoy their viruses, worms, and other security bugs.  I am not out
to compete with Outlook and Eudora in garbage digesting.

The burnt hand teaches best.  After that, advice about fire goes to the heart.