Subject: Problem using MIME attachments

From: [email-masked] (Jonathan Kruger)
Date: Fri, 5 Feb 93 8:54:01 EST

Why is it that when I send a NeXT WordPerfect document or a .GIF file
as a MIME attachment it works, but if I send a WP5.1 or .TIFF file the
attachment only contains a few bytes like "II*" and nothing else?
-------------------------------------------------------------------------------
      Jonathan Kruger          "If trees could scream, would we be so cavalier
   [email-masked]        about cutting them down?  We might, if they
 Chesapeake Biological Lab      screamed all the time, for no good reason."

---

From: Ron Pool <[email-masked]>
Date: Fri, 5 Feb 1993 10:05:51 -0500 (EST)

> attachment only contains a few bytes like "II*" and nothing else?

We just noticed this happen with a DOS WordPerfect 5.1 document here
yesterday.  I looked into why it was happening and decided that
file_type() in pine/send.c was deciding it was quotable-text, when it
wasn't.  I'm not sure why, but for us encoding all enclosures as base64
works, so I replaced the call to file_type() with an assignment to the
value IsBinary.  This is a bad fix as it means some files that could have
been directly viewable with MIME tools now won't be, but we had a couple
of users who really needed to get going.

I did notice while reading send.c that there are special checks for
certain file types that just look for a signature of just a few characters at
the beginning of a file to determine file type.  This allows pine to
recognize some certain files (GIF and TIFF in particular), but it's
certainly possible for files to be recognized as being of a certain
type when they're really not of that type.
--
Ron Pool; Electronic Technology Group; B-15 Wing Hall; Ithaca, NY 14853
Internet: [email-masked]

---

From: Terry Gray <[email-masked]>
Date: Fri, 5 Feb 1993 10:21:48 -0800 (PST)

Ron et al,
This is a really important issue, and one we've been wrestling with for
some months.  (In fact, I've taken the liberty of CCing the MIME authors
in case they wish to comment.)

Putting aside for the moment the possibility that there may still be bugs
in either Pine's base64 or quoted-printable encoding/decoding routines, the
design problem is: when to use b64 encoding and when to use q-p encoding.

The issues:

  -Above all else, the integrity of MIME attachments must be preserved.
   That is, MIME is a failure if people can't have certainty that the
   MIME-attached bits will be delivered intact.  This would argue for
   using B64 encoding all the time.  (You don't want q-p newline
   translation to occur on something that wasn't really a text file.)

  -At odds with the above is the fact that text line delimiters differ from
   one system to the next, and if you know you are dealing with a text
   file, it would be nice if the system would handle newline translation
   automatically.  This argues for using q-p whenever you think you have
   a text file.  (If b64 was used for text, the destination system might
   have trouble reading the text, if the nl conventions were different.
   The translation couldn't be automated because the MIME typing doesn't 
   currently differentiate text files by originating OS.)

  -If you think you have a text file, q-p is also desirable because it
   is semi-readable by non-MIME mail readers.

  -One lesson of FTP is that we don't want a system where the user has
   to explicitly declare the type of file and therefore the encoding type.
   (That's not to preclude the possibility of override, but in general,
   we don't want people to have to say the equivalent of "binary" or "type
   image" in order to guarantee correct delivery.  I've seen too many
   help desk problems from getting this wrong; I've even heard the claim
   that you can't reliably send binary data on the Internet, presumably
   because of ftp mode ignorance!!)

  -The behavior (choice of encoding) should be predictable.  

So one problem is how to decide whether or not you really have a text file.
One can define a text file to be a file that only uses certain bit
patterns within the character space, but to be certain you must scan the
entire file.  But that's inefficient, so right now Pine looks at the first
chunk and tries to make the right decision.

It would appear that the current test is not sufficiently conservative,
and may also fail the predictability goal.  Some Unix systems have a very
sophisticated "file" program that could help, but others are less
sophisticated, and DOS users may have nothing of the kind.  So what is the
minimum acceptable test for "text-ness"?

Of if efficiency concerns imply that there will always be some uncertainty
in the correctness of the encoding decision --a bad thing-- should we
consider defining subtypes text/mac, text/dos, text/unix so b64 could be
used in all cases, and the mail *reader* could do the appropriate newline
translation, if any was needed?   (This assumes that everyone will soon
have MIME-capable mail readers available, which is arguable, but perhaps
not that unreasonable a premise if it would improve the certainty that
MIME attachments are reliable.)

Comments/suggestions are welcome.

-teg

---

From: Nathaniel Borenstein <[email-masked]>
Date: Fri, 5 Feb 1993 15:28:04 -0500 (EST)

Excerpts from mail: 5-Feb-93 Re: Problem using MIME atta.. Terry
[email-masked] (4704*)

>  (If b64 was used for text, the destination system might

Actually, a scrupulously correct MIME implementation shouldn't have this
trouble, because MIME says that text should be converted to CRLF form
before it goes into base64; however, I anticipate that many folks will
have trouble getting this detail right.

>    is semi-readable by non-MIME mail readers.

Yes, that's another good reason not to use base64 for text.

> translation, if any was needed?  

I think that's a bad idea; if you're going to put text in base64, you
should simply convert to CRLF conventions.  It's tricky to understand
the necessity of this, perhaps, but simple to implement.

Just my two cents....  -- Nathaniel

---

From: Mark Crispin <[email-masked]>
Date: Fri, 5 Feb 1993 13:51:00 -0800 (PST)

It's important to realize that there are two separate issues here:
 1) determining the difference between text and binary
 2) choosing between quoted-printable and base64

I will contend that the (2) choice is one of esthetics.  Quoted-printable is
quite capable of carrying arbitrary binary.  However, to make this choice
intelligently, you have to be able to do (1).

My NeXT MailManager application punts on both of these, and just transmits
everything as binary files.  This has the newline problem that Terry noted,
although it's not as serious as it might be since it's tagged as binary (there
is a MailManager bug in that it fails to do newline conversions when
unattaching segments declared as text).

The kludge suggested for Pine would make Pine do what MailManager does with
attaching (but not have the MailManager bug).  It certainly seems to be what's
needed to get your users back on the air.

The behavior you've reported with damaged attachments is *definitely* a bug,
and we'll look into it.  Either there is something wrong with c-client's
quoted-printable encoder, or Pine may be doing something wrong with preparing
the data for encoding.  Either way, we'll fix it.

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 6 Feb 1993 20:24:59 -0500 (EST)

First a couple of points:

 * If you follow the MIME canonical MIME encoding model as Nathaniel suggests
and first convert to CRLF and then B64 you have to make sure the body part
is tagged as text so the client knows to convert to the local end of line
convention. (This is related to other MIME discussions about what type to
use for an attached text file text/xxxx or application/o-s, or a newly
defined type). If you don't B64 encode it you let the mail transport
(sendmail in the case of UNIX) do the end of line conversion. This is what
Pine does now. 

 * Pine actually reads the whole file to make it's determination as to
whether it's text or not. If it has lines over some limit long or 
more than 10% characters with the 8th bit on it concludes the file is
binary. (It has to allow for ISO-8859-1 text which might have a few 8 bit
characters). 

I believe the bug in question here may be because the files have NULLs in
them and the above test for binary fails for the NULL character -- if it
has NULLs it should be binary. Then Pine tries to treat them as strings
and of course they wind up a lot shorter than they ought to. I believe this
is fixed in an unreleased version of Pine. 

I'm trying to remember exactly what the reliability problem of q-p
encoding is. I know one problem is going through an EBCDIC gateway, but I
can't think of any other. A file attachment that is text with some
non-EBCDIC characters will be corrupted as it goes through the gateway. 

It seems to me we have three choices now:

   1 Encoding everything in B64 including text attachments tag as text/plain
      + Increased reliability
      - Loss distinction between message text parts and text attachments
      - Loss of readability of text attachments by non MIME users 
     
   2 Encode text with q-p and tag as application/o-s (Current Pine)
      + Readability for non-MIME folks
      - Less reliable

   3 Create a text/file subtype for file attachments and B64 them
      + Increased reliability
      - Loss of readability for non-MIME folks

I've assumed that not handling the end of line conversion is not an option.
Three is the best if you don't worry about non-MIME folks. It is reliable
and distinguishes file attachments from text parts of the message, but it
will require an extension to MIME.

At this point I feel I'm sort of lacking data to decide which is best. 
What are the relative risks and importance of:
  
   - Is the reliability of B64 or q-p encoding really substantial? 
     What is the real risk here? How many EBCDIC gateways are there? 

   - How important readability for non-MIME folks?

   - How important is it to distinguish text parts from text attachments

We can eliminate the third trade off with a new MIME type, but we have to
choose between the other two. Personally in Pine I often will use a MIME
attachment for a text file while sending to someone without MIME because I
know they can read it, and then if they do have MIME we're encouraging
it's use. I'm also skeptical that there are that many reliability problems
with q-p encoding, though I'd certainly like to hear about them. How many
EBCDIC gateways are there, and what other problems are there? 

Lastly, it is often nice to display attachments when the user requests.
Right now if the type is application/o-s you have to use a hueristic like
Pine does when creating an attachment to determine if it is text or not to
know it it can be displayed. If the text/file subtype (or application/
text-file) where added then it would be clear which was which.

Hope this makes sense!

LL

---

From: Terry Gray <[email-masked]>
Date: Sat, 6 Feb 1993 18:15:19 -0800 (PST)

>      - Loss of readability for non-MIME folks

This is a comment just on the typing specification, not whether or not
to use b64 encoding...

As we've discussed separately, the problem with "text/file" is that it
breaks the typing model.  "file" is not a type, it is a specification of
the way the bits happen to be stored, and is useful only to distinguish
text that came from a file from text that was entered directly into the
composer. 

In order to make the text/file approach work, you would need to create a
parallel universe of "file" subtypes for all other *real* subtypes of type
text.  That is, you need to have text/plain and text/plainfile, text/rich
and text/richfile, etc.  (That's again because "file" has no bearing on
the semantic type of the information.)

So I think this is the wrong direction.  It is clear that the current
strategy in Pine of using application/o-s is also wrong, as it should
certainly be possible to display attached text files base on their MIME type.
So I believe attached text files should use the normal MIME text type.

This is orthogonal to the original question I posed of how to reliably
determine when you really have a text file, as it would be disastrous to
apply newline transformations to something that just *seemed* like it was
a text object...  obviously I didn't realize that Pine was already looking
at the entire file to make its decision, so maybe that's a non-issue.

Then the more interesting question becomes when to use b64 and when to
use q-p once you've decided you've got a text file.  Is it, as Mark
suggests, strictly an aesthetic decision, or are there cases (e.g. the
EBCDIC gateway) where q-p will lose data integrity.  (I'm assuming q-p
would be done in a way that quotes NL-From sequences that would otherwise
be munged by Bky mail delivery software.)

Since I'm personally convinced that data integrity is *infinitely* more
important than backwards compatibility for non-MIME mailers, this seems
to me to be the crucial issue.  (You can keep using Ctl-R text inclusion
until all your friends are MIME-capable!)

-teg

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 6 Feb 1993 21:56:01 -0500 (EST)

Hello Terry,

Since the two issues here are orthogonal let me reply twice. 

Yes, a bunch of subtypes like text/plaintext would clearly be a bad idea.
That's an interesting observation about the "text" major type. What
exactly does it mean? Seems that it can be taken to indicate that it is
text to be displayed as part of the initial message the user sees when we
opens the message, or does it indicate the format of the data? I'm not
sure if this is clear.

The main problem I see with tagging an attachment that happens to be text
as "text/plain" is that the receiver of will see little indication that
that part of the message is an attachment. This is the case in the way
that Pine works now and I believe is the case with Andrew. Instead the
potentially 100's of Kb will be text displayed when the user opens the
message as Pine (and Andrew I think) currently display all text parts of a
message right up front. We could change this so it only displays the first
text part, but there are many MIME messages (Barber shop MIME) for example
that wouldn't be displayed in this case. It was my impression that one of
the intents of MIME was to be able to map multimedia mail composed on a
GUI desk top into something that was manageable on a character terminal.

Would a subtype application/text-file, sort of a parallel to application/
octet-stream, solve this problem? There certainly aren't any other sub
types that we'd have to worry about. 

My personal use isn't certainly not important here. It would be nice to
hear what some of the other MIME folks think about the meaning of the
"text" type. 

LL

---

From: Terry Gray <[email-masked]>
Date: Sat, 6 Feb 1993 19:13:33 -0800 (PST)

> that part of the message is an attachment.

Or said differently, there is little indication that text entered directly
into the composer is *not* an attachment from the user's view point.  So
maybe we're looking at this backwards: the problem is distinguishing the
text that is entered directly into the composer so that it will be
displayed automatically, as opposed to displayed "on demand".

Is there some clever use of "text" and/or "message" types that could be used
to distinguish the composed text part of the message?

-teg

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 6 Feb 1993 22:20:33 -0500 (EST)

> to distinguish the composed text part of the message?

Yes, displayed on demand versus automatic is exactly the problem I have in
mind! Sorry if that wasn't clear.

I'm not sure that message type would fool Pine as it tries to display all
the sub messages and their text too. I don't see any subtypes under
message that look good though it seems like a clever idea. 

Another possiblity that's come to my mind is to use application/
X-text-file for now and as an experiment. Of course it would only work
with Pine, but it's legal MIME and other MIME implementations should just
treat it as application/octet-stream. (Better check the RFC on this one to
be sure). (Actually last I saw one MIME implementation was getting away
sending Application/octet-data which isn't even legal MIME). 

LL

---

From: Mark Crispin <[email-masked]>
Date: Sat, 6 Feb 1993 19:33:04 -0800 (PST)

Hi -

I'm staying out of the discussion about TEXT/FILE vs. APPLICATION/TEXT because
I think it's religious and I'm pretty ecumenical on that particular question.
I agree with most of the assessments of the pros and cons on each, and I'd go
with concensus.  Of course, that's easy for me to say since it's not up to me
to implement it!  ;-)

> non-EBCDIC characters will be corrupted as it goes through the gateway.

Please try to make sure Nathaniel and Ned have the details of this, because
supposedly q-p is immune to this.

>       - Loss distinction between message text parts and text attachments

I'm not sure this ``minus'' is all that important, as long as you have the
ability to save any text part to a file.  A clever user interface can do a lot
here, and Pine certainly qualifies.

>    2 Encode text with q-p and tag as application/o-s (Current Pine)

This rumor of unreliability is of concern to us.  I think that application/o-s
should indicate that newline processing should not be done, hence Pine vs
MailManager's cat-fight over text attachments (Pine attaches as text,
MailManager attaches as binary).

Whether the data is in q-p or base64 should be irrelevant to the text vs
binary issue; the typing should indicate this

Here is my feeling on the matter of the ``right'' thing to do:

ATTACHING:
 - Select binary vs text.  Text data is a sequence of mostly printable
   characters and formatters (e.g. CTRL/L) terminated by the *local* system's
   newline convention (CTRL/J on Unix, CTRL/M CTRL/J on DOS, CTRL/M on Mac,
   etc.).  So this is an osdep decision.  Text that exhibits some other
   line convention (e.g. CTRL/M's appearing in the data on Unix) should be
   automatically disqualified as text and labelled as binary.
   - if text, apply newline conversion to internet standard CR/LF.  Set type
     to be TEXT/FILE, APPLICATION/TEXT, whatever is decided as P.C. for this.
   - if binary, set type to APPLICATION/OCTET-STREAM
 - Determine base64 vs q-p encoding
   - if text, default to q-p, unless there is an excessive amount of 8th bit
     and non-printable CTRL characters (e.g. nulls, CTRL/C, etc.) that would
     make a q-p segment likely to be longer than base64.
   - if binary, default to base64, unless there is a paucity of 8th bit and
     non-printable CTRL characters that would make a q-p segment likely to be
     shorter than base64.

DETACHING:
 - apply q-p or base64 decoding as possible.
 - process the resulting data:
   - if APPLICATION/TEXT (or whatever) convert all CR/LF to local newline.
   - if APPLICATION/OCTET-STREAM, write without conversions

Essentially, the encoding format should say nothing about newlines.  It is the
type that should say this, as follows:
 - APPLICATION/OCTET-STREAM should say that ``this is a sequence of octets
   that should be written to a file without any attempt to change it to local
   data conventions.''
 - APPLICATION/TEXT should say ``this is text in Internet newline format that
   you may need to convert to local data conventions.''
TOPS-20 ;-) and DOS machines will happily ignore the difference, but everyone
else would have to do newline conversions for APPLICATION/TEXT.

I think that MIME made a big mistake in trying to mesh encodings (base64, q-p,
7bit, 8bit) with the newline problem.  I should have understood this back
then, but I didn't.  It's a terrible mixing of layers to have q-p try to
understand ``newlines'' vs. CR, LF, and CR LF.  I wonder if we can fix this in
MIME without endangering its standards track process by saying ``this is
merely a deletion of what we've realized is a bad idea''.

-- Mark --

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 6 Feb 1993 23:04:55 -0500 (EST)

On Sat, 6 Feb 1993, Mark Crispin wrote:

> ....
> .... 

Agreed!!!

LL

---

From: Nathaniel Borenstein <[email-masked]>
Date: Sun, 7 Feb 1993 10:57:00 -0500 (EST)

Rather than another type, how about adding a parameter to
application/octet-stream to indicate whether the attachment is readable
as text, e.g.

Content-type: application/octet-stream; readable=yes

or something like that?  Seems to me that this captures the one bit of
difference that we're trying to get at here with the smallest possible
change to MIME.... -- Nathaniel

---

From: Terry Gray <[email-masked]>
Date: Sun, 7 Feb 1993 09:21:31 -0800 (PST)

> Content-type: application/octet-stream; readable=yes
 
Not a bad idea, but I would turn it around:

 Content-type: text/plain; source=file

or

 Content-type: text/plain; source=message

I think it is imperative that text objects get typed as "text", lest the NL
stuff gets messed up...

-teg

---

From: Mark Crispin <[email-masked]>
Date: Sun, 7 Feb 1993 11:21:59 -0800 (PST)

> I think it is imperative that text objects get typed as "text", lest the NL

And now we've gone full circle, because now we have TEXT/PLAIN being used both
for text which is intended to be read by a human in the mail reader, and
application data which just happens to be in text/Internet new line form.

At the very least, we need a separate subtype that clearly states that:
 1) it is application data
 2) it is text using Internet newlines

It is a matter of religion whether or not it is a subtype of APPLICATION or of
TEXT, and I am ecumenical on this question; although I somewhat think that
APPLICATION is more appropriate (note that MESSAGE also has ``text'').

We can delete from q-p all the crap about newline handling if we separate text
vs. binary at the upper levels.

So, I see it as the addition on one subtype (as a bug fix) and the deletion of
text (as unnecessary); hopefully miminal damage and not enough to endanger its
progression to Draft.

---

From: Laurence Lundblade <[email-masked]>
Date: Sun, 7 Feb 1993 15:00:05 -0500 (EST)

>  Content-type: text/plain; source=file

This looks like it would work great for Pine! Possibly the parameter
should be name=<file> since that's what is used with application/o-s.
Pine will display all text without the name parameter and treat that with
the name parameter as a displayable attachment.

This will solve the problem for Pine, but for the sake of ineroperability
it seems like it would be good to persue something more formal like Mark
has suggested, so other implementors figure out how do deal with this. 

LL