Subject: Warning: File Attachments in Pine are encrypted

From: [email-masked] (Paul Robinson)
Date: 29 May 1994 07:33:43 GMT

This is a small screed and warning about the Pine mailer (version 3.89)
used here at Digital Express, and explains why some people were
complaining to that they could not read files I sent to them. I did want 
to warn people here about this (IMO *very bad*) behavior.  Followups
have been directed to comp.mail.pine so as not to clog up the general
newsgroup here at Digex with specific information about that application,

The Pine Mailer, if you use the 'Attchmnt: ' option to include a file on 
a message - as I do many times - instead of simply attaching an ASCII 
text file to the message (even when the file is in plain 7-bit ASCII and 
has no special characters) will "encrypt" the file using MIME Base64.  
Even if there is no binary material in the file and you did not ask for 
the file to be changed.  

It means if I mail a file to someone who can't support MIME I can't use
the attachment feature.  It means I can't mail something using attachments
unless I *know* they have a MIME compliant mailer. 

If this is a local change or caused by a set/setenv option, then it should
be changed back to set the default to not use Base64 under ordinary
circumstances.  If it is, as I suspect, a standard practice of the
program, it represents a serious, perhaps fatal mistake on the part of the
designers of the application. 

Technically Base64 is not an encryption but as far as I am concerned it
is, because it means people who do not have MIME based tools CANNOT read
files sent using the "Attachmnt: " request.  It means that for almost all
cases where you want to include a file in a message, one *must* use the ^R
function of the Pine Mailer's editor, and load the whole file into the
buffer instead of using the MIME capability to mark files with a
separator.  This defeats the automatic separator capability which Pine can
generate for individual, separate files. 

If you were trying to mail hundred-thousand byte files to people, as I do 
on a routine basis (I am releasing an update to Internet RFC 1394 and 
mailing people without FTP capability a copy of the file, which is 145K) 
this sort of behavior is *totally* unacceptable because you end up having 
to use system memory to load in a file which could simply have been file
copied into a spool file directly for mailing.

Encrypting files which are plain text when the user (like me) does not 
request it and where the encryption causes problems constitutes a 
*showstoppingly bad* feature as to effectively represent a defect in the 
program. 

To check that perhaps the file I was mailing had (unknown to me) some
hidden binary characters (values 0-31 or 128-255) and Pine was correctly
compressing the file into Base64 to allow transport, I created two small
files, consisting of simply "Line one" and "Line one/Line two of 2" (where
the / is a CR.)  It did, and when I checked (using "tail" to look 
directly at the last 100 lines of my mail spool file), sure enough, these 
7-bit Ascii files were now encrypted unreadable MIME Base64.  Pine won't 
even save this message directly, I have to use the View command to look 
at the message or to save it.  Other systems do send out unencrypted MIME 
with separators, so it is possible to *not* be doing this. 

This misfeature means if I mail a file to someone who can't support MIME I
cannot use the attachment feature.  It means I can not mail something using
attachments unless I know *beyond a shadow of a doubt* that they have a
MIME compliant mailer.  Since this may not be true, it makes the
attachment feature of the Pine Mailer effectively worthless. 

To me, this represents a major damaging attack upon the alleged claims 
that MIME would merely be an enhancement to RFC821 mail, in that if one's 
mailer was not MIME compatible, as long as you did not send binary files 
or use any special features, they would not have to upgrade.

Attaching a plain flat 7-bit-ASCII text file to a message does *not*, IMO 
constitute a special feature that requires that it be changed into a 
method that is incompatible with ALL mail systems that do not support MIME
Base64 decompression/decryption.

So this is to warn people that if you are not *absolutely certain* your 
correspondent's mailer is MIME compatible, to ONLY use the ^R read file 
feature unless and until this *showstoppingly bad* practice of Pine is 
changed to not make such a dredful mistake.  

"This is a foul and inequitious rule and it must be expunged!"
   - The Mill and the Floss

--
Reports on Security Problems: To Subscribe write [email-masked]
Paul Robinson - [email-masked] / [email-masked] / [email-masked]
Voted "Largest Polluter of the (IETF) list" by Randy Bush <[email-masked]>
Voted "Largest Polluter of digex.general" by Mike <[email-masked]>

---

From: Terry Gray <[email-masked]>
Date: Sun, 29 May 1994 08:54:13 -0700 (PDT)

Paul,
Welcome (I think :) to the Pine list/newsgroup, and congratulations on 
having the first and longest flame since the creation of the newsgroup!

Now let's talk about the issue...

In considering how to handle text attachments, we had a very clear choice
between: backwards compatibility for recipients who did not yet have MIME
compliant software, and the integrity of the attachments.  (Without some
encoding, attachments would certainly be corrupted by certain mail
gateways and/or mail delivery agents.  Even text attachments.)

In the end, the decision was made to favor integrity over backward 
compatibility.  We considered the following issues:

 o Pine is a very popular program.  We didn't want Pine to undermine
   the MIME standard by our using unsafe MIME practices.  MIME attachments
   need to be just as reliable as FTP, else users will not be able to
   depend on MIME.  This is just as true for text files as for binary,
   even though in many cases users might find the corruption of text
   files to be innocuous.

 o There is a workaround for non-MIME recipients, namely using ^R file
   inclusion, rather than attachments.  I agree that this is less 
   convenient, and for large messages less efficient... but we live in
   a world of compromises.  Note that there are also non-Pine 
   alternatives for sending files the way you want, such as the send 
   program that is part of the Metamail MIME package.

 o The problem is transient, until such time as most recipients have
   access to MIME-capable software.  All of the major mailer vendors
   have committed to MIME support, though they haven't all delivered
   yet.  And I think most service providers understand that they can't 
   honestly claim Internet mail compatibility without MIME support anymore.

 o It is common and uncontroversial for files to be uuencoded for
   Internet mail transmission, and then to be uudecoded outside of
   the recipient's mailer.  The equivalent functionality is widely
   available for MIME encoding as well.  (An example is John Myer's
   mpack/munpack program.)

The real goal should be to get everyone into the MIME game as quickly
as possible, so rather than trying to get Pine to move backwards and
risk undermining the effort because of corrupted attachments, how 'bout 
leaning on the sites that don't yet provide a way for their clients to
handle MIME?

-teg

---

From: Alan Barrett <[email-masked]>
Date: Sun, 29 May 1994 18:35:52 +0200 (GMT+0200)

> gateways and/or mail delivery agents.  Even text attachments.)

It is possible to have a mail user agent do something like this when
deciding what Content-Transfer-Encoding to use for an attachment:

        if atachment looks vaguely like text then
            ask user if attachment should be treated as text
        endif
        if attachment is not text then
            use Content-Transfer-Encoding: base64
        else
            /* attachment is text: it looks like text, and the user
             * has confirmed that it should be treated as text */
            if it would be safe to use 7bit C-T-E then
                use Content-Transfer-Encoding: 7bit
            else if it would be safe to use 8bit C-T-E then
                use Content-Transfer-Encoding: 8bit
            else
                /* fallback for text that can't use 7bit or 8bit C-T-E */
                use Content-Transfer-Encoding: quoted-printable
            endif
        endif

The user would always have the option of forcing base64 encoding if
he thought that a message with another encoding would be damaged in
transit, but normal text attachments would be sent with the encoding
most friendly to non-MIME-capable recipients.

--apb (Alan Barrett)

---

From: [email-masked] (Paul Robinson)
Date: 30 May 1994 00:30:16 GMT

Terry Gray <[email-masked]> wrote in <[email-masked]> 
as follows:

> Paul, Welcome (I think :) to the Pine list/newsgroup, and 
> congratulations on having the first and longest flame since the 
> creation of the newsgroup! 

If the system wasn't broken, the whole commentary wouldn't have been 
needed.  Whether it's a "flame" or not is another issue.  The software 
acts in a way that is IMO defective for this purpose.  That it can be
used in other purposes is irrelevant, it's not much good as far as I 
am able to work with it.

> In considering how to handle text attachments, we had a very clear 
> choice between: backwards compatibility for recipients who did not yet 
> have MIME compliant software, and the integrity of the attachments.  
> (Without some encoding, attachments would certainly be corrupted by 
> certain mail gateways and/or mail delivery agents.  Even text 
> attachments.)

Fine, but give *me* the ability to turn it off when either I know I don't 
need this capability or I can't use it.

> compatibility.  We considered the following issues:

How about both?

>   the MIME standard by our using unsafe MIME practices.  

Which it does *out of the box* by not allowing non-mime systems to read 
files it creates.

>  MIME attachments need to be just as reliable as FTP, else users will 
>  not be able to depend on MIME.  

True.  But your program now makes them *non reliable*.

>  This is just as true for text files as for binary, even though in many 
>  cases users might find the corruption of text files to be innocuous.

True, but when there is a possibility that I don't want it that way, let 
me decide if I want to not use Base64.

>   inclusion, rather than attachments.  

But then, there is no inter-file separator.

>   I agree that this is less 

Nonetheless it would have been possible to do both: to provide reliable 
Base64 protection for most files, and allow people to override that 
protection when needed.  Additionally, they could identify a file 
directly.

>   honestly claim Internet mail compatibility without MIME support anymore.

Which may be years.  Beyond that, perhaps I want to create a file 
specifically for displaying *including* the separators; I can't do that.

>   available for MIME encoding as well.  (An example is John Myer's

The difference being the user's explicit uuencoding of a file.  Here, 
Pine is deciding on its own to encrypt my file.  For most users that 
might be acceptable, but I should be able to decide how my mail goes
out if I need to.

> leaning on the sites that don't yet provide a way for their clients to

Of all the impudent statements!  'Lean on' someone else to change their 
system to work with a mailer that doesn't follow the standards!  That's a 
twist.  

The MIME standards are supposed to make the changes transparent to 
someone using non-MIME mail, which this doesn't.

Now, there is a simple fix to the problem to add the capability to 
identify files as text files that you don't want translated.

I figured out what to do; change the routine that reads the file name; 
since it doesn't allow < or [ in the name, those can be used to mark 
options for the file, e.g.

Attachmnt: 1. .pinerc[ascii] (2KB) ""

Would send the file ".pinerc" and force it to assume 7-bit ascii even if 
wrong.

Attachmnt: 1. rfc1394.zip[binary,type=application/zip] (45K) ""

Would tell the system to use base64 compression and set the header
type to application/zip to indicate a zip format file in the header.  
This would also allow people to mark files for a particular application, 
e.g. 

Attachmnt: 1.  BUTERFLY.BMP [binary, type=application/windows-bmp] (234K) ""

---

From: Roman Czyborra <[email-masked]>
Date: Mon, 30 May 1994 03:45:27 +0200

> The Pine Mailer, if you use the 'Attchmnt: ' option to include a
> file on a message - as I do many times - instead of simply attaching
> an ASCII text file to the message (even when the file is in plain
> 7-bit ASCII and has no special characters) will "encrypt" the file
> using MIME Base64.  Even if there is no binary material in the file
> and you did not ask for the file to be changed.

First of all, base64 is neither an encryption nor a compression
method, so please don't call it like that.  It is a transfer-encoding,
very simple, yet the most reliable there is for mailing files.  Pine's
documentation explains when and why base64 was chosen.  Even if you
haven't read the Technical Notes, you should've noticed what an
attachment is if you've used it that many times.

> [^R] is *totally* unacceptable because you end up having to use
> system memory to load in a file which could simply have been file

I would assume that attachments use even more system memory.

> To me, this represents a major damaging attack upon the alleged
> claims that MIME would merely be an enhancement to RFC821 mail, in
> that if one's mailer was not MIME compatible, as long as you did not
> send binary files or use any special features, they would not have
> to upgrade.

Minus the hype, you have a point here.  Perhaps a future Pine could
ask "Are you willing to risk some data corruption and attach the file
unencoded?"  Until then you might want to format your mass mailings
with elm or mailto which give you more control over the encoding.  Or
how about using the earlier versions of Pine?  (I'm sure Pine 2000
will have a configurable list of hosts known to be 8bit-clean, and
decode all the quoted-printables when filing messages).

---

From: Terry Gray <[email-masked]>
Date: Sun, 29 May 1994 19:33:25 -0700 (PDT)

> system to work with a mailer that doesn't follow the standards!  That's a 

Paul, 
Pine *does* follow the MIME standard.  The problem is with legacy 
mailers that do not yet.  Pine's attachment feature was not designed
to be used when sending to non-MIME-capable recipients, for the reasons 
outlined previously.

You are a sophisticated user focused on a very specific requirement
that is at odds with some of our requirements.  Sophisticated users
have many options...

-teg

---

From: [email-masked] (Jeff Bishop)
Date: 30 May 1994 03:55:33 GMT

In article <[email-masked]>,

>> creation of the newsgroup! 

Pointing out an obvious mistake in the program is hardly what I consider a
flame.  MIME encryption should be an optional feature; to assume that every
receipient will have MIME-aware tools is a mistake, and Paul was right to
point that out.

>> compatibility.  We considered the following issues:

Integrity isn't worth much if the recipient can't read it.  Pine may be a
popular program, but it isn't popular enough that we can automatically
assume that everyone we send email to will use it or even have it available
to them.  If/when MIME becomes a universal standard, it will be reasonable
to make it automatic and drop ASCII altogether.  Until then it makes a lot
more sense to allow file transmissions everyone else can read.

---

From: Paul Tarr <[email-masked]>
Date: Mon, 30 May 1994 09:56:23 -0700 (PDT)

> receipient will have MIME-aware tools is a mistake, and Paul was right to

: Baloney - It is not an obvious mistake. Maybe you want an enhancement 
but the pine group has graciously supplied us with a nice mail program at 
no charge and doesn't deserve this kind of flak. The pine group has 
clearly stated their objectives and criteria for development and have
delivered a good product meeting those objectives. The pine group is 
flexible and has modified their criteria over the last nine months that I 
have subscribed to the list, as a result of user requests. You may 
disagree with their objectives but if your request doesn't fit into the 
pine charter then build your own program instead of flameing the pine 
development team! I don't believe that the pine development team is in 
business to customize pine for every user for free.

> more sense to allow file transmissions everyone else can read.

: Ditto the above comment. One can always use ^R if you want ASCII. The 
susprise to me was that uuencode has integrity problems but that being the 
case, I see no reason why pine should not make the break and make the 
default MIME. I gather that the pine team is open to considering 
enhancements that allow one to send attachments using methods other than 
pine but, if I understand the pine charter correctly, every enhancement 
pushes the pine mailer further from the stated objective of keeping the 
program simple for the new or novice user. Give the pine team some 
credit. I think they have done a good job balancing requirements and 
"power" user wishes and from the mail messages I see posted thay are 
certainly open to suggestions.

				CHEERS from
				Paul Tarr 
				([email-masked])

---

From: Jeff Bishop <[email-masked]>
Date: Mon, 30 May 1994 12:22:54 -0500 (CDT)

On Mon, 30 May 1994, Paul Tarr wrote:
> : Baloney - It is not an obvious mistake. 

Assuming that every recipient will have a particular format at their 
disposal is an obvious mistake.  Most non-Pine users I know don't have 
MIME-aware tools and it is silly to pretend that they do.

> Maybe you want an enhancement 
> no charge and doesn't deserve this kind of flak. 

If they didn't want "flak" they need not have established this list or 
pine-bugs.  That they did implies that they are interested in knowing 
what aspects can stand improvement.

> default MIME.

No one objects to making MIME the default.  It's the fact that the user 
cannot override it without loading the file into the editor which is 
annoying.  Adding an override option to be specified in .pinerc would not 
interfere with this objective at all.

> I gather that the pine team is open to considering 
> program simple for the new or novice user.

See above.  The novice user need not know the override option is 
available, while the power user would specifically enable it.  To anyone 
who doesn't Pine would function exactly the way it does now.  Hardly 
grounds for saying "write your own program" rather than fixing what's broke.

> Give the pine team some 

Like labeling as "flames" any specific criticisms?  Somehow that doesn't 
strike me as particularly "open."

---

From: Terry Gray <[email-masked]>
Date: Mon, 30 May 1994 11:40:45 -0700 (PDT)

On Mon, 30 May 1994, Jeff Bishop wrote:

> flame.  

Jeff,
My opening comment to Paul R. was meant humorously in the spirit of his
own signature lines about being voted "Largest polluter... " I had thought
one smiley in my sentence was sufficient, but I misjudged.  Sorry about
that. 

As to Base64 encoding of all attachments being "an obvious mistake in the
program", we disagree.  MIME mailers should be able to deliver attachments
with the same robustness as FTP.  This goal is best achieved with Base64.

If you'd like to propose a new MIME type that explicitly conveys the
intent that file integrity (and therefore C-T-E) doesn't matter, e.g. 
TEXT/APPROXIMATION, then we would have some common ground... but helping
recipients get MIME-capable s/w --which they are going to need for other 
things anyway-- seems like a better place to invest effort.

-teg

---

From: Jeff Bishop <[email-masked]>
Date: Mon, 30 May 1994 14:38:46 -0500 (CDT)

On Mon, 30 May 1994, Terry Gray wrote:

> TEXT/APPROXIMATION, 

That'd be fine, as long as it doesn't assume anything about the 
recipient's tools.  Many, probably most file transmissions are text-based 
and an approximation would be fine.

> then we would have some common ground... but helping

No argument there - I'd be the last to argue that MIME capability should 
be dropped or even relegated to non-default status.  This is why I 
suggested ASCII override as an option to be set in pine.conf or .pinerc, 
neither of which is likely to be tampered with by a novice user anyway.

BTW - any particular reason why Pine doesn't allow a separate "Reply-to:" 
field?  That too could be incorporated without confusing the newbies, 
either by adding it to the .pinerc options or by telling Pine to use the 
environmental variable REPLYTO.

I know some will say "if you want that use Elm" but it seems like a few 
such changes to Pine would render Elm wholly redundant. 

_________________________________________________________________________
Jeff Bishop				"I hate quotations.  Tell me what
Graduate Student			 you know."
Department of Linguistics		
Northwestern University			-Ralph Waldo Emerson

---

From: Terry Gray <[email-masked]>
Date: Mon, 30 May 1994 13:20:12 -0700 (PDT)

> environmental variable REPLYTO.

Pine 3.90 will support customized headers, including Reply-To:

We're hoping to have a beta version available by the end of June
(that's "late spring" to us in Seattle :)

-teg

---

From: [email-masked] (Bill Card)
Date: Mon, 30 May 1994 15:38:47 GMT

I used the "attachment" function for txt files when I was new at PINE.  
It seemed to make sense.

Unfortunately, I did not know that MIME would be used, and neither did my 
correspondents.  I got upset messages back, saying things like "please 
don't encode the attachments".  So now I use ^R, and everyone is pleased.

If MIME is to be the future standard, how do we all learn to use it?

One posting mentions the documentation for PINE, which certainly ought to 
include a discussion of MIME.  How can I get the documentation?

[email-masked]
-- 
----------------------------------------
Bill Card
[email-masked]

---

From: [email-masked] (Michael Storz)
Date: Tue, 31 May 1994 18:47:31 GMT

In article <[email-masked]>, [email-masked] (Paul Robinson) writes:
|> Terry Gray <[email-masked]> wrote in <[email-masked]> 
|> as follows:

Unfortunately I cannot find Terrys message, therefore I must comment Pauls
one.

|> 
|> > Paul, Welcome (I think :) to the Pine list/newsgroup, and 
|> > congratulations on having the first and longest flame since the 
|> > creation of the newsgroup! 

The mail from Paul is very long but is definitely no flame, but describes
a very annoying feature of pine. 

|> > In considering how to handle text attachments, we had a very clear 
|> > choice between: backwards compatibility for recipients who did not yet 
|> > have MIME compliant software, and the integrity of the attachments.  
|> > (Without some encoding, attachments would certainly be corrupted by 
|> > certain mail gateways and/or mail delivery agents.  Even text 
|> > attachments.)

This absolutely nonsense. If this would be true, then you MUST encode 
the first body part of every mail in addition. This is not done, therefore
there is definitely no requirement to do it for every attachment.

If you do not offer the possibility to specify the content-type of the
attachment, then you MUST automatically determine the kind of the attachment.
In case of text files used as attachment you have to scan the file which
will be attached and determine if it is a text file or not. If the file
uses only chars out of US-ASCII then no encoding should be done. If some
of the chars are from an 8-bit code then quoted-printable has to be used.
BASE64 should be used only for the rest of the files.

Using BASE64 always as the encoding means the progam is just not smart
enough. 

|> 
|> > o Pine is a very popular program.  We didn't want Pine to undermine
|> >   the MIME standard by our using unsafe MIME practices.  

Using no encoding for US-ASCII, quoted-printable for ISO-... and
BASE64 for others is absolutely compliance with the MIME standards. 
quoted-printable was exactly defined for such purpose. There is no sentence
in the whole standard which implies or suggests to use BASE64 for every
body part.

|> 
|> >  MIME attachments need to be just as reliable as FTP, else users will 
|> >  not be able to depend on MIME.  
|> 

They are. FTP has two modes for transfering files, if you remember. One
for text files, which is the default and one for binary files and that
one you have to request explicitly.
|> >  This is just as true for text files as for binary, even though in many 
|> >  cases users might find the corruption of text files to be innocuous.
|
|> > o There is a workaround for non-MIME recipients, namely using ^R file
|> >   inclusion, rather than attachments.  
|> >   I agree that this is less 
|> >   convenient, and for large messages less efficient... but we live in
|> >   a world of compromises.  Note that there are also non-Pine 
|> >   alternatives for sending files the way you want, such as the send 
|> >   program that is part of the Metamail MIME package.
|> 

A workaround always means something is broken. 

|> 
|> > o The problem is transient, until such time as most recipients have
|> >   access to MIME-capable software.  All of the major mailer vendors
|> >   have committed to MIME support, though they haven't all delivered
|> >   yet.  And I think most service providers understand that they can't 
|> >   honestly claim Internet mail compatibility without MIME support anymore.

Non MIME aware user agents will be around for years. The aim of all extensions
made by IETF working groups is backward compatibility. pine is not committed to
this.

In addition using BASE64 as encoding will increase the size of files at about
30 % .

|> > o It is common and uncontroversial for files to be uuencoded for
|> >   Internet mail transmission, and then to be uudecoded outside of
|> >   the recipient's mailer.  The equivalent functionality is widely
|> >   available for MIME encoding as well.  (An example is John Myer's
|> >   mpack/munpack program.)

No, definitely no. I have seen no one which encodes their text files
with uuencode if the charset is US-ASCII, as it is for most mail messages.
uuencode is ONLY used for non-US-ASCII text files like files from 
Wordperfect or MS-Word or for binary files.

|> 
|> > The real goal should be to get everyone into the MIME game as quickly
|> > as possible, so rather than trying to get Pine to move backwards and
|> > risk undermining the effort because of corrupted attachments, how 'bout 
|> > leaning on the sites that don't yet provide a way for their clients to
|> > handle MIME?

It is wishable that everybody uses MIME capable UAs, but this has nothing
to do with using the wrong encoding.

|> 
|> The MIME standards are supposed to make the changes transparent to 
|> someone using non-MIME mail, which this doesn't.

-- 

Michael Storz
================================================================================
                                      !   X.400 : G=Michael;S=Storz;OU=lrz;
   Leibniz-Rechenzentrum              !           P=lrz-muenchen;A=d400;C=de
   Barer Str. 21                      !   RFC822: [email-masked]
   80333 Muenchen                     !   Fax   : +49 89 2809460
   Germany                            !   Tel   : +49 89 2105-8720
================================================================================