Subject: Re: CHARSET considerations

From: [email-masked] (Steve Summit)
Date: Fri, 14 May 93 17:29:24 -0400

In <[email-masked]>, Rick wrote:
>         Any user of Pine 3.05 (and as far as I can tell 3.07 or 2.x)
> can shoot themself in the foot  (head if you prefer)  by setting
> character-set = Zeldas_private_codepage.

This is almost certainly a bad idea, especially if (as Rick
implied in another part of the referenced message) the user can do
so by setting a default charset value in a user configuration file
somewhere.  (If users dink with the message headers themselves,
all bets are off.)

> Should the Pine developers remove this feature?

I'm not sure what the feature in question is, but if it's
something which lets users specify the value to be sent out as the
MIME Content-Type: charset, I think it's a bad idea, and should be
removed or significantly altered.

An easy mistake to make (I speak from experience) is to assume
that the charset parameter on a MIME Content-Type: line encodes
the character set used by the entity composing the message, or
the character set to be used by the entity displaying the
message. I find that the best way to think about charset is that
it is *neither*.  charset is an octet-based encoding used during
message transfer; it need bear no relation to the composing or
viewing character sets.  In the most general case, a message will
be composed using some native character set, translated
automatically to a MIME-registered charset, and translated at the
other end into a native display character set.  It should be more
likely that the charset value be selected by an automaton, not by
a human.

(If anyone finds the above paragraph startling, you're welcome to
write to me for clarification.  I'm not going to prolong this
message with additional explanations right now.)

It's not necessarily *wrong* to think of charset as having
something to do with the composing or viewing character set (in
many cases, not coincidentally, all three will be identical), but
it is very easy to make conceptual mistakes, implement
nonconformant software, or just generally misunderstand how MIME
is supposed to work if you don't explicitly separate in your mind
the concepts of composing/viewing character sets and transmission
charsets.  (You'll notice that I reinforce this distinction in my
own head and in this message by using the terms "character set"
and "charset" noninterchangeably.)

The charset situation is much like the canonical CRLF situation:
the fact that the canonical representation is identical to some
but not all of the available local representations guarantees
misunderstandings.

To be sure, automated selection of and translation to a registered
MIME charset is a non-trivial task, and mailers which are trying
to adopt MIME right away cannot be faulted for deferring
development of such functionality for a while.  However, just
letting users specify non-default, non-7-bit-US-ASCII, (non-MIME)
charsets is an open invitation to misunderstanding and
noninteroperability.

For now, composition agents which wish to allow users to use
extended character sets (such as Latin-1), but which elect to
relegate character set and/or charset selection to the user,
should either present the user with a menu of registered MIME
charsets from which to select (presumably it will be up to the
user to ensure that the editor or composition tool is actually
using a character set corresponding to the selected charset), or
(in the case of what it sounds like PINE is doing) at least filter
the user's open-ended charset selection against the list of
registered values (and perhaps also the X- pattern).

I've copied this message to the IETF character sets mailing list
([email-masked], subscription requests to
[email-masked]); any followup traffic should
be sent there, and *not* to the ietf-822 list.

					Steve Summit
					[email-masked]

P.S. to [email-masked]: despite my e-mail address,
I'm actually in Seattle, near UW.  I'd be glad to stop by one day
and talk with you guys in person about this stuff.

---

From: Mark Crispin <[email-masked]>
Date: Fri, 14 May 1993 20:47:02 -0700 (PDT)

Steve -

     Thanks for your comments.  You needn't convince me; I've been involved
with the MIME charset issue for a long time (you'll note that I am one of the
authors of the ISO-2022-JP spec).  In defense of the other Pine team members,
the sin is one of omission rather than of comission; they wanted to do
something about character sets and didn't want to wire in a table of legal
values (since it might change).

     I've suggested that as a first pass the charset should only be settable
in the system config file, and that the charset always be coerced to US-ASCII
unless the text contains 8-bit characters and/or has ``funny'' control
characters such as ESC or SI/SO.  More work would definitely be needed in this
area, but you'll appreciate that there are other, higher priorities just
now...

-- Mark --

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 17 May 1993 09:06:59 -0700 (PDT)

I haven't followed this discussion, but Pine does do a few things with
character sets.  For example if you set Pine to use ISO-8859-1 and send an
all ASCII message it will tag it US-ASCII (instead of ISO-8859-1).  Also,
it's smart enough to display the lower 128 for all incoming messages in
the ISO-8859-X characters sets and greek the ones it can't if the
character set of the receiving Pine is US-ASCII or ISO-8859-X.  Thought
that was what required for minimal MIME compliance.

Hope I haven't upgraded the sin from omission to comision....

LL

---

From: Mark Crispin <[email-masked]>
Date: Mon, 17 May 1993 15:05:50 -0700 (PDT)

Hi Laurence -

     Perhaps all that is needed is a list in the system .pinerc file of all
the valid charsets, and not let the user set her charset to one that is not in
the list.  So, perhaps Pine could have US-ASCII, ISO-2022-JP, and the various
ISO-8859-x sets wired in as an initial list, and the system file specify
additional valid sets?

     The concern is to avoid letting users do things like set it to things
such as ``Latin-1'' or ``ASCII'' or similar bogons...

     What do you think?

-- Mark --

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 18 May 1993 09:12:46 -0700 (PDT)

Yes, I think that's exactly right.  We can probably adjust the comment in
the .pinerc for now.  In the long term the .pinerc needs a lot of good
error checking.  It just hasn't been done because limited resources. 

LL

---

From: Virtual Dave Lankes <[email-masked]>
Date: Tue, 18 May 1993 13:58:37 -0400 (EDT)

It turns out is a problem in the terminal server (a cisco server by the
way). I was connecting via a telnet session that was doing odd translation
of return characters. I needed to "rlogin" and bring the default vt100
environment with me. Thanks Syracuse Computing (John Wobus) for the answer.

.........................................
: R.David Lankes (a.k.a."Virtual Dave") :
: Technical Consultant AskERIC Project  :
:   Coordinator Oliver/AEGIS Project    :
:         Syracuse University           :
:     School of Information Studies     : 
:       [email-masked]         :
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^