Subject: Extended ISO-8859-1 and Pine

From: Arthur Snoke <[email-masked]>
Date: Thu, 25 Jun 1998 09:33:44 -0400 (EDT)

On 27 May I sent the Discussion Forum a note describing problems I, as a
new PINE user, was having with some characters which were mistranslated,
and I thought that the problem was with my configuration of pine not
handling the iso-8859-1 character set translation.

I was wrong.  PINE on my sunos system handles normal iso-8859-1 fine.

But I still get some characters mistranslated, and after some sleuthing
by myself and John Berthels <[email-masked]>, I think we have
an answer and that it is likely of interest to members of the list and the
authors of pine.

In the iso-8859-1 8-bit extension of US-ASCII, the first 32 character
codes (hex 80-9F) are not used.  These ARE used in two "extensions" of the
8-bit character set, one is in the Windows Character Set and in the html
which is written by "Save as" to html in clarisworks (and maybe others)
and interpreted in Netscape 4 (but not Netscape 3).  (HTML v3.2 also does
not use those 32 character codes, so Netscape 4 is "nonstandard" in this 
respect unless there is a more recent "accepted" html character set).

Hence if you get mail using pine (or eudora) which was written by a
netscape 4.x mailer (or others, perhaps), you will get several characters
mistranslated.  The most obvious of these is the apostrophe used in
contractions like "Physician's".  Instead of being transmitted as 39
decimal (27 hex), it comes across as a curly apostrophe with the code
decimal 146 (hex 92).  Here is a line from my most recent troublesome
document: 

     The Physician^Òs Dilemma:

On my xterm window, this comes across as the character with iso-8859-1
code 210 decimal (D2 hex) -- an offset of 64 decimal (40 hex) from what I
see when I use my Sun Mailtool and look at the untranslated quoted
printable rendition.  (John writes concerning the truly unprintable
characters hex 0-31, "A common way to print these to screen is as a caret
'^' followed by a character. Carriage return (ASCII 13 decimal) is printed
as ^M - and the 'M' character is ASCII decimal 77 = 13 + 64. So Pine is
being helpful and using the same convention for the 'upper control
characters' "unprintable" characters is commonly done." 

Others from among the 32 characters which can be bothersome are smart
quotes (single and double), various length dashes and the ... (ellipsis).

Assuming we have identified the problem correctly, the question is what to
do about it?  It may be that some mail server translators (like the one
for the PINE List) can or do some renditions in 8-bit translation.  (But a
recipient using netscape would like to get his curly quotes so might not
appreciate any other translation.)  As I can see the curly quotes, etc.,
within Netscape on the same machine I use for pine, there is a font set
which knows about them, so maybe pine could not add 64 decimal to these
characters but just display them.  Or one could put in a simple filter
which translates the characters into printable ones -- hex 92 gets changed
to hex 27, etc.

I (and John) would appreciate comments and suggestions.  If someone
already has such a filter, please share.  If you want references
documenting any of the above, we have it.  (The PC Character Map and the
extended html set are not quite identical, for example.)

-----------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/pine/pine-info/
-----------------------------------------------------------------

---

From: "Adam H. Kerman" <[email-masked]>
Date: Thu, 25 Jun 1998 09:41:55 -0500 (CDT)

>From: Arthur Snoke <[email-masked]>
>Date: Thu, 25 Jun 1998 09:33:44 -0400 (EDT)

>an answer and that it is likely of interest to members of the list and the

That was fascinating; thank you.

>The most obvious of these is the apostrophe used in contractions like
>"Physician's". Instead of being transmitted as 39 decimal (27 hex), it comes
>across as a curly apostrophe with the code decimal 146 (hex 92).

In fact, I was having just such a discussion with someone from Germany a few
weeks ago about this very miscoded character!

---

From: Dennis Gurgul <[email-masked]>
Date: Thu, 25 Jun 1998 10:54:17 -0400 (EDT)

I get the same apostrophe problem logging into a DEC unix box via
SmartTerm telnet on a W-95 PC.  

Dennis J. Gurgul
Helix System Management
617.724.3169

---

From: Raja BrewHaHa <[email-masked]>
Date: Thu, 25 Jun 1998 22:57:25 -0600 (MDT)

On Thu, 25 Jun 1998, Arthur Snoke wrote:

> PINE on my sunos system handles normal iso-8859-1 fine.

That would be because it does nothing to it, except for display.

> 8-bit character set, one is in the Windows Character Set and in the html

That's code page 850, not iso-8859-1. There have been no changes to
iso-8859-1 regarding use of characters below 0x20, or for about 32
characters above 0x80.

> respect unless there is a more recent "accepted" html character set).

As I mentioned, this is code page 850. Yes. NetScape is folowing M$
towards possible backwards incompatibility if the IETF or ISO bodies
extend iso-8859-1 and others. 

>      The Physician^Òs Dilemma:

Decimal 146 is within an excluded range of iso-8859-1. It's okay for 8bit,
as in binary files. It shouldn't be printable.

> appreciate any other translation.)  As I can see the curly quotes, etc.,

In order to do that, you would have to make everyone register with the
character set that they use. Asking NetScape to document these
non-standards as

Content-Type: text/plain; charset=code-page-850

would make more sense. Pine might then know how to translate such
characters into printables for display.

> which translates the characters into printable ones -- hex 92 gets changed

I'm not sure why Pine is adding 64 to characters between 0x80 and 0xBF
inclusive. Possibly, those are already used for terminal control, which
means that you could change that code in Pine yourself if those characters
are printable on your terminal (with that font you mention loaded).

---

From: John Berthels <[email-masked]>
Date: Fri, 26 Jun 1998 12:21:27 +0100 (BST)

> That would be because it does nothing to it, except for display.

Almost...see 'control character' bit below.

> iso-8859-1 regarding use of characters below 0x20, or for about 32

Cool (I didn't know the name for it). Its code page 850. Unfortunately
this is what Netscape 4 (at least)  will produce and label as text/plain;
charset=iso-8859-1 [It and IE also understand these codes as &XXX;
'ampersand tags' in HTML documents.]

> extend iso-8859-1 and others. 

I'm not sure, but I think this range is already used in iso-8859-1 for
non-printable control characters.

> as in binary files. It shouldn't be printable.

Yup. But as above, Netscape labels this bodypart as iso-8859-1. Bad
netscape.

> characters into printables for display.

This would be the best solution, naturally.

> are printable on your terminal (with that font you mention loaded).

PINE seems to do the same thing for the 80->bf control characters as it
does for the 00->1f ones. Namely print a caret '^', followed by a
character which has the code of the control character + 64.

[e.g. ^M for carriage return. This practice is common - is it standardised
anywhere?]

Attached is a little perl script which maps a couple of these CP 850
characters to similar glyphs in iso-8859-1. I've installed it with a

display-filters=_CHARSET(ISO-8859-1)_ /path/to/the/filter/8859filt

line. Seems to work, but I don't get much of this mail.

In the spirit of "liberal in what you expect", would a config option to
perform similar translations to characters in this illegal range in a
iso-8859-1 text be a good thing to add to pine? [Would remove the overhead
of running the display filter.]

regards,

jb

-- 
John Berthels
Email: [email-masked]
X.400: /G=john/S=berthels/O=nexor/P=nexor/A=cwmail/C=gb/
#!/usr/local/bin/perl -w
#
# Small PINE filter to handle non-iso-8859-1 characters
# incorrectly produced by some mailers (eg Netscape 4)
# (PINE presents these chars to us as a caret-(code + 64) sequence)
#
# v1.0, generally untested
# 
# [email-masked], Thu Jun 25 18:42:28 BST 1998
#
# thanks to Arthur Snoke <[email-masked]> for
# ideas + determination
#
# use: _CHARSET(ISO-8859-1)_ /path/to/filter/8859filt
# in your 'display-filter' setting
#

#
# Map decimal code for character to iso-8859-1 representation.
#
%map = (
	"133" => "...",
	"145" => "'",
	"146" => "'",
	"147" => "\"",
	"148" => "\"",
	"150" => "-",
	"151" => "-",
);
while( defined( $char = getc() ) )	{
	if( $char eq "^" )	{
		$char = getc();
		if( defined $char )	{
			$num = (ord $char) - 64;
			if( exists( $map{$num} ) )	{
				print $map{$num};
			} else {
				print "^$char";
			}
		} else {
			print "^";
		}
	} else {
		print $char;
	}
}