Subject: Support for Unicode punctuation

From: Andrew Daviel <[email-masked]>
Date: Tue, 11 Mar 2003 12:32:56 -0800 (PST)

Certain mailers (OK, Microsoft) seem to generate Unicode punctuation
within otherwise ISO-8859-1 HTML documents, e.g. &#8217; instead of "'"
== &#44;
Pine 4.44 currently displays these as-is, though it decodes characters in 
the range 0-255.

Unfortunately, this seems to be legal. As far as I can see it is
considered correct to use an HTML character reference to encode a
character not within the current encoding. Why they just can't use the
ASCII subset of the ISO-8859-1 subset of ISO10646 for English I really
don't know (well, OK, there's no Euro symbol but I don't use that...)

It would be neat if Pine could map these punctuation marks into
ISO-8859-1 in the same was as Netscape 4 does; see below.

I'm upgrading to RedHat 8 shortly which has UTF-8 charsets by default so
no doubt I'll have to contend with fixing various old software that
needs C locale etc. etc.  :-(

http://www.unicode.org/charts/PDF/U2000.pdf

http://www.w3.org/TR/html4/charset.html
HTML uses the .. Universal Character Set (UCS), defined in [ISO10646]
..
Conforming user agents must correctly map to ISO 10646 all characters in 
any character encodings that they recognize (or they must behave as if
they did).
.. tools may employ any convenient encoding .. provided the encoding is 
correctly labeled. Occasional characters that fall outside this encoding 
may still be represented by character references . 
 "&#D;", where D is a decimal number, "&#xH;" or "&#XH;", where H is a 
 hexadecimal number,

  dec  hex
       2002 EN space
       2003 EM space
- 8208 2010 hyphen
- 8209 2011 hyphen
- 8210 2012 figure dash
- 8211 2013 EN dash
- 8212 2014 EM dash
` 8216 2018 left single quotation mark
' 8217 2019 right single quotation mark
, 8218 201A low single comma quotation mark
" 8220 201C left double quotation mark
" 8221 201D right  double quotation mark
,, 8222 201E low double comma quotation mark
* 8226 2022 bullet   (or ISO8859-1 00B7 "·")
... 8230 2026 ellipsis 
< 8249 2039 single left-pointing angle quiotaiton mark
> 8250 203A single right-pointing angle quotation mark
/      2044 fraction slash
EUR 8364
[tm] 8482

etc. etc. possible mappings/kludges

-- 
Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376
[email-masked]

-- 
-----------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/pine/pine-info/
-----------------------------------------------------------------

---

From: Andrew Daviel <[email-masked]>
Date: Wed, 12 Mar 2003 23:53:24 -0800 (PST)

> within otherwise ISO-8859-1 HTML documents, e.g. &#8217; instead of "'"
...
> ISO-8859-1 in the same was as Netscape 4 does; see below.

I have written a (probably ugly) patch for pine4.53 filter.c
which handles some of this. Things like quote marks, hyphens
& space are translated into iso-8859-1 single bytes and handled 
normally. Some complex characters (!?, ..., ''' ) are
handled using the "alternate" mechanism used for e.g. Æ -> AE
in ASCII

ftp://andrew.triumf.ca/pub/linux/pine4.53.unicode.filter.patch
http://andrew.triumf.ca/pub/linux/pine4.53.unicode.filter.patch