Subject: ISO to CP translation in Pine 394

From: Barry Landy <[email-masked]>
Date: Mon, 8 Jul 1996 13:09:57 +0000 (GMT)

There seems to be a bug in the behaviour of the translation of
ISO-8859-1 to PC code pages. 

The technical notes specify as follows:

==============================   
   In the case of PC-Pine, the character values cannot be passed through
   to the display device unaltered since MS-DOS uses various non-standard
   character sets called "Code Pages".
   
   The mapping between DOS Code Page and standard character set is
   controlled by the "character-set" variable in the PINERC file and the
   PC's installed Code Page. PC-Pine will automatically map common
   characters in IBM Code Pages 437, 850, 860, 863, and 865 to ISO-8859-1
   and back when the PINERC has "character-set=ISO-8859-1". Pine will
   also map common characters for IBM Code Page 866 to ISO-8859-5 and
   back when "character-set=ISO-8859-5". The mappings are bi-directional,
   and applied to all saved text attachments in the defined character
   set, messages exported, etc.
   
   Alternatively, the translation tables can be configured externally and
   applied at run time whenever the "character-set=" variable is set to
   something other then "US-ASCII" (the default). PC-Pine looks in the
   text file pointed to by the environment variable "ISO_TO_CP" for the
   table to use for mapping text matching the type defined by the
   "character-set=" variable into the local Code Page value. PC-Pine
   looks in the text file pointed to by the environment variable
   "CP_TO_ISO" for the table to use for mapping text in the local Code
   Page into outbound text tagged with the "character-set=" variable's
   value.
   
   A text file containing a character set mapping table is expected to
   contain 256 elements where each element is a decimal number separated
   from the next element by white-space (space, tab or newline, but no
   commas!). The index of the element is the character's value in the
   source character set, and the element's value is the corresponding
   character's value in the destination character set. 
======================= 

I have sent myself a trial message contain a few pound signs. I have
tried displaying it in various versions of Pine (OS2 391 or 394, PC-WAT
391/394) with or without the ISO_TO_CP settings. I get the following
results, where "No ISOTOCP" means what it says, "X<->X" means an
ISOTOCP setting mapping all characters onto themselves, and "163->156"
means an ISOTOCP table which maps 163 onto 156, and all other chacaters 
onto themselves. Note that 163 is pound in 8859-1 and 156 is pound in
CP850/CP437, whereas u-acute is 163 in CP850/CP437.
"163-65" maps 163 onto 65 and leaves all others alone (65 is A).

                    394 WAT     391 WAT      394 OS2     394 WIN
No ISOTOCP          ?            pound        u-acute    pound
X<->X               u-acute      u-acute      u-acute    pound
163->156            ?            pound        u-acute    ?
163->65             A            A            u-acute    A

To check that the SET variables influence the OS/2 port at all, I
experimented with the other direction (CP_TO_ISO). With an X<->X table,
pound is translated (bizarrely?) as ^blob (where by blob I mean a black
square); with a 156-163 table, pound becomes 163 (displayed as u-acute).
The details dont matter; the fact there is a change means that Pine is
noticing the CPTOISO table.

The following problems are evident: 
a) The OS2 port is not honouring the promise to translate to the
codepage, and is not reading ISO_TO_CP
b) The WAT port isnot getting the ISO to CP right, and is outputting ?
for £ (I dont know where that ? comes from!)
c) I dont understand why forcing 163-> 156 causes a ? out of 394Win and
394WAT, but gets a pound from 391WAT.

(The W16 port of 394 gets the output of pound right, but Windows is
defined as using a mapping of 8859-1 (Windows code page 1252))

-------------------------------------------------------------------------------
Barry Landy                        Computer Laboratory:+44 1223 334600
Head of Systems and Development    Direct line:        +44 1223 334713
University of Cambridge Computing Service
New Museums Site                   Email:[email-masked]
Pembroke Street, Cambridge CB2 3QG