Subject: pine charset translation patch with improvements for UTF-8

From: Bernhard Kaindl <[email-masked]>
Date: Sat, 3 Jan 2004 13:31:33 +0100

Hi!

I've made a number of updates to the iconv patch from Jungshik Shin
which allows charset translation from/to as many charsets as your libiconv
or glibc iconv() implementation supports(which are many, at least on Linux).

These updates are mostly UTF8-only and all of the UTF-8 changes could
be applied even if your platform does not support the iconv() function
(in which case it shoud be sufficient you just do not add -DHAVE_ICONV,
 the iconv() part will not be used then)

I saw two possible approaches to improve multibyte charset
support in pine:

1) Change vitually all char's to wchar's and use wchar-functions.
   Would be a really big job, and as the patch would be huge as
   well, I think it would be also hard to maintain with other
   patches and pine updates. It would be a major rewrite. It may
   be something for the pine maintainers if they have much time
   and I guess nobody will want to invest this time.

2) Something like UTF-8 could be used internally and if somebody
   cannot use UTF-8, external encoding converters for the terminal
   like luit(but the other way around, if one exists) could be used
   to translate to and from UTF-8. It can be done step by step.

So I went towards approach 2.

Based on UTF-8 sequence and character width checking code from
Bob Rasmussen(which he has allowed to distribute with pine),
I worked on further UTF-8 display and input-oriented improvements.

His code changed the Writechar routine to count UTF-8 character
widths instead of byte counts(thereby wrapping lines with UTF-8
characters too early).

I extended this to the message text wrapper and the index view data
generator to do the same, thereby extending the reach to the most
important parts for message browsing and viewing.

I then went on to apply the same to the message header editor so
that you can edit and enter the message header(at least the subject)
using an UTF-8 terminal.

Known problems/Todo:

1 The message header editor wraps long lines into multible lines,
  and the FormatLines also uses byte positions to cut the line into
  two parts. Of course there is a good probabity that it chooses to
  split the lines in the middle of an UTF-8 sequence, thereby making
  the sequence invalid.

2 Buffers with many invalid charaters are not yet correcty converted
  to strings of question-marks(?) on output. It works mostly but not
  yet completely for extreme cases.

3 If message headers (such as Subject:) do not identify their
  character set by using RFC2047-encoded words, they display as
  garbage. In these cases Pine could find the charset of the first
  attachment and use that for the header display.

4 When the displayed message part is text/html, the line wrap may be wrong
  because the html filter also wraps lines and may wrap in the middle of a
  Chinese character sequence. It may also occur that it does now wrap some
  lines which it should, so I appened the fixed message wrap filter after
  the html filter to prevent that it looses output by not wrapping such
  lines. But this is a workaround which does not work for all cases and
  it does not help for the case where it wraps in the middle of an UTF-8
  sequence.

5 sometimes, the To: and Cc: fields are corrupted when postponing a message

6 For certain points to work(correct UTF-8 display of the message headers in
  message view and postponing subject fields with UTF-8 characters)
  the pine config setting "pass-control-characters-as-is" has to be enabled.

7 editing address fields in the header editor which contain UTF-8 characters
  corrupts these addresses when leaving the input field(or line) (seems not
  to be a problem when testing as user root for some reason in my config)

While the first five could be rare or bearable, this last problem is bad.

If the last two problems would be fixed, it could be claimed that such pine
mostly supports UTF-8 for message view and edit if an external editor is used.

I could look at problems #1, #2 and #6, but problem #4(would be better
addressed by somebody who knows the html filter) and I'm not sure of #5.

#7 should be in the vicinity of the header editor, I suspect the "builder"s
which is called from many places in composer.c

Bernhard
--
The patch to which this mail refers is at:
http://www.suse.de/~bk/pine/iconv/4.58/pine-iconv-8h.patch.gz

If you read this mail later, there may be newer patches in this directory:
http://www.suse.de/~bk/pine/iconv/4.58/

More info on the iconv patch which also adds UTF-8 support for
message view and editing: http://www.suse.de/~bk/pine/iconv/FAQ.html
-- 
-----------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/pine/pine-info/
-----------------------------------------------------------------

---

From: Bernhard Kaindl <[email-masked]>
Date: Sat, 3 Jan 2004 15:46:52 +0100

>   split the lines in the middle of an UTF-8 sequence, thereby making

I assume this is done in pico/composer.c:FormatLines(), didn't look sharp.

> 5) sometimes, the To: and Cc: fields are corrupted when postponing a message
...
> 7) editing address fields in the header editor which contain UTF-8 characters

7) was a side effect of setting the new assumed_charset setting provided by
the patch, the attached trivial diff fixes it, I hope it fixes 5) as well.

> mostly supports UTF-8 for message view and edit if an external editor is used.

Ok, this means one of them is fixed, item 6 remains,
which should be not too hard, I assume.

Bernhard

--- pine4.58.full/pine/pine.h	2004-01-03 15:25:55.000000000 +0100
+++ pine4.58.full/pine/pine.h	2003-12-31 17:27:34.000000000 +0100
@@ -3006,6 +2978,7 @@
     } data;
 } REPLY_S;
 
+#define pico(F) call_pico(F)
 #define	REPLY_PSEUDO	1
 #define	REPLY_FORW	2	/* very similar to REPLY_PSEUDO */
 #define	REPLY_MSGNO	3
--- pine4.58.full/pine/send.c	2004-01-02 18:35:40.000000000 +0100
+++ pine4.58.full/pine/send.c	2004-01-03 15:33:16.000000000 +0100
@@ -10766,3 +10781,16 @@
 {
     return(0L);
 }
+
+int
+call_pico (ps)
+	struct pico_struct * ps;
+{
+	int ret;
+	char * assumed_save = ps_global->VAR_ASSUMED_CHAR_SET;
+	ps_global->VAR_ASSUMED_CHAR_SET = NULL;
+#undef pico
+	ret = pico(ps);
+	ps_global->VAR_ASSUMED_CHAR_SET = assumed_save;
+	return ret;
+}