Subject: pine utf-8 summary and patch

From: Bernhard Kaindl <[email-masked]>
Date: Mon, 26 May 2003 23:28:05 +0200

Hi,
   I want to report the latest state of UTF-8 support for pine which
I am aware of. Please add anything I might be missing or missing to
refer to, thanks!

UTF-8 is only possible if your terminal supports it, this should be
the case for recent XFree86 Code if the Unix-Locale used ends with
.UTF-8 like en_US.UTF-8 this has to be set in the environment when
starting e.g. xterm. Other termials capable of UTF-8 like mlterm
should be able to switch character encodings on the fly.

For complete utf-8 display it seems to be neccesary to enable the pine
config option pass-control-characters-as-is to prevent recent versions
of pine from dropping characters in the range of 0x80 to 0x9f which are
needed to send many UTF-8 characters to the terminal.

If all mails are converted to utf-8, e.g. by using procmail rules,
then it should be sufficient to set the charset option in the config
to utf-8 to prevent pine from attempting to translate the mail and
have all pre-converted mails displayed.

pico seems to support utf-8 as well, so even the standard editor
should work just in case, to send utf-8 mail then.

If you want to pine to convert mails to utf-8 for mail view and reply,
you can use the display-filters config option, but it's slow and you
need to convert the mail headers(to, from, subject) separately.

Jungshik Shin sent a patch it make it work better with UTF-8 and i18n
in general:

 http://mail.nl.linux.org/linux-utf8/2002-07/msg00012.html

It allows to convert the mail headers and provides new config options
to configure mime charset names names to charset names recognized by
iconv.

It also includes a good set of display filters, but using it adds
a delay to the message diplay in the viewer for me.

I updated this patch to 4.55 and made the use of display-filters obsolete
by replacing them with a streaming iconv() filter used by pine directly.

This patch should work on most modern Linux Distributions, I developed
it on SuSE Linux 8.2, but should work as soon as your xterm supports
UTF-8 correctly and your GLIBC has a working implementation of iconv().

This patch should work on Solaris as well if you define HAVE_ICONV
for your target:

./build EXTRACFLAGS="-DHAVE_ICONV" target

Bernhard

=======================================================================

PS: This patch only addresses the character set conversion issue, it
does not address the fact that pine thinks that the output of one byte
to the terminal moves the cursor by one at the same time. This is of
course not the case with any multibyte character encoding.

To fix this, a separate patch which addresses this issue is needed,
but this patch can be developed completely independent of this patch
because it addresses entirely different issues.

But if the bulk of your mail includes only a few non-ASCII characters,
this is not a major problem, the only thing you'll see that lines with
non-ASCII characters end too early and are instead wrapped to the next
line.

PPS: I've noticed a bug in the patch which is not yet fixed:

If I set assumed-charset in the config and view a message with
mailing list management information which pine appends in an
editoral at to bottom of the mail body, the mailing list
management editoral is completely messed up.

PPPS: Improvements which are missing:

1)

If the mail does not specify an encoding, US-ASCII is assumed by default
which does not always match your most common case.

You can change this by setting the assumed-charset config setting which
is added by this patch, but it would be good to see if e.g. the subject
contains a locale tag for for some encoding and use this encoding then
so that assumed-charset is not required to be set in this case to see
the mail converted using the best conversion.

2)

Some Windows Mailers mislabel their mails with an ISO-8859 charset
but are indeed encoded in Windows-1252.

It would be good to recogize these Windows Mailers and provide
automatic mappings from ISO-8859 to Windows encondings in this
case.

The charset-aliases setting provides such aliases already, but
they are not predefined.

I think it would be good to to make these mappings dependent of
the mailer and provide some mappings which are known to be good
by default which can be overridden by the user just in case.

Example config settings for use with this patch are:

charset-aliases=iso-8859-1:windows-1252,
        iso-8859-12:windows-1250,
        iso-8859-15:windows-1252,
        iso-8859-2:windows-1252,
        iso-8859-11:windows-874,
        tis-620:windows-874,
        gb2312:gb18030,
        gbk:gb18030,
        euc-cn:gb18030,
        ks_c_5601-1987:x-windows-949,
        5601:x-windows-949

iconv-aliases=x-windows-949:mscp949,
        euc-kr:mscp949

PPPPS: If you have an UTF-8 Terminal but your pine OS does not have iconv():

libiconv like in Linux glibc:
 http://www.gnu.org/software/libiconv/

The portable iconv implementation by Bruno Haible:
 ftp://ftp.ilog.fr/pub/Users/haible/gnu/libiconv-1.5.1.tar.gz

The portable iconv implementation by Konstantin Chuguev:
 ftp://ftp.urc.ac.ru/pub/local/OS/Unix/converters/iconv-0.4.tar.gz

PPPPPS: Why Unicode? (Edited from the Unicode HOWTO)

People in different countries use different characters to represent the words
of their native languages. Nowadays most applications, including email systems
and web browsers, are 8-bit clean, i.e. they can operate on and display text
correctly provided that it is represented in an 8-bit character set, like
ISO-8859-1.

There are far more than 256 characters in the world - think of cyrillic,
hebrew, arabic, chinese, japanese, korean and thai -, and new characters
are being invented now and then. The problems that come up for users are:

* It is impossible to store text with characters from different character
  sets in the same document.
* As long as every document has its own character set, and recognition
  of the character set is not automatic, manual user intervention is
  inevitable.
* New symbols like the Euro are being invented. ISO has issued a new
  standard ISO-8859-15, which is mostly like ISO-8859-1 except that it
  removes some rarely used characters (the old currency sign) and replaced
  it with the Euro sign. If users adopt this standard, they have documents
  in different character sets on their disk, and they start having to think
  about it daily. But computers should make things simpler,
  not more complicated.

The solution of this problem is the adoption of a world-wide usable
character set. This character set is Unicode http://www.unicode.org/

PPPPPPS: More Information:

Introduction to i18n - Coded Character Sets And Encodings in the World:

 http://www.debian.org/doc/manuals/intro-i18n/ch-codes.en.html

This text is a very comprehensive one-stop information resource on UTF-8
on Linux/Unix Operating systems:

 http://www.cl.cam.ac.uk/~mgk25/unicode.html

-- 
-----------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/pine/pine-info/
-----------------------------------------------------------------

---

From: Bernhard Kaindl <[email-masked]>
Date: Mon, 26 May 2003 23:39:15 +0200

Here comes the patch (attached)

Note: UTF-8 is not only good when you recieve mails in different
locales which are not 100% compatible.

I do UTF-8 only because operating systems and mailers start to
switch to utf-8 and using utf-8 myself is the only way to be
compatible with it.

I also start to see nice benefits of this, for example, this patch
finally allows me to see all kinds of annotations which Windows Users
use to their mails:

„upper-lower quotes“ (I have some more but could not find them atm)

And some random signatures:

睡眠不足はいい仕事の敵だ。

Gáspár・ガーシュパール・Гашпар・가스팔・Γασπαρ
ᏱᎦᏊ ᎣᏌᏂᏳ ᎠᏓᏅᏙ ᎠᏓᏙᎵᎩ
ᏂᎪᎯᎸᎢ ᎾᏍᏋ ᎤᏠᏯᏍᏗ ᏂᎯ.

G̳á̳s̳p̳á̳r̳ ・ ɡaːʃpaːr ・ גאשפאר・

If you have SuSE Linux 8.2 and you set your locale to en_US.UTF-8,
start an xterm, select "Unicode Best" from the xterm font menu,
and show this mail with patched pine configured with character-set utf-8
and pass-control-character-as-is on and you see all the above!

And you can edit this with pico and vim!

Best Regards,
Bernhard Kaindl

> ./build EXTRACFLAGS="-DHAVE_ICONV" target

diff -ru pine4.55/pine/filter.c pine4.b1/pine/filter.c
--- pine4.55/pine/filter.c	2003-04-07 21:36:32.000000000 +0200
+++ pine4.b1/pine/filter.c	2003-05-26 20:45:04.000000000 +0200
@@ -65,6 +65,9 @@

 #include "headers.h"
+#ifdef HAVE_ICONV
+#include <iconv.h>
+#endif

 /*
@@ -2164,6 +2167,123 @@
     }
 }
 
+#ifdef HAVE_ICONV
+static iconv_t iconv_desc;
+
+/*
+ * This filter converts characters in one character set (the character
+ * set of a message, for example) to another (the user's character set)
+ * and is also capable of translating a single byte and multibyte character
+ * sets into other character sets.
+ */
+void
+gf_convert_utf8(f, flg)
+    FILTER_S *f;
+    int       flg;
+{
+    static int einval_inbytesleft;
+    GF_INIT(f, f->next);
+
+    switch (flg) {
+    case GF_DATA: {
+	int conv, inbytesleft = eob - op, outbytesleft = eib - ip;
+	while (1) {
+	    if (!outbytesleft || !inbytesleft) {
+		dprint(2,(debugfile, "gf_convert_utf8: flushing output\n"));
+		GF_FLUSH(f->next);
+		outbytesleft = eib - ip;
+	    }
+	    if (!inbytesleft) {
+		dprint(2,(debugfile, "gf_convert_utf8: need more input!\n"));
+		GF_CH_RESET(f);
+		break;
+	    }
+	    if (inbytesleft == einval_inbytesleft) {
+		GF_FLUSH(f->next);
+		dprint(2,(debugfile, "utf8 abort, no more valid input!\n"));
+		op = eob;
+		GF_CH_RESET(f);
+		break;
+	    }
+	    conv=iconv(iconv_desc, &op, &inbytesleft, &ip, &outbytesleft);
+	    if (conv > 0) {
+		dprint(2,(debugfile, "conv: %d, il: %d, ol: %d\n",
+			    conv, inbytesleft, outbytesleft));
+	    } else switch(errno) {
+	    case E2BIG:
+		dprint(2,(debugfile, "2big: outbytesleft=%d\n", outbytesleft));
+		outbytesleft = 0;
+		break;
+	    case EILSEQ: {
+#if 1
+		unsigned char c;
+		GF_PUTC(f->next, '[');
+		if (GF_GETC(f, c)) {
+		    char hexout[2];
+		    inbytesleft--;
+		    sprintf(hexout, "%2x", c);
+		    GF_PUTC(f->next, hexout[0]);
+		    GF_PUTC(f->next, hexout[1]);
+		}
+		GF_PUTC(f->next, ']');
+#else
+		GF_PUTC(f->next, '?');
+#endif
+		dprint(2,(debugfile, "eilseq: 0x%02x ibl: %d, obl: %d\n",
+			    c, inbytesleft, outbytesleft));
+		iconv(iconv_desc, NULL, NULL, NULL, NULL);
+		break;
+	    }
+	    case EINVAL:
+		dprint(2,(debugfile,
+			    "einval: %d, ol: %d, incomplete input: 0x%02x\n",
+			    inbytesleft, outbytesleft, (unsigned char) *op));
+		einval_inbytesleft = inbytesleft;
+		inbytesleft = 0;
+		break; 
+	    } /* switch (errno) */
+	} /*  while (1) */
+	GF_END(f, f->next);
+	break;
+    } /* GF_DATA */
+    case GF_RESET:
+    	einval_inbytesleft = 0;
+	break;
+    case GF_EOD:
+	if (iconv_desc)
+	    iconv_close(iconv_desc);
+	iconv_desc = NULL;
+	GF_FLUSH(f->next);
+	(*f->next->f)(f->next, GF_EOD);
+	dprint(1, (debugfile, "-- gf_eod convert_utf8: %d\n", iconv_desc));
+    } /* switch (flg) */
+}
+
+/*
+ * Resolves the charset to convert and allocates the character set
+ * conversion descriptor for gf_convert_utf8()
+ */
+void *
+gf_convert_utf8_opt(charset)
+    unsigned char *charset;
+{
+    dprint(3, (debugfile, "-- charset before resolution: %s\n", charset));
+    charset = (unsigned char*) 
+	resolve_charset_alias(charset,ps_global->VAR_CHAR_SET_ALIASES);
+    /* moved after resolve_charset_alias, to allow aliasing X-UNKNOWN
+       to another charset and not have AR_ASSUMED_CHAR_SET. Another idea
+       would be to check if the subject or from had charset tags and use
+       this charset (we could just use the last charset variable from
+       conv_sstrcpy() in mailview.c) */
+    if (ps_global->VAR_ASSUMED_CHAR_SET &&
+	(! strucmp(charset, "US-ASCII") || ! strucmp(charset, "X-UNKNOWN"))) 
+	charset = (unsigned char*) ps_global->VAR_ASSUMED_CHAR_SET ;
+    dprint(3, (debugfile, "-- charset after resolution: %s\n", charset));
+
+    iconv_desc = get_iconv_d(ps_global->VAR_CHAR_SET, charset);
+    return((void *) &iconv_desc);
+}
+#endif
 
 /*
  * This filter converts characters in one character set (the character
diff -ru pine4.55/pine/init.c pine4.b1/pine/init.c
--- pine4.55/pine/init.c	2003-04-16 08:20:52.000000000 +0200
+++ pine4.b1/pine/init.c	2003-05-26 16:24:33.000000000 +0200
@@ -214,6 +214,14 @@
 
 CONF_TXT_T cf_text_character_set[] =	"Reflects capabilities of the display you have. Default: US-ASCII.\n# Typical alternatives include ISO-8859-x, (x is a number between 1 and 9).";
 
+CONF_TXT_T cf_text_assumed_charset[] =	"When MIME charset information is missing in Content-Type header field.\n# Message is assumed to be in this charset. Default: US-ASCII. Typical values\n# include ISO-8859-x, ISO-2022-JP, EUC-KR, GB2312, and Big5. The value of\n# header fields which are not encoded per RFC 2047\n# is also assumed to be\n# in this charset. Currently, this only works for header fields. For message\n# body, it doesn't work yet. However, you can get the same effect by\n# defining a display filter for US-ASCII as if US-ASCII is your assumed charset.";
+
+CONF_TXT_T cf_text_charset_aliases[] =	"List of charset aliases. Each alias is a pair of charsets delimetered by a\n# single colon, the first one being an alias to the second one. The latter is\n# usually standard/prefered MIME name while the former is non-standard name used\n# by some email clients. For instance, you may have 'x-big5:big5,x-x-big5:big5,euc-cn:gb2312'";
+
+#ifdef HAVE_ICONV
+CONF_TXT_T cf_text_iconv_aliases[] =	"List of charset aliases to use with iconv(). Each alias is a pair of\n# charsets delimetered by a single colon, the first one being an alias to the\n# second one. The former is usually standard/prefered MIME name while the\n# latter is non-standard name used by iconv(3) on your system.\n#For example,\n# your iconv may use non-standard 'UTF8' for the standard 'UTF-8'. In that\n # case, you can put 'UTF-8:UTF8' here.";
+#endif
+
 CONF_TXT_T cf_text_editor[] =		"Specifies the program invoked by ^_ in the Composer,\n# or the \"enable-alternate-editor-implicitly\" feature.";
 
 CONF_TXT_T cf_text_speller[] =		"Specifies the program invoked by ^T in the Composer.";
@@ -503,6 +511,14 @@
 				cf_text_thread_lastreply_char},
 {"character-set",			0, 1, 0, 1, 1, 0, 0, 0, 0, 0,
 				cf_text_character_set},
+{"assumed-charset",			0, 1, 0, 1, 1, 0, 0, 0, 0, 0,
+				cf_text_assumed_charset},
+{"charset-aliases",			0, 1, 0, 1, 1, 1, 0, 0, 0, 0,
+				cf_text_charset_aliases},
+#ifdef HAVE_ICONV
+{"iconv-aliases",			0, 1, 0, 1, 1, 1, 0, 0, 0, 0,
+				cf_text_iconv_aliases},
+#endif
 {"editor",				0, 1, 0, 1, 1, 1, 0, 0, 0, 1,
 				cf_text_editor},
 {"speller",				0, 1, 0, 1, 1, 0, 0, 0, 0, 0,
@@ -2014,6 +2030,11 @@
     set_current_val(&vars[V_SIGNATURE_FILE], TRUE, TRUE);
     set_current_val(&vars[V_LITERAL_SIG], TRUE, TRUE);
     set_current_val(&vars[V_CHAR_SET], TRUE, TRUE);
+    set_current_val(&vars[V_ASSUMED_CHAR_SET], TRUE, TRUE);
+    set_current_val(&vars[V_CHAR_SET_ALIASES], TRUE, TRUE);
+#ifdef HAVE_ICONV
+    set_current_val(&vars[V_ICONV_ALIASES], TRUE, TRUE);
+#endif
     set_current_val(&vars[V_GLOB_ADDRBOOK], TRUE, TRUE);
     set_current_val(&vars[V_ADDRESSBOOK], TRUE, TRUE);
     set_current_val(&vars[V_FORCED_ABOOK_ENTRY], TRUE, TRUE);
diff -ru pine4.55/pine/mailindx.c pine4.b1/pine/mailindx.c
--- pine4.55/pine/mailindx.c	2003-04-09 21:01:48.000000000 +0200
+++ pine4.b1/pine/mailindx.c	2003-05-26 16:24:33.000000000 +0200
@@ -4885,8 +4885,10 @@
 	  }
 
 	  if(cdesc->adjustment == Left)
+/*	 strncpy_pad(p,cdesc->string,width,1); */
 	    sprintf(p, "%-*.*s", width, width, cdesc->string);
 	  else
+/*  	strncpy_pad(p,cdesc->string,width,-1); */
 	    sprintf(p, "%*.*s", width, width, cdesc->string);
 
 	  /*
@@ -5855,7 +5857,15 @@
 	if(l = prefix ? strlen(prefix) : 0)
 	  strcpy(s, prefix);
 
-	sprintf(buftmp, "%.75s", addr->personal);
+/*	Used to be 
+ *	   sprintf(buftmp, "%.75s", addr->personal); 
+ *
+ *  Some implementations of sprintf() are locale-dependent and
+ *  don't pass through an invalid sequence of bytes blindly. 
+ *
+ *  Use strncpy() instead */
+	strncpy(buftmp,  addr->personal,75);
+	buftmp[75]='\0';
 	istrncpy(s + l,
 		 (char *) rfc1522_decode((unsigned char *)tmp_20k_buf,
 					 SIZEOF_20KBUF, buftmp, &dummy),
diff -ru pine4.55/pine/mailpart.c pine4.b1/pine/mailpart.c
--- pine4.55/pine/mailpart.c	2003-03-11 19:51:00.000000000 +0100
+++ pine4.b1/pine/mailpart.c	2003-05-26 16:24:33.000000000 +0200
@@ -4000,7 +4000,8 @@
 		    fs_give((void **) &p);
 		}
 		else
-		  passed = !strucmp(test + 9, "us-ascii");
+		  passed = !strucmp(test + 9, 
+                    ps_global->VAR_ASSUMED_CHAR_SET ? ps_global->VAR_ASSUMED_CHAR_SET : "us-ascii");
 	    }
 	    else
 	      dprint(1, (debugfile,
diff -ru pine4.55/pine/mailview.c pine4.b1/pine/mailview.c
--- pine4.55/pine/mailview.c	2003-03-19 01:10:30.000000000 +0100
+++ pine4.b1/pine/mailview.c	2003-05-26 16:29:12.000000000 +0200
@@ -1495,6 +1495,7 @@
 #endif
 
         if(text2 = (void *)mail_fetchtext(ps_global->mail_stream, msgno)){
+	    FILTLIST_S	filters[8], *filtpnt;
  	    if(!gf_puts(NEWLINE, pc))		/* write delimiter */
 	      goto write_error;
 #if	defined(DOS) && !defined(WIN32)
@@ -1503,11 +1504,13 @@
 	    gf_set_readc(&gc, text2, (unsigned long)strlen(text2), CharStar);
 #endif
 	    gf_filter_init();
-	    /* link in filters, similar to what is done in decode_text() */
-	    if(!ps_global->pass_ctrl_chars){
-		gf_link_filter(gf_escape_filter, NULL);
-		gf_link_filter(gf_control_filter, NULL);
-	    }
+
+	    /* get the character set filters and link them in */
+	    get_cset_filters(filters, body);
+	    for(filtpnt = filters ; filtpnt->filter; filtpnt++)
+		gf_link_filter(filtpnt->filter, filtpnt->data);
+
+	    /* link in more filters, similar to what is done in decode_text() */
 	    gf_link_filter(gf_tag_filter, NULL);
 
 	    if((F_ON(F_VIEW_SEL_URL, ps_global)
@@ -4979,6 +4982,55 @@
     return(0);
 }
 
+int
+get_cset_filters(filters, body)
+    FILTLIST_S	*filters;
+    BODY        *body;
+{
+    int		 filtcnt = 0;
+#ifndef HAVE_ICONV
+    if(F_OFF(F_DISABLE_2022_JP_CONVERSIONS, ps_global))
+      filters[filtcnt++].filter = gf_2022_jp_to_euc;
+    }
+#endif
+    if(body && body->parameter && F_OFF(F_DISABLE_CHARSET_CONVERSIONS, ps_global)) {
+	unsigned char *charset;
+	if((charset=rfc2231_get_param(body->parameter,"charset", NULL, NULL))) {
+	    unsigned char *use_charset =
+		resolve_charset_alias(charset, ps_global->VAR_CHAR_SET_ALIASES);
+
+	    if (ps_global->VAR_ASSUMED_CHAR_SET && !strucmp(use_charset, "X-UNKNOWN")) 
+		use_charset = (unsigned char*) ps_global->VAR_ASSUMED_CHAR_SET ;
+
+	    dprint(3, (debugfile, "get_cset_filters: cset %s resolved to %s\n",
+				charset, use_charset));
+	    if (strucmp(charset, ps_global->VAR_CHAR_SET)) {
+		/* Charset strings differ */
+		unsigned char *tab;
+		if(tab = conversion_table(charset, ps_global->VAR_CHAR_SET)){
+		    filters[filtcnt].filter = gf_convert_charset;
+    		    filters[filtcnt++].data = gf_convert_charset_opt(tab);
+		}
+#ifdef HAVE_ICONV
+		else {
+		    filters[filtcnt].data   = gf_convert_utf8_opt(use_charset);
+		    if (filters[filtcnt].data)
+			filters[filtcnt++].filter = gf_convert_utf8;
+		}
+#endif
+	    } /* resolved charset strings differ */
+	    fs_give((void **) &charset);
+	} /* rfc2231_get_param success */
+    } /* body && body->parameter && F_OFF(F_DISABLE_CHARSET_CONVERSIONS, ps_global)) */
+    if(!ps_global->pass_ctrl_chars &&
+	/* UTF-8 needs to send bytes between 0x80 and 0x9f to the utf-8 terminal */
+       (ps_global->VAR_CHAR_SET && !strucmp(ps_global->VAR_CHAR_SET, "utf-8"))) {
+	filters[filtcnt++].filter = gf_escape_filter;
+	filters[filtcnt++].filter = gf_control_filter;
+    }
+    filters[filtcnt].filter = NULL;
+    return filtcnt;
+}

 /*----------------------------------------------------------------------
@@ -5005,33 +5057,12 @@
 {
     FILTLIST_S	filters[13];
     char       *err, *charset;
-    int		filtcnt = 0, error_found = 0, column, wrapit;
     int         is_in_sig = OUT_SIG_BLOCK;
+    int		error_found = 0, filtcnt, wrapit;
+    int         column = wrapit = (flags & FM_DISPLAY) ?
+					ps_global->ttyo->screen_cols : 80;
 
-    column = (flags & FM_DISPLAY) ? ps_global->ttyo->screen_cols : 80;
-    wrapit = column;
-
-    memset(filters, 0, sizeof(filters));
-    if(F_OFF(F_DISABLE_2022_JP_CONVERSIONS, ps_global))
-      filters[filtcnt++].filter = gf_2022_jp_to_euc;
-
-    if(charset = rfc2231_get_param(att->body->parameter,"charset",NULL,NULL)){
-	if(F_OFF(F_DISABLE_CHARSET_CONVERSIONS, ps_global)){
-	    unsigned char *tab;
-
-	    if(tab = conversion_table(charset, ps_global->VAR_CHAR_SET)){
-		filters[filtcnt].filter = gf_convert_charset;
-		filters[filtcnt++].data = gf_convert_charset_opt(tab);
-	    }
-	}
-
-	fs_give((void **) &charset);
-    }
-
-    if(!ps_global->pass_ctrl_chars){
-	filters[filtcnt++].filter = gf_escape_filter;
-	filters[filtcnt++].filter = gf_control_filter;
-    }
+    filtcnt = get_cset_filters(filters, att->body);
 
     if(flags & FM_DISPLAY)
       filters[filtcnt++].filter = gf_tag_filter;
@@ -5124,7 +5155,11 @@
      * local charset's undefined or it's not the same as the specified
      * charset, put up a warning...
      */
-    if(charset = rfc2231_get_param(att->body->parameter,"charset",NULL,NULL)){
+    if(
+#ifdef HAVE_ICONV
+       strucmp(ps_global->VAR_CHAR_SET, "utf-8") &&
+#endif
+      (charset = rfc2231_get_param(att->body->parameter,"charset",NULL,NULL))){
 	int rv = TRUE;
 
 	if(strucmp(charset, "us-ascii")
@@ -5144,9 +5179,10 @@
 	if(!rv)
 	  goto write_error;
     }
+    filters[filtcnt].filter = NULL; /* Terminate filters list */
 
     err = detach(ps_global->mail_stream, msgno, att->number,
-		 NULL, pc, filtcnt ? filters : NULL);
+		 NULL, pc, filters);
     if(err) {
 	error_found++;
 	if(style == QStatus) {
diff -ru pine4.55/pine/osdep/os-lnx.h pine4.b1/pine/osdep/os-lnx.h
--- pine4.55/pine/osdep/os-lnx.h	2003-03-19 01:57:38.000000000 +0100
+++ pine4.b1/pine/osdep/os-lnx.h	2003-05-26 16:24:33.000000000 +0200
@@ -214,6 +214,7 @@
  ----*/
 #define DF_DEFAULT_PRINTER        ANSI_PRINTER
 
+#define HAVE_ICONV

 /*----- The usual sendmail configuration for sending mail on Unix ------*/
diff -ru pine4.55/pine/pine.h pine4.b1/pine/pine.h
--- pine4.55/pine/pine.h	2003-04-16 08:22:12.000000000 +0200
+++ pine4.b1/pine/pine.h	2003-05-26 16:24:33.000000000 +0200
@@ -582,6 +582,11 @@
 		, V_THREAD_EXP_CHAR
 		, V_THREAD_LASTREPLY_CHAR
 		, V_CHAR_SET
+		, V_ASSUMED_CHAR_SET
+		, V_CHAR_SET_ALIASES
+#ifdef HAVE_ICONV
+		, V_ICONV_ALIASES
+#endif
 		, V_EDITOR
 		, V_SPELLER
 		, V_FILLCOL
@@ -812,6 +817,11 @@
 #define GLO_FLD_SORT_RULE	     vars[V_FLD_SORT_RULE].global_val.p
 #define VAR_CHAR_SET		     vars[V_CHAR_SET].current_val.p
 #define GLO_CHAR_SET		     vars[V_CHAR_SET].global_val.p
+#define VAR_ASSUMED_CHAR_SET	vars[V_ASSUMED_CHAR_SET].current_val.p
+#define VAR_CHAR_SET_ALIASES	vars[V_CHAR_SET_ALIASES].current_val.l
+#ifdef HAVE_ICONV
+#define VAR_ICONV_ALIASES 	vars[V_ICONV_ALIASES].current_val.l
+#endif
 #define VAR_EDITOR		     vars[V_EDITOR].current_val.l
 #define GLO_EDITOR		     vars[V_EDITOR].global_val.l
 #define VAR_SPELLER		     vars[V_SPELLER].current_val.p
@@ -3879,6 +3889,8 @@
 void	   *gf_html2plain_opt PROTO((char *, int, HANDLE_S **, int));
 void	    gf_2022_jp_to_euc PROTO((FILTER_S *, int));
 void	    gf_euc_to_2022_jp PROTO((FILTER_S *, int));
+void	    gf_convert_utf8 PROTO((FILTER_S *, int));
+void	   *gf_convert_utf8_opt PROTO((unsigned char *));
 void	    gf_convert_charset PROTO((FILTER_S *, int));
 void	   *gf_convert_charset_opt PROTO((unsigned char *));
 void	    gf_escape_filter PROTO((FILTER_S *, int));
@@ -4577,6 +4589,7 @@
 char	   *strrindex PROTO((char *, int));
 void	    sstrcpy PROTO((char **, char *));
 void	    sstrncpy PROTO((char **, char *, int));
+char	   *strncpy_pad PROTO((char *, char *, int, int));
 char	   *istrncpy PROTO((char *, char *, int));
 unsigned char *conversion_table PROTO((char *, char *));
 char	   *month_abbrev PROTO((int));
@@ -4674,6 +4687,12 @@
 unsigned char *trans_euc_to_2022_jp PROTO((unsigned char *));
 unsigned char *trans_2022_jp_to_euc PROTO((unsigned char *));
 
+#ifdef HAVE_ICONV
+#include <iconv.h>
+iconv_t        get_iconv_d PROTO((char *, char *));
+unsigned char* resolve_charset_alias PROTO((char *, char **));
+#endif
+
 
 /*-- takeaddr.c --*/
 char	  **detach_vcard_att PROTO ((MAILSTREAM *, long, BODY *, char *));
diff -ru pine4.55/pine/reply.c pine4.b1/pine/reply.c
--- pine4.55/pine/reply.c	2003-03-26 19:18:16.000000000 +0100
+++ pine4.b1/pine/reply.c	2003-05-26 16:24:33.000000000 +0200
@@ -1512,18 +1512,32 @@
        && (decoded[0] == 'R' || decoded[0] == 'r')
        && (decoded[1] == 'E' || decoded[1] == 'e')){
 
-        if(decoded[2] == ':')
-	  sprintf(buf, "%.200s", subject);
+        if(decoded[2] == ':'){
+	    strncpy(buf, subject,l);  
+            buf[l]='\0';
+        }
 	else if((decoded[2] == '[') && (p = strchr(decoded, ']'))){
 	    p++;
 	    while(*p && isspace((unsigned char)*p)) p++;
-	    if(p[0] == ':')
-	      sprintf(buf, "%.200s", subject);
+	    if(p[0] == ':'){
+	      strncpy(buf, subject,l);  
+              buf[l]='\0';
+	    }   
 	}
     }
-    if(!buf[0])
-      sprintf(buf, "Re: %.200s", (subject && *subject) ? subject : "your mail");
-
+    if(!buf[0]) {
+	/*
+	sprintf(buf, "Re: %.200s", (subject && *subject) ? subject : 
+		"your mail"); 		 
+	Used to be 
+	Some implementations of sprintf() are locale-dependent and
+	don't pass through an invalid sequence of bytes blindly. 
+	Use strncpy() instead 
+	*/
+	strcpy(buf,"Re: ");
+	strncpy(buf+4, (subject && *subject) ? subject : "your mail",l);  
+	buf[l+4]='\0';
+    }
     fs_give((void **) &tmp);
     return(buf);
 }
diff -ru pine4.55/pine/send.c pine4.b1/pine/send.c
--- pine4.55/pine/send.c	2003-03-26 20:25:36.000000000 +0100
+++ pine4.b1/pine/send.c	2003-05-26 16:24:33.000000000 +0200
@@ -7349,7 +7349,8 @@
 		src = pf->scratch ? pf->scratch
 				  : (*pf->text) ? *pf->text : "";
 
-		len = strlen(src)+1;
+    /* raw legacy encoding->utf-8 can increase the size up to 3 */
+		len = strlen(src)*3+1;  
 		p = (char *)fs_get(len * sizeof(char));
 		if(rfc1522_decode((unsigned char *)p, len, src, &charset)
 						   == (unsigned char *) p){
@@ -8085,6 +8086,23 @@
 	    if(new_encoding != ENCBINARY)
 	      new_encoding = ENC8BIT;  /* short lines, < 30% 8 bit chars */
 	}
+	else if(max_line < 300L || (eight_bit_chars * 100L)/len < 80L){
+	    /*
+	     * The previous test misses East Asian, Greek and Russian text
+       * in ISO-8859-7, KOI8-R, EUC-KR, Big5, and GB2312
+	     * with a lot higher percentage of 8bit chars than Western European text
+       * in ISO-8859-x. For them, use a relaxed condition for the 
+			 * percentage of 8bit chars along with a more strict condition
+			 * on the maximum line length. 
+	     */
+
+	    can_be_ascii--;
+	    if(body->type == TYPEOTHER)
+	      body->type = TYPETEXT;
+
+	    if(new_encoding != ENCBINARY)
+	      new_encoding = ENC8BIT;  /* short lines, < 30% 8 bit chars */
+	}
 	else{
 	    can_be_ascii--;
 	    if(body->type == TYPEOTHER){
diff -ru pine4.55/pine/strings.c pine4.b1/pine/strings.c
--- pine4.55/pine/strings.c	2003-04-03 02:13:32.000000000 +0200
+++ pine4.b1/pine/strings.c	2003-05-26 16:29:12.000000000 +0200
@@ -62,6 +62,7 @@
       - strindex       Replacement for strchr/index
       - strrindex      Replacement for strrchr/rindex
       - sstrcpy        Copy one string onto another, advancing dest'n pointer
+      - strncpy_pad    Copy one string onto another with padding
       - istrncpy       Copy n chars between bufs, making ctrl chars harmless
       - month_abbrev   Return three letter abbreviations for months
       - month_num      Calculate month number from month/year string
@@ -80,6 +81,10 @@
 
  ====*/
 
+#ifdef HAVE_ICONV
+#include <iconv.h>
+#endif
+
 #include "headers.h"
 #include "../c-client/utf8.h"
 
@@ -663,6 +668,170 @@
       (*d)++;
 }
 
+/*-----------------------------------------------------------------------
+  copy at most n chars of the source string onto the destination string.
+  if p==0, no padding, just calls strncpy.
+  if p >0, the destination buffer is filled from the left(start).
+  if p <0, the destination buffer is filled from the right(end).
+  return value: d
+  -> This function is not used at the moment
+  could be also done with sprintf() but some implementations truncate
+  a character sequence wich is invalid in the current locale.
+ ----*/
+#if 0
+char *
+strncpy_pad(d, s, n, p)
+    char *d;
+    char *s;
+    int  n;
+    int  p;
+{
+    char *start = d;
+    if (!p) 
+	return strncpy(d, s, n);
+    if ( p > 0){    /* left justification */
+	while(n-- > 0 && (*d = *s++) != '\0')
+		d++;
+	while(n-- > 0) {
+		*d=' ';
+		d++;
+	}
+    }
+    else {         /* right justification */
+	int np;
+	if ( (np = n-strlen(s)) > 0){
+		n -= np;
+		while (np--) {
+			*d = ' ';
+			d++;
+		}
+	}
+	while(n-- > 0 && (*d = *s++) != '\0')
+		d++;
+   }
+   return start;
+}
+#endif
+    
+/*
+ *  * * * * * *  Character set translation helpers  * * * * * * * *
+ */
+
+/*
+ * Like sstrncpy but with charset conversion(if possible) and null termination.
+ *  cset: from charset for conversion
+ * *dest: a pointer to dest. buffer
+ * rawin: a pointer to raw source buffer
+ * deced: a pointer to rfc2047-decoded source buffer
+ * outbytesleft: the length of dest. buffer
+ */ 
+void
+conv_sstrncpy(cset, dest, rawin, decoded, outbytesleft)
+    char  *cset;
+    char **dest;
+    char  *rawin;
+    char  *decoded;
+    int	   outbytesleft;
+{
+#ifndef HAVE_ICONV
+    sstrncpy(dest, decoded, outbytesleft);
+#else
+    static char   *charset = 0;
+    static iconv_t iconv_d = 0;
+    /* if charset allocated and new cset differs free old charset */
+    if (charset && cset && strcmp(charset, cset)) {
+	fs_give((void **)&charset);
+	if (iconv_d)
+	    iconv_close(iconv_d);
+    }
+    if (cset) {
+	if (charset) {
+	    /* old charset to reuse, just reset the conversion state */
+	    if (iconv_d)
+		iconv(iconv_d, NULL, NULL, NULL, NULL);
+	}
+	else {
+	    /* no charset to reuse, get new conversion descriptor */
+	    charset = fs_get(strlen(cset)+1);
+	    strcpy(charset, cset);
+	    iconv_d = get_iconv_d(ps_global->VAR_CHAR_SET, cset);
+	}
+    }
+    if (!iconv_d || !cset)
+        /* We have nothing to convert, so we just copy */
+	sstrncpy(dest, rawin, outbytesleft);
+    else {
+	size_t inbytesleft = strlen(decoded);
+	int ret = iconv(iconv_d, &decoded, &inbytesleft, dest, &outbytesleft);
+	dprint(4, (debugfile, "iconv_sstrncpy: ret=%d il=%d, ol=%d\n",
+		ret, inbytesleft, outbytesleft));
+    }
+#endif
+    **dest = '\0';
+}
+unsigned char*
+resolve_charset_alias(cs, aliases)
+    char  *cs;
+    char **aliases;
+{
+    int i;
+    char *bdry;
+
+    if ( !aliases )
+	return cs;
+    for (i=0; aliases[i] && *(aliases[i]); i++) {
+	if (bdry=strchr(aliases[i],':')) {
+	    *bdry='\0';
+	    if (!strucmp(aliases[i], cs)) {
+	  	*bdry=':';
+	  	return *(bdry+1) ? bdry+1 : cs;
+	    }
+	    *bdry=':';
+	}
+    }
+    return cs;
+}
+#ifdef HAVE_ICONV
+iconv_t get_iconv_d(toset, fromset)
+  char *toset;
+  char *fromset;
+{
+  char *fromcode, *tocode;
+  iconv_t iconv_d;
+
+  fromset = resolve_charset_alias(fromset,ps_global->VAR_ICONV_ALIASES);
+  toset   = resolve_charset_alias(toset,ps_global->VAR_ICONV_ALIASES);
+
+  if (!fromset ||
+      !strucmp(fromset, "US-ASCII") ||
+      (toset && !strucmp(fromset,toset)))
+       return NULL; 
+
+  /* 8 = strlen("US-ASCII"), 11 = strlen ("//TRANSLIT")+1; */
+
+  tocode   = (char *) fs_get((size_t)((toset   ? strlen(toset)   : 8) + 11 )); 
+  fromcode = (char *) fs_get((size_t)((fromset ? strlen(fromset) : 8) + 1  ));
+
+  strcpy(tocode, toset ? toset : "US-ASCII"); 
+  if (strucmp(tocode,"UTF-8"))   // No need of transliteration for UTF-8
+      strcat(tocode,"//TRANSLIT");
+
+  strcpy(fromcode, fromset ? fromset : "US-ASCII"); 
+
+  if ( (iconv_d = iconv_open(tocode, fromcode)) != (iconv_t) -1 ) {
+       dprint(2, (debugfile,"iconv open successful: from %s to %s\n",fromcode,tocode));
+  } 	
+  else {
+       dprint(2, (debugfile,"iconv open failed: from %s to %s\n",fromcode,tocode));
+       iconv_d = NULL;
+  }    
+  fs_give((void **) &tocode);
+  fs_give((void **) &fromcode);
+
+  return iconv_d;
+
+}
+#endif
 
 /*----------------------------------------------------------------------
   copy at most n chars of the source string onto the destination string
@@ -2950,6 +3119,7 @@
     while(s && (sw = strstr(s, RFC1522_INIT))){
 	/* validate the rest of the encoded-word */
 	if(rfc1522_valid(sw, &cset, &enc, &txt, &ew)){
+	    unsigned char *cset_r;    /* cset with alias resolution */
 	    if(!rv)
 	      rv = d;				/* remember start of dest */
 
@@ -2967,6 +3137,8 @@
 	    if(lang = strchr(cset, '*'))
 	      *lang++ = '\0';
 
+	    cset_r = resolve_charset_alias(cset,ps_global->VAR_CHAR_SET_ALIASES);
+
 	    /* Insert text explaining charset if we don't know what it is */
 	    if(F_OFF(F_DISABLE_2022_JP_CONVERSIONS, ps_global)
 	       && !strucmp((char *) cset, "iso-2022-jp")){
@@ -2993,8 +3165,9 @@
 		/* else, just translate it silently */
 	    }
 	    else if((!ps_global->VAR_CHAR_SET
-		     || strucmp((char *) cset, ps_global->VAR_CHAR_SET))
-	            && strucmp((char *) cset, "US-ASCII")){
+		     || strucmp((char *) cset_r, ps_global->VAR_CHAR_SET))
+		    && strucmp((char *) cset_r, "US-ASCII")){
+
 		dprint(5, (debugfile, "RFC1522_decode: charset mismatch: %s\n",
 			   cset));
 		if(!cs)
@@ -3004,6 +3177,7 @@
 		    if(!*charset)		/* only write first charset */
 		      *charset = cpystr(cset);
 		}
+#ifndef HAVE_ICONV
 		else{
 		    if(d-rv<len-1)
 		      *d++ = '[';
@@ -3014,6 +3188,7 @@
 		    if(d-rv<len-1)
 		      *d++ = SPACE;
 		}
+#endif
 	    }
 
 	    /* based on encoding, write the encoded text to output buffer */
@@ -3039,12 +3214,9 @@
 		  q = NULL;
 
 		if(p = rfc822_qprint((unsigned char *)txt, strlen(txt), &l)){
-		    strncpy((char *) d, (char *) p, len-1-(d-rv));
-		    d[len-1-(d-rv)] = '\0';
+		    conv_sstrncpy(cset_r, (char **)&d, txt, (char *)p, len-1-(d-rv));
 		    fs_give((void **)&p);	/* free encoded buf */
-		    d += l;			/* advance dest ptr to EOL */
-		    if(d-rv > len-1)
-		      d = rv+len-1;
+		    break;
 		}
 		else{
 		    if(q)
@@ -3065,12 +3237,8 @@
 	      case 'B' :			/* 'B' encoding */
 	      case 'b' :
 		if(p = rfc822_base64((unsigned char *) txt, strlen(txt), &l)){
-		    strncpy((char *) d, (char *) p, len-1-(d-rv));
-		    d[len-1-(d-rv)] = '\0';
+		    conv_sstrncpy(cset_r, (char **)&d, txt, (char *)p, len-1-(d-rv));
 		    fs_give((void **)&p);	/* free encoded buf */
-		    d += l;			/* advance dest ptr to EOL */
-		    if(d-rv > len-1)
-		      d = rv+len-1;
 		}
 		else
 		  goto bogus;
@@ -3100,24 +3268,35 @@
 	     * Found intro, but bogus data followed, treat it as normal text.
 	     */
 
+            l = (sw - s) + RFC1522_INIT_L;
+ 
+#ifdef HAVE_ICONV
+	    if (!rv) rv=d; 
+	    conv_sstrncpy(ps_global->VAR_ASSUMED_CHAR_SET, (char **)&d, s, s,
+		(int) min(l,len-1-(d-rv)));
+#else  
 	    /* if already copying to destn, copy it */
 	    if(rv){
-		strncpy((char *) d, s,
-			(int) min((l = (sw - s) + RFC1522_INIT_L),
-			len-1-(d-rv)));
-		d += l;				/* advance d, tie off text */
-		if(d-rv > len-1)
-		  d = rv+len-1;
+		sstrncpy((char *) d, s, (int) min(l, len-1-(d-rv)));
 		*d = '\0';
-		s += l;				/* advance s beyond intro */
 	    }
-	    else
-	      s += ((sw - s) + RFC1522_INIT_L);
+#endif // HAVE_ICONV
+	    s += l;				/* advance s beyond intro */
 	}
     }
 
-    if(rv && *s)				/* copy remaining text */
-      strncat((char *) rv, s, len - 1 - strlen((char *) rv));
+     if ( s && *s) { /* copy remaining text */
+#ifndef HAVE_ICONV
+	if(rv && *s)
+	    strncat((char *) rv,
+		s, len - 1 - strlen((char *) rv));
+#else 
+	if (!rv)
+	    rv = d; 
+	conv_sstrncpy(ps_global->VAR_ASSUMED_CHAR_SET, (char **)&d, s,
+		s, len - 1 - strlen((char *) rv));
+#endif // HAVE_ICONV
+    }
 
     if(translate_2022_jp){
 	unsigned char *trans;

---

From: Bernhard Kaindl <[email-masked]>
Date: Tue, 27 May 2003 22:29:39 +0200

Hello,

   I've updated the UTF-8 patch:

http://www.suse.de/~bk/pine/iconv/pine4.55.iconv.diff.6d

People with SuSE Linux 8.1 or better around can use this rpm:

http://www.suse.de/~bk/pine/iconv/RPMS/8.1/pine-4.55-0.i586.rpm

I tested it for a few hours while cleaning up a big mailbox.

Fixed issues:

- fixed memory problem (Segmentation fault) in streaming iconv filter
- fixed handling of incomplete conversions in streaming iconv filter
- fix rfc822_q header decoding problem which only showed on second view

Best Regards,
Bernhard Kaindl

PS: Additional Information:

 Configuration:

  - set the character-set option in the pine config to: "utf-8"
  - enable pass-control-characters-as-is in the pine config to
    see some characters like german umlauts in the config correctly.
  - See the bottom of this mail for some more examples

 Special instructions for Users of SuSE Linux 8.2:

  If you have SuSE Linux 8.2, all you have to do in addition is
  to set your locale to an UTF-8 locale e.g. by putting entering
  export LANG=en_US.UTF-8 start an xterm from this shell to have
  the xterm started with UTF-8 character I/O. Then select
  "Unicode Best" from the xterm font menu and start the patched
  and configured pine in it.

  If you use pico or vim as editor also support UTF-8, but without
  any additional patches as of SuSE Linux 8.2.

 Compilation:

  This patch should work on most modern Linux Distributions, I developed
  it on SuSE Linux 8.2, but should work as soon as your xterm supports
  UTF-8 correctly and your system has iconv() the case for Solaris and
  Linux with glibc-2.2 and newer.

  On Solaris and other systems use:

  The patch uses a #define called HAVE_ICONV which indicates if the iconv()
  library call which is available on this system. If no () is not provided
  by the system, one of three freely available implementations can be used.

Current Status:

  There is more to do to make pine's display engine understand that
  byte which is sent to the terminal does not always move the cursor,
  but that's a completely separate issue from the charset conversion
  which this patch addresses.

Known Bug:

  If assumed-charset in the config and you view a message with
  mailing list management information which pine appends in an
  editoral at to bottom of the mail body, the mailing list
  management editoral is completely messed up.

Environment:

  UTF-8 like en_US.UTF-8 this has to be set in the environment when

 The original patch from Jungshik Shin sent with documentation,
 the display-filters described are obsolete now:

-

Free implementation of iconv, if your system doesn't provide one:

-

Config settings for use with this patch are:

Required if use with UTF-8 is intented:

 character-set=utf-8

Recommended:

 pass-control-chars-as-is

Other Examples, highy personal:

 assumed-charset=windows-1252

---

From: Bob Rasmussen <[email-masked]>
Date: Tue, 27 May 2003 22:21:07 -0700 (PDT)

I have tested Bernhard's updated patch, and can report that it works quite
well. This is a big step forward in internationalizing Pine! Gone are the
garbage characters resulting from character set mismatches. I can display
European diacritics, Chinese spam, and so forth. I was testing with Anzio,
our telnet program, which is UTF-8 compatible.

Regards,
....Bob Rasmussen,   President,   Rasmussen Software, Inc.

personal e-mail: [email-masked]
 company e-mail: [email-masked]
          voice: (US) 503-624-0360 (9:00-6:00 Pacific Time)
            fax: (US) 503-624-0760
            web: http://www.anzio.com