Subject: Problem decoding base64 encoded subject fields?

From: Kolbjørn Barmen <[email-masked]>
Date: Mon, 8 Mar 2004 10:52:50 +0100 (CET)

It seems that pine has a problem decoding base 64 encoding subjects every
now and then, and the common factor to me seems to be the length of the
subject field.  The result is that the subject field is not decoded at all.

Is this a known problem?  If not, I can provide examples :)

I'm using pine 4.58 on Linux.

-- 
Kolbjørn Barmen
UNINETT Driftsenter
-- 
-----------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/pine/pine-info/
-----------------------------------------------------------------

---

From: Paulo Matos <[email-masked]>
Date: Mon, 8 Mar 2004 18:25:50 +0000 (WET)

Hi folks!

> subject field.  The result is that the subject field is not decoded at all.

	This is indeed a problem but not pine's problem, at least for my 
research on the matter.	

	This is an RFC2047 (or the obsoleted RFC1522) violation while 
encoding the Subject, the size of the encoded Subject is over 75 chars 
(see bellow).

Quoting RFC2047:
| 2. Syntax of encoded-words
| (...)
| encoded-word = "=?" charset "?" encoding "?" encoded-text "?="
| (...)
| An 'encoded-word' may not be more than 75 characters long, including
| 'charset', 'encoding', 'encoded-text', and delimiters. If it is
| desirable to encode more text than will fit in an 'encoded-word' of
| 75 characters, multiple 'encoded-word's (separated by CRLF SPACE) may
| be used.
|
| While there is no limit to the length of a multiple-line header
| field, each line of a header field that contains one or more
| 'encoded-word's is limited to 76 characters.
|
| The length restrictions are included both to ease interoperability
| through internetwork mail gateways, and to impose a limit on the
| amount of lookahead a header parser must employ (while looking for a
| final ?= delimiter) before it can decide whether a token is an
| "encoded-word" or something else.

	Pine is RFC2047 strict and does not decode an invalid 
encoded-word.

	If this is not the case then you may have found a bug...

--
	Paulo Matos
 ----------------------------------- ----------------------------------
|Sys & Net Admin                    | Serviço de Informática           |
|Faculdade de Ciências e Tecnologia | Tel: +351-21-2948596             |
|Universidade Nova de Lisboa        | Fax: +351-21-2948548             |
|P-2829-516 Caparica                | e-Mail: [email-masked]          |
 ----------------------------------- ----------------------------------

---

From: Kolbjørn Barmen <[email-masked]>
Date: Wed, 10 Mar 2004 11:40:08 +0100 (CET)

> encoding the Subject, the size of the encoded Subject is over 75 chars 
[snip]
> 	Pine is RFC2047 strict and does not decode an invalid 

Not more strict than when I tell pine to mark messages based on content of
subject, it does indeed also pick those with bogus encoding :)

> 	If this is not the case then you may have found a bug...

Well, you seem to be correct, allthough the above behaviour is perhaps
strange.  (pine being able to "read" the bogus encoded subject when it does
search, allthough not being able to display them in index list or when
viewing message)

-- 
Kolbjørn Barmen
UNINETT Driftsenter

---

From: Paulo Matos <[email-masked]>
Date: Thu, 11 Mar 2004 18:34:50 +0000 (WET)

> subject, it does indeed also pick those with bogus encoding :)

I can confirm it! This also happens to me.

> search, allthough not being able to display them in index list or when

IMHO this is a misbehavior since it ruins strictness coherency. I call 
pine developers' attention for this point.

Regards,

---

From: Bernhard Kaindl <[email-masked]>
Date: Fri, 12 Mar 2004 00:10:17 +0100

> I can confirm it! This also happens to me.

A maybe not fully matching example are encoded attachment file names.

Some programs encode some parts of them while it apparently was not
written in the RFC and lately, newer versions of pine started to
decode them also even if it wasn't required by the RFC(or maybe
even forbidden, don't know...).

> pine developers' attention for this point.

I wouldn't want to promote to use longer encoded words, but OTOH, it does
not seem to have fatal consequencies to be able to read them.

There is a patch to relax the length check(allow double lengh) at decoding in:

http://www.suse.de/~bk/pine/iconv/4.58/pine-iconv-8m.patch.gz

The chunks for this look like this:

-int	       rfc1522_valid PROTO((char *, char **, char **, char **,
+int	       rfc1522_valid PROTO((char *, int, char **, char **, char **,

@@ -2949,7 +3347,7 @@ rfc1522_decode(d, len, s, charset)
 
     while(s && (sw = strstr(s, RFC1522_INIT))){
 	/* validate the rest of the encoded-word */
-	if(rfc1522_valid(sw, &cset, &enc, &txt, &ew)){
+	if(rfc1522_valid(sw, 1, &cset, &enc, &txt, &ew)){
 	    if(!rv)
 	      rv = d;				/* remember start of dest */
 
@@ -3222,10 +3646,14 @@ rfc1522_valenc(c)
 
 /*
  * rfc1522_valid - validate the given string as to it's rfc1522-ness
+ * if relaxchk is true, double the maximum length of an encoded word.
+ * this is necessary to decode overlong encoded words generated by 
+ * numerous incompliant implementations of RFC 2047 (1522).
  */
 int
-rfc1522_valid(s, charset, enc, txt, endp)
+rfc1522_valid(s, relaxchk, charset, enc, txt, endp)
     char  *s;
+    int  relaxchk;
     char **charset;
     char **enc;
     char **txt;
@@ -3237,7 +3665,11 @@ rfc1522_valid(s, charset, enc, txt, endp
     rv = rfc1522_token(c = s+RFC1522_INIT_L, rfc1522_valtok, RFC1522_DLIM, &e)
 	   && rfc1522_token(++e, rfc1522_valtok, RFC1522_DLIM, &t)
 	   && rfc1522_token(++t, rfc1522_valenc, RFC1522_TERM, &p)
-	   && p - s <= RFC1522_MAXW;
+	   && p - s <= RFC1522_MAXW * (relaxchk ? 2 : 1);
+    /*
+     * relax the length condition by doubling the max length of an
+     * encoded word. It's is needed for some longer encoded words.
+     */
 
     if(charset)
       *charset = c;
@@ -3288,7 +3720,7 @@ rfc1522_encode(d, len, s, charset)
       }
       else if(*p == RFC1522_INIT[0]
 	      && !strncmp((char *) p, RFC1522_INIT, RFC1522_INIT_L)){
-	  if(rfc1522_valid((char *) p, NULL, NULL, NULL, (char **) &q))
+	  if(rfc1522_valid((char *) p, 0, NULL, NULL, NULL, (char **) &q))
 	    p = q + RFC1522_TERM_L - 1;		/* advance past encoded gunk */
       }
       else if(*p == ESCAPE && match_escapes((char *)(p+1))){
@@ -3288,7 +3720,7 @@ rfc1522_encode(d, len, s, charset)
       }
       else if(*p == RFC1522_INIT[0]
 	      && !strncmp((char *) p, RFC1522_INIT, RFC1522_INIT_L)){
-	  if(rfc1522_valid((char *) p, NULL, NULL, NULL, (char **) &q))
+	  if(rfc1522_valid((char *) p, 0, NULL, NULL, NULL, (char **) &q))
 	    p = q + RFC1522_TERM_L - 1;		/* advance past encoded gunk */
       }
       else if(*p == ESCAPE && match_escapes((char *)(p+1))){

It's originally from Jungshik Shin, looks plausible to me so far.

I didn't carefully re-verify it now, but: "go figure" :-)

Bernhard

---

From: Kolbjørn Barmen <[email-masked]>
Date: Fri, 12 Mar 2004 11:15:25 +0100 (CET)

> The chunks for this look like this:

Thanks, I generated a new smaller patch from those chunks and put them
into my own pine ebuild script (using gentoo), and it works nicely :)

-- 
Kolbjørn Barmen
UNINETT Driftsenter