Subject: [Imap-uw] two ipop3d "bugs"

From: Adrian Buciuman <[email-masked]>
Date: Fri, 11 Nov 2005 05:34:51 +0200

I want to use ipop3d to access an IMAP server.

So I've done some "torture" tests, to see if it works well or fails
safe in different strange conditions.

Interesting logs:
1.from the pop3 client:

retr 1
+OK 239 octets
Date: Thu

.
retr 2
+OK 10030 octets
Status: RO

.

For the first message Status: should have its own line. Two lines  (or
one?) have been appended by ipop3d at the end of the message.

Should the second retr give an error?

2. ipop3d to imap server

00000005 FETCH 1 BODY.PEEK[HEADER]
* 1 FETCH (BODY[HEADER] {227}
Return-Path: <>
Received: from test ([10.80.80.80])
. by xx-xx-xx-xxx (Cyrus v2.2.12) with LMTPA;
. Thu, 16 Jun 2005 21:21:32 +0300
X-Sieve: CMU Sieve 2.2
Date: Thu
Message-ID: <cmu-lmtpd-1714-1118946092-0@xx-xx-xx-xxx>
)
00000005 OK Completed (0.000 sec)
00000006 FETCH 1 BODY[TEXT]
* 1 FETCH (BODY[TEXT] "")
00000006 OK Completed (0.000 sec)
00000007 FETCH 2 BODY.PEEK[HEADER]
* OK Message 2 no longer exists
* 2 FETCH (BODY[HEADER] "")
* 2 FETCH (FLAGS (\Deleted \Seen))
00000007 OK Completed (0.000 sec)
00000008 FETCH 2 BODY[TEXT]
* OK Message 2 no longer exists
* 2 FETCH (BODY[TEXT] "")
00000008 OK Completed (0.000 sec)

(First message has no body, nor the line between headers and body.
Second message has been deleted and the mailbox expunged.)
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Mon, 14 Nov 2005 09:44:10 -0800 (PST)

I considered your two issues carefully.

I agree that the Status: line should always be at the start of a line and 
that an extra blank line should not be written if the message has no body. 
The attached ipop3d.patch file has both changes, and these changes will 
appear in the next imap-2005 development snapshot.

The second issue is more problematic.

The POP3 protocol does not have any provision for "holes" in the mailbox 
("message no longer exists").  In my opinion, the IMAP protocol does not 
have "holes" either.

The problem is that the Cyrus server disagrees with that opinion; I 
understand the technical issues why they made that decision even though I 
continue to disagree with their chosen solution.  The bottom line here is 
that ipop3d is doing the best that it reasonably can given that, in 
effect, it's been dealt a lousy hand of cards.

There will be one minor change: instead of the previous
 	retr 2
 	+OK 10030 octets
 	Status: RO

 	.
behavior you will now see
 	retr 2
 	+OK 10030 octets
 	Status: RO

 	.
That is, you'll only see one blank line after the Status: line.

The octet count will continue to be bogus; that came from the IMAP server. 
That is why I believe that Cyrus' behavior here is non-compliant with the 
specification.  Regardless of whether there can be a "hole" in the 
sequence space due to expunging (I say there can not, but I am not the 
only voice), IMAP size counts must be exact.  Since the IMAP server 
already informed the client of a non-empty message, it can't withdraw the 
message contents since that would change the size.

This will probably end up being hashed out with the Cyrus developers 
offline from this list, and eventually we (meaning them and me) will 
figure out something better.  But for the nonce, I hope that the attached 
patch will suffice for you.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

*** ipop3d.c~	Wed Apr 13 17:44:10 2005
--- ipop3d.c	Mon Nov 14 09:21:41 2005
***************
*** 10,16 ****
   *		Internet: [email-masked]
   *
   * Date:	1 November 1990
!  * Last Edited:	13 April 2005
   * 
   * The IMAP toolkit provided in this Distribution is
   * Copyright 1988-2005 University of Washington.
--- 10,16 ----
   *		Internet: [email-masked]
   *
   * Date:	1 November 1990
!  * Last Edited:	14 November 2005
   * 
   * The IMAP toolkit provided in this Distribution is
   * Copyright 1988-2005 University of Washington.
***************
*** 343,361 ****
  	      sprintf (tmp,"+OK %lu octets\015\012",
  		       (elt = mail_elt (stream,msg[i]))->rfc822_size + SLEN);
  	      PSOUT (tmp);
! 				/* output header */
  	      t = mail_fetch_header (stream,msg[i],NIL,NIL,&k,FT_PEEK);
! 	      blat (t,-1,k,NIL);
! 				/* output status */
  	      sprintf (tmp,STATUS,elt->seen ? "R" : " ",
  		       elt->recent ? " " : "O");
! 	      PSOUT (tmp);
! 	      CRLF;		/* delimit header and text */
  				/* output text */
  	      t = mail_fetch_text (stream,msg[i],NIL,&k,
  				   FT_RETURNSTRINGSTRUCT);
! 	      blat (t,-1,k,&stream->private.string);
! 	      CRLF;		/* end of list */
  	      PBOUT ('.');
  	      CRLF;
  	    }
--- 343,370 ----
  	      sprintf (tmp,"+OK %lu octets\015\012",
  		       (elt = mail_elt (stream,msg[i]))->rfc822_size + SLEN);
  	      PSOUT (tmp);
! 				/* get header */
  	      t = mail_fetch_header (stream,msg[i],NIL,NIL,&k,FT_PEEK);
! 	      blat (t,-1,k,NIL);/* write up to trailing CRLF */
! 				/* build status */
  	      sprintf (tmp,STATUS,elt->seen ? "R" : " ",
  		       elt->recent ? " " : "O");
! 				/* normal header ending with CRLF CRLF? */
! 	      if (t[k-3] == '\012') {
! 		PSOUT (tmp);	/* write status */
! 		CRLF;		/* then write second CRLF */
! 	      }
! 	      else {		/* abnormal - no blank line at end of header */
! 		CRLF;		/* write CRLF first then */
! 		PSOUT (tmp);
! 	      }
  				/* output text */
  	      t = mail_fetch_text (stream,msg[i],NIL,&k,
  				   FT_RETURNSTRINGSTRUCT);
! 	      if (k) {		/* only if there is a text body */
! 		blat (t,-1,k,&stream->private.string);
! 		CRLF;		/* end of list */
! 	      }
  	      PBOUT ('.');
  	      CRLF;
  	    }
***************
*** 402,415 ****
  				/* update highest message accessed */
  	      if (i > last) last = i;
  	      PSOUT ("+OK Top of message follows\015\012");
! 				/* output header */
  	      t = mail_fetch_header (stream,msg[i],NIL,NIL,&k,FT_PEEK);
! 	      blat (t,-1,k,NIL);
! 				/* output status */
  	      sprintf (tmp,STATUS,elt->seen ? "R" : " ",
  		       elt->recent ? " " : "O");
! 	      PSOUT (tmp);
! 	      CRLF;		/* delimit header and text */
  	      if (j) {		/* want any text lines? */
  				/* output text */
  		t = mail_fetch_text (stream,msg[i],NIL,&k,
--- 411,431 ----
  				/* update highest message accessed */
  	      if (i > last) last = i;
  	      PSOUT ("+OK Top of message follows\015\012");
! 				/* get header */
  	      t = mail_fetch_header (stream,msg[i],NIL,NIL,&k,FT_PEEK);
! 	      blat (t,-1,k,NIL);/* write up to trailing CRLF */
! 				/* build status */
  	      sprintf (tmp,STATUS,elt->seen ? "R" : " ",
  		       elt->recent ? " " : "O");
! 				/* normal header ending with CRLF CRLF? */
! 	      if (t[k-3] == '\012') {
! 		PSOUT (tmp);	/* write status */
! 		CRLF;		/* then write second CRLF */
! 	      }
! 	      else {		/* abnormal - no blank line at end of header */
! 		CRLF;		/* write CRLF first then */
! 		PSOUT (tmp);
! 	      }
  	      if (j) {		/* want any text lines? */
  				/* output text */
  		t = mail_fetch_text (stream,msg[i],NIL,&k,
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Adrian Buciuman <[email-masked]>
Date: Sat, 14 Jan 2006 00:46:04 +0200

http://mailman1.u.washington.edu/pipermail/imap-uw/2005-November/000292.html

http://mailman1.u.washington.edu/pipermail/imap-uw/2005-November/000295.html

> appear in the next imap-2005 development snapshot.
[............]
> The octet count will continue to be bogus; that came from the IMAP server.
[............]
> -- Mark --

Thank you for the patch and for your clear explanation. I am sorry for
having delayed so long to answer your email. I've thought it is better to
only answer after I will have time to apply the patch, and to perform some
more tests.

I've tried the patch "ported"  to imap-2004; i don't know from where to
download the development version of imap-2005.

A problem with this patch is that k may be 0, 1 or 2 at this line:
if (t[k-3] == '\012')

So depending on what garbage is in memory I may get:
(12 is expunged)
retr 12
+OK 531 octets

Status:
.

(notice a line above Status:)

I would rather put something like:
if ((k==0) || ((k>2)  &&  (t[k-3]=='\012')))

If the header has size 1 or 0 or the body has size 1 the octet count is
wrong, even if the imap server doesn't leave holes in the mailbox; I don't
see an easy way around it. Such messages are however not valid emails. I
hope robust clients will not crash or truncate if they receive some garbage
"message" bigger than the expected size, nor timeout if the "message" is
smaller.

TOP sometimes still gives an extra line for messages with no body.

retr 5
+OK 26 octets
Status: RO
Status:

.
top 5 0
+OK Top of message follows
Status: RO
Status: R

.
top 5 1
+OK Top of message follows
Status: RO
Status: R

.
top 5 99999
+OK Top of message follows
Status: RO
Status: R

.
retr 4
+OK 247 octets
Date: Thu
Status:
.
top 4 0
+OK Top of message follows
Date: Thu
Status: R
.
top 4 1
+OK Top of message follows
Date: Thu
Status: R

.

Another issue I've found: if i give a bad username/password, ipop3d
will try three times to login to the imap server with that bad pair,
than close the pop3 connection. It will make more sense to give an
pop3 error after every failed imap login attempt (except perhaps when
the error is caused by a busy mailbox, or ipop3d only gets readonly
access to the imap server);  maybe the pop3 client will ask the user
for a new user/pass if given the opportunity.

Thank you,
Adrian Buciuman

---

From: Mark Crispin <[email-masked]>
Date: Fri, 13 Jan 2006 17:10:05 -0800 (Pacific Standard Time)

I believe that I have addressed all your issues in the ipop3d in the 
latest imap-2006 development tarball:
 	ftp://ftp.cac.washington.edu/mail/imap-2006.DEV.tar.Z

Specifically:

Status: header is no longer written if the RFC 822 header is less than 4 
bytes.

Extra blank line is no longer written in TOP with messages that have no 
RFC 822 body.

Password is only sent once.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw