Subject: Re[2]: pop3 patch to cclient

From: Vadim Zeitlin <[email-masked]>
Date: Wed, 20 Jun 2001 20:35:42 +0200 (CET)

[replying to the list]

On Wed, 20 Jun 2001 09:40:40 -0700 (PDT) Mark Crispin <[email-masked]> wrote:

MC> This message may be bad news for you, or at least make you unhappy.  Please be
MC> patient reading all this, since at the end there's a glimmer of hope.

 Thanks for the detailed reply - and the glimmer of hope, of course! :-)

MC> Unfortunately, the LIST command in most POP3 servers sends the message size in
MC> UNIX (LF newline) format.

 Ah, I didn't know/think about this. Thanks for explaining this! I really can
agree much more easily when I understand what I am agreeing with :-)

MC> Also, prior to the CAPA command there was no way of knowing whether or not the
MC> server supported TOP.  Presumably, a server which supports CAPA and TOP uses
MC> the good definition of TOP, but many POP3 servers don't support CAPA.

 This is something I did think of. I believe that the approach of the pop3
code in imap2001 is the best one can do (use CAPA if available, assume it
supports TOP and UIDL otherwise). There are not many servers still used which
don't support TOP and using TOP with them is not a fatal error anyhow - it is
just not efficient, but being slightly less efficient with 0.1% of servers
that don't support TOP at the cost of being vastly more efficient with the
rest of them seems a reasonable choice.

MC> I don't know how to solve this problem.  If you do, by all means, share your
MC> solution with me.

 I surely don't as I've never thought about this LF/CR LF issue. Right now I
can only think about assuming that the server is broken and retrieve the first
message entirely, then compute its size locally (and correctly) and compare
with LIST output. If they match I think it should be safe to assume that the
server is ok and so we can use LIST for all the other messages.

 It doesn't seem neither elegant nor simple nor even efficient (as you'd
probably want to do it only once and not each time you connect to the server)
and so I think your solution with flags is much better.

MC> This isn't to say that you shouldn't develop your patch and distribute it
MC> yourself.  I encourage third parties to develop/distribute/support patches
MC> that, for various reasons, can't be in the official distribution.

 The reason I proposed my patch was that we were using a patch by someone else
before which was just setting elt->rfc822_size to 1 in pop3_header which was
really stupid - it doesn't cost that much to write 3 extra lines of code to
send a LIST! So I'll probably indeed make it available somewhere/somehow as I
think my patch is better.

MC> If you do distribute a patch, you should have a documentation file outlining
MC> these issues.

 Ok, thanks again for explaining them!

MC> I would consider the use of new FT_xxx flags to say "alright to use POP3
MC> LIST" and "alright to use POP3 TOP", which could be passed to
MC> pop3_fetchfast() and pop3_header().  In other words, the code would have
MC> both methods, but only applications which indicate that they're willing to
MC> take the risk will do the dangerous ones.

 Sorry, I thought mail_fetch_fast() was deprecated in favour of
mail_fetch_structure() - isn't it? So shouldn't the latter accept this flag as
well?

 Then my current patch to it

--- imap-2000b/src/c-client/mail.c      Thu Jan  4 11:04:19 2001      
+++ M/extra/src/c-client/mail.c Tue Jun 19 15:26:34 2001
@@ -1198,5 +1199,11 @@
       hdr = (char *) memcpy (fs_get ((size_t) hdrsize+1),s,(size_t) hdrsize);
       hdr[hdrsize] = '\0';     /* tie off header */
+
+      /*** MAHOGANY PATCH ***/
+      /* don't retrieve the entire body to just get the size of the header for
+         POP3 driver, this is *awfully* inefficient */
+      if (body || !elt->rfc822_size)
       (*stream->dtb->text) (stream,msgno,&bs,(flags & ~FT_INTERNAL) | FT_PEEK);
+      /*** END OF MAHOGANY PATCH ***/
       if (!elt->rfc822_size) elt->rfc822_size = hdrsize + SIZE (&bs);
       if (body)                        /* only parse body if requested */

could be applied as is as pop3_header() would only fill rfc822_size if flags
contained FT_POP3_TOP_OK (or whatever).

 OTOH, maybe it would be better to have a parameter instead of a flag? It
would allow existing programs use the TOP more easily just by setting this
parameter to TRUE. Then instead of testing the flags the code would just
test something like pop3_parameters(GET_POP3USETOP, NULL) && LOCAL->cap.top.
What do you think?

MC> If you decide to send me a suggested patch which goes this route, it is
MC> probably too late for it to get into imap-2001, but it can be in an update
MC> release (e.g. imap-2001a).

 I'll try to do it. What is the dead line for 2001a?

 Thanks,
VZ

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2001 23:51:54 -0700 (PDT)

On Wed, 20 Jun 2001 20:35:42 +0200 (CET), Vadim Zeitlin wrote:
> can only think about assuming that the server is broken and retrieve the
> first message entirely, then compute its size locally (and correctly) and
> compare with LIST output. If they match I think it should be safe to assume
> that the server is ok and so we can use LIST for all the other messages.

In *theory* that should work, but "theory" and "practice" can be two different
things.  RFC 1939 says that the count should be exact, but that doesn't wasn't
followed by POP3 servers in the past.

Given the sad experience with IMAP, where vendors of broken servers say "there
is nothing wrong with my server, it works just fine with Netscape and Outlook
Express", I don't think that we can assume that LIST counts will ever be well
tested.  All it needs is for one message to calculate differently in LIST than
it does in RETR.

So, it's best to go with a FT_??? flag saying "OK to use POP3 LIST command for
rfc822_size".

> The reason I proposed my patch was that we were using a patch by someone
> else before which was just setting elt->rfc822_size to 1 in pop3_header
> which was really stupid

I've seen that patch.  Someone once wondered why I didn't use it.  "Well,
starting with the small matter that it'd break Pine..."  :-)

> mail_fetch_structure() - isn't it?

Not at all.

If all you want is the flags/internaldate/size, then mail_fetch_fast() is the
function to call.  Generally, most programs also want the envelope, which gets
the mail_fetch_fast() data.  If you're going to call mail_fetch_structure()
anyway, there's no point to calling mail_fetch_fast() as well.

> So shouldn't the latter accept this flag as well?

Looking at the flag wouldn't be in mail_fetch_fast() nor
mail_fetch_structure(); it'll be in the methods called by these functions in
pop3.c.

> test something like pop3_parameters(GET_POP3USETOP, NULL) && LOCAL->cap.top.

In general, flags have been used to affect fetching behavior, not parameters.
I don't think that making the code less consistant is a good thing.

>  I'll try to do it. What is the dead line for 2001a?

2001a hasn't even started, much less acquired a deadline.