Subject: IMAP messages "disappear"

From: Michael Østergaard Pedersen <[email-masked]>
Date: Fri, 18 Jan 2002 17:29:13 +0100

I am using a web based email program (IMP 3.0) with a PHP backend
compiled with the c-client library (--with-imap). My setup is:

Courier-IMAP 1.4.1
C-Client: CVS-Snapshot from 10-01-2002 (have also tried others)
PHP 4.1.1
IMP 3.0

My problem is that when I log on to IMP using an IMAP server, I can see the
new messages in my Inbox. As soon as I click one of them I'm told that the
message does not exist and all new messages "disappear".

The problem is that apparently I can only see messages in Maildir/new,
and as soos as they are moved to /Maildir/cur I can no longer see
them. Netscape and any other IMAP mail client I have tried does not
have this problem with my IMAP server.

Still, in IMP it says that I have a total number of messages equal to
what I do have, although I can't see any of them with IMP. Since IMP
relies on the c-client library provided by PHP I was thinking that
perhaps this could be where the problem is?

I have asked on the IMP mailing list, but they said it wasn't IMP's
fault. Some PHP people say that they just rely on c-client, so the
problem could not be with PHP. When I asked about my IMAP server, they
told me that it's a client problem and that the server works fine with
other IMAP clients. That's why I'm trying here now.

Regards,
Michael

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Fri, 18 Jan 2002 08:50:23 -0800 (PST)

On Fri, 18 Jan 2002 17:29:13 +0100, Michael Xstergaard Pedersen wrote:
> them.

Sorry to pass the buck again, but Maildir is an internal Courier-IMAP issue.
There is nothing in IMAP that exports Maildir-specific information to an IMAP
client.  An IMAP client would never hear of a Maildir/new or a Maildir/cur,
much less care about such server-internal details.

> Netscape and any other IMAP mail client I have tried does not

That suggests that the problem, whatever it is, is triggered by something that
IMP does as opposed to Netscape.

When you say "any other IMAP mail client" do you include Pine?  If not, you
should.  Netscape and Outlook are incredibly poor tests of an IMAP server
since they use only a fraction of IMAP's capabilities.  Pine on the other hand
uses almost everything in IMAP.  Also, Pine is based upon c-client; so if Pine
works that eliminates c-client as a cause and would refocus the issue upon IMP
and/or PHP.;

> When I asked about my IMAP server, they
> other IMAP clients.

The Courier-IMAP server has been known to have bugs in the past in which it
did not comply with the IMAP specification.  However, this report is a new
one; and a large number of people use Courier-IMAP with Pine successfully.
Also, the author of Courier-IMAP is not likely to be helpful on a bug report
in his server, so it's worth going the extra mile to prove that it's his
server's fault before speaking further with him.

What you need to do is get a telemetry of an IMAP session which demonstrates
the problem.  That will allow people to focus fairly quickly on whether the
bug is in the server or in the client.

I'll help you analyze the telemetry once you get it, but I very much doubt
that it'll result in a problem found in c-client.  If you're lucky, it'll be a
bug in PHP or IMP; if you're unlucky, it'll be a bug in Courier-IMAP.

---

From: Mark Edwards <[email-masked]>
Date: Mon, 28 Jan 2002 16:30:47 -0800

I'm trying to figure out what this error in my maillog means:

Jan 28 16:22:30 antsclimbtree imapd[82031]: Unable to accept SSL connection,
host=UNKNOWN

I'm using a self-signed certificate right now, so perhaps that's all this
means.  The client, which is Outlook Express set to use SSL, doesn't
complain after the initial complaint about the certificate not being from a
proper authority.

Thanks.

-- 
Mark Edwards
Engineer
Mr. Toad's
San Francisco

---

From: Mark Crispin <[email-masked]>
Date: Mon, 28 Jan 2002 16:45:40 -0800 (Pacific Standard Time)

> Jan 28 16:22:30 antsclimbtree imapd[82031]: Unable to accept SSL connection,

That error message means that OpenSSL's SSL_accept() routine failed.
I've never seen this before.  The host=UNKNOWN part is very suspicious.
It means that getpeername() failed on stdin (file designator 0), which is
wierd since it should return the IP address of the other end.

How is imapd being invoked?  Is it being invoked directly from inetd, or
is there something between it and the network?

> complain after the initial complaint about the certificate not being from a

This problem has nothing to do with a self-signed certificate.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

---

From: Mark Edwards <[email-masked]>
Date: Mon, 28 Jan 2002 17:02:09 -0800

It is being invoked via FreeBSD 4.4's inetd, which has tcp wrapper 
functionality built-in.  I don't think there's any way to disable it.  
However, my hosts.allow file is set to

imapd : ALL : allow

so it shouldn't be blocking access at all.  However, I do know that the 
readme files say not to use tcpwrappers with ipop3d and imapd.

Perhaps I should run imapd and ipop3d as daemons rather than from inetd?

Does this error message indicate that imapd traffic is happening 
unencrypted?

Thanks.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 28 Jan 2002 17:52:21 -0800 (PST)

> functionality built-in.  I don't think there's any way to disable it.

I have never seen SSL IMAP (port 993) service work with TCP wrappers.  Regular
IMAP (port 143) service seems to work OK, even if you subsequently do a
STARTTLS command.

> Perhaps I should run imapd and ipop3d as daemons rather than from inetd?

No, because imapd and ipop3d don't run as separate daemons.

> Does this error message indicate that imapd traffic is happening

That error message is a fatal error.  That server session died forthwith.

---

From: Dag Nygren <[email-masked]>
Date: Thu, 18 Apr 2002 00:00:52 +0300

> other IMAP clients. That's why I'm trying here now.

Did you ever find a solution to this?

I see the same problem with IMP 3.0 and IMAP and just traced it down
to IMP using the "SORT" imap command and this seems to behave
strangely.

BRGDS

-- 
Dag Nygren                               email: [email-masked]
Oy Espoon NewTech Ab                     phone: +358 9 8024910
Träsktorpet 3                              fax: +358 9 8024916
02360 ESBO                              Mobile: +358 400 426312
FINLAND

---

From: Mark Crispin <[email-masked]>
Date: Wed, 17 Apr 2002 15:45:14 -0700 (Pacific Daylight Time)

On Thu, 18 Apr 2002, Dag Nygren wrote:
> > My setup is:

I strongly suggest that you try some IMAP server other than Courier.

If the problem goes away (and I suspect it will) with some non-Courier
IMAP server, then you know that the problem is in Courier.

Courier does not implement IMAP.  It implements what the author of Courier
thinks IMAP should be, which is somewhat different from the IMAP
specification.  I gave up on him a long time ago, and I'm not particularly
interested in wasting any more time going through the cycle of:
 1) problem gets reported to me
 2) I respond that it's due to a wrong thing that Courier does
 3) the author of Courier insists that I am wrong
 4) I quote the section of the IMAP specification that Courier violates
 5) the author of Courier calls me an idiot, and says that the IMAP
     specification is wrong

-- Mark --

---

From: Dag Nygren <[email-masked]>
Date: Thu, 18 Apr 2002 12:30:13 +0300

Hi again,

found the problem(s) and thought that everybody would be interested
in the result and perhaps learn from it.

The problem was that IMP3.0 didn't show any messages, just said that there
were n messages, but no header info at all.

First a want to clear the makers of both IMAP and IMP from all
accusations (?), it was NOT their faults.

Checking the traffic between IMP and IMAPD  (ethereal is a very
good tool for this BTW) after putting in some debug statements in
the imapd mailbox driver showed that IMP3.0 was using a UID SORT
command and that IMAPD never answered this.
What took me some time to find out was that imapd actually crashed
and there was no traces of this.
Using imapd through gdb showed me that the culprit was the format
string of my debug statement trying to sprintf() an interger with a %s,
ehhhh.....
Fixing that I still got no headers, but now the traffic between IMP and IMAPD
looked sane, and was indeed reading the headers.

Mr. Michael stergaard Pedersen mailed me and told me that he had
seen the same problems with some versions of imapd and suggested
imapd-2000c.
As I have patched my imapd-4.7.2c to recognize MH-inboxes I didn't
really want to change that and though I would find out the reason instead.

I traced the problem down to a PHP-call to imap_fetch_overview, which really
did not return anything.

After a lot of tracing and filling up system logs I thought of one thing:
Versions.

Checking what version of mail.h and libc-client.a my PHP-4.1.2 compilation
used I found out that they were definitely not using the 4.7.2c version and 
additionally
the version of the two files didn't even match....
Fixing that and recompiling PHP fixed the problem !

The moral of the story:
- Perhaps Mark could provide a "make install" that updates all the files in 
the system:
  imapd, mail.h (or newer) and the library ? At least echo a reminder to make 
sure
  that they match.
- When installing a new version of something ALWAYS make sure that there is 
nothing
  old lying around (knew that ;-) )

Anyway IMP3.0 works fine now :-)))

BRGDS