Subject: errors with pop3

From: Timo Veith <[email-masked]>
Date: Fri, 5 Nov 2004 11:28:37 +0100

Hi c-client subscribers,

one week ago I wrote the appended message to the debian-isp mailing list 
but got no reply. So I thought to try it here. I also have searched the 
list archive of c-client and found a similar problem posted on 19th of 
march 2003 by "Harondel J. Sibble" <[email-masked]>, subject "uid 
errors with pop3". It seems that there was no real answer to the 
problem or am I wrong?

I have a strange problem with the uw-imap ipop3d daemon. Yesterday four 
users told me indepently that they've gotten duplicate mails. I asked 
them how many mails they've gotten doubled and when that strange 
behaviour started.

One of them answered that it started around 17:00. This is about the 
same time when I noticed the following log entries:

Oct 25 08:48:41 zuse ipop3d[19125]: Message 4 UID 0 less than 4926
Oct 27 17:02:03 zuse ipop3d[27988]: Message 7 UID 32 less than 14107
Oct 27 17:02:08 zuse ipop3d[28012]: Message 124 UID 32 less than 146
Oct 27 17:02:22 zuse ipop3d[28255]: Message 196 UID 32 less than 3908
Oct 27 17:02:54 zuse ipop3d[28640]: Message 25 UID 32 less than 9984
Oct 27 17:02:54 zuse ipop3d[28578]: Message 425 UID 32 less than 2270
Oct 27 17:04:14 zuse ipop3d[28951]: Message 129 UID 32 less than 9912
Oct 27 17:04:26 zuse ipop3d[29009]: Message 427 UID 32 less than 427
Oct 27 17:06:03 zuse ipop3d[29343]: Message 2695 UID 32 less than 8961
Oct 27 17:06:12 zuse ipop3d[29401]: Message 660 UID 32 less than 3135
Oct 27 17:06:26 zuse ipop3d[29458]: Message 325 UID 32 less than 18611
Oct 27 17:06:49 zuse ipop3d[29485]: Message 523 UID 32 less than 43890
Oct 27 17:23:17 zuse ipop3d[31933]: Message 2084 UID 32 less than 2084
Oct 27 17:23:52 zuse ipop3d[32119]: Message 5 UID 32 less than 4110
Oct 27 17:33:07 zuse ipop3d[898]: Message 4 UID 32 less than 7412
Oct 27 18:05:50 zuse ipop3d[5562]: Message 2 UID 32 less than 5630
Oct 27 18:06:35 zuse ipop3d[5750]: Message 2 UID 32 less than 9098
Oct 27 20:20:38 zuse ipop3d[21608]: Message 12 UID 32 less than 12667
Oct 27 20:26:41 zuse ipop3d[22461]: Message 1586 UID 32 less than 28339
Oct 28 06:39:04 zuse ipop3d[14223]: Message 323 UID 32 less than 29023
Oct 28 06:55:35 zuse ipop3d[15684]: Message 2 UID 32 less than 3362
Oct 28 07:37:09 zuse ipop3d[19740]: Message 3 UID 32 less than 2813
Oct 28 07:42:09 zuse ipop3d[20356]: Message 4 UID 32 less than 3526
Oct 28 07:47:56 zuse ipop3d[21078]: Message 3 UID 32 less than 6210
Oct 28 07:57:20 zuse ipop3d[22515]: Message 331 UID 32 less than 331
Oct 28 09:24:04 zuse ipop3d[3396]: Message 186 UID 32 less than 187
Oct 28 09:30:48 zuse ipop3d[4640]: Message 20 UID 32 less than 5723
Oct 28 09:37:48 zuse ipop3d[5901]: Message 12 UID 32 less than 9318
Oct 28 10:03:01 zuse ipop3d[11465]: Message 2 UID 32 less than 17721
Oct 28 10:29:19 zuse ipop3d[16774]: Message 3 UID 32 less than 13953
Oct 28 10:30:47 zuse ipop3d[17217]: Message 2 UID 32 less than 8010

You can see that they appear cumulatively around 17:00. And there were 
never that much of this kind before. The FAQ from uw-imap calls these 
log messages harmless 
( http://www.washington.edu/imap/IMAP-FAQs/index.html#7.14 ) but I am 
not sure if this can still be called harmless.

My question is what circumstances can bring ipopd to suddenly behave 
this way? Could it be users popping their mailbox too often? File 
system corruption? Has anybody else experienced such a behaviour?

My system is Debian woody, ipopd-ssl version  2001adebian-6.

TIA and kind regards,

Timo
-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Fri, 5 Nov 2004 07:56:22 -0800 (PST)

Every message has a 32-bit unique identifier (UID) value, which is 
strictly ascending in the mailbox (on other words, message n must have a 
higher UID than message n-1).  This is needed for IMAP, but since ipop3d 
uses the same library UIDs are enforced there too.

Consider one of these messages: "Message 124 UID 32 less than 146".  This 
is mathematically impossible.  Message 123 has a UID of 146 (possible), 
but message 124 has a UID of 32; not only is this wrong (since 146 > 32) 
but it is impossible for message 124 to ever have a UID of 32 no matter 
what message 123's UID is!

One way that this can happen is if some tool other than c-client 
manipulated the mailboxes, e.g. copied messages from one mailbox to 
another.

However, given that all these "UID 32" messages appeared, there is 
another, darker, probable cause.

In traditional UNIX mailbox format, message metadata is kept in headers in 
the message.  The original metadata header was Status:.  Now there are 
several others, one of these being X-UID.

Spammers and virus writers have taken to forging metadata headers, either 
out of stupidity or with malicious intent.  If your mail delivery delivers 
a message with one of these forged headers intact, it will be perceived as 
message metadata with the resulting confusion.

In other words, I believe that a spam/virus came in with
 	X-UID: 32
in the header.  There is no legitimate reason for any incoming message to 
do this; the only reason is stupidity or evil.

The tmail and dmail programs in modern versions of UW imapd will recognize 
this and automatically convert these forged headers into something 
harmless.  In the above case, it would be:
 	X-Original-X-UID: 32

If you decide to try tmail or dmail, be sure to get a modern version. 
This is a relatively recently added facility.  The latest UW IMAP toolkit 
sources (which include tmail and dmail as well as imapd and ipop3d) are 
always on:
 	ftp://ftp.cac.washington.edu/mail/imap.tar.Z

For what it's worth, most other mailbox formats store message metadata 
out-of-band from the message, where spammers and virus writers can't 
attack it.  mbx format, for example, does not have this problem.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Timo Veith <[email-masked]>
Date: Mon, 8 Nov 2004 15:22:46 +0100

Hello Mark,

thank you very much for making this clear. I have another question. See 
below.

Am Freitag, 5. November 2004 16:56 schrieben Sie:
> in the header.  There is no legitimate reason for any incoming
> message to do this; the only reason is stupidity or evil.
>
> The tmail and dmail programs in modern versions of UW imapd will
> recognize this and automatically convert these forged headers into
> something harmless.  In the above case, it would be:
>   X-Original-X-UID: 32
>
> This is a relatively recently added facility.  The latest UW IMAP
> toolkit sources (which include tmail and dmail as well as imapd and
> ipop3d) are always on:

Because I don't have a recent version of uw-imap (not even modern) and I 
would prefer to leave uw-imap as it is, do you think it is a good idea 
to use exim for filtering out the X-UID header?

I would set "headers_remove = X-UID" in the procmail_pipe driver.

TIA and kind regards

Timo

---

From: Mark Crispin <[email-masked]>
Date: Mon, 8 Nov 2004 08:03:13 -0800 (PST)

> to use exim for filtering out the X-UID header?

If you can do this, yes that is an excellent idea.

-- Mark --

---

From: Timo Veith <[email-masked]>
Date: Tue, 9 Nov 2004 12:15:02 +0100

Am Montag, 8. November 2004 17:03 schrieb Mark Crispin:
> > Because I don't have a recent version of uw-imap (not even modern)
> > and I would prefer to leave uw-imap as it is, do you think it is a
> > good idea to use exim for filtering out the X-UID header?
>

Although it's my first time to remove a header with exim, I think I did 
it. :)  My test showed that a manually inserted X-UID (telnet localhost 
25) was removed. Hopefully it really helps to prevent it from happening 
again. We will see.

Thanks again for helping me out. :)

Kind regards,

Timo