Subject: Re: "pseudo messages" from ipop3d

From: Lars Hallberg <[email-masked]>
Date: Sun, 26 Oct 2003 16:00:50 +0100

Sory Mark for getting this twice... i gofed up... now to the list:

Great!

The below text answers most my questions. Maybe it shuld be included in 
the dokumentation :-) Becose I could not find out for what purpos the 
pseudo messages was ther. That might decres the ranting about them too :-)

I'm resorting mail with formail + procmail and I was wandering if i 
shuld preserv the pseudo message even when the mailbox is rebuild with 
much different contents (possably empty - I do move all read mail from 
inbox to varius destination). The answer is Yes then - becouse the 
pseudo message is about new messages uid, and have nothing directly to 
do whith the curent messages. Good.

But It's one thing that still wories me. Do they have to be the first 
message in the file?

I curently

  mv mbox .mbox.old
  cat /dev/null >> mbox
  sleep 1
  formail-procmail stuf (that will put the pseudo message back and 
maybe more messages)
  rm .mbox.old

Ther is a risk that something can deliver to mbox before the pseudo 
message is written back.

The sleep is to make it likly that any progressing dilivery to mbox 
finish befor I start reading, but it also increes the risk that somthing 
manage to deliver a mesage before the pseudo message.

I could .lock the mbox first... but then i would lock out my own 
procmail... right?

Sounds a bit problematic. So I want to know if it at all is a problem 
first... do the pseudo message need to be the first message?

Mark Crispin wrote:

>All UIDs must be unique
>
>IMAP UIDs consist of the mailbox name, a 32-bit integer which is constant for
>the mailbox, and a 32-bit integer which is per-message.  Generally, most
>people think of the IMAP UID as the per-message 32-bit integer.  This value
>must be strictly ascending in the mailbox, a very useful property which makes
>it possible to avoid downloading the entire UID map.
>
>The per-mailbox 32-bit integer (called the "UID validity") should remain
>constant, but if it ever changes, the new value must be larger.  Since the
>uniqueness is the combination of the UID validity and per-message UID, it is
>permitted to renumber the per-message UIDs.  This is what happens if something
>happens to corrupts the UIDs, or if a mailbox is deleted then recreated.
>
>To preserve the qualities of uniqueness and strict ascendency, UID validity
>and the "last assigned UID" must be stored with the mailbox.  That is the
>primary function of the pseudo-message.  It also holds such things as IMAP
>keywords.
>
>POP3 UIDs are a string of up to 70 printable (0x21 - 0x7e) ASCII characters.
>There is no requirement for any sort of ordering.  The other technique that I
>referred to is to use an MD5 checksum of the message text.
>
>So, IMAP UIDs can be used as POP3 UIDs, but not the other way around.  IMAP
>UIDs are implemented by the underlying c-client library, so the c-client based
>POP3 server uses IMAP UIDs for POP3 UIDs.
>
>The whole point of the c-client based POP3 server is to provide POP3 service
>that interoperates with the IMAP service.  So, there is no chance of it ever
>using the MD5 technique; the result would be even *more* cruft in the mailbox
>since it would be "both and", not "either or".
>
>If you don't care about interoperability with the IMAP service, then perhaps
>some other POP3 server, such as Qualcomm's qpopper, is a better choice for
>you.
>
>This is an excellent example of why "which is better, ipop3d or qpopper?" type
>questions are meaningless.  There is rarely an unconditional "better",
>otherwise the world would be full of monopolies.  More often, it is a question
>of "which is a better choice considering my needs?"
>
>People move from qpopper to ipop3d all the time, because ipop3d turns out to
>suit their needs better.  I imagine that Qualcomm sees people moving from
>ipop3d to qpopper for the same reason.

---

From: Lars Hallberg <[email-masked]>
Date: Sun, 26 Oct 2003 22:40:12 +0100

All UIDs must be unique

>> primary function of the pseudo-message.  It also holds such things as 
BTW

Do this mean I have to remove the X-UID header when moving mail between 
mailfolders?

What bad things will happen otherwise?

Is it better or worse that *all* X-UID headers are removed (also for 
mail that return to the original folder)?

Right now, mail is moved to other folders with the X-UID header, and 
while resorting - new mail without any X-UID header can be inserted 
betwin messages with X-UID header. Scary stuff...

Thanks /LaH

---

From: Mark Crispin <[email-masked]>
Date: Sun, 26 Oct 2003 15:31:31 -0800 (PST)

> Do this mean I have to remove the X-UID header when moving mail between

If you use the c-client library to move messages (e.g. mail_copy()) all of
these details are taken care of for you.  Why aren't you doing this?

No, you should not copy X-UID headers.

> What bad things will happen otherwise?

Either you will write useless junk headers, or you will screw up the UID
regime and cause c-client to purge it and recreate a completely new UID
regime, causing your POP3 users to complain bitterly that all these
messages got downloaded again.

> mail that return to the original folder)?

You can't return messages.  All you can do is add new messages.

> betwin messages with X-UID header. Scary stuff...

You can't resort messages in the mailbox or insert messages in the middle
with c-client managed mailboxes.  This is a permanent restriction since it
is required behavior in IMAP.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Lars Hallberg <[email-masked]>
Date: Mon, 27 Oct 2003 02:52:11 +0100

>these details are taken care of for you.  Why aren't you doing this?

Becose I'm sorting mail with procmail / foremail.

>No, you should not copy X-UID headers.

OK

>messages got downloaded again.

This means fireing upp mutt over the mailbox directly and move mail is 
harmfull to? So basicly - everyone complaining about seeing the pseudo 
message is potentialy doing something bad?

But I don't need to care about POP, this is imap over ssl exlusivly. How 
harmfull is recreated UID:s for IMAP (if I shuld remove them all)?

>You can't return messages.  All you can do is add new messages.

I move the folder, and filter the mailes, some goes back to the old 
folder but any new delivery during the formail - procmail run vill come 
in between. This is bad if I understan You corectly. I belive I can fix it!

>is required behavior in IMAP.

OK, so I shuld make sure that:

 * Mail that stays in the same folder keep ther UID and order.

 * That every new mail end up last (or that the mailfolder is locked 
during the processing).

 * That mail that move to other folders have ther X-UID removed.

 * Anything more?

somthing like this

.lock mbox
mv mbox mbox.old
sort mbox.old > mbox.new (with formail & procmail, removing every X-UID 
on mail not going to mbox.new,
                                              and that pseudo message 
and X-UID is preserved in mbox.new)
mv mbox.new mbox
rm mbox.old
release .lock

One of the points of this whole sorting is to keep mbox small - so this 
lock vill be reasnoble short!

For other folders this is harder, but in this specific case I can prety 
safely asume that they will not get any new mail during the sort. After 
all, ther is no point in moving mail manually to folders that is 
automaticly sorted. If I move mail manually it will be to folders that 
is not automaticly sorted - or to mbox to be automaticly sorted to the 
right place. I just have to make sure the sorting of mbox don't happens 
while another sort is going on. Can do that!

But it realy looks like I'm using the wrong tools :-) Still, I belewe I 
will get away with it!

Thanks for Your reply!

/LaH

---

From: Mark Crispin <[email-masked]>
Date: Sun, 26 Oct 2003 18:49:16 -0800 (Pacific Standard Time)

> Becose I'm sorting mail with procmail / foremail.

You should run procmail as part of the mail delivery process, before
messages reach into any mailboxes.

It sounds like your problem is that you are running procmail on mailboxes
after POP3 or IMAP servers have gotten to it.  That seems to be a very bad
idea.

> harmfull to?

If it copies X-UID headers, yet.

> So basicly - everyone complaining about seeing the pseudo

Yes.

> harmfull is recreated UID:s for IMAP (if I shuld remove them all)?

The same warning about bad things apply to IMAP.  In fact, it is IMAP's
requirements that cause the necessity to recreate UIDs.

-- Mark --

---

From: Lars Hallberg <[email-masked]>
Date: Mon, 27 Oct 2003 04:45:52 +0100

>after POP3 or IMAP servers have gotten to it.  That seems to be a very bad

Yes, it seems to take a lot of extra considerations :-/

But what I want is to sort the mail *after* i read them in the INBOX 
(over IMAP) so it's a bit hard to avoid. And to keep inbox small, so the 
task of checking for new mail is fast, also over a slow link. If I realy 
need to look up an old message for this list, then I go to the c-client 
mailfolder (who is probable huge). But to read the new mails i only have 
to read INBOX.

>requirements that cause the necessity to recreate UIDs.

Yes, but redownloading over POP3 will curupt the lokal mailfolder 
(duplikate mails)?

Over IMAP it will (only) waste bandwith, resources and the users (my) 
time? I beliv I will gain more in all thes 3 aspects then I'm loosing. 
But still, I want to do it as good as possably :-)

And I sertenly want to do the right thing for the mbox, becos one reason 
to keep it smal is that I might need to read it ower gprs, wher I pay 
for each KB downloaded. So alot of extra download ther is not wanted.

Spam and viruses is an exception. That I want to sort on delivery. And I 
can wait with reading and verifeing the spamfilters work until i have a 
fast link :-)

Thanks again!