Subject: location of uid of a message in a mbx folder

From: Darren Jacobs <[email-masked]>
Date: Fri, 15 Apr 2005 12:21:13 -0400

Hi All,

I'm trying to track down where message uid is encoded in a message in a 
mbx mail folder?

Darren....

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Fri, 15 Apr 2005 09:31:36 -0700 (PDT)

On Fri, 15 Apr 2005, Darren Jacobs wrote:
> I'm trying to track down where message uid is encoded in a message in a mbx 
> mail folder?

In general, you should not worry about the format of an mbx mailbox unless 
you need to repair one.  In particular, you should use the c-client 
library to access mbx mailboxes and not write your own code to do so; the 
mbx mailbox locking protocol is quite complex and it is easy to make a 
mistake.

The UID is stored in the per-message line as the last field (following the 
hyphen).  That is, with a per-message-line of:
 	16-Mar-2000 00:51:24 -0800,6726;000000100009-0000087b

INTERNALDATE:	16-Mar-2000 00:51:24 -0800
RFC822.SIZE:	6726
FLAGS:		000000100009
UID:		0000087b

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Darren Jacobs <[email-masked]>
Date: Thu, 21 Apr 2005 09:27:57 -0400

Mark Crispin wrote:

>> I'm trying to track down where message uid is encoded in a message in 
>> a mbx mail folder?
>
>
> In general, you should not worry about the format of an mbx mailbox 
> unless you need to repair one.  In particular, you should use the 
> c-client library to access mbx mailboxes and not write your own code 
> to do so; the mbx mailbox locking protocol is quite complex and it is 
> easy to make a mistake.
>
> The UID is stored in the per-message line as the last field (following 
> the hyphen).  That is, with a per-message-line of:

To do what I'm trying to do the c-client will be only partially 
helpful.  Let me explain. 

We're considering doing a mass conversion of user INBOX's into mbx 
format (from mbox format).  Using the 'mailutil copy' command to do the 
conversion works fine, but it zero's out the message uid values.  We 
still have a lot of pop users who leave their mail on the server and 
we're concerned that simply doing the conversion they will end up 
downloading duplicates of their messages the next time they log in.

What I'm writing is a utility to parse the mbox INBOX prior to 
conversion to get all the message uid's (this is where I use the 
c-client api) and write these uids into the new mbx INBOX messages.

I've tried two ways of getting to the message headers in the mbx INBOX's:

(1) looping through the folder searching the regex: 
'/(\d|\d\d)\-[JFMASOND][a-z][a-z]\-\d\d\d\d\s[0-1]\d\:[0-5]\d\:\d\d/'
(2) starting with a 2048 byte offset and using the sizes from the 
message headers to advance forward to each message header

Option 1 is risky in that it might come across a message who's body 
contains that regex

Option 2 looks to be that answer but so far my calculated offset's are 
not lining up with message header locations.  They always seem to be off 
by varying amounts.

Any suggests on this?

Regards,

Darren Jacobs

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Thu, 21 Apr 2005 08:41:50 -0700 (PDT)

I understand what you're trying to do.  You are correct that, since 
mailutil creates a new mailbox, the UIDVALIDITY and message UIDs are not 
preserved.

You have to do much more than your proposed procedure though.  You have to 
copy the UIDVALIDITY and UIDLAST values from the traditional UNIX mailbox 
(either in the hidden message or in an X-IMAPbase: header).  You have to 
process internaldate, flags, and keywords and convert those to mbx form; 
the Status:, X-Status:, X-Keywords:, and X-UID: headers are *not* used in 
mbx format and must not appear.

I don't know what you mean by "the sizes from the message headers" but if 
you mean the bogus SVR4 Content-Length: header, that header is completely 
useless.  Ignore it; you have to locate the internal start-of-message 
header, and manually calculate the message size.  Note that traditional 
UNIX format uses LF-only newlines as in normal UNIX files, whereas mbox 
format uses CRLF-newlines; you have to account for that (you can not just 
copy the texts either).  mbx format sizes are quite different.

This is not an easy task, and there are lots of things that can go wrong.

The easiest way would be just to use mailutil copy, and then manually 
patch the UID data (remember you must copy UIDVALIDITY and UIDLAST as 
well!) into the file.  A program to do the latter task would not be easy, 
but it will be much easier than what you're proposing.

-- Mark --

---

From: Darren Jacobs <[email-masked]>
Date: Thu, 21 Apr 2005 14:35:35 -0400

Mark Crispin wrote:

>
> I don't know what you mean by "the sizes from the message headers" but 
> if you mean the bogus SVR4 Content-Length: header, that header is 
> completely useless.  Ignore it; you have to locate the internal 
> start-of-message header, and manually calculate the message size.  
> Note that traditional UNIX format uses LF-only newlines as in normal 
> UNIX files, whereas mbox format uses CRLF-newlines; you have to 
> account for that (you can not just copy the texts either).  mbx format 
> sizes are quite different.

I was refering to the RFC822 message size specified in the mbx header.  
For instance:
----
18-Apr-2005 15:57:59 -0400,*2762*;000000000001-00000000
---
for this header its 2762.  I thought that I could simply read this, add 
the previous offset and get the location of the next message header in 
the folder.  For instance, to get the location of the 2nd message in the 
folder I'd add 2048 to the RFC822 message size of the first message (say 
2762), and get the location in the file of the next message, that is 
4810.  Then I'd do a seek with this number to get to the appropriate 
spot in the file. 

However it seems that this message size number (a) is not what I think 
it is or (b) not accurate as when I try doing the above I never get a 
number that gets me to the right spot in the file.  Its probably (a).  I 
was hoping to avoid having to calculate the message size manually by 
doing this.

Darren....

---

From: Darren Jacobs <[email-masked]>
Date: Thu, 21 Apr 2005 14:50:59 -0400

Darren Jacobs wrote:

> Mark Crispin wrote:
>
>>
>> I don't know what you mean by "the sizes from the message headers" 
>> but if you mean the bogus SVR4 Content-Length: header, that header is 
>> account for that (you can not just copy the texts either).  mbx 
>> format sizes are quite different.
>
>
> I was refering to the RFC822 message size specified in the mbx 
> header.  For instance:
> ----
> for this header its 2762.  I thought that I could simply read this, 
> add the previous offset and get the location of the next message 
> header in the folder.  For instance, to get the location of the 2nd 
> message in the folder I'd add 2048 to the RFC822 message size of the 
> first message (say 2762), and get the location in the file of the next 
> message, that is 4810.  Then I'd do a seek with this number to get to 
> the appropriate spot in the file.
> number that gets me to the right spot in the file.  Its probably (a).  
> I was hoping to avoid having to calculate the message size manually by 
> doing this.
>

...and I just discovered what I was doing wrong.  I was not adding the 
length of the message header to the RFC822 message size when I 
calculated the offset (i.e. [previous offset + RFC822 message size + 
length of msg hdr] ).  Seems to work quite nicely now.  Thanks Mark.

Darren....

Darren.....