Subject: [Imap-uw] MBX file format specification

From: Richard Westlake <[email-masked]>
Date: Mon, 13 Mar 2006 16:34:36 +0000 (GMT Standard Time)

Hi
Please could someone tell me where to find the specification for the MBX 
mailbox file format? Sorry it is somewhere obvious, I tried Google and The 
University of Washington IMAP Information Centre, maybe I missed 
something.

Thanks in advance

Richard Westlake

School of Crystallography, Birkbeck College, Malet Street, London WC1E 7HX
Tel: 020-7631-6859
----------------------------------------------------------------------
                Truth endures but spelling changes    --  Anon.
----------------------------------------------------------------------

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Mon, 13 Mar 2006 08:54:16 -0800 (PST)

On Mon, 13 Mar 2006, Richard Westlake wrote:
> University of Washington IMAP Information Centre, maybe I missed something.

The current mbx specification is the source code, although there is some 
information in the FAQ (where it discusses how to repair a damaged mbx 
file).

Or you can ask me a question.  What is it that you need to know about it?

In general, I do not recommend writing a separate implementation of code 
to support mbx format.  The locking rules for mbx are complex and easy to 
miss.  Instead, it's better to use the c-client library.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Richard Westlake <[email-masked]>
Date: Thu, 16 Mar 2006 21:48:25 +0000 (GMT Standard Time)

On Mon, 13 Mar 2006, Mark Crispin wrote:

> information in the FAQ (where it discusses how to repair a damaged mbx file).
>
> In general, I do not recommend writing a separate implementation of code to 
> support mbx format.  The locking rules for mbx are complex and easy to miss. 
> Instead, it's better to use the c-client library.

Mark
Sorry this message is rather long.

I have taken note of your comments on file locking. I don't intend to work 
on any live file that another process might want to access. I would be 
making and working on private copies of the files. I also accept that I 
cant blame you or your software, if my own code makes a mess of a mail 
file.

I am interested in the format of UW imapd mbx files for several reasons, 
including:

1) Curiosity
  Does anything else use same mbx file format? With Google I get lots of 
hits for tools to repair outlook express mbx files.  There was also a 
document on "MBX Mailbox file format" by Michael Santovec; however this is 
very different to UW imapd

2) Repairing corrupt mbx format files. Hopefully this will be a very rare 
event, but we would probably need to recover the persons email quickly 
when it happens. I was thinking of producing some caned procedures or 
scripts which junior members of the support group could use to recover 
most if not all of the messages, if I wasn't around. I have since found 
some repair scripts, which might help with this, including Tim Mooney's 
mbx-repair.

3) One-off bulk conversion of UNIX mbox format files to mbx format.
I have used "mailutil copy" for this, however it discards, the X-Keyword: 
flags it doesn't recognise (e.g. Junk, NonJunk, $Forward, $MNDSent). I had 
thought of scanning a file to build a list of X-Keyword flags, and using 
that to create the header for an mbx file with no messages. Then using 
"mailutil append" to append the messages from the old mbox to the new mbx 
file. Would mailutil recognise the X-Keyword values, if I put the flags 
strings in the header and set then appropriate flags?

I have some idea about how the file format works after looking at FAQ, 
examining the code; unfortunately C isn't one of my languages.  I also 
examined mailbox files, as I made changes via the mail client.

There still something I am unsure of.

The first 2048 bytes of the file are the mbx header, which starts with

*mbx*
ttttttttnnnnnnnn
flag-1
flag-2
flag-3
.
.
.

where
tttttttt is a time value; this doesn't seem to change, so I think it is 
when the file was first created?
nnnnnnnn it the largest message UID used in the file
flag-1, flag-2,  are locally defined flags. e.g. Junk, NonJunk, $Forward, 
$MNDSent

The mail messages follow the header
Each message starts with a line of the form

dd-mmm-yyyy hh:mm:ss +zzzz,ssss;ffffffffFFFF-UUUUUUUU

This should be followed by a message of ssss bytes
FFFF is the standard flags, with each bit representation a flag (e.g. 
1-Read, 2-Deleted, 4-Important, 8-Answered)
I assume the locally defined flags are similarly recorded in ffffffff
UUUUUUUU is a hexadecimal number, the unique ID (UID) of the message. The 
messages seem to be stored in the file in UID order.
What happens if the UIDs get out of sequence or are duplicated, will imapd 
replace the UIDs with new ones, in sequence or would this require outside 
intervention?

Something that puzzles me, I made a new file by asking the mail client to 
save messages in a new folder. The first message starts with the usual 
line

dd-mmm-yyyy hh:mm:ss +zzzz,ssss;ffffffffFFFF-00000000

Next I have asked the mail client (pine via imap) to open and list this 
new folder, the UID is replaced by 00000001 and a hexadecimal number has 
been added after the file header and before the first message header. I am 
curious, what this number does and how important it is?

Richard Westlake

---

From: Tim Mooney <[email-masked]>
Date: Thu, 16 Mar 2006 15:56:00 -0600 (CST)

In regard to: [Imap-uw] Re: MBX file format specification, Richard Westlake...:

> happens. I was thinking of producing some caned procedures or scripts which 
> junior members of the support group could use to recover most if not all of 
> the messages, if I wasn't around. I have since found some repair scripts, 
> which might help with this, including Tim Mooney's mbx-repair.

I'll accept any and all feedback on that script (including "what were you
thinking?!") .  Note that it uses "mailutil" exclusively for creating an
new, pristine folder and for appending messages from corrupted folders
onto it the new mailbox, so it pushes the whole locking issue onto Mark's
routines (thankfully!).

The script is pretty rough around the edges, but I'll definitely fix any
problems you report.

Tim
-- 
Tim Mooney                              [email-masked]
Information Technology Services         (701) 231-1076 (Voice)
Room 242-J6, IACC Building              (701) 231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Thu, 16 Mar 2006 19:17:57 -0800 (Pacific Standard Time)

On Thu, 16 Mar 2006, Richard Westlake wrote:
> Does anything else use same mbx file format?

The mbx format is private to the UW c-client library, used by UW imapd, 
ipop3d, mailutil, and of course Pine.  It was designed by me, and is a 
descendant of the mailbox format used on the Tenex and TOPS-20 systems of 
the 1970s and 1980s.

I have heard rumors of a version of postfix that knows how to append in 
mbx format; but I do not (will not, can not) promise that it will work 
today or in the future.

> 2) Repairing corrupt mbx format files.

Have you read this particular FAQ?
 	http://www.washington.edu/imap/IMAP-FAQs/index.html#7.15

> flags it doesn't recognise (e.g. Junk, NonJunk, $Forward, $MNDSent).

Hmm; mailutil is supposed to create the keywords as needed.  Do you have 
the most recent version?

> tttttttt is a time value; this doesn't seem to change, so I think it is when 
> the file was first created?

tttttttt is the IMAP UIDVALIDITY.  It is set to the UNIX date when the 
file is created, and normally won't change; but if the UID regime ever 
gets invalidated (shouldn't happen) then a new UIDVALIDITY will be 
assigned

> nnnnnnnn it the largest message UID used in the file

More accurately, it's the last-assigned UID.  This always increases; it 
never gets smaller, even if the message with that UID is expunged.

> flag-1, flag-2,  are locally defined flags. e.g. Junk, NonJunk, $Forward, 

Correct.  They're what IMAP calls keywords.

> dd-mmm-yyyy hh:mm:ss +zzzz,ssss;ffffffffFFFF-UUUUUUUU

All correct.  The date is the IMAP INTERNALDATE, the ssss is

> What happens if the UIDs get out of sequence or are duplicated, will 
> imapd replace the UIDs with new ones, in sequence or would this require 
> outside intervention?

This invalidates the UID regime, which will cause a new UIDVALIDITY to be 
assigned and every message issued a new UID.

This is bad juju, since it will cause clients which cause messages to toss 
their cache and reload the entire mailbox.

> save messages in a new folder. The first message starts with the usual line
>
> Next I have asked the mail client (pine via imap) to open and list this new 
> folder, the UID is replaced by 00000001

This is old (imap-2004g) behavior.  It uses "lazy" UID assignment, in 
which the message is delivered with a zero (unassigned) UID and a UID is 
assigned and written when the mailbox is opened.

The new behavior (in the development imap-2006) assigns UIDs when the 
message is delivered, in order to support the IMAP UIDPLUS extension.

> and a hexadecimal number has been 
> added after the file header and before the first message header. I am 

This is the most recent UNIX process ID to have written into the file.  It 
helps when multiple sessions have the file open; it tells other sessions 
that the mailbox got written by some other process and it needs to look at 
the file.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Sirota <[email-masked]>
Date: Thu, 16 Mar 2006 22:42:02 -0500

--On Thursday, March 16, 2006 7:17 PM -0800 Mark Crispin 
<[email-masked]> wrote:
>> I have used "mailutil copy" for this, however it discards, the
>> X-Keyword:  flags it doesn't recognise (e.g. Junk, NonJunk, $Forward,
>> $MNDSent).
>
> Hmm; mailutil is supposed to create the keywords as needed.  Do you
> have the most recent version?

We've found the same thing with imap-2004g, and came up with a workaround
involving using the IMAP "STORE" command.

Haven't tested the development version to see whether this problem 
remains.

Mark
-- 
Mark Sirota, Associate Director, Network Engineering and Services
University of Pennsylvania, Information Systems and Computing
[email-masked], 215/573-7214
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Alan Thew <[email-masked]>
Date: Fri, 17 Mar 2006 10:42:33 +0000 (GMT)

On Thu, 16 Mar 2006 19:17 , Mark Crispin <[email-masked]> said:

> descendant of the mailbox format used on the Tenex and TOPS-20 systems of the 
> 1970s and 1980s.
>
> I have heard rumors of a version of postfix that knows how to append in mbx 
> format; but I do not (will not, can not) promise that it will work today or 
> in the future.

exim can deliver to an mbx mailbox. We ran this without problems.
<snip>

-- 
Alan Thew                                       [email-masked]
Computing Services, University of Liverpool     Fax: +44 151 794-4442
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw