Subject: repairing "mbx" formatted files

From: Mike Trank <[email-masked]>
Date: Wed, 18 Oct 2000 09:46:16 -0600 (CST)

Hi there.

Anyone know a way that I can repair an "mbx" type mailbox file, manually
or otherwise?

It got ruined because I did not take Mark's advice about not using "mbx"
with NFS.....

Now, no c-client programs can open it up....

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Wed, 18 Oct 2000 08:49:37 -0700 (PDT)

> Anyone know a way that I can repair an "mbx" type mailbox file, manually

Some people have developed automated scripts, but if you're comfortable using
emacs it's pretty easy to fix it manually.  DON'T USE vi OR ANY OTHER EDITOR
UNLESS YOU ARE CERTAIN THAT EDITOR CAN HANDLE BINARY!!!

A c-client program would have issued an error message saying which byte number
it got to when it got unhappy with it the file, usually after the word "at",
e.g. something like:
	Unable to parse internal header at 43921: ne bombastic blurdybloop
This byte number ("43921") is the location of the error in the file.  That's
the point you need to fix.

c-client is expecting an internal header at that byte number, looking
something like:
 6-Jan-1998 17:42:24 -0800,1045;000000100001-00000001
The format of this internal line is:
dd-mmm-yyyy hh:mm:ss +zzzz,ssss;ffffffffFFFF-UUUUUUUU

The only thing that is variable is the "ssss" field, it can be as many digits
as needed.  All other fields (inluding the "dd") are fixed width.

So, the easiest thing to do is to look forward in the file for the next
internal header, and delete everything from the error point to that internal
header.

Here's what to do if you want to be smarter and do a little bit more work.
Generally, you're in the middle of a message, and there's nothing wrong with
that message.  The problem happened in the *previous* message.

So, search back to the previous internal header.  Now, remember that "ssss"
field?  That's the size of that message.  Mark where you are in the file, move
the cursor to the line after the internal header, and skip that many bytes
("ssss") forward.  If you're at the point of the error in the file, then that
message is corrupt.  If you're at a different point, then perhaps the previous
message is corrupt and has a too long size count that "ate" into this message.

Basically, what you need to do is make sure that all those size counts are
right, and that moving "ssss" bytes from the line after the internal header
will land you at another internal header.

Usually, once you know what you're looking at, it's pretty easy to work out
the corruption, and the best remedial action.  I generally don't use repair
scripts because I prefer the flexibility of manual repair.  Repair scripts
will make the problem go away but may not always do the smartest/best salvage
of the user's data.

> Now, no c-client programs can open it up....

Sigh, sorry to hear this.

---

From: Rich Graves <[email-masked]>
Date: Sun, 22 Oct 2000 23:43:06 -0400 (EDT)

I found the message in src/osdep/unix/unix.c and friends but I'm not sure
what it means. Can you reassure me that this is nothing to worry about?

I'm running imap-2000.RELEASE-CANDIDATE.8 on Linux, ipop3d slurping mail
from a Berkeley mbox to mbx-formatted /var/imap/hash/user/INBOX. Got two
sets of messages like this.

Oct 22 23:06:26 shemp ipop3d[24673]: pop3 service init from 129.64.222.58
Oct 22 23:06:26 shemp ipop3d[24673]: Message 1 UID 300710995 greater than\
 last 2622
Oct 22 23:06:26 shemp ipop3d[24673]: Auth user=joeschmo\
 host=dhcp-129-64-222-58.dorm.brandeis.edu [129.64.222.58] nmsgs=1/1
Oct 22 23:06:26 shemp ipop3d[24673]: Logout user=joeschmo\
 host=dhcp-129-64-222-58.dorm.brandeis.edu [129.64.222.58] nmsgs=0 ndele=1

I don't have the message, but the pseudoheader is:

>From MAILER-DAEMON Sun Oct 22 23:06:26 2000
Date: 22 Oct 2000 23:06:26 -0400
From: Mail System Internal Data <[email-masked]>
Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA
Message-ID: <[email-masked]>
X-IMAP: 0972270386 0000002623
Status: RO

---

From: Mark Crispin <[email-masked]>
Date: Sun, 22 Oct 2000 20:50:54 -0700 (PDT)

> Oct 22 23:06:26 shemp ipop3d[24673]: Message 1 UID 300710995 greater than\

This is nothing to worry about.  It is a debugging warning.

What happened is that someone sent a message to the user with a header
containing "X-UID: 300710995".  The "X-UID" header line isn't supposed to
appear in mail sent over the wire; it is an internal header to c-client.  Very
often, however, spammers write such headers as part of their obfuscated fake
headers.

Unlike X-UIDL (which is used by qpopper for POP3 only), the X-UID header
follows IMAP UID conventions (which are good for POP3 and IMAP).  IMAP UIDs
are assigned strictly ascending.  At the time, the highest UID ever assigned
in the mailbox was 2622.  300710995 is much larger, and hence bogus.

c-client also assigned UIDs monotonically (although this is not required by
any standard).  It is very unlikely that over 300 million messages have been
delivered to that mailbox in its history... :-)

Fortunately, although this is clearly bogus data generated by a hacker (and
probably a spammer), the effect is harmless.  c-client detected the bogon,
wrote the syslog() message, and replaced the bogus UID with a newly-assigned
one (2623).

The worst impact is if a bogon duplicates a previously-assigned UID belonging
to a message that was subsequently deleted, and the ascending order in the
mailbox is not otherwise impacted.  Such an occurance would probably cause the
client to ignore the message...which in the case of spam is probably a
feature, not a bug!