Subject: mailutil appenddelete silently loses mail?

From: Tim Mooney <[email-masked]>
Date: Thu, 30 Sep 2004 14:29:09 -0500 (CDT)

All-

I have a vague recollection of reading that a bug in mailutil was recently
fixed where mailutil would miss the last message in a folder when doing
a `move' or `appenddelete', but I can't find any reference to that in
2004a and the 2004b dev doesn't include any kind of ChangeLog, so I don't
know if I'm just dreaming...

In any case, I've run into a strange situation with mailutil that seems
similar to the problem I might have imagined:

$ ls -al INBOX
-rw-------    1 mooney   sysadmin 55801103 Sep 30 12:02 INBOX
$ file INBOX
INBOX: MBX mail folder
$ mailutil check INBOX
Unable to find CRLF at 53775404 in 64 bytes, text: From [email-masked]  Fri Aug 13 12:54:08

It's corrupt, we have utilities to deal with that (all of which
use mailutil to some extent).

I split the folder at the corruption point, into:

$ ls -al INBOX.????
-rw-------    1 mooney   sysadmin 53775404 Sep 30 12:08 INBOX.head
-rw-------    1 mooney   sysadmin  2025699 Sep 30 12:08 INBOX.tail

I check INBOX.head, which should now be a valid MBX-format mailbox:

$ mailutil check INBOX.head
No new messages, 262 total in INBOX.head

I create a new, empty mailbox in MBX format:

$ file INBOX
INBOX: MBX mail folder
$ mailutil check INBOX
No new messages, 0 total in INBOX

Now I appenddelete the messages from INBOX.head into INBOX:

$ mailutil appenddelete INBOX.head INBOX
$ echo $?
0
$ mailutil check INBOX
262 new message(s) (0 unseen), 262 total in INBOX
$ ls -al INBOX
-rw-------    1 mooney   sysadmin 31356329 Sep 30 12:08 INBOX

Ok, where did the other roughly 20 Megabytes go?

If I compare a "before and after", by viewing the 53 MB INBOX.head and
the 30 MB INBOX in two different pine windows, the message index shows
the exact same messages, in the same order, and with the same size
displayed.

If I pull off the 2K header from each mailbox so that I can use ``diff -u''
on them (it doesn't like the null padding), I see that the `appenddelete'
has changed each of the MBX index lines, so that the last eight characters
are all zeros:

-27-Nov-2002 00:30:04 -0600,1245;000000000009-00000003
+27-Nov-2002 00:30:04 -0600,1245;000000000009-00000000

It also appears to have rearranged the order that some of the messages
are written into the file, so that it's very difficult to find what's
truly missing.

Any suggestions to try figure out what the missing 20 MB is, and what
may have happened?

Two asides:

- I've noticed that the MBX "index" line,

 	03-Aug-2004 16:06:06 -0500,4355;000000008003-00000ee6

   sometimes has a date that is zero-padded, and sometimes its
   space-padded:

 	 4-Aug-2004 00:02:58 -0500,696;000000008003-00000eed

   Why is that?

- It would be useful to have a `version' command available from within
   mailutil, so you can see some kind of version info, so you can tell
   what imapd kit mailutil came out of.

Tim
-- 
Tim Mooney                              [email-masked]
Information Technology Services         (701) 231-1076 (Voice)
Room 242-J6, IACC Building              (701) 231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164
-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Thu, 30 Sep 2004 12:40:11 -0700 (PDT)

If by "the other 20 megabytes" you are referring to the difference between 
53775404 and 31356329, that difference is probably in expunged messages 
which never got garbage collected because the expunge was always from a 
shared-access session.

The way to test that is to expunge INBOX.head without multiple access.  I 
bet that you'll find that it shrinks by that about that amount.

The last 8 bytes in the internal header are the message UID.  Newly 
delivered messages get a UID of 0, which causes a UID to be assigned the 
next time the mailbox is opened (as opposed to append).  Otherwise, append 
would have to go to a lot more work.

" 3-Aug-2004" and "03-Aug-2004" are exactly equivalent.  One happens when 
the date is provided, the other happens when the date is defaulted.  It's 
not worth fixing.

On Thu, 30 Sep 2004, Tim Mooney wrote:

> Unable to find CRLF at 53775404 in 64 bytes, text: From 
> [email-masked]  Fri Aug 13 12:54:08
>
>

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Tim Mooney <[email-masked]>
Date: Thu, 30 Sep 2004 15:28:08 -0500 (CDT)

In regard to: Re: mailutil appenddelete silently loses mail?, Mark Crispin...:

> 53775404 and 31356329,

Yes.

> that difference is probably in expunged messages which 
> never got garbage collected because the expunge was always from a 
> shared-access session.
>
> The way to test that is to expunge INBOX.head without multiple access.  I bet 
> that you'll find that it shrinks by that about that amount.

You're exactly correct.  I wasn't expecting that there would be a case
where messages wouldn't show up in the index (even marked for deletion)
but would be hanging around in the folder.  Pressing x within pine even
indicated "no messages expunged", but the size shrunk to match the other
folder exactly.  I understand now though -- the messages were already
expunged, they just weren't excised from the folder yet.

> The last 8 bytes in the internal header are the message UID.  Newly delivered 
> messages get a UID of 0, which causes a UID to be assigned the next time the 
> mailbox is opened (as opposed to append).  Otherwise, append would have to go 
> to a lot more work.

Ok.

What's the best source for information on the MBX internal header?  I
think it's time I learn the signficance of the various bytes in the
internal header after the ;.

> " 3-Aug-2004" and "03-Aug-2004" are exactly equivalent.  One happens when the 
> date is provided, the other happens when the date is defaulted.  It's not 
> worth fixing.

Thanks for the info.

Tim

---

From: Mark Crispin <[email-masked]>
Date: Thu, 30 Sep 2004 16:24:29 -0700 (Pacific Daylight Time)

On Thu, 30 Sep 2004, Tim Mooney wrote:
> What's the best source for information on the MBX internal header?

Some information is in the IMAP FAQs:
 	http://www.washington.edu/imap/IMAP-FAQs/index.html#7.15

> I

The 12 bytes between the ; and the - are the flags mask; 8 hex digits of 
keyword flags and 4 hex digits of system flags.  The bit definitions are 
in the source code.

-- Mark --