Subject: mailutil transfer and bad messages

From: Nicolas Kowalski <[email-masked]>
Date: Wed, 08 Dec 2004 19:17:36 +0100

Hello.

When I use mailutil transfer to copy a whole mailbox hierarchy from a
uw-imap server to a cyrus one, it sometimes fails with the error
"message contains invalid header", and then stops.

Is there a way to make mailutil ignore these badly formatted messages
and continue the transfer ?

Thanks.

-- 
Nicolas

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Dec 2004 10:39:16 -0800 (PST)

> and continue the transfer ?

That problem (the check for "invalid header") is not in mailutil; it's in 
the Cyrus server.  The Cyrus server is refusing to accept the message in 
question.

Mailutil, in turn, stops when it ceases to be able to do what it was told 
to do.  Mailutil has no way of knowing that the problem is just that silly 
check, or some more serious matter.  I doubt very much that you would want 
to have mailutil tell you that a copy was done, only to find that most 
messages were not copied due to a serious problem.

I suggest that you ask the maintainers of the Cyrus server if there is a 
way to disable that check.  Otherwise, there is no way that you can truly 
copy your messages.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Nicolas Kowalski <[email-masked]>
Date: Wed, 08 Dec 2004 20:18:52 +0100

Mark Crispin <[email-masked]> writes:

> That problem (the check for "invalid header") is not in mailutil; it's
> in the Cyrus server.  The Cyrus server is refusing to accept the
> message in question.

Yes, but the message really seems to be bad. When I look the offending
mailbox (mbx format) with Pine, I get this:

 R   1999-01-28 <obscured>      (4530) Re: <subject obscured>
 RA  1999-05-07 <obscured>      (4063)
 R                              (1155)
 R   1999-05-08 <obscured>      (4883) Re:

The third message contains the text body from the previous message,
and begins with a '>From ' line. I believe some crap happened when I
converted from unix format to mbx two years ago.

> Mailutil, in turn, stops when it ceases to be able to do what it was
> told to do.  Mailutil has no way of knowing that the problem is just
> that silly check, or some more serious matter.  I doubt very much that
> you would want to have mailutil tell you that a copy was done, only to
> find that most messages were not copied due to a serious problem.

You are right, there is a serious problem in this case. 

-- 
Nicolas

---

From: Nicolas Kowalski <[email-masked]>
Date: Wed, 08 Dec 2004 22:58:48 +0100

Cliff Green <[email-masked]> writes:

> Today at 7:17pm, Nicolas Kowalski wrote:

> NK> When I use mailutil transfer to copy a whole mailbox hierarchy from a
> NK> uw-imap server to a cyrus one, it sometimes fails with the error
> NK> "message contains invalid header", and then stops.
> NK> 
> NK> Is there a way to make mailutil ignore these badly formatted messages
> NK> and continue the transfer ?

[...]

> So, no - you can't make mailutil ignore them, but you can remove them so 
> it doesn't have to.

Yes, that's what I will do.

Thanks.

-- 
Nicolas

---

From: [email-masked]
Date: Wed, 8 Dec 2004 21:08:01 -0600

I have a program you may download from sourceforge which will usually correct
corrupt MBX files.

The url is   http://sourceforge.net/projects/mbxuncorrupter/

You may be able to fix the MBX file then mailutil won't gripe.  Any
unrecoverable mail fragments will be put into separate text files.

Hope this helps.

TomC

> Yes, that's what I will do.

-------------------------------------------------
Email solutions, MS Exchange alternatives and extrication,
security services, systems integration.  
Contact:    [email-masked]

---

From: Nicolas Kowalski <[email-masked]>
Date: Thu, 09 Dec 2004 12:08:30 +0100

Thanks.

However, I ran your utility on the specified mailbox, and it did not
repair the badness. That's probably because the MBX headers are not
corrupted, but because the unix format mailbox (used to create this
mbx-format one) already had badness in it.

Also, when running test transfers, I am experiencing more weird error
messages from mailutil, such as "message contain NUL character". It
seems that my users mailboxes contain a lot of crap.

-- 
Nicolas

---

From: [email-masked]
Date: Thu, 9 Dec 2004 06:51:39 -0600

> seems that my users mailboxes contain a lot of crap.

So these mailboxes were first converted from something else?  Interesting. 
Would you be willing to send one or two of the corrupt files to my email
address?  I have a morbid curiousity about corrupt files.

Send them to [email-masked],  not to the list.

thanks

tc

---

From: Cliff Green <[email-masked]>
Date: Thu, 09 Dec 2004 09:43:27 -0500 (EST)

On Dec 8 at 10:58pm, Nicolas Kowalski wrote:

NK> Cliff Green <[email-masked]> writes:
NK> 
NK> > Today at 7:17pm, Nicolas Kowalski wrote:
NK> 
NK> > NK> When I use mailutil transfer to copy a whole mailbox hierarchy from a
NK> > NK> uw-imap server to a cyrus one, it sometimes fails with the error
NK> > NK> "message contains invalid header", and then stops.
NK> > NK> 
NK> > NK> Is there a way to make mailutil ignore these badly formatted messages
NK> > NK> and continue the transfer ?
NK> 
NK> [...]
NK> 
NK> > So, no - you can't make mailutil ignore them, but you can remove them so 
NK> > it doesn't have to.
NK> 
NK> Yes, that's what I will do.

Just a note - that's okay for an mbox-format folder, mbx folders are a 
different story;  you'll probably need to copy those messages to an mbox 
folder, remove the extra header, then move them back to the mbx folder.

c
-- 
Clifford Green               Internet -  [email-masked]
Academic Computing Services                    UMDNJ-IST
Never wrestle with a pig, for if you do, you will both get dirty, but the 
pig will enjoy it. -- Charlie Munger

---

From: Mark Crispin <[email-masked]>
Date: Thu, 9 Dec 2004 08:05:29 -0800 (PST)

> folder, remove the extra header, then move them back to the mbx folder.

What's OK for an mbox format mailbox is also OK for mbx; the problem is 
that it is not OK for Cyrus.

-- Mark --

---

From: Nicolas Kowalski <[email-masked]>
Date: Thu, 09 Dec 2004 17:26:50 +0100

Mark Crispin <[email-masked]> writes:

> What's OK for an mbox format mailbox is also OK for mbx; the problem
> is that it is not OK for Cyrus.

I found on the web some tips about it, it requires patching
imap/message.c in the cyrus source directory. It shows that Cyrus
reject unconditionnaly:

1 - message body containing the '\0' character

2 - header name not terminated by a ':', such as '>From '

3 - empty line in headers

4 - null-name header like ': <value>'

Mark, which above conditions are accepted or rejected by UW-IMAP ?

Thanks.

-- 
Nicolas

---

From: Mark Crispin <[email-masked]>
Date: Thu, 9 Dec 2004 08:51:37 -0800 (PST)

> Mark, which above conditions are accepted or rejected by UW-IMAP ?

UW imapd does not reject any of these; it will accept whatever garbage you 
throw at it.  Whether it does anything useful with that garbage 
subsequently depends upon the nature of that garbage.

-- Mark --

---

From: Nicolas Kowalski <[email-masked]>
Date: Fri, 10 Dec 2004 18:39:12 +0100

On Wed, 8 Dec 2004, Mark Crispin wrote:

> I suggest that you ask the maintainers of the Cyrus server if there is 
> a way to disable that check.  Otherwise, there is no way that you can 
> truly copy your messages.

I disabled myself these checks in the cyrus source, and then all the 
transfers went fine, very quickly, perfectly keeping all mail attributes 
(internal dates, flags, ...). :-)

Thanks Mark, you showed me the right direction.

Regards.

-- 
Nicolas

---

From: Lars Immisch <[email-masked]>
Date: Tue, 22 Feb 2005 22:01:42 +0100

Hi,

> seems that my users mailboxes contain a lot of crap.

I remember, I had those a lot when I converted my old Unix mailboxes 
that had NeXTMail attachments in them.

- Lars