Subject: Big messages and layout = dir (maildir, mh, mx)

From: Luca Olivetti <[email-masked]>
Date: Mon, 31 Aug 1998 15:09:07 +0200

My users tend to abuse the mail system (e.g. they don't know ftp and use
it to send big files).
The problem is aggravated by the fact that the imap server seems to read
each and every message when the client requests (or should request) just
the headers, with horrendous performance and sometimes crashing (out of
memory).
I'm not sure the problem lies within the server, since the client is
Communicator (4.03 spanish), but I gleaned through the code for mx,mh
and maildir (this last one is what I'm using, and I known that's not
part of the standard c-client distribution), and effectively the
"fetchheader" routine of each of those seems to read the whole
file/message.
I'm not a C wizard but wouldn't a rewrite of that routine improve
performance and reduce the risk of overusing memory? (probably not if
the messages are short, but if they are long messages with
attachments...).

I'm using the imap server 4.2 with the maildir driver by Mattias Larsson
(and a couple of fixes by me) under redhat linux 4.2.

-- 
Luca Olivetti
Wetron Automatización S.A. http://www.wetron.com/
Tel. +34 93 5883004      Fax +34 93 5883007

------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Mon, 31 Aug 1998 10:26:09 -0700 (PDT)

> My users tend to abuse the mail system (e.g. they don't know ftp and use

I'm not sure what you mean by "abuse".  We have administrative users here with
thousands of messages and tens of megabytes in a single mailbox (they also
tend to have lots of mailboxes).  With the right mailbox format, you can
handle that kind of load.

> the headers, with horrendous performance and sometimes crashing (out of

c-client does not keep more than a single message in memory at a time in any
local file format.

You may have unrealistic expectations of what you can accomplish on a small
memory machine.  Memory is cheap; disk bandwidth is not.  You can see this for
yourself by comparing the performance of 16-bit Windows PC Pine with 32-bit
Windows PC Pine; the 32-bit version uses memory while the 16-bit version
constantly reads from disk.

The entire message text needs to be available when parsing MIME, and (in the
case of Maildir) for calculating the Internet standard message length (it is
an absolute requirement of the IMAP protocol that this value be correct).

The underlying problem is that you use Maildir.  Maildir does not scale well
to large mailboxes.  This is a reason why we do not offer support for that
format (there are other reasons).  Some of these problems are inherant in the
"dir" type layout, others are specific to Maildir.

It may be possible to tweak things somewhat to make Maildir work a bit better,
but you're likely to get mixed results at best.  I am totally disinterested in
doing so.

If performance is important to you, use mbx format instead.

---

From: Tom <[email-masked]>
Date: Mon, 31 Aug 1998 20:46:29 -0700 (PDT)

On Mon, 31 Aug 1998, Luca Olivetti wrote:

...
> the messages are short, but if they are long messages with

  Even better, use a mail format that keeps index info (like headers)
separate from the messages.  mbx is probably best for uw-imapd.
cyrus-imapd uses a fast opening mailbox format too.  At least with Cyrus
you can repair mailboxes with the reconstruct tool.  I have no idea how I
might repair a damaged mbx.  I have one that uw-imapd crashed upon
reading...

Tom

---

From: Luca Olivetti <[email-masked]>
Date: Wed, 02 Sep 1998 08:49:51 +0200

> handle that kind of load.

Would you consider a single message of 60 megabytes an abuse ? ;-)
 
> an absolute requirement of the IMAP protocol that this value be correct).

Yes, I didn't spot the call to strcrlfcpy. I just assumed that the
message was already in crlf format. I was wrong.

> If performance is important to you, use mbx format instead.

Is there an easy way to migrate existing mailboxes to mbx format? If I
change something I would like the change to be transparent to the users.
When I switched to imap and decided to use maildir it was because it was
easy to migrate existing eudora mailboxes to maildir.
As a side note, I had to remove the dummy driver otherwise communicator
(4.03) would recognize all files as mail folders!. 4.5 doesn't exhibit
the same behavior, but I'm the only one using it at the moment (it's
beta and not in spanish).

---

From: Mark Crispin <[email-masked]>
Date: Tue, 1 Sep 1998 23:52:40 -0700 (PDT)

> Would you consider a single message of 60 megabytes an abuse ? ;-)

Not necessarily.  It depends upon the content.

> Is there an easy way to migrate existing mailboxes to mbx format?

Take a look at mbxcvt, which is one of the imap-utils.

> (4.03) would recognize all files as mail folders!

If you remove the dummy driver, then you can't list folders at all.

---

From: Luca Olivetti <[email-masked]>
Date: Wed, 02 Sep 1998 09:16:35 +0200

> If you remove the dummy driver, then you can't list folders at all.

It has been working since last May.
It works because the folders are listed by the maildir driver itself,
and those are the only folders I'm interested in (at the moment).