Subject: What to do with VERY large mailbox files?

From: Frank Joerdens <[email-masked]>
Date: Tue, 29 Apr 2003 19:08:15 +0200

Greetings. I am a little stuck with my thinking on how to deal with
mailboxes (flat-file ones, obviously) growing to several hundred
megabytes in size. Doesn't this create a performance issue with the
server and/or client programs at some point? Should I instruct my users
to clean their inboxes regularly? Should I not worry at all? Is this the
reason why people think they want file/message formats rather than
flat-file ones?

I actually do have a specific issue with my files under /var/spool/mail
becoming too big, since my backup scheme includes looking for files
which have changed in the past 24 hours, and copying those to a backup
server in another section of the building. Those files are then kept for
14 days. With a few dozen mailboxes having grown to several hundred
megs, this scheme already becomes pretty impractical.

Should I use logrotate to start a new spoolfile for each user if his/her
inbox has grown over a certain size? Should I use procmail to create a
separate file which contains only the previous day's mails for each
user, which is then deletete/replaced every day (but how would I then
reconstruct the entire several-hundred-meg-mailbox once the mailserver's
filesystems has crashed?)?

Seeing that this issue of backing up large mailbox files should be (I
think) of very general import, I am sure I must be missing something.

Regards, Frank
-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Sebastian Hagedorn <[email-masked]>
Date: Tue, 29 Apr 2003 19:15:15 +0200

-- Frank Joerdens <[email-masked]> is rumored to have mumbled on 
Dienstag, 29. April 2003 19:08 Uhr +0200 regarding What to do with VERY 
large mailbox files?:

> server and/or client programs at some point?

Yes.

> Should I instruct my users
> to clean their inboxes regularly?

Yes.

> Is this the

Yes and no. There are more choices than just flat-file and file/message.
--
Sebastian Hagedorn M.A. - Postmaster - RZKR-R1 (Flachbau), Zimmer 18
Zentrum für angewandte Informatik - Universitätsweiter Service RRZK
Universität zu Köln / Cologne University - Tel. +49-221-478-5587

---

From: Frank Joerdens <[email-masked]>
Date: Tue, 29 Apr 2003 19:34:21 +0200

On Tue, Apr 29, 2003 at 07:15:15PM +0200, Sebastian Hagedorn wrote:
[]
> Yes and no. There are more choices than just flat-file and file/message.

According to Mark Crispin's text

http://www.washington.edu/imap/documentation/formats.txt.html

. mbx is the best format to use with c-client, which is a flat-file
format. And he lists different formats under either flat-file or
file/message. Are there still other choices? What do you mean exactly?

Regards, Frank

---

From: Mark Crispin <[email-masked]>
Date: Tue, 29 Apr 2003 10:40:12 -0700 (Pacific Daylight Time)

> . mbx is the best format to use with c-client, which is a flat-file
                                       ^^^^^^^^
> file/message. Are there still other choices? What do you mean exactly?

Note that the text in question refers to a specific piece of software.

There are, indeed, other alternatives to flat-file and file/message
formats.  We are working on such an alternative in c-client, and hope to
offer it soon (it's in testing here).

If you are still using traditional UNIX mailbox format, switching to mbx
will make a substantial improvement in performance and scaleability.  It
isn't the ultimate solution, but it will buy you time.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

---

From: Sebastian Hagedorn <[email-masked]>
Date: Tue, 29 Apr 2003 19:46:10 +0200

-- Frank Joerdens <[email-masked]> is rumored to have mumbled on 
Dienstag, 29. April 2003 19:34 Uhr +0200 regarding Re: What to do with VERY 

> file/message. Are there still other choices? What do you mean exactly?

OK, semantics. To me "flat file" signifies a BSD style mailbox (mbox). 
Technically mbx is also a flat-file, but it carries sufficient meta 
information to be a whole lot faster than mbox files ...

mbx files the size of several hundred megabytes are still pretty unwieldy, 
but you'd be much better off with mbx than with mbox. file/message in 
itself doesn't help if there are many mails. For that to be efficient you 
also need meta information, the way the Cyrus server does it.
--
Sebastian Hagedorn M.A. - Postmaster - RZKR-R1 (Flachbau), Zimmer 18
Zentrum für angewandte Informatik - Universitätsweiter Service RRZK
Universität zu Köln / Cologne University - Tel. +49-221-478-5587

---

From: John Dennis <[email-masked]>
Date: 29 Apr 2003 13:50:38 -0400

The Cyrus IMAP server (http://asg.web.cmu.edu/cyrus/imapd) supports a
quota mechanism to address exactly this issue. To the best of my
knowledge (I'm sure I'll be corrected if I'm wrong) the UW IMAP does not
support quotas. Cyrus IMAP also supports other mail storage formats. The
O'Reilly book "Managing IMAP" book does a good job of explaining the
pros and cons of selecting a server implementation and configuration
issues.

John

---

From: Mark Crispin <[email-masked]>
Date: Tue, 29 Apr 2003 10:51:23 -0700 (Pacific Daylight Time)

On Tue, 29 Apr 2003, John Dennis wrote:
> To the best of my
> support quotas.

If UW imapd does not support quotas on RedHat systems, then that is
because RedHat systems do not support quotas.

UW imapd inherits its system management from the underlying UNIX system.
Quotas are part of that.  UW imapd does its part; it properly detects a
quota exceeded condition and backs out changes to prevent trashed mailbox
files.

-- Mark --

---

From: Frank Joerdens <[email-masked]>
Date: Tue, 29 Apr 2003 20:00:00 +0200

On Tue, Apr 29, 2003 at 10:51:23AM -0700, Mark Crispin wrote:
[]
> quota exceeded condition and backs out changes to prevent trashed mailbox

Yes. I'll have to use quotas. Educating users to recognize the fact that
some behaviour is incompatible with existing solutions is often the most
difficult part of the job . . . ;)

Regards & Thanks For All the Useful Input,

Frank

---

From: Oskar Teran <[email-masked]>
Date: Tue, 29 Apr 2003 13:00:08 -0500

>isn't the ultimate solution, but it will buy you time.
this thread is interesting since I'm also in this same 'boat'.  I have 
sites that have outgrown their outlook .pst files...several 2GB pst 
files worth of emails to migrate (2GB is the max file size for pst files 
- by then you're at your own risk).  I was wondering about using 
something like dbmail for storing these emails.  What is the recommended 
max. file size limit for an .mbx file?  I setup the mailutil and the new 
imap setup to use mbx by default. 
I gave the people at those sites the rant about using unix/linux for 
email instead of exchange (they were considering exchange).  So, I got 
them to use unix/linux but now I have to worry about the integrity of 
their email data store.

Regards,

Oskar

---

From: John Dennis <[email-masked]>
Date: 29 Apr 2003 16:17:59 -0400

> quota exceeded condition and backs out changes to prevent trashed mailbox

You are referring to user/group quotas at the system level which is
supported on RedHat systems. I was talking about a higher level quota
mechanism implemented in the IMAP server which monitors the size of a
individual user's mail store at any level in the users mailbox hierarchy
and is capable of notifying users of when they are close to their quota
limit allowing them the opportunity to manage their mail store prior to
reaching a point where mail delivery fails. I believe this was what the
original poster was looking for. RFC 2087 provides *some* of this but I
don't believe UW IMAP supports the QUOTA capability defined in RFC 2087
and thats what I meant when I said UW IMAP does not support quotas. 

John

---

From: Nicolas Kowalski <[email-masked]>
Date: Tue, 15 Feb 2005 17:25:43 +0100 (MET)

Sorry, this is really back in time....

> offer it soon (it's in testing here).

Are there any news about this alternative format ?

Thanks.

-- 
Nicolas

---

From: Mark Crispin <[email-masked]>
Date: Tue, 15 Feb 2005 08:48:13 -0800 (PST)

> Are there any news about this alternative format ?

Not yet.  I've been bogged down in other tasks.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.