Subject: [Imap-uw] Mail file fragmentation on Linux

From: Andrew Daviel <[email-masked]>
Date: Mon, 5 Dec 2005 12:17:14 -0800 (PST)

I am running UW imapd and dmail with MBX mailbox format. I noticed using
debugfs "stat <file>" that the mail file can get quite badly fragmented
over time.

I wondered if there was any way to discourage this in the MBX code.
One thing that occurred to me was that maybe when the file is growed to
append new messages, that the requested allocation have "room for
expansion", e.g. rounded up to the next 20 blocks or something.
(maybe it already does this)

Just copying the mail file to a new location and moving it back seems to
get the fragmentation under control.

As an example, after this procedure, my mailbox had fragments of about 1k
blocks. Before, a heavy user had fragments of about 3 blocks each, not
all contiguous or even in order. Seems to me this must slow down disk
access a bit

-- 
Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376  (Pacific Time)
[email-masked]
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Thu, 8 Dec 2005 18:11:16 -0800 (Pacific Standard Time)

> debugfs "stat <file>" that the mail file can get quite badly fragmented

Since most access to a mbx-format mailbox is random access, I'm not 
convinced that fragmentation is a major issue.  Certainly, fragmentation 
is less of a problem overall on modern filesystems than it is on FAT 
filesystems under DOS.

I don't think that there's any good way to prevent fragmentation from 
happening given the way that mbx files work.  I agree that copying to 
another file and moving it back is a good way to defragment the file.

I'd like to see some more definite statistics before I believe that 
fragmentation is much to worry about.  I'm of the opinion that 
defragmentation wastes more resources than fragmentation does, but I may 
be wrong.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: [email-masked]
Date: Fri, 16 Dec 2005 13:48:35 -0500

> debugfs "stat <file>" that the mail file can get quite badly fragmented

We run a large predominately mbx-format, with some Unix format, server. We 
have observed that if we massive copy all files off a large LUN, then copy 
them back--ie de-fragmenting the file system--the number of IOPs drops.
(Such a massive copy requires taking the server out for 6 hours or more.) 
Thereafter there is a regular increase in the IOPs as the file system 
fragments over time.

To prevent this, we run a nightly defrag: a Perl script finds files in 
many fragments. It copies each such file. If the result is fewer 
fragments, we keep the new file; otherwise we keep the original file.
This helps to keep the IOPs down. However, over time we find this is
less and less effective, because the Linux ext2 filesystem becomes
more and more fragmented; copying a file frequently results in more
fragments then you started with.

If you google the fragmentation issue you will find many sources that 
perpetuate the urban legend that Linux ext2/ext3 filesystems don't
need defragmenting ...

Perhaps the solution lies in having the operating system automatically 
defragment files (as Apple MacOS X does with files under 10MB.)

Alex Nishri
University of Toronto

---

From: Mark Crispin <[email-masked]>
Date: Tue, 20 Dec 2005 13:33:32 -0800 (Pacific Standard Time)

> defragment files (as Apple MacOS X does with files under 10MB.)

Hi Alex -

Thank you for the information.  Over the years, I have watched the 
discussions on fragmentation from the sidelines, and in particular the 
question of whether fragmentation is actually harmful on ext2/ext3 
filesystems.  There's been more heat than light in those discussions, so 
I'm glad to see some factual information.

The cause of the fragmentation is not the overwrite-on-expunge behavior of 
the drivers in c-client (the library used by UW imapd to access mailbox 
files), but rather that new messages are appended.  There is no good way 
to prevent fragmentation in a flat file format (either traditional UNIX or 
mbx format) because the new mail append is when the fragmentation occurs.

The assumption made by filesystems is that software will periodically 
replace the file with a new copy; and that if fragmentation is avoided 
when writing new files this will suffice to cure any excessive 
fragmentation problems.

This assumption is false when c-client is used, since it never writes a 
new copy of the mailbox; it always updates in place.  Normally, updating 
in place is much better (far fewer writes and no additional disk space 
consumed), but it doesn't cure the fragmentation and the filesystem was 
depending upon it to do so.

I don't think that it would be a good idea for c-client to attempt to 
defragment.  It is a time-consuming process, and an unsafe one.  But I 
would consider building a manual defragmentation tool which would be as 
close to safe with c-client programs (imapd, ipop3d, Pine, etc.) as 
possible if the community considers it to be worthwhile.

-- Mark --

---

From: Erik Kangas <[email-masked]>
Date: Tue, 20 Dec 2005 20:31:02 -0500

Mark Crispin wrote:

> would consider building a manual defragmentation tool which would be
> as close to safe with c-client programs (imapd, ipop3d, Pine, etc.) as

I would vote for this and I would use this.  In particular, I would
imagine a periodic process that

* See if there is contiguous space available to defragmentation on the
system (and how big?)
* finds "large" folders (especially INBOXES)
* Sees if they are fragmented
* If they are "sufficiently fragmented", defragment

This would help performance somewhat for people to keep large inboxes --
until they can be taught the pleasures of small inboxes.  So, if such a
tool was to be developed, I would love it if, in addition to actually
defragmenting a mbx-formatted folder, it could:

* Tell how how fragmented the folder file is in the first place
* Tell you about contiguous space available for defragmenting

so that the controlling script could decide if it makes sense to bother
defragmenting the folder.

Best,
-Erik Kangas
LuxSci.com

---

From: Mark Crispin <[email-masked]>
Date: Tue, 20 Dec 2005 17:47:11 -0800

On Tue, 20 Dec 2005, Erik Kangas wrote:
>> I
> I would vote for this and I would use this.

I have built a single file defragmentation tool on:
 	ftp://ftp.cac.washington.edu/mail/powertools/defrag.c

This program was written for Linux, and may need modification on other 
operating systems.  It takes a single argument, the file name of a file to 
defragment, and will defragment it as best as the filesystem will allow. 
It will decline to act on any file that is currently open and locked.

It does this by opening, exclusive-locking, copying the file to a temp 
file, truncating the file to zero bytes, then copying it back.  This has 
the benefit of preserving the inode number in case anything cares.

There is a known failure case which will cause problems for the user and 
require manual remedy.  If, after starting the copy-back, an I/O error 
occurs, the program will stop and report an error.  However, it will not 
delete the temp file in this case.

There are other failure cases which will also stop and report an error, 
but no change was made to the original file

Note that although the atime and mtime are preserved, it is not possible 
to prevent the ctime from being changed.  This will cause the file to be 
flagged as \Marked in IMAP lists.  I can't do anything about this, sorry.

As for the more ambitious program that you outlined, that's dependent upon 
the underlying filesystem and is more research than I care to undertake. 
All I tried to do is to build a program that will safely (or rather, as 
safely as possible) defragment a file on a live system without risking 
possible loss of mail.

-- Mark --

---

From: Richard Westlake <[email-masked]>
Date: Wed, 21 Dec 2005 19:34:05 +0000 (GMT Standard Time)

On Tue, 20 Dec 2005, Erik Kangas wrote:
> until they can be taught the pleasures of small inboxes.

I would be interest in what people consider being a reasonable maximum 
INBOX size and if there is any online resources covering the problems 
caused by large INBOX.

Most of the accounts on our email serve have an INBOX smaller then 
150Mbyte, the rest are smaller then 500Mbyte, which is an improvement as 
we did have, files larger then 1Gbyte.

The Apple email client seems to be a particular antisocial with a large 
INBOX. We used to be able to tell when one particular Apple user had 
arrived, because the mail server would be unusable while their client 
connected and scanned for new mail.

I would be curious if Erik Kangas or anyone else has any suggestions on 
how to educate senior staff on the "pleasures of small inboxes"

Richard  Westlake

School of Crystallography, Birkbeck College, Malet Street, London WC1E 7HX
Tel: 020-7631-6859
----------------------------------------------------------------------
                Truth endures but spelling changes    --  Anon.
----------------------------------------------------------------------
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Sirota <[email-masked]>
Date: Wed, 21 Dec 2005 15:02:09 -0500

> how to educate senior staff on the "pleasures of small inboxes"

I realize I'm not directly answering your question, but unless these users
are running disconnected from the network, you can (and perhaps should)
configure Apple Mail not to cache (I think you set the cache size to zero).

Apple Mail is an odd duck in that it is a very modern client which appears
to be designed to run over slow network connections -- it maintains a
cache of data about every message in every mailbox, which it downloads in
the background, and it uses that cache for searching and the like rather
than asking the server to do it.

Real IMAP clients should have no need for local data at all, except when
running in disconnected mode (see the late, great Mulberry for an
excellent implementation of this).

Mark

---

From: scott hollatz <[email-masked]>
Date: Wed, 21 Dec 2005 14:43:18 -0600 (CST)

> we did have, files larger then 1Gbyte.
	[stuff deleted]

We limit the inbox to about 15MB (+/- 5MB depending on who you are) via
the lock+move+unlock+inform method.  This may sound draconian but keeps
inbox sizes down, the user knows how to get the moved mail, and all are
generally happy.

We have four two-node Sun V400 clusters typically with 1K users on each
cluster.

-- 
scott hollatz                                        net [email-masked]
information technology systems and services          tel +1 218 726 8851
university of minnesota duluth mn usa                fax +1 218 726 7674
                                                                         --
                                            "gabba gabba hey" - the ramones

---

From: Tim Mooney <[email-masked]>
Date: Wed, 21 Dec 2005 14:50:25 -0600 (CST)

In regard to: [Imap-uw] INBOX size [was Mail file fragmentation on Linux],...:

> I would be interest in what people consider being a reasonable maximum INBOX 
> size and if there is any online resources covering the problems caused by 
> large INBOX.

Reasonable depends on format.  MBX format INBOXes give significantly
improved performance, even at much larger sizes and message counts, than
traditional "From " format mailboxes.

I still recommend keeping the INBOX small (and try to lead by example),
but the format does make a huge difference.

My gripe with both traditional and MBX INBOXes relates more to backups
than it does to client access.

Tim
-- 
Tim Mooney                              [email-masked]
Information Technology Services         (701) 231-1076 (Voice)
Room 242-J6, IACC Building              (701) 231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Erik Kangas <[email-masked]>
Date: Wed, 21 Dec 2005 16:22:16 -0500

> we did have, files larger then 1Gbyte.
...
> how to educate senior staff on the "pleasures of small inboxes"

Hello,

What is a "too large" inbox depends on many factors.  Lets assume you use 
mbx-formatted mailboxes:

* The throughput of your I/O system is the central variable.  I recommend 
very fast SCSI drives in a RAID array that gives you a boost from striping
* How physically large the files are
* How fragmented files are
* How many messages are in the files.
* How busy the server [I/O] is

I.e. you can have a "small" mailbox (say 75 Mb) that has many thousands of 
messages that will be much much slower than a larger mailbox with many 
fewer messages due to all the work the IMAP server has to do.

You will also see that as your server gets busier with more users and more 
fragmented files, the bigger folders become slower and slower for obvious 
reasons.

For education, I would recommend pitching small mailboxes as in the users' 
best interests as they will be more productive, access things faster, etc.

Watch out for:
* People who use POP to check their email, leave it on the server, and let 
it build up.  This is REALLY inefficient as the INBOX grows.
* People who subscribe to all folders in their account - thus having their 
email client select them all every few minutes (Entourage will do this if 
you are not careful).

I heard Mark Crispin say that they were working on a new email folder 
format that would be much faster ... i.e. one where the metadata and/or 
header data would be stored in separate index files so that access would be 
much less dependent on folder size or number of messages.  Can we have a 
progress report? :)

Best,

-Erik Kangas
LuxSci.com

---

From: Mark Crispin <[email-masked]>
Date: Wed, 21 Dec 2005 13:22:55 -0800 (Pacific Standard Time)

On Wed, 21 Dec 2005, Erik Kangas wrote:
> I heard Mark Crispin say that they were working on a new email folder format 
> that would be much faster ... i.e. one where the metadata and/or header data 
> would be stored in separate index files so that access would be much less 
> dependent on folder size or number of messages.  Can we have a progress 
> report? :)

The short answer is "stalled due to other tasks, but probably going to 
return back to it shortly".

We have a prototype that worked well, but it wasn't "ready for prime time" 
in the distribution.

-- Mark --