Subject: [Imap-uw] imapd expunge trouble

From: Shaun Ferguson <[email-masked]>
Date: Tue, 14 Jun 2005 16:48:01 -0500

Greetings,

We run a clustered student email system, using uw-imapd on the back-end
for mailbox storage and retrieval.  We recently expanded our storage for
student accounts, which are held on a GFS file-system (RedHat GFS from
CVS, circa May 2005).  In the process of doing this, we upgraded our
backend from a uw-imapd-2001 branch on RedHat 9 to uw-imapd-2004d on
Debian Sarge (we are NOT using the Debian version of uw-imapd, since we
require different compile flags). 

It recently came to our attention that the messages being marked for
deletion in users' mbx folders are shown to be expunged without error
(we've run by-hand imapd connection tests), and subsequent reads on the
mbx file show no existence of the "expunged" messages, but the data
still remains in the file.  Thus, the messages have become "lost" in the
file, as far as the uw-imapd utilities know.  It doesn't appear to be
corruption of the file format, per se, since other non-expunged messages
are still accessable to the users (as are new messages that arrive). 
The expunge is simply not removing the messages from the file.

We have noticed that when the message is "expunged" but doesn't actually
get deleted, the flags in the mbx header of the message changes as follows:

Before expunge:  14-Jun-2005 15:54:53 -0500,1377;000000000003-00000048

After expunge:  14-Jun-2005 15:54:53 -0500,1377;000000008003-00000048

Once the new '8' flag is set, the message goes "invisible" to uw-imapd.

We've run local filesystem tests in the exact same fashion as we have on
the GFS filesystem.  We can consistently and properly expunge messages
from the mbx folders stored on the ext3 filesystem, without issue.  We
have not yet been able to do so on the GFS filesystem.  We have also
experienced trouble with tmail and locking under a clustered
environment, which causes corruption of the mbx folder when two messages
are delivered at the same time (common in a clustered environment), and
have been working on that issue to near-fruition with another coworker
who is very familiar with the uw-imapd source (who has unfortunately
been called away on another project).  This may be a related locking
problem, depending on the expunge operation's steps, and we have not
been able to research or otherwise discover a solution.

Any assistance that can be rendered is very much appreciated.

Thank you,

Shaun Ferguson

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Tue, 14 Jun 2005 15:50:59 -0700 (Pacific Daylight Time)

Shaun -

There are two things going on:

The first thing is that in recent versions of the Linux kernel, the 
developers of Linux have "improved" the flock() system by changing the 
definition of how flock() works on network filesystems such as NFS.

The traditional definition, for the past 20 or so years, has been that 
flock() on a network filesystem is a successful no-op.  The recent 
"improvement" in Linux is that doing so always returns ENOLCK.

Suffice it to say that there have been howls of protest from application 
developers who have depended upon the traditional behavior for the past 
two decades.  The answer has been a Stallmanesque "the new version is 
better, fix your software."

The effect of this incompatible change to flock() is that attempts to 
upgrade a shared lock to an exclusive lock on an NFS file always appear to 
fail due to some other process having a shared lock.  In reality, neither 
a shared lock nor an exclusive lock is acquired.

This is what that strange "8" flag that you observed means: the message is 
expunged, but the space could not be reclaimed because (imapd falsely 
believed that) some other session had the mailbox opened.  It will reclaim 
the space when it can lock the mailbox exclusive (which will never happen 
because of the incompatible flock() change).

I discovered the problem after imap-2004d was released.  The necessary 
workaround is in imap-2004e:
 	ftp://ftp.cac.washington.edu/mail/imap-2004e.DEV.tar.Z

Although this is presented as a development snapshot, this version is 
really just imap-2004d with a small number of limited-audience but 
critical changes since imap-2004d.  Put another way, it's safe to install 
in place of imap-2004d.

The second thing that is going on was alluded to above.  I don't know much 
about GFS, but it seems to be that for the purposes of flock() locking 
that it is a type of network filesystem.  As such, it is not suitable for 
the mbx format; shared access and locking can not work on such systems and 
you *will* have mailbox corruption.

Put another way: the mbx format *MUST* be on a local filesystem that 
supports flock() and instantaneous (and atomic) update of data and inode 
data across all sharers.

The corrolary to this is that you can't use mbx on your GFS filesystem; 
and as you have noticed corruption will result.

Some "experts" will claim that the "fix" for this problem is to use 
fcntl() locking.  It's a long story why not; so trust me, you do not want 
to do this.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw