Subject: MH again

From: Dag Nygren <[email-masked]>
Date: Thu, 30 Aug 2001 20:26:31 +0300

Hi,

I am trying to get imapd to understand INBOX
as MH inbox and it seems to work pretty well
now.

But there is some parts of the code that would need
some clarification:

1. mh_ping tries to get messages from sysinbox() ??
  This must surely be the task of MH itself,otherwise
  it will bypass all procmail etc. stuff.
  Mark ?

2. Is there a good reason for not using setcasecmp
  instead of the comlicated construct in the current
  code ?

As it seems that we need both MH and Mailbox support
the code would try to be transparent in the way that
INBOX will mean the MH inbox if there exists a $HOME/.mh_profile
file, otherwise it would work the way it did before.
The only change will be in mh.c.
Could Mark imagine accepting the patch in the
main code when I get it done ?

BRGDS

-- 
Dag Nygren                               email: [email-masked]
Oy Espoon NewTech Ab                     phone: +358 9 8024910
Träsktorpet 3                              fax: +358 9 8024916
02360 ESBO                              Mobile: +358 400 426312
FINLAND

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Thu, 30 Aug 2001 10:48:36 -0700 (PDT)

>   it will bypass all procmail etc. stuff.

I don't know what you mean by "bypass all procmail, etc. stuff".  Procmail is
run at mail delivery time, not by the MUA (including c-client).

Presumably, an IMAP user wants to see new mail when it comes in.  That's why
mh_ping() does that.  You can always open #mh/inbox instead of #mhinbox if you
don't want mh_ping() to do that; this behavior is a special behavior of the
#mhinbox name.

>   instead of the comlicated construct in the current

Presumably you mean strcasecmp(), and the answer is yes.  Not all systems have
strcasecmp().  Actually, the c-client version is compare_cstring() but not all
existing code has been hacked to use it yet.

> main code when I get it done ?

No.  It was once that way, and it won't be changed back.  There is a reason
why #mhinbox is used for that purpose instead of using INBOX.

MH format does not support full c-client semantics.  If INBOX was automatic
for MH, then there's a nasty timebomb waiting for anyone who tried mh years
and years ago, decided they didn't like it, but didn't go and delete the
.mh_profile file.  Their mail will get converted to MH, but a number of things
(such as sticky flags, UIDs, etc.) will stop working.

This actually happened back when INBOX was automatic for MH, and it bit
people.  Let's just say that the experience, after a Very High Up individual
at my workplace was bit, was sufficiently unpleasant for me that it will be a
cold day in hell before I ever let INBOX be automatic for mh.

The relatively few number of people who really want MH must use #mhinbox
instead of INBOX.  Most people don't, since so many things won't work
afterwards.

---

From: Dag Nygren <[email-masked]>
Date: Thu, 30 Aug 2001 21:11:30 +0300

> don't want mh_ping() to do that; this behavior is a special behavior of the

But MH will deliver my mail to #mh/inbox automatically and that's where
I can find the newly arrived mail's, not in sysinbox.
If imapd would do that "delivery" part for me, it would just move it there
and bypass the additional processing.

> existing code has been hacked to use it yet.

OK.

> (such as sticky flags, UIDs, etc.) will stop working.

That's true.

> cold day in hell before I ever let INBOX be automatic for mh.

OK, I accept that. What about a compilation option then ?

> instead of INBOX.  Most people don't, since so many things won't work

The problem is that not very many clients give you that option....
And additionally if you want to use the POP-part of imapd it will
always access INBOX.

BRGDS

Dag

---

From: Vadim Zeitlin <[email-masked]>
Date: Thu, 30 Aug 2001 20:20:22 +0200 (CET)

On Thu, 30 Aug 2001 10:48:36 -0700 (PDT) Mark Crispin <[email-masked]> wrote:

MC> Their mail will get converted to MH, but a number of things
MC> (such as sticky flags, UIDs, etc.) will stop working.

 I wanted to ask this question since a long time ago, so please excuse me
if it's slightly off topic to the current discussion: just why is there no
support for these features in MH? How is it more difficult to support flags
for MH than for MBOX, for example? MH is IMHO much better format than MBOX,
but it is completely unusable in c-client :-( Why not enhance MH driver to
store the flags in the MH files?

 Regards,
VZ

---

From: Mark Crispin <[email-masked]>
Date: Thu, 30 Aug 2001 11:20:59 -0700 (PDT)

> I can find the newly arrived mail's, not in sysinbox.

If newly arrived messages are delivered to #mh/inbox automatically, then there
will never be mail in sysinbox.

If newly arrived messages are in sysinbox, then that is where they were
delivered.  It is then the job of an MUA, such as mh or imapd, to move the
messages from sysinbox to #mh/inbox.

Procmail is run by mail delivery (sendmail, etc.), not by the MUA.

---

From: Mark Crispin <[email-masked]>
Date: Thu, 30 Aug 2001 11:23:57 -0700 (PDT)

> store the flags in the MH files?

There is no usable standard for storing c-client flags in mh, and more
importantly c-client can not unilaterally create one that will not be trashed
by a non-c-client mh application.

I would certainly dispute the claim that mh is a "much better format than
mbox."  Have you done performance testing with mh?  It's very slow!

---

From: Dag Nygren <[email-masked]>
Date: Thu, 30 Aug 2001 21:39:54 +0300

> mbox."  Have you done performance testing with mh?  It's very slow!

It is indeed, but it has some advantages which my users like,
not in any particular order:

1. You can grep your mails searching for something
2. You can organize your mail in hierarchical structures
  (Probably the most important feature)
3. You can use the excellent exmh MUA when working locally

I/they will change format anyday, when someone offers one
with the same features.

Dag

---

From: Vadim Zeitlin <[email-masked]>
Date: Thu, 30 Aug 2001 20:40:21 +0200 (CET)

On Thu, 30 Aug 2001 11:23:57 -0700 (PDT) Mark Crispin <[email-masked]> wrote:

MC> There is no usable standard for storing c-client flags in mh, 

 Why not use someone else's standard? There are a few quite nice MUAs for
Unix which use MH as their (primary) mailbox format. E.g. I was using
XFMail for a long time and it worked very nicely with MH (including flags,
very fast access time &c). And all it did was to add "XFMStatus:" header to
MH files.

 I know that Sylpheed also uses MH and I guess it supports flags for it as
well, although I don't know how do they do it.

MC> and more
MC> importantly c-client can not unilaterally create one that will not be trashed
MC> by a non-c-client mh application.

 Using "X-CC-Flags:" header should be sufficiently safe, no? Although, of
course, being compatible with at least one other popular MH mailer would be
nice.

MC> I would certainly dispute the claim that mh is a "much better format than
MC> mbox."  Have you done performance testing with mh?  It's very slow!

 Sorry, is it a joke? My main problem with c-client MBOXes is that opening
them is *horribly* slow. XFMail was blazingly fast even compared to
c-client MBX files (and I used both on the same computer, under the same OS
and on the same partition) and fairly sized MBOXes are just unusable with
c-client. Did you try opening an MBOX with ~10000 messages? It takes
_ages_! And in XFMail I had no troubles at all with such folders (except
that the stupid thing has a hardcoded limit of 12000 messages per mailbox).

 MH is not slow by itself, although c-client driver for it may be. However
I'd like to know how can you open MBOX folder faster than MH one
considering that you have to scan the entire file? Of course, if c-client
maintained some sort of summary files for MBOXes it could be different (I'd
really like to have time to do this one of these days...) but right now
there is no local mailbox format in c-client which is usable for folders
containing more than, let's be generous, 3000 messages.

 Regards,
VZ

---

From: Nerijus Baliunas <[email-masked]>
Date: Fri, 31 Aug 2001 17:50:40 +0200 (EET)

On Thu, 30 Aug 2001 20:40:21 +0200 (CET) Vadim Zeitlin <[email-masked]> wrote:

VZ> MC> I would certainly dispute the claim that mh is a "much better format than
VZ> MC> mbox."  Have you done performance testing with mh?  It's very slow!
VZ> 
VZ>  Sorry, is it a joke? My main problem with c-client MBOXes is that opening
VZ> them is *horribly* slow. XFMail was blazingly fast even compared to
VZ> c-client MBX files (and I used both on the same computer, under the same OS
VZ> and on the same partition) and fairly sized MBOXes are just unusable with
VZ> c-client. Did you try opening an MBOX with ~10000 messages? It takes
VZ> _ages_! And in XFMail I had no troubles at all with such folders (except
VZ> that the stupid thing has a hardcoded limit of 12000 messages per mailbox).
VZ> 
VZ>  MH is not slow by itself, although c-client driver for it may be. However
VZ> I'd like to know how can you open MBOX folder faster than MH one
VZ> considering that you have to scan the entire file? Of course, if c-client
VZ> maintained some sort of summary files for MBOXes it could be different (I'd
VZ> really like to have time to do this one of these days...) but right now
VZ> there is no local mailbox format in c-client which is usable for folders
VZ> containing more than, let's be generous, 3000 messages.

Hello,

Have you read http://www.courier-mta.org/mbox-vs-maildir/ ?
There are some benchmarks, and it seems Mark's claims about maildir
slowness are not true anymore (and maybe he accepts some patch for
maildir support in c-client one of these days, although I doubt it...).

Regards,
Nerijus

---

From: Mark Crispin <[email-masked]>
Date: Fri, 31 Aug 2001 09:34:59 -0700 (PDT)

> Have you read http://www.courier-mta.org/mbox-vs-maildir/ ?

Hardly an unbiased source.

> maildir support in c-client one of these days, although I doubt it...).

There are lies, damn lies, and benchmarks.  In this case, what the benchmark
chooses to measure: what some MUA which was designed for maildir needs.

My benchmarks use what is needed for IMAP.  That is the only thing that is
important to me.

The simple fact is that it is impossible to do a basic IMAP operation (load
mailbox message metadata) without opening every file in a maildir (or MH)
mailbox and reading every byte.

Not all systems are like Linux with fast file opens.  On most UNIX systems,
particularly those with robust filesystems, open() is a very expensive call.
We're talking about each open() taking milliseconds.

There is also additional disk seek time because each message is likely to be
on a different spot on the disk (multiple files are naturally more likely to
present a fragmented database over single files).

True, in mbox format you also have to read every byte of the file.  But, since
it is one file, it is more likely to be contiguous on the disk, and there are
no filesystem operations that need to be done.  It is quite a bit faster than
maildir or mh.

But mbox format has not been the preferred format for over 10 years.  It is
the default format in UW imapd because it is the default format on UNIX.

In the preferred mbx format (a different format than mbox), only the metadata
needs to be read and not message data.  It is much faster than mbox format,
and allows shared access in ways that maildir can only fantasize about.  In a
proposed follow-on format to mbx, the metadata will be read even faster.

The only way to maintain metadata in a one-file/one-message format is to have
an flat index file, such as used by Cyrus, which contains all of the data used
by IMAP.  In such a mechanism, the actual message files are only ever used to
get message text data, never to calculate metadata.  This in turn means that
most message files are never looked at or opened.

You can't do an index file in mh, because other mh software doesn't use it.
The index file mechanism depends upon all software using it, in order to avoid
having to look at the message data to ensure that the metadata is up to date.

You can't do an index file in maildir either, because maildir's entire purpose
is to eliminate locking.  An index file would sabotage maildir's sole benefit.
Nor can you squeeze the index information into the filename.

By the way, maildir's trick of putting a limited amount of metadata into the
filename also is a performance issue, because most filesystems serialize
directory operations.  You can't generalize from Linux to other UNIX
platforms.  Nor can the filename hold all the needed metadata.

There are three or four different maildir drivers for c-client written by
maildir fans since I don't provide one.  I've tested them all.  None of them
work well.  All of them demonstrate the (slow) performance results that I
report.  Most importantly, all of them have been the cause of problem reports
to me.

Given that the maildir fan community can't write a working c-client maildir
driver with reasonable performance, there is no reason to believe that a
maildir skeptic such as myself would do any better.

---

From: Nerijus Baliunas <[email-masked]>
Date: Sat, 1 Sep 2001 03:24:37 +0200 (EET)

On Fri, 31 Aug 2001 09:34:59 -0700 (PDT) Mark Crispin <[email-masked]> wrote:

MC> On Fri, 31 Aug 2001 17:50:40 +0200 (EET), Nerijus Baliunas wrote:
MC> > Have you read http://www.courier-mta.org/mbox-vs-maildir/ ?
MC> 
MC> Hardly an unbiased source.

Maybe, but your claims at http://www.washington.edu/imap/documentation/formats.txt.html:

" The Maildir format used by qmail has all of the performance
 disadvantages of mh noted above, with the additional problem that the
 files are renamed in order to change their status so you end up having
 to rescan the directory frequently the current names (particularly in
 a shared mailbox scenario).  It doesn't scale, and it represents a
 support nightmare; it will therefore never be supported in the
 official distribution.  Maildir support code for c-client is available
 from third parties; but, if you use it, it is entirely at your own
 risk (read: don't complain about how poorly it performs or bugs)."

What does it mean "it doesn't scale, and it represents a support nightmare"?
Is it an unbiased? Do you have any benchmarks?

MC> > There are some benchmarks, and it seems Mark's claims about maildir
MC> > slowness are not true anymore (and maybe he accepts some patch for
MC> > maildir support in c-client one of these days, although I doubt it...).
MC> 
MC> There are lies, damn lies, and benchmarks.  In this case, what the benchmark
MC> chooses to measure: what some MUA which was designed for maildir needs.

No, it says "this benchmark takes both measurements into account":

"It should be noted that some of these numbers reflect the overall system
performance that may differ from the apparent performance seen by a mail
client. When running the benchmark, the UW-IMAP server did not actually take
much longer to open a 2,000 message folder than Courier-IMAP -- it postponed
the mbox file rewrite until the folder was closed. However, this benchmark
takes both measurements into account. From the user's standpoint, some of the
delay in opening a large folder is postponed until the folder is closed. This
results in a slightly faster response when opening a folder, but from the
system's viewpoint the load's the same. This is why both measurements are
important. Whether you take the load up front, or spread it around, the grand
total is still the same.  The decision to postpone rewriting the mbox file can
result in some savings in time (mostly by consolidating multiple rewrites into
one). However, there's also a down side to this approach. An IMAP server can
always be killed by an abnormal system event, for example. When that happens
to the UW-IMAP server, any unsaved changes to the folder will be lost."

Mark, the point is that maildir may not be better than mbox, but maybe it's time
to recognize (and support) it?

MC> My benchmarks use what is needed for IMAP.  That is the only thing that is
MC> important to me.
MC> 
MC> The simple fact is that it is impossible to do a basic IMAP operation (load
MC> mailbox message metadata) without opening every file in a maildir (or MH)
MC> mailbox and reading every byte.
MC> 
MC> Not all systems are like Linux with fast file opens.  On most UNIX systems,
MC> particularly those with robust filesystems, open() is a very expensive call.
MC> We're talking about each open() taking milliseconds.

Do you have any benchmarks? What are these "robust filesystems"?

MC> There is also additional disk seek time because each message is likely to be
MC> on a different spot on the disk (multiple files are naturally more likely to
MC> present a fragmented database over single files).

I think it's time for independent benchmarks.

MC> True, in mbox format you also have to read every byte of the file.  But, since
MC> it is one file, it is more likely to be contiguous on the disk, and there are

Why should it be contiguous? If it's POP3, then probably yes, because client
just downloads and deletes messages, but if it's IMAP, then a client can have
messages even a year or more old.

MC> no filesystem operations that need to be done.  It is quite a bit faster than
MC> maildir or mh.
MC> 
MC> But mbox format has not been the preferred format for over 10 years.  It is
MC> the default format in UW imapd because it is the default format on UNIX.
MC> 
MC> In the preferred mbx format (a different format than mbox), only the metadata
MC> needs to be read and not message data.  It is much faster than mbox format,
MC> and allows shared access in ways that maildir can only fantasize about.  In a

Sorry, I don't know a better format for shared access than maildir. What do you
mean exactly? And what does it mean "It is much faster than mbox format"?
Do you have any benchmarks?

MC> You can't do an index file in mh, because other mh software doesn't use it.

We don't talk now about other software, we talk about c-client's mh driver.
If uw-imapd could support message flags with mh driver, what's a problem
with other software here (Vadim proposed X-CC-Flags header)?

MC> You can't do an index file in maildir either, because maildir's entire purpose
MC> is to eliminate locking.  An index file would sabotage maildir's sole benefit.
MC> Nor can you squeeze the index information into the filename.

I'm not qualified to talk about architectural (dis)advantages of one or other format,
I just want you to finally accept and maybe do some benchmarks with maildir.

MC> By the way, maildir's trick of putting a limited amount of metadata into the
MC> filename also is a performance issue, because most filesystems serialize
MC> directory operations.  You can't generalize from Linux to other UNIX
MC> platforms.  Nor can the filename hold all the needed metadata.

As in http://www.courier-mta.org/mbox-vs-maildir/:

"Documentation included with the University of Washington IMAP server
(UW-IMAP) states that maildirs have many "performance disadvantages" and that
the maildir format "doesn't scale." Furthermore, maildirs are supposedly
vulnerable to "filesystem trashing" due to multiple "open() and stat()" calls,
because "just about every filesystem in existence serializes" file creation
and access.  The document makes a conclusion
that this results in performance degradation for "moderately sized" mailboxes
of about 2,000 messages.
Painting "just about" every filesystem in existence with the same brush,
and assuming that every filesystem works pretty much in the same way, is very
misleading. Many contemporary high performance filesystem are designed
explicitly for parallel access.  For example, consider the SGI XFS
filesystem:
The free space and inodes within each AG are managed independently and
  in parallel so multiple processes can allocate free space throughout the
  file system simultaneously."

So is SGI XFS filesystem a Linux one (I know it's being ported to Linux BTW)?

MC> There are three or four different maildir drivers for c-client written by
MC> maildir fans since I don't provide one.  I've tested them all.  None of them
MC> work well.  All of them demonstrate the (slow) performance results that I
MC> report.  Most importantly, all of them have been the cause of problem reports
MC> to me.
MC> 
MC> Given that the maildir fan community can't write a working c-client maildir
MC> driver with reasonable performance, there is no reason to believe that a
MC> maildir skeptic such as myself would do any better.

I understand that. But do you understand that all your claims about maildir slowness
are without (almost) any proof? Do you have any benchmarks like in
http://www.courier-mta.org/mbox-vs-maildir/ ? For example, you say
"most filesystems serialize directory operations". Have you done any benchmarks?

Regards,
Nerijus

---

From: David Carter <[email-masked]>
Date: Sat, 1 Sep 2001 15:32:13 +0100 (BST)

On Sat, 1 Sep 2001, Nerijus Baliunas wrote:

> Mark, the point is that maildir may not be better than mbox, but maybe
> it's time to recognize (and support) it?

Can't you use the Courier IMAP server if you want to use maildir? I don't
see why Mark should have to support a mailbox format that he is clearly
has grave doubts about.

Maildir can be an excellent choice if all of your user agents are using
POP or the subset of the IMAP protocol which corresponds to POP with
access to multiple mail folders. This doesn't automatically make maildir a
good choice for user agents which use more advanced features in the IMAP
command set. I imagine that maildir would have a particularly hard time
with user agents which can be configured to sort or thread messages.

-- 
David Carter                             Email: [email-masked]
University Computing Service,            Phone: (01223) 334502
New Museums Site, Pembroke Street,       Fax:   (01223) 334679
Cambridge UK. CB2 3QG.

---

From: Nerijus Baliunas <[email-masked]>
Date: Sat, 1 Sep 2001 16:43:56 +0200 (EET)

On Sat, 1 Sep 2001 15:32:13 +0100 (BST) David Carter <[email-masked]> wrote:

DC> On Sat, 1 Sep 2001, Nerijus Baliunas wrote:
DC> 
DC> > Mark, the point is that maildir may not be better than mbox, but maybe
DC> > it's time to recognize (and support) it?
DC> 
DC> Can't you use the Courier IMAP server if you want to use maildir? I don't
DC> see why Mark should have to support a mailbox format that he is clearly
DC> has grave doubts about.

I do use Courier IMAP server. But it would be nice to have support for maildirs
in c-client, as c-client is used in client applications.

Regards,
Nerijus

---

From: Vadim Zeitlin <[email-masked]>
Date: Sat, 1 Sep 2001 23:39:12 +0200 (Romance Daylight Time)

Because it would be useful in MUAs using c-client.

DC> I imagine that maildir would have a particularly hard time
DC> with user agents which can be configured to sort or thread messages.

 Could you please elaborate on this? I'm having _very_ hard time
implementing efficient sorting/threading right now and I would like very
much to know how can it be optimized (on the client or server side).

 Thank you,
VZ

---

From: David Carter <[email-masked]>
Date: Mon, 3 Sep 2001 10:32:33 +0100 (BST)

On Sat, 1 Sep 2001, Vadim Zeitlin wrote:

> implementing efficient sorting/threading right now and I would like
> very much to know how can it be optimized (on the client or server
> side).

Well, a quick (and not very scientific) experiment indicates that the SORT
and THREAD extensions in the UW server are an order of magnitude faster
than the same extensions in the Courier server on my workstation. This
appears to be true regardless of whether I use MBX or Unix mail folders.

At the mail folder access level, this is comparing n lots of "lseek() +
read()" against n lots of "open() + read() + close()", where n is the
number of messages in the folder. I was using rather large values of n,
around twenty thousand random small messages. In retrospect, this might
have been rather unfair as ext2 doesn't cope well with large directories.

However, its easy enough to set up this sort of benchmark for yourself.