Subject: I'm done with UWIMAP, Mark doesn't follow the spirit of open source

From: Marc MERLIN <[email-masked]>
Date: Sat, 17 Aug 2002 14:01:34 -0700

In a nutshell, after rejecting patches that I sent him a while ago (which is
ok, mind you), and  refusing to make any mention of my  patches and/or put a
link in  his documentation and/or  the web site (which  is rude, if  you ask
me), Mark silently  re-implemented one of my patches, without  giving me any
credit whatsoever, and ignoring an inquiry Email I sent him about it.

Mark, you are very allowed to do that, your patch was a re-implementation of
mine, you  merely took the  idea and re-wrote  it a different  (and probably
better way).
But:
- refusing to  acknowledge patches that  you don't  want to include  is rude
  (and  I've been  told  that you've  refused  perfectly acceptable  feature
  patches from others than me too)
- taking the idea of  one of the patches, and silently  including it in your
  code after  having said that you  aren't interested in it  and refusing to
  even link to it, is demonstrating very poor ethics.

I'm done  with uwimap, the security  of my system  will only be better  as a
result. And BTW, your  mailbox comparison essay is  misleading. I've run MBX
on big  systems (with  autodetection of  MBX users  and dynamic  changing of
home, as written in  one of the patches I sent you and  you ignored), and it
doesn't scale.
Maildir _is_ the way to go, and I'm going to courier imap with no regrets.
(even   linux  with   a  slow   ext2  FS   can  handle   multiple  directory
add/scan/delete requests  a lot faster that  uwimap scans and deals  with an
MBX file)

Marc

This is the mail that I sent to Mark and that he never acknowledged:
(I gave him a chance to explain himself)

----- Forwarded message from Marc MERLIN -----

From: Marc MERLIN
To: [email-masked]
Subject: restrictBox a re-implementation of my patch
X-Sysadmin: BOFH
X-URL: http://marc.merlins.org/
X-Operating-System: Proudly running Linux 2.4.14-lvm1.0.1rc4-ext3-0.9.15-servers11-up/Debian woody

Please have a look at
http://marc.merlins.org/linux/uwimap/files/imap-2001aRC2.blockfswandering.patch
from
http://marc.merlins.org/linux/uwimap/

When I sent you the patches, you answered:
----------------------------------------------------------------------------
From: Mark Crispin <[email-masked]>
Subject: Re: [offlist] Several patches for uwimap, please review
To: Marc MERLIN
Date: Mon, 5 Nov 2001 18:25:21 -0800 (PST)

Thank you for your suggested patches.

Although these patches would undoubtably be useful for some people, I think
that it would be better if you distribute them as third-party patches
instead
of me.
----------------------------------------------------------------------------

and then, what do I find in a later release of uwimap?
restrictBox

Err, this more or less a re-implementation of my idea and my patch (yours is
most likely better,  but I was hoping/expecting that you'd  improve my patch
as you included it)

It would have been  nice to let me know you had included  that and to credit
me for the idea and original patch.

Marc

----- End forwarded message -----

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/   |   Finger [email-masked] for PGP key

---

From: Feiyu Xie <[email-masked]>
Date: Mon, 19 Aug 2002 11:01:03 -0400

We are using courier imap with Maildir as well, for security, speed, and
*cough* NFS. I'd consider the Cyrus (which is basicly a Maildir with
meta-data files for each user) if we don't use NFS for mail storage... Marc,
could you tell us more about what kind of comparison/experience you had with
MBX vs. Maildir? This will definitely help people out there... 

thanks,
--Feiyu

-----Original Message-----
From: Marc MERLIN [mailto:[email-masked]]
Sent: Saturday, August 17, 2002 5:02 PM
To: [email-masked]
Cc: [email-masked]
Subject: I'm done with UWIMAP, Mark doesn't follow the spirit of open source

Marc

----- Forwarded message from Marc MERLIN -----

From: Marc MERLIN
To: [email-masked]
Subject: restrictBox a re-implementation of my patch
X-Sysadmin: BOFH
X-URL: http://marc.merlins.org/
X-Operating-System: Proudly running Linux
2.4.14-lvm1.0.1rc4-ext3-0.9.15-servers11-up/Debian woody

Please have a look at
http://marc.merlins.org/linux/uwimap/files/imap-2001aRC2.blockfswandering.pa
tch
from
http://marc.merlins.org/linux/uwimap/

Marc

----- End forwarded message -----

-- 
"A mouse is a device used to point at the xterm you want to type in" -
A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet
cooking
Home page: http://marc.merlins.org/   |   Finger [email-masked] for PGP
key

---

From: Marc MERLIN <[email-masked]>
Date: Mon, 19 Aug 2002 18:42:50 -0700

On Mon, Aug 19, 2002 at 05:10:57PM -0400, Alexy Khrabrov wrote:
> Marc -- apparently, Mark is very much overloaded by
> supporting UW imap singlei-handedly for so many platforms
> and so many years.  In my brief time spent searching archives,

I understand that. I maintain a bunch of things myself and have more than a
full time job.
That said, I try to at least answer my personal mail.

> scary, scary, scary docs:).  Minimalism is the only way to
> make it possible, so you're probably overreacting in your
> assessment.  Mark might have also considered patches like
> yours before, and forcing people to explain the wrongs they
> didn't commit usually are sent to /dev/null, especially if
> there's a lot of real problems and it looks like a non-issue
> to the other side.  This is not to say patches should not be
> acknowledged, but rather it stems from the feature of the UW imap

Mmmh, he just answered he wasn't interested in integrating any of the
patches and that I should maintain them on my side and distribute them
myself.
That's his right, mind you.

However, never answering my request to somehow link to the patches or
mention them somewhere is not being very helpful.
(note that  this didn't happen  to just  me. Jauder Ho, from  Transmeta back
then, also wrote patches, some of which mine were based on, and met the same
result than me  when he tried to  submit them (with nice  #define to exclude
all the code by default))

> If there's a community of mail administrators crucially
> dependent on UW imap and knowing it to the level of patching, which
> seems to be the case, the proper thing to do is to form an Apache
> like group for community maintained code, put it onto Sourceforge,
(...)

Sounds like a reasonable idea.

Note that I only  became unhappy after Emailing him and  giving him a chance
to explain  himself, or say  something like "sorry,  I never really  had the
time to look  at your patch, and just ended  up rewriting it independantly",
or "oh yeah, I meant to write you about this", or something...
But no, nothing, hence the default answer of "I don't care, go away"

Anyway, doesn't matter, I moved on

Marc
(note that I contribute to many open source projects, this is not just the
one patch I happened to write and sent to my first maintainer)
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking 
Home page: http://marc.merlins.org/   |   Finger [email-masked] for PGP key

---

From: Marc MERLIN <[email-masked]>
Date: Mon, 19 Aug 2002 18:54:40 -0700

> MBX vs. Maildir? This will definitely help people out there... 

Well, I had  MBX break down in a reasonably  sized organization (200 people)
with mailboxes going up to 100MB with 5-10k messages. Not small, but not out
of the ordinary either. You see that in ms exchange every day
uwimap at  that stage became quite  slow, and by design,  expunging messages
from a 5-10k mesg mailbox is excruciatingly slow.

With maildir+ you  only have to do one  readdir to get a pointer  to all the
messages, and the size of each of them.
Deleting one message in the middle is one unlink.
You can get all the new messages or all the read message with one readdir

Mark's paper on directory operations bogging down and being slowed down and
serialized by the kernel is just outdated.
FS that have directories in hashes or btrees can be looked up quite quickly,
and some can apparently support concurrent operations.

Unfortunately, I  haven't had the time  to setup a  system where I can  do a
real MBX vs Maildir comparison (it'd take a  lot of time to do it fairly for
both sides, and you'd have to redo it with different filesystems)

My other  big beef  with MBX  is that  pretty much  no mail  client directly
supports it without going through an  imap server. With Maildir, I can point
mutt at  my Maildir  directory, and just  navigate it, or  just use  grep or
grepmail to find messages.
(granted, it's  not MBX's  fault, but  if there  is no  tool support  for it
outside of c-client, it's not very useful to me)

Marc

---

From: "Jeff Breitner" <[email-masked]>
Date: Tue, 20 Aug 2002 09:32:59 -0400

> Well, I had  MBX break down in a reasonably  sized 
> organization (200 people) with mailboxes going up to 100MB 
> with 5-10k messages. Not small, but not out of the ordinary 
> either. You see that in ms exchange every day uwimap at  that 
> stage became quite  slow, and by design,  expunging messages 

This has been a recuring theme, especially with half of the Outlook
world sending klez-related attachments to everyone Outlook as ever seen
in its life.

However, I disagree that expunging is *that* terribly slow.  This
account, which subscribes to no less than 10 mailing lists routinely
accesses mbx files that have hundreds of mail messages.  It isn't
lickety-split like an Exchange server, but it's acceptable.

> 
> With maildir+ you  only have to do one  readdir to get a 
> pointer  to all the messages, and the size of each of them. 
> Deleting one message in the middle is one unlink. You can get 
> all the new messages or all the read message with one readdir
> 

I agree with you on this point, and it's helpful for killing off the
occasional 400 megabyte video file attachment sent by a well-meaning
individual.  Nothing like circumventing everything, going into the file
system and simply finding the offending file and killing it on sight.
MBX simply isn't that nice to deal with.

> My other  big beef  with MBX  is that  pretty much  no mail  
> client directly supports it without going through an  imap 
> server. With Maildir, I can point mutt at  my Maildir  
> directory, and just  navigate it, or  just use  grep or 
> grepmail to find messages. (granted, it's  not MBX's  fault, 
> but  if there  is no  tool support  for it outside of 
> c-client, it's not very useful to me)

This may be a problem for people like you and me, but for the majority
of people who read mail, their only exposure to the guts is the
occasional system rebuild where they have to hit
Tools->Accounts->default-mail in their Outlook/Outlook Express.  They
get access to their mail *only* through the services of a server.

I've never experimented with using MySQL databases as the mail spool,
but I have to believe that if one is going to make a move from mbx, that
might be the preferred direction.  I would be interested in knowing how
that compares to mbx and maildir.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 20 Aug 2002 13:31:46 -0700 (Pacific Daylight Time)

Sometimes, when people do not answer mail, it is because they are on
vacation.  I just returned from a one-month email-free vacation in Alaska.

I am not going to refute Marc Merlin's message point-by-point, except to
point out that the IMAP and c-client mailing lists archives shows that his
patch was merely one of a long series of patches which have the notion of
restricting a user's file naming access above and beyond what the
filesystem provides.  The first such patch was written by me over 10 years
ago, and I have written several others for people who asked me nicely.
I've also received any number of such patches from other people, including
Merlin.

The new restrictBox functionality is the culmination of this 10+ years
worth of study of what various sites do, and a distillation of it into
something that is reasonably simple.

I spent a considerable amount of time hand-holding Merlin through matters
that he could have found out for himself if he had read the documentation
and various books available.  If this is the way that he chooses to repay
my efforts, we are both better off with him using a different product.

Scalability is something which is a good deal more complex that he
presents.  What scales well on a Linux based server to Outlook clients is
not necessarility the same as what scales well to a Solaris based server
to Pine or Mulberry clients.

I recommend that if a person really wants the one-file per message type
format that they use Cyrus instead of maildir.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

---

From: Marc MERLIN <[email-masked]>
Date: Wed, 21 Aug 2002 17:55:26 -0700

> vacation.  I just returned from a one-month email-free vacation in Alaska.
 
Ok, that would be why you didn't answer that mail. Understand that after not
getting an answer for two weeks, it's unfortunately become natural to decide
that your mail was ignored.
Note  too that  when  you leave  that  long,  it is  customary  to leave  an
autoresponder.

That does not  explain why you have  yet to answer my mail  asking nicely if
you  would  care  to  add  a  pointer  to my  patches  to  a  file  in  your
distribution,  or  your web  page  (that  was  after  you said  you  weren't
interested in including them, which I'll  repeat is ok, I never got offended
by that)

You brushed me off, and we both know that.
 
> filesystem provides.  

Uh?
1) 4 patches, split and documented separately, and two of them were almost
   one liners (actually, I think you also ended up re-implementing the one
   that lets you hide files that start with a dot).

2) Only one patch, the "anti FS wandering" patch, restricted access to the
   FS. That's what you re-implemented as restrictBox

So no, it's not "a long series of patches which have the notion of of
restricting a users' file naming access..."

> The first such patch was written by me over 10 years

That's fine.
1) it wasn't in the distribution, so not very useful
2) I didn't ask you to write a feature I needed if I could have coded it and
   contributed it back
3) you made no  mention of that when I Emailed  you. Your answer amounted to
   "thank  you for  your  submission,  I'm not  interested  in including  or
   bundling these  patches, and I'm  not interesting  in linking to  them or
   mentioning them in my documentation"

The last two people I talked to about submitting patches to you told me both
"good luck". Coincidence?

> I've also received any number of such patches from other people, including
 
Great, then I guess that it only took a  long time for any of them to go in,
that can  happen. Mentioning this when  people end up  writing independently
the same thing because it's not in the distribution would have been helpful
 
> something that is reasonably simple.

And that's  fine. I did not  claim ownership of that  idea or said  that you
stole my code and put your  name on it. For that matter, your implementation
is probably better.
However, the lack  of response and info  when you are asked  about this sure
didn't lead anyone to believe that several people had written the same patch
and that you were  hoping to include the feature at some  point (it was sent
to the list too,  and no one else seemed to know because  no one answered to
say the feature was on its way or that they had written something similar)

> my efforts, we are both better off with him using a different product.

Uh?
You are  confused. I Emailed you once  a personal question, maybe  2-3 years
ago, on  whether imapd was  able to  jump from header  to header in  an mbox
file. You nicely answered, that no, it was not the case.

After  that,  I have  Emailed  the  list a  few  times  to discuss  a  patch
implementation  or announce  one, I  reported a  problem with  syslogging in
imap-2001.BETA.SNAP-0105251616  which was  fixed in  2001aRC2, and  the only
question I asked that was in the  documentation (well, it wasn't when I read
it, but the documentation had been  updated since my last read) was building
with SSLTYPE=nopwd

So, yes,  shoot me, I asked  one question that was  in the docs, but  I sent
more patches  and answered questions than  I wasted people's time  with that
one instance.
So since you are calling this "hand holding", I see now that you are full of
shit (forgive  the expression, English isn't  my native tongue, and  I don't
know of a more polite equivalent), it's clear that I don't need to deal with
you further.

Do reconsider your dealing with people  who offer to contribute code to your
project. I've contributed  to more  than 20,  and you were  the worst  by an
other of  magnitude: even for a  linux kernel patch, people  answered me and
told me why they  did or did not like the patch and  whether it had a chance
to go in or not and why.

Have a nice day.

Marc

---

From: Mark Crispin <[email-masked]>
Date: Wed, 21 Aug 2002 18:29:42 -0700 (Pacific Daylight Time)

> Note  too that  when  you leave  that  long,  it is  customary  to leave  an

Not everybody cares to publish their vacation plans for everybody in the
world (including potential burglars) to see.

> interested in including them, which I'll  repeat is ok, I never got offended

By policy, we do not advertise any third-party patches.  We can not accept
such liability.  There are many third-party patches, and it is the
responsibility of their authors

If you feel "brushed-off", that is your problem.

> restricting a users' file naming access..."

Yes it is.  Your patch is merely one of perhaps dozens such patches by
many people (myself included), with similar purpose, which have been
posted on the lists or newsgroups, emailed to me as suggestions, or even
written by me for someone.  I even posted a "if you want to do something
like this, here is the type of this you should do in your patch" message.

I am sorry if you felt that you had an original idea that nobody had
thought of before.  You didn't.

Nor were you the most recent person in the series.  You were just one of
many.

> 1) it wasn't in the distribution, so not very useful

I did not want to support any of those patches, and I do not incorporate
*any* patches which I do not want to support.

The new code in imap-2002 is something that I decided can be supported.

>    mentioning them in my documentation"

I did not incorporate your patch.  I wrote new code.  The fact that it
does something similar is covered in the free-fork license:

 (5) the University of Washington may make modifications to the
 Distribution that are substantially similar to modified versions of
 the Distribution, and may make, use, sell, copy, distribute, publicly
 display, and perform such modifications, including making such
 modifications available under this or other licenses, without
 obligation or restriction;

Even when I reject a patch, I keep notes about the type of functionality
that people write patches to implement.  I use these notes (or "suggestion
box" if you prefer) to guide me when I implement new functionalities.

As it so happens, I recently completed a security project in imap-2002,
and thus all security-related suggestions were considered during that
development cycle.

> "good luck". Coincidence?

I accept only a small number of non-bugfix related patches.

Some people would call that "quality control."

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Wed, 21 Aug 2002 19:00:06 -0700 (Pacific Daylight Time)

On Wed, 21 Aug 2002, Alexy Khrabrov wrote:
> Actually, I'd like to see what kind of control I can have
> over the list of mailboxes presented to the client.  Is
> this patch hiding the files starting with a dot implemented?

This is controlled by the hideDotFiles variable in env_unix.c

I do not recommend doing this, because some clients and/or users may
actually want to see the dot files.  For example, Pine wants to see the
dot files, and has its own feature to control hiding.  Years ago, my
co-workers on the Pine team got very angry with me when an earlier version
hid dot files from Pine.

> Or, better still, showing/hiding only mailboxes whose names
> match a regexp?

That is something you would have to write yourself.  The routine that you
would need to modify is dummy_list_work() in dummy.c, perhaps by adding
code to the "if"  instruction that checks for HIDEDOTFILES and
MXINDEXNAME.

> The problem is, some mailers, notoriously KDE's Kmail, create
> a .<name>.index file for each <name> file in your Mail directory.
> Then it all shows up on top of a sorted list of folders in any
> typical imap client.

Unfortunately, this is one of a wide variety of such things which people
may want (or *not* want) hidden... :-)

> I also read some imap servers have ~/.mailboxlist, but apparently
> this is not the case with UW imapd.

Actually, the ~/.mailboxlist file is an artifact of UW imapd.  It holds
the IMAP subscription list of non-newsgroup mailboxes (of course,
subscribed newsgroups are in .newsrc).  If you don't use a client such as
Outlook or Netscape that unilaterally subscribes all names, you probably
don't have one.

> Can I use namespaces to control this?  What is the "right" way?
> Or it should be done on the client side (e.g. hacking PHP clients)?

The answer to your question is "it all depends".

If you do something server-side, then it applies to all users whether or
not they want it.  There's no way for the user to control it.

If you do something client-side, then it only applies to the users of that
client even if all your users want it.  The user is forced to control it.

As indicated above with the example of Pine, even the choice to hide dot
files can be controversial.  Add to this such details as artifacts of
particular clients (e.g. your Kmail example).

Fortunately, as a sysadmin, you have considerably more leeway in deciding
things than I do.  You only have to worry about users strangling you.  I
have to worry about users *AND* sysadmins strangling me, not to mention
their often conflicting demands.  :-)

Sysadmins who have non-shell access IMAP server systems also have more
leeway than sysadmins of shell access systems.  You can even decide to
rewrite env_unix.c's mailboxfile() routine completely to make mailbox
names be completely different than files.  As distributed, mailboxfile()
basically converts a home-directory relative path name to an absolute path
name, with a few bells and whistles.  You could do something quite
different.  If you do this, though, be sure to change dummy_list_work() in
dummy.c to return names that match what mailboxfile() expects.

-- Mark --

---

From: Marc MERLIN <[email-masked]>
Date: Wed, 21 Aug 2002 19:15:16 -0700

> thought of before.  You didn't.
 
You are correct. I thought I did, I  didn't, and like the other problems, it
was  mostly due  to having  little  or no  feedback after  sending the  said
patches.

There was  a 5  line description of  what the patch  did. It was  trivial to
reply that you  were considering the idea and/or had  gotten similar patches
without even having to spend the time to read the patch.

My misunderstanding  were due to  lack of information which in some cases
led to pretty obvious conclusions, whether they were right or not.
 
> does something similar is covered in the free-fork license:
 
Just for the record, I never claimed  that you "stole" (for lack of a better
word)  my  patch. It  was  clear  that yours  was  written  and  implemented
differently.
What's still  clear however,  is that your  response was  clearly dismissive
while you say that in reality  you are considering the ideas and patches. If
that's the  case, please say so,  people are not going  to guess, especially
from the replies (or lack thereof) you send.
 
> box" if you prefer) to guide me when I implement new functionalities.
 
A little note to say this would have gone a long way to avoid problems.
 
> Some people would call that "quality control."

or  possessiveness, but  that's  not relevant. It's  your  project, you  are
entitled to wanting to re-write every single feature that is sent to you.
Again, saying so would have helped.

Anyway, no need to waste more of your time or mine, matter closed.

Marc