Subject: Transition from /var/mail/user to ~user/mbox

From: Erik R Leo <[email-masked]>
Date: Tue, 18 Aug 1998 14:57:53 -0400 (EDT)

Greetings;

	I sent this same message (more-or-less) directly to Mark Crispin,
since it may describe a problem with the mbox driver.  Now I am sending it
to this list, since it is more likely a problem with my understanding of
the mbox driver :).

	I am using the pine4.02 (including the pine4.02A patch)
distribution for POP, IMAP and local access to my mail spool.  I also use
sendmail (8.1.9a) with procmail (3.11pre7) as a local delivery agent on a
BSD/OS (3.1) platform.

	I am in transition from a shared spool (/var/mail/user) to a home
directory (~user/mbox) delivery scheme.  Following advice gleaned from the
imap mailing list ([email-masked]), I have used the default
behavior of the mbox driver to move the existing contents of
/var/mail/user to ~user/mbox, and I have configured procmail to deliver
new mail to ~user/mbox.  Everything seems to work fine, but I considered
this to be a temporary configuration, since the move from the old spool to
the home directory should be a one-time operation.

	I assumed the final configuration would simply be a matter of
pointing the c-client programs (pine, imapd, ipop2d, ipop3d) at the new
spool location, and following advice gleaned from .../imap/docs/CONFIG, I
modified sysinbox() to return ~user/mbox on my workstation.

	Unfortunately, this configuration broke (neither pine nor
ipop3d--the 2 that I tested--was able to open my inbox).

	Any advice would be appreciated.

TIA,
-Erik
-- 
Erik R. Leo, Net Worker       SoVerNet
Tel:    +1(802)463-2111       Vermont's Sovereign Internet Connection
Fax:    +1(802)463-2110       Post Office Box 495, 5 Rockingham Street
Email:  [email-masked]       Bellows Falls, Vermont 05101

------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 18 Aug 1998 12:44:01 -0700 (PDT)

> to this list, since it is more likely a problem with my understanding of

Yes, I got your first message, which you sent less than 6 hours ago.  Please
allow a reasonable amount of time for a response.

> ipop3d--the 2 that I tested--was able to open my inbox).

What, precisely, do you make sysinbox() return?  sysinbox() must return an
absolute path, not anything with a "~" in it.

What, precisely, happened?  It does not suffice to say "was [not] able to open
my inbox".  Did you get an error message?  Did you get zero messages?

I strongly recommend that you either disable the mbox driver in c-client or
you use some name other than "mbox" (I recommend "INBOX").  The mbox driver
exists specifically to move mail *from* the sysinbox() to ~/mbox.  Although
locking should protect you from moving from ~/mbox to ~/mbox, it is playing
with fire.

Note that any mail which does not go through procmail will go to
/var/mail/user.  There have been instances where this has happened, so your
tinkering may end up creating more problems than the trivial bit of efficiency
that it gains (in other words, you may be better off leaving it alone).

---

From: Geir Johannessen <[email-masked]>
Date: Tue, 18 Aug 1998 22:43:15 +0200 (MET DST)

> absolute path, not anything with a "~" in it.

We deliver mail to ~/mail/INBOX with procmail and use the following patch
to make imapd find the correct inbox and use ~/mail for everything related
to mail. I enclose the patches we use below.

I would appreciate some comment on what we are doing. It seems to be
working just fine, but are we doing something "wrong"?

--- src/osdep/unix/env_unix.c.orig	Tue Jul 14 03:55:36 1998
+++ src/osdep/unix/env_unix.c	Tue Jul 21 22:04:41 1998
@@ -548,7 +548,8 @@
 {
   char tmp[MAILTMPLEN];
   if (!sysInbox) {		/* initialize if first time */
-    sprintf (tmp,"%s/%s",MAILSPOOL,myusername ());
+/*  sprintf (tmp,"%s/%s",MAILSPOOL,myusername ()); */
+    sprintf (tmp,"%s/mail/INBOX",myhomedir ());
     sysInbox = cpystr (tmp);	/* system inbox is from mail spool */
   }
   return sysInbox;
@@ -572,7 +573,8 @@
 				/* validate name, return its name */
     if (!mailboxfile (dst,tmp)) return NIL;
   }
-  else strcpy (dst,myhomedir());/* no arguments, wants home directory */
+/*else strcpy (dst,myhomedir());*//* no arguments, wants home directory */
+  else sprintf (dst,"%s/mail",myhomedir());
   return dst;			/* return the name */
 }
 
@@ -646,7 +648,8 @@
     }
   }
 				/* build resulting name */
-  sprintf (dst,"%s/%s",dir,name);
+/*sprintf (dst,"%s/%s",dir,name);*/
+  sprintf (dst,"%s/mail/%s",dir,name);
   return dst;			/* return it */
 }
 
--- Makefile.orig	Tue Jul 21 22:07:42 1998
+++ Makefile	Tue Jul 21 22:04:41 1998
@@ -51,7 +51,7 @@
 # mbox	if file "mbox" exists on the home directory, automatically moves mail
 #	 from the spool directory to "mbox" and uses "mbox" as INBOX.
 
-EXTRADRIVERS=mbox
+EXTRADRIVERS=

 # The following plaintext login types are defined:
@@ -72,7 +72,7 @@
 
 # Miscellaneous command line options passed down to the c-client Makefile
 
-EXTRACFLAGS=
+EXTRACFLAGS=-DSVR4_DISABLE_FLOCK
 EXTRALDFLAGS=
 
 # Normal commands

That was it. In someone would like to see what we did with procmail I
enclose that patch also:

--- src/authenticate.c.orig	Wed May 20 15:41:55 1998
+++ src/authenticate.c	Sat Jun 13 14:18:20 1998
@@ -41,7 +41,7 @@
 #ifndef MAILSPOOLHASH
 #define MAILSPOOLHASH	0      /* 2 would deliver to /var/spool/mail/b/a/bar */
 #endif
-/*#define MAILSPOOLHOME "/.mail"		      /* watch the leading / */
+#define MAILSPOOLHOME "/mail/INBOX"		      /* watch the leading / */
 						  /* delivers to $HOME/.mail */
 #define STRLEN(x)	(sizeof(x)-1)
 
--- config.h.orig	Wed May 20 15:39:50 1998
+++ config.h	Fri May 22 17:58:47 1998
@@ -45,9 +45,9 @@
 #define TRUSTED_IDS	{"root","daemon","uucp","mail","x400","network",\
 			 "list","slist","lists","news",0}
 
-/*#define NO_fcntl_LOCK		/* uncomment any of these three if you	     */
-/*#define NO_lockf_LOCK		/* definitely do not want procmail to make   */
-/*#define NO_flock_LOCK		/* use of those kernel-locking methods	     */
+#define NO_fcntl_LOCK		/* uncomment any of these three if you	     */
+#define NO_lockf_LOCK		/* definitely do not want procmail to make   */
+#define NO_flock_LOCK		/* use of those kernel-locking methods	     */
 
 /*#define RESTRICT_EXEC		    /* don't allow everyone to fork programs */
 
@@ -71,7 +71,7 @@
 	is not found, maildelivery will proceed as normal to the default
 	system mailbox. */
 
-#define ETCRC	"/etc/procmailrc"	/* optional global procmailrc startup
+#define ETCRC	"/etc/mail/procmailrc"	/* optional global procmailrc startup
 					   file (will only be read if procmail
 	is started with no rcfile on the command line). */
 
--- Makefile.orig	Wed May 20 15:40:59 1998
+++ Makefile	Mon May 18 19:54:32 1998
@@ -2,7 +2,7 @@
 
 # BASENAME should point to where the whole lot will be installed
 # change BASENAME to your home directory if need be
-BASENAME	= /usr
+BASENAME	= /usr/local
 # For display in the man pages
 VISIBLE_BASENAME= $(BASENAME)

--
Geir Johannessen          #  [email-masked]
E B Schieldropsvei 35-25  #  http://www.stud.ntnu.no/~joge/
N-7033 TRONDHEIM, NORWAY  #  Tlf private +47-73888989, job +47-73598048
"Due to circumstances beyond your control, you are master of your fate
 and captain of your soul."

---

From: Mark Crispin <[email-masked]>
Date: Tue, 18 Aug 1998 14:07:21 -0700 (PDT)

> working just fine, but are we doing something "wrong"?

If it works, then it is very likely to be right, or "close enough" to right
that you needn't worry.  But I'll give you some comments about your specific

> +    sprintf (tmp,"%s/mail/INBOX",myhomedir ());

This is the correct patch to change the location of where mail is delivered
from the mail spool to ~/mail/INBOX, and is probably what you want.

>    return dst;			/* return it */

These patches are probably not correct.  The second patch in particular is
probably a bug; it will always insert "mail/" in a namespace or "~" names.
Unless you want to force all non-absolute mailbox references to insert a
"mail/", but have certain other references continue to use the user's home
directory, this is probably not what you want.

The correct patch is in env_init().  That is, you probably want to change:
      myHomeDir = cpystr (home);/* use real home directory */
to the directory that you want, e.g.
      sprintf (tmp,"%s/mail",home);
      myHomeDir = cpystr (tmp);

> +EXTRADRIVERS=

This patch is correct if you want to disable moving mail from the sysinbox to
mbox.  On your system, that is probably what you want.

> +EXTRACFLAGS=-DSVR4_DISABLE_FLOCK

You probably do not want to do this, since this disables the protection
between multiple processes accessing the mailbox.  It is only an issue if you
access NFS files this way (the bottom line advice is: don't!), but the code
does a runtime test for NFS to disable locking automatically.

You almost certainly *do* want locking for local files.

If you have NFSv3, be sure that you have a recent version of the c-client
library (one with flocksun.c) and be sure to build on your NFSv3 system.
Binaries built on an NFSv2 system will run on an NFSv3 system, but the runtime
test will not work on the NFSv3 system.

---

From: Erik R Leo <[email-masked]>
Date: Tue, 18 Aug 1998 19:04:02 -0400 (EDT)

> allow a reasonable amount of time for a response.

	Sorry.  I didn't mean to be rude, its just that after thinking
about the "problem" a little more, I decided the wider audience might help
me understand the purpose/behavior of the mbox driver a little better.

> absolute path, not anything with a "~" in it.

	I changed the line in .../imap/src/osdep/unix/env_unix.c 
sysinbox() from:

sprintf (tmp,"%s/%s",MAILSPOOL,myusername ());

to:

sprintf (tmp,"%s/mbox",myhomedir ());

> my inbox".  Did you get an error message?  Did you get zero messages?

	Pine displayed the "[Opening "INBOX" <|> ]" message indefinitely
with the spinning cursor.  It also created ~user/mbox.lock.  I finally
killed the process.  No error messages, no completion of the "Opening
"INBOX" stage.

> locking should protect you from moving from ~/mbox to ~/mbox, it is playing

	I guess I am abusing the mbox driver's behavior, but I was hoping
to use it for a transition phase, where the current spool (/var/mail)
could be NFS-mounted on my new mailserver, with the new mailserver
delivering to ~user/mbox.  Then the spool could be moved incrementally, as
needed and on demand without mail service downtime.

> that it gains (in other words, you may be better off leaving it alone).

	We use procmail as a local delivery agent, so its use is pretty
wired into the delivery system, but your point is taken.

	Other ideas for accomplishing the transition and any hints about
the "best" final configuration are truly appreciated.

-Erik

---

From: Mark Crispin <[email-masked]>
Date: Tue, 18 Aug 1998 16:46:14 -0700 (PDT)

> me understand the purpose/behavior of the mbox driver a little better.

The sole purpose of the mbox driver is to service an environment in which mail
is delivered to the spool directory and the user program is expected to move
it to the mbox file in his home directory.

> sprintf (tmp,"%s/mbox",myhomedir ());

OK, that is correct.  You should have also disabled the mbox driver, since
otherwise it would try to move mail from mbox to mbox.

> > What, precisely, happened?

That is very different from "not able to open".  Since you say NFS is
involved, the most obvious culprit is locking.  Is your system NFSv2 or NFSv3?
If it is NFSv3, do you have a recent version of the software with the
flocksun.c file?

> needed and on demand without mail service downtime.

If you want such a transition, then you should just use the mbox driver and
not play around with changing the sysinbox.  By changing the sysinbox, you
prevent mail from ever being moved from the mail spool.

I also strongly recommend against using NFS for any kind of mail access.  NFS
is not a scaleable, efficient, or reliable way of accessing mail.  That's what
IMAP is for.

---

From: brian moore <[email-masked]>
Date: Wed, 19 Aug 1998 15:12:48 -0700

> prevent mail from ever being moved from the mail spool.

Well, what we did doing here:

/var/mail became a mess with several thousand mailboxes, competition for
.lock creation, deletion, etc and all the fun of Berkeley mailboxes.

So I hacked procmail to check the existence of
/var/mail/u/s/username/inbox : if it existed, mail was delivered there,
else, it went to the /var/mail/username.

I did the same sort of thing with c-client (to take care of ipop3d and
imapd) in sysinbox:

char *sysinbox ()
{
  struct stat buf;
  char user_name[20];
  char tmp[MAILTMPLEN];
  if (!sysInbox) {              /* initialize if first time */
    sprintf (tmp,"%s/%s",MAILSPOOL,myusername ());
    sysInbox = cpystr (tmp);    /* system inbox is from mail spool */
    if ( stat(sysInbox, &buf) < 0) {
        fs_give((void **) &sysInbox);      /* slight memory leak otherwise */
        strncpy(user_name,myusername(),15);
        sprintf(tmp,"%s/%c/%c/%s/inbox", MAILSPOOL, user_name[0],
                                         user_name[1], myusername() );
        sysInbox =cpystr(tmp);
    } 
  }
  return sysInbox;
}

I then moved a few mailboxes and things worked, then a few more.  The
whole move was done over a week or so in little bursts of activity.
("I'm bored, let's move the A's....").  When I get around to it, I'll
kill the extra stat() and force both procmail and c-client to use the
proper thing.

This is done in the made-up directory because our users don't have shell
accounts or home directories.

> I also strongly recommend against using NFS for any kind of mail
> access.  NFS is not a scaleable, efficient, or reliable way of
> accessing mail.  That's what IMAP is for.

That's true as well.  Any chance UW will let you explain your mail
architecture in depth, as Earthlink has done?

You've got an impressive structure from what I've seen, and it would be
nice to see how to handle the sort of hammering you have.

-- 
Brian Moore                             Kill A Spammer For Jesus
Sysadmin, C/Perl Hacker, Usenet Vandal

---

From: Mark Crispin <[email-masked]>
Date: Wed, 19 Aug 1998 15:20:01 -0700 (PDT)

> nice to see how to handle the sort of hammering you have.

I'm not sure how much there is to explain.  I'll refer you to Terry Gray
<[email-masked]> for specific details.