Subject: Re: tmail/mbx/imapd truncate message body

From: Mark Crispin <[email-masked]>
Date: Tue, 15 Sep 1998 12:30:50 -0700 (PDT)

I took a look at the sample data that you submitted.

There is a spurious CR in every line of the message.  That is, each line of
the message ends with CR CR LF.  CR CR LF CR CR LF is not recognized as an end
of line.

The reason that it works when you mbxcvt it to unix format is that unix format
gets rid of all the CRs.  If you mbxcvt it back to mbx format you'll discover
that it works that way.

Are you seeing a "tmail called with LF-only newlines" warning in your syslog?
If so, I suspect that exim is sending tmail a "From " line that ends with a
bare LF, followed by message text in which every line ends with CR LF.  Try
fixing your mailer so that it gives a consistant newline convention to tmail.

---

From: Malcolm Beattie <[email-masked]>
Date: Wed, 16 Sep 1998 10:58:42 +0100 (BST)

> fixing your mailer so that it gives a consistant newline convention to tmail.

Wonderful. Thanks. This turns out to be a gotcha in configuring exim.
The exim documentation I used to have was an old copy and it didn't
provide a way to use \r\n terminators for transports (e.g. appending
to files or delivering to pipes--I used the latter to deliver via
tmail). As a result, lines were all \n-terminated, tmail moaned about
it (in fact, I chopped the error message out of the source) but
delivery worked, which was the main thing.

When I printed out a new copy of the exim documentation, I discovered
the "use_crlf" option which uses \r\n-termination. Oh good, I thought,
and turned it on (and went back to an out-of-the-box tmail). That's
when the problem started. Mail delivered with that exim had the
"body-vanishing" problem. It turns out that one of the other options
in the pipe transport is "prefix" which is used to put the Berkeley
"From ..." at the beginning of the message and defaults to

    prefix = "From ...\n"

That constant \n was the first line terminator seen by tmail and so it
went into "add-\r" mode as you spotted with nasty consequences. Setting
    prefix =
to remove any prefix make everything work nicely (since there's no need
for a "From " line when delivering with tmail. Things are now looking
better for mbx.

I'm still having a problem, though, trying to make sure *everything*
is mbx format. I've compiled imapd with
    DEFAULTDRIVERS=mbx unix
but when I do an IMAP
    create foo
it still creates it in unix format, not mbx. I do an initial
  create #driver.mbx/INBOX
for each user so that tmail delivers to the right place but I can't
ensure that new mailboxes created by IMAP clients are always mbx. The
only reason I compiled in unix was because there's a chance that I
may want to use some special unix-format mailboxes to store some
configuration info. I'd like to be able to force a default of mbx for
everything, is possible, though if necessary I can compile out unix
format and hope that does the trick. Can I force mbx? Am I likely to
find any other problems setting up an all-mbx system: all mail is
delivered to ~/INBOX and all folders are in the user's home directory.

It would be nice if I could make that ~/Mail in such a way that users
don't have to tell their IMAP clients about it but the blackbox option
only allows /foo/username whereas we have to use /foo/group/username.
While I'm asking questions, then, can I force that ~/Mail
transparently to clients? Thanks for your quick responses, by the way.
I've been running UW imapd for years on our two central systems with
nothing fancy (delivery to /var/spool/mail/username, unix-format
mailboxes, clients have to set Mail as server-subdirectory etc.). Now
that I'm setting up a new mail cluster from scratch, I want to try to
do it according to best practice, which I'm given to understand is mbx.

--Malcolm

-- 
Malcolm Beattie <[email-masked]>
Unix Systems Programmer
Oxford University Computing Services

---

From: Jauder Ho <[email-masked]>
Date: Wed, 16 Sep 1998 03:33:12 -0700 (PDT)

hey malcom.. I use sendmail + procmail but the attached patch might help
out quite a little bit.

the way to make mbx standard is to change unixproto to mbxproto in the
makefile I believe. I am attaching two files. one's a patch to do things
the way we do it here, which imho works real well, the only change I would
do would be to change in to a /var/imap/<1st char>/<2nd char>/<username>
format in the future but right now that would cause uncessary problems
right now. it is not black box but close enough plus it retains all the
features I want. a) I don't need to use tmail, imapd will snarf at connect
time. b) it's pretty clean. I've also included a script which is in bad
need of subroutinization (HINT HINT) but works real nice to collect
results. I've got a third script that you can have if you want that
creates the dirs. 

code is not guaranteed... yadda yadda. ymmv. I'm trying to convince mrc to
put the getrusage() code into the mainline imap dist itself.. maybe if
enough people use it, mark will :)

--Jauder
diff -ruN imap.old/Makefile imap/Makefile
--- imap.old/Makefile	Tue Apr 28 13:51:32 1998
+++ imap/Makefile	Fri Jul  3 23:32:02 1998
@@ -72,7 +72,7 @@
 
 # Miscellaneous command line options passed down to the c-client Makefile
 
-EXTRACFLAGS=
+EXTRACFLAGS=-DVERBOSELOG
 EXTRALDFLAGS=
 
 # Normal commands
diff -ruN imap.old/src/imapd/imapd.c imap/src/imapd/imapd.c
--- imap.old/src/imapd/imapd.c	Fri Jul  3 14:08:43 1998
+++ imap/src/imapd/imapd.c	Fri Jul  3 23:32:02 1998
@@ -46,6 +46,11 @@
 #include <signal.h>
 #include "misc.h"
 
+#ifdef VERBOSELOG
+#include <sys/types.h>
+#include <sys/resource.h>
+#include <time.h>
+#endif
 
 /* Autologout timer */
 #define LOGINTIMEOUT 60*3
@@ -111,6 +116,12 @@
 char *lastid = NIL;		/* last fetched body id for this message */
 SIZEDTEXT lastst = {NIL,0};	/* last sizedtext */
 
+#ifdef VERBOSELOG
+time_t *timer;			/* timer */
+time_t starttime;		/* record start time of process */
+double uutime,sstime;		/* user and system time */
+#endif
+
 /* Response texts which appear in multiple places */
 
 char *win = "%s OK %.80s completed\015\012";
@@ -215,6 +226,9 @@
 
 void main (int argc,char *argv[])
 {
+#ifdef VERBOSELOG
+  struct rusage sysinfo;
+#endif
   unsigned long i,j,k,m,uid;
   long f;
   char *s,*t,*u,*v,tmp[MAILTMPLEN];
@@ -242,6 +256,9 @@
   default:
     fatal ("Unknown state from myusername_full()");
   }
+#ifdef VERBOSELOG
+  starttime = time(timer);
+#endif
   printf ("* %s %s IMAP4rev1 v%s server ready\015\012",t,tcp_serverhost (),
 	  version);
   fflush (stdout);		/* dump output buffer */
@@ -1016,6 +1033,34 @@
   } while (state != LOGOUT);	/* until logged out */
   syslog (LOG_INFO,"Logout user=%.80s host=%.80s",user ? user : "???",
 	  tcp_clienthost ());
+#ifdef VERBOSELOG
+  /* This is for additional logging information so that we can do analysis */
+  getrusage(RUSAGE_SELF,&sysinfo);
+
+  uutime = (double) sysinfo.ru_utime.tv_sec +
+          sysinfo.ru_utime.tv_usec / 1000000.0;
+  sstime = (double) sysinfo.ru_stime.tv_sec +
+          sysinfo.ru_stime.tv_usec / 1000000.0;
+
+  syslog (LOG_INFO,"u %s from %s start %d et %d ut %.3f st %.3f mss %ld iss %ld minf %ld majf %ld sw %ld bi %ld bo %ld mi %ld mo %ld nsig %ld vcs %ld ics %ld",           user ? user : "???",
+          tcp_clienthost (),
+          starttime,
+          time(timer) - starttime,
+          uutime,
+          sstime,
+          sysinfo.ru_maxrss,
+          sysinfo.ru_idrss,
+          sysinfo.ru_minflt,
+          sysinfo.ru_majflt,
+          sysinfo.ru_nswap,
+          sysinfo.ru_inblock,
+          sysinfo.ru_oublock,
+          sysinfo.ru_msgrcv,
+          sysinfo.ru_msgsnd,
+          sysinfo.ru_nsignals,
+          sysinfo.ru_nvcsw,
+          sysinfo.ru_nivcsw);
+#endif
   exit (0);			/* all done */
 }
 
@@ -1194,12 +1239,43 @@
 
 void clkint (void)
 {
+#ifdef VERBOSELOG
+  struct rusage sysinfo;
+#endif
   alarm (0);			/* disable all interrupts */
   server_traps (SIG_IGN,SIG_IGN,SIG_IGN,SIG_IGN);
   if (!quell_events)
     fputs ("* BYE Autologout; idle for too long\015\012",stdout);
   syslog (LOG_INFO,"Autologout user=%.80s host=%.80s",user ? user : "???",
 	  tcp_clienthost ());
+#ifdef VERBOSELOG
+  /* This is for additional logging information so that we can do analysis */
+  getrusage(RUSAGE_SELF,&sysinfo);
+
+  uutime = (double) sysinfo.ru_utime.tv_sec +
+          sysinfo.ru_utime.tv_usec / 1000000.0;
+  sstime = (double) sysinfo.ru_stime.tv_sec +
+          sysinfo.ru_stime.tv_usec / 1000000.0;
+
+  syslog (LOG_INFO,"u %s from %s start %d et %d ut %.3f st %.3f mss %ld iss %ld minf %ld majf %ld sw %ld bi %ld bo %ld mi %ld mo %ld nsig %ld vcs %ld ics %ld",           user ? user : "???",
+          tcp_clienthost (),
+          starttime,
+          time(timer) - starttime,
+          uutime,
+          sstime,
+          sysinfo.ru_maxrss,
+          sysinfo.ru_idrss,
+          sysinfo.ru_minflt,
+          sysinfo.ru_majflt,
+          sysinfo.ru_nswap,
+          sysinfo.ru_inblock,
+          sysinfo.ru_oublock,
+          sysinfo.ru_msgrcv,
+          sysinfo.ru_msgsnd,
+          sysinfo.ru_nsignals,
+          sysinfo.ru_nvcsw,
+          sysinfo.ru_nivcsw);
+#endif
   fflush (stdout);		/* make sure output blatted */
   if (critical) state = LOGOUT;	/* badly hosed if in critical code */
   else {			/* try to gracefully close the stream */
diff -ruN imap.old/src/osdep/unix/Makefile imap/src/osdep/unix/Makefile
--- imap.old/src/osdep/unix/Makefile	Mon Jun 29 16:40:49 1998
+++ imap/src/osdep/unix/Makefile	Fri Jul  3 23:32:02 1998
@@ -34,7 +34,7 @@
 # Command line build parameters
 
 EXTRAAUTHENTICATORS=
-EXTRADRIVERS=mbox
+EXTRADRIVERS=
 PASSWDTYPE=std

@@ -68,7 +68,7 @@
 SPOOLDIR=/usr/spool
 MAILSPOOL=$(SPOOLDIR)/mail
 NEWSSPOOL=$(SPOOLDIR)/news
-STDPROTO=unixproto
+STDPROTO=mbxproto
 RSHPATH=/usr/ucb/rsh

@@ -83,7 +83,7 @@
 # Standard distribution build parameters
 
 DEFAULTAUTHENTICATORS=log
-DEFAULTDRIVERS=imap nntp pop3 mh mx mbx tenex mtx mmdf unix news phile
+DEFAULTDRIVERS=imap pop3 mh mbx unix news phile

 # Normally no need to change any of these
diff -ruN imap.old/src/osdep/unix/dummy.c imap/src/osdep/unix/dummy.c
--- imap.old/src/osdep/unix/dummy.c	Thu Jun  4 18:53:11 1998
+++ imap/src/osdep/unix/dummy.c	Fri Jul  3 23:35:14 1998
@@ -238,11 +238,9 @@
 				/* list it if not at top-level */
     if (!level && dir && pmatch_full (dir,pat,'/'))
       dummy_listed (stream,'/',dir,LATT_NOSELECT,contents);
-				/* scan directory, ignore . and .. */
+				/* scan directory, ignore . files */
     if (!dir || dir[strlen (dir) - 1] == '/') while (d = readdir (dp))
-      if ((d->d_name[0] != '.') ||
-	  (d->d_name[1] && (((d->d_name[1] != '.') || d->d_name[2]) &&
-			    strcmp (d->d_name+1,MXINDEXNAME+3)))) {
+      if (d->d_name[0] != '.') {
 				/* see if name is useful */
 	if (dir) sprintf (tmp,"%s%s",dir,d->d_name);
 	else strcpy (tmp,d->d_name);
diff -ruN imap.old/src/osdep/unix/env_unix.c imap/src/osdep/unix/env_unix.c
--- imap.old/src/osdep/unix/env_unix.c	Sun Jun 28 10:22:53 1998
+++ imap/src/osdep/unix/env_unix.c	Fri Jul  3 23:32:02 1998
@@ -445,7 +445,8 @@
       nslist[0] = &nshome,nslist[1] = &nsblackother,nslist[2] = &nsshared;
     else {			/* not a black box */
       nslist[0] = &nshome,nslist[1] = &nsunixother,nslist[2] = &nsshared;
-      myHomeDir = cpystr (home);/* use real home directory */
+      sprintf (tmp,"/var/imap/%s",myUserName); /* DON'T use real home dir */
+      myHomeDir = cpystr (tmp);                /* use /var/imap/<userid> */
 				/* make sure user rc files don't try this */
       blackBoxDir = blackBoxDefaultHome = "";
     }

---

From: Mark Crispin <[email-masked]>
Date: Wed, 16 Sep 1998 10:53:08 -0700 (PDT)

On Wed, 16 Sep 1998 10:58:42 +0100 (BST), Malcolm Beattie wrote:
> Setting

I'm glad to hear that that is now working out.

> is mbx format. I've compiled imapd with

The only thing that does is remove support for other formats and such things
as being able to proxy to other servers (e.g. access news via IMAP).

The simplest thing is to set STDPROTO to mbxproto in the Makefile.  In the
next version (imap-4.5, unreleased), set CREATEPROTO to mbxproto and leave
EMPTYPROTO alone.