Subject: Re: INBOX folder construction, impact on existing mail.txt and mbox

From: Mark Crispin <[email-masked]>
Date: Sat, 13 Jan 1996 10:54:33 -0800

Chris, just a few comments from your message.  Other people will
undoubtably answer other points:

On Sat, 13 Jan 1996, Chris Koenigsberg wrote:
> I just did some experiments regarding Pine's algorithm for
> construction of the "INBOX" folder, to report for a meeting on Monday.

Actually, this is c-client's algorithm, not Pine's, and kicks in only if
you do not specify an override with Pine's inbox-path variable.

> 2. If you have an existing ~/mbox file, Pine will prefer to use that,
> instead of mail.txt, as the place where it appends truly new
> /var/spool/mail/$USER messages, before displaying the whole bunch as
> your INBOX.

This only happens if c-client is built with the mbox driver enabled.  By
default, it is not enabled in the imap-3.5 toolkit used in Pine 3.91 or in
the imap-3.6 toolkit used in Pine 3.92.  It is enabled in imap-4, but we
haven't decided whether that will go into Pine or not.

The enabling is done by the setting of the EXTRADRIVERS variable in the
c-client Makefile.

> We're also considering setting
> inbox-path=/var/spool/mail/$USER

This is usually the preferred technique to get the behavior that you say
you want.

> We had been assuming that we would have to do a massive "conversion"
> of all these existing legacy "mtxt" format ~/mail.txt files, into
> "mbox" format instead, to eliminate the corruption problem.

If you do decide to convert, the mbxcvt tool in the imap-utils toolkit may
be very useful for you.

> Now I have recently decided to try and trust the Pine/c-client team,
> that they have fixed many "mtxt" bugs since the days of old MM
> c-client, and NOT convert the file formats after all..... wise or
> foolish? (we'll find out :-)

Well, speaking as a highly biased observer, I would say that this is the
path of wisdom...  ;-)

> A major question for us will be whether Pine's handling of "mtxt"
> format (in ~/mail.txt as part of INBOX, and/or an "imported" folder
> moved into the mail subdirectory) is more robust than MM's.

Undeniably.  But there is an important caveat:

> One concern is that the "corrupt" mtxt files may come during NFS
> outages, if people run the mail program (MM or Pine) on a host other
> than the one where their home directory is actually physically
> located.

And here's the caveat:
	Don't even think of using the mtxt (a.k.a. tenex) format over NFS
	with c-client (Pine, imapd, etc)!!

The problem here is that Pine uses update mode (O_RDWR) random access I/O,
based on the presumption that the buffer and inode caches reflect what is
actually on the disk.  This is very fast; however, NFS violates most of
the underlying assumptions.  There is no guarantee of synchronization of
the kernel buffer caches between different machines when an update
happens, and worse the inode cache can get out of synchronization with the
buffer cache on the same machine!!!

Even worse, there is *no* locking when NFS is involved, so there is no
guarding of critical code when simultaneous read/write access happens.

*Usually*, c-client will detect that something is hosed, and decline to
proceed any further; there haven't been too many clobberage incidents.
But, "verbum sat sapenti", a word to the wise is sufficient...

My advice; if you use tenex format, use it on a real filesystem that
follows the documented Unix semantics (certain operations documented to be
atomic are, certain operations documented to update the disk do, certain
operations documented to semaphore do) and not an imitation filesystem
such as NFS that simulates some of the semantics and throws up its hands
on others.  This means that if remote access is desired to tenex files,
you should use IMAP.

If you must use NFS, you're better off with the Unix mbox format.  You
lose some safeguarding with mbox as well (and it's slower to boot), but
you still have a basic locking semaphore in place.  Also, the mbox code
doesn't keep the file open except when doing actual operations, which
permits the crufty NFS kernel to get the buffer and inode caches back into
synchronization.

-- Mark --

DoD #0105, R90/6 pilot, FAX: (206) 685-4045  ICBM: N 47 39'35" W 122 18'39"
Science does not emerge from voting, party politics, or public debate.

---

From: [email-masked] (Chris Koenigsberg)
Date: Sat, 13 Jan 1996 01:54:40 GMT

We're introducing official support for Pine at the U. of Chicago, in
the beginning of February, and are working out some configuration
details. If I'm forgetting anything, please let me know (post a
followup here, or email me if you prefer, [email-masked]).

We are starting out with the existing /var/spool/mail/$USER mailboxes,
NFS-mounted from a server, not introducing IMAP yet. (sorry I couldn't
make it to the IMAP Consortium meeting in Seattle this past week, I
would have loved to have gone, maybe next time...)

1. If you have an existing "mtxt" format ~/mail.txt file, Pine wants
to use that as part of the INBOX folder, i.e. it will slurp new mail
out of /var/spool/mail/$USER, append it to mail.txt, and display the
entire contents of mail.txt as your INBOX.

3. If you have neither an "mtxt" mail.txt file, nor an mbox file, Pine
will just use the contents of /var/spool/mail/$USER as its INBOX.

These considerations are important to people who have existing
~/mail.txt files (from MM) and/or existing mbox files (from Berkeley
Mail etc.), perhaps with large numbers of old messages there.

If they try Pine, it seems that Pine is going to show them all their
many old messages, as being new and unread. (I could say, that's what
they get for leaving all their old messages in one monolithic mail
file :-) but I've got to try and ease their transitions.

We want to minimize the number of site-specific configuration
changes. But we are interested in setting the system-wide Pine fixed
config file combination:

read-message-folders=saved-messages
feature-list=auto-move-read-msgs

to sweep mail, after it's read, OUT of the INBOX folder (i.e. out of
the files /var/spool/mail/$USER and ~mail.txt and/or ~mbox).

which would tell Pine to ignore the mail.txt and mbox files. That is,
this will force Pine's INBOX folder construction to use ONLY the
/var/spool/mail file, NOT the existing legacy ~/mail.txt or ~/mbox
files.

If we do set this inbox-path, and a user does want to "import" their
existing mail.txt or mbox file into Pine, they can simply move the
whole file, into their mail subdirectory, and Pine will then see it as
a folder.

Currently many people still use "MM" with their "mtxt" format
~/mail.txt files, and we have bad problems with "corrupted" mail.txt
files, due to bugs in old MM where it doesn't quite finish writing the
file out before dying, and the new invocation can't read it. The
"mtxt" format is inherently more delicate than the "mbox" format,
since it has the binary flags which include one for the message
length....

I am hoping/betting that Pine will not "corrupt" mtxt files the way MM
frequently does. But if we do still get a lot of corrupt mtxt files
from Pine, we'll have to think about converting them.

One concern is that the "corrupt" mtxt files may come during NFS
outages, if people run the mail program (MM or Pine) on a host other
than the one where their home directory is actually physically
located. If this is the case, then even Pine might not be able to
prevent the mtxt file corruption problems? I haven't been able to test
this reliably yet though. (we have made an effort to urge people to
login & run their MM on the same host where their home directory
lives, rather than go across NFS, and we will continue to do so with
Pine)

Am I forgetting anything? :-)

Chris Koenigsberg
[email-masked]

---

From: [email-masked] (Larry)
Date: 18 Jan 1996 22:28:47 -0500

<grovel mode>

In an attempt to gain efficiency, we decided to try using the tenex
mailbox features of pine.  

To do this, I created a mail.txt file in the home directory of a bunch
of users, which I had read was the way to go about this.  

Of course, now we have the problem -- for most of these users, the
this mail.txt has been ignored completely -- mail comes from
/usr/spool/mail/$USER and goes back there when they quit.

For a few of them, their mailboxes were converted to tenex in their
home directories, but now Pine doesn't recognize the mail.txt file at
all, and has gone back to /usr/spool/mail/$USER ... 

If I copy their mail.txt to their mail/ directory, pine CANNOT open it
as a folder.

The users .pinerc's are indentical between those who are affected and
those who aren't.  /usr/local/lib/pine.conf and their .pinerc's DO NOT
have incoming-folder set to anything.

% uname -a
OSF1 front V3.2 148 alph

frontier: fran % uname -a
OSF1 frontier.wilpaterson.edu V3.2 148 alph

It was built for OSF1 - the only changes to the source was the 1 line
"no runaways" patch.  It is not running over NFS.  Nothing was done to
the source to activate tenex (ie EXTRADRIVERS.)

I've read the thread below, and the FAQ, and done dejanews searches --
I can't find anything relevent..  Please help!!

Thanks a lot!

--Larry

</grovel>

Mark Crispin <[email-masked]> writes:
...

---

From: Mark Crispin <[email-masked]>
Date: Fri, 19 Jan 1996 18:40:09 -0800

There are two things to check:

1) if you created the ~/mail.txt file as an empty file, are you ABSOLUTELY
   sure that it is empty?  If it is non-empty, it must be in proper tenex
   data format, otherwise it will be ignored.

2) what is the setting of inbox-path in the users' .pinerc?  It must be
   INBOX, not /usr/spool/mail/something.

-- Mark --