Subject: IMAPd problem...core dumping

From: [email-masked] (Marc G. Fournier)
Date: 21 Oct 1994 01:23:18 -0400

Hi...

	Hate to ask it...but when we use pine to try and
read a relatively large mailbox (ie. 22Meg), rimapd dumps
core and closes connection.

	the mail folder is fine, elm can read it in, just 
pine fails.

	I have the newest versions...imapd-3.5 and pine-3.91.

	anyone experience this?  any way to fix it?

Thanks...

-- 
Marc G. Fournier  | Internex Online: Toronto's First Public Internet Connection
System Admin      | Data: 363-4151        Telnet: io.org        Voice: 363-8676
[email-masked]    | Worldwide Chat Through IRC      Over 9400 Usenet Newsgroups
Toronto, Canada   | Limited Free Access   Telnet   Ftp   Gopher    Online Games

---

From: Mark Crispin <[email-masked]>
Date: Sun, 23 Oct 1994 14:15:03 -0700

Marc -

The most likely explanation for the problem is that Pine (local mail) or
imapd (remote mail) ran out of virtual memory.  c-client (the internal
engine used by both Pine and imapd) does not tolerate malloc() failure; if
the single malloc() in all of c-client/Pine/imapd returns 0, it issues an 
``Out of free storage'' message and core dumps.

c-client's drivers for the two primary UNIX formats of mail -- bezerk and 
mmdf -- both read the entire mail folder into memory.  Thus, a 22MB mail 
folder requires at least 22MB (plus overhead) of virtual memory for the 
process.

The reason this is done has to do with rewriting the mail folder (as in
changing flags, or expunging).  Since a message can grow as a result of
flag setting (writing the Status:/X-Status: lines), it is necessary to
have a scratch area to hold the mailbox.  There are two possible scratch
areas: virtual memory and the disk.  With virtual memory, you do a single
pass write of the new in-memory state to the disk. With the disk as a
scratch area, the entire mail folder in its new state needs to be written
to a scratch file, then it needs to be copied back on top of the original
mail folder.  You can't rename() since several other UNIX mail tools
require that you not change the inode of your mail folder.

With either choice, you face the risk of running out of resources.  You
either run out of virtual memory, or you run out of disk.  My belief is
that many more people run out of disk (due to disk quotas) than run out of
virtual memory; this is based upon the number of people who send messages
reporting conditions when you exceed disk quota compared to the number of
reports such as yours.  Furthermore, the virtual memory method is faster. 
My suspicion is that Elm made the opposite choice. 

Fortunately, there is a way out.  If you want to have godzilla mail
folders (and 22MB fits that description) and lack the virtual memory to
support it, you can use an alternate format supported by c-client that
does not have this restriction, such as tenex or mh format.  Tenex format
has a fixed-width field to allocate space for the flags, so c-client is
able to do mailbox updates in place without ever having to have the entire
folder in memory.  The tenex driver will automatically read new mail from
your /usr/spool/mail file and convert it on the fly to your ~/mail.txt
file.  The disadvantage of Tenex format is that it is a different (read:
incompatible) format and Elm won't be able to read it. 

There is a tool called mbxcvt on our FTP server which can convert between
mailbox formats.  I suspect that mbxcvt will also encounter the same out
of memory program (since it too uses c-client), but you can use Elm to
split the godzilla mailbox in halve or quarters, convert with mbxcvt, then
rejoin them.  If you need any help in figuring out how to do this, please
let me know. 

I regret the problems that you are having.  As you can surmise, it was a
calculated gamble that was made, and most of the time we win it.  Users
who have nasty disk quotas that won't even let them make temp files
appreciate the current behavior.  Unfortunately, there are times when we
lose the gamble, and are stuck with having to offer the consolation prize 
of ``you can do it if you change your mail format''.

Thank you for your interest in Pine, and again, sorry for the hassles!

-- Mark --