Subject: Does pine read all the mail into memory?

From: [email-masked] (Jef Poskanzer)
Date: Fri, 25 Aug 1995 19:08:07 GMT

I don't use pine myself, haven't done more than browse through
the man page.  I'm a sysadmin on a public access system where it's
installed.  One of our users has a 17meg mail file (he likes mailing
lists), and I've noticed that when he fires up pine he gets a ~25meg
process, we run out of memory and start swapping, and system performance
goes to hell.

Is pine really reading the whole mail file into memory?  Is there
a way to configure it to not do this?
---
Jef

              Jef Poskanzer  [email-masked]  [email-masked]
                      http://www.river.org/~jef/
     "Trust is good, but control is better." -- Feliks Dzerzhinski

---

From: David L Miller <[email-masked]>
Date: Fri, 25 Aug 1995 14:30:57 -0700 (PDT)

On Fri, 25 Aug 1995, Jef Poskanzer wrote:

> Date: Fri, 25 Aug 1995 19:08:07 GMT
> From: Jef Poskanzer <[email-masked]>
> To: [email-masked]
> Subject: Does pine read all the mail into memory?
>

When using the default Berkeley (bezerk) mailbox format, Pine does
indeed read the entire folder into memory.  This can be avoided by
switching to a format, such as Tenex, that only keeps an index in
memory.  See the Pine Technical Notes for more information.  They can
be found in the doc directory of the source distribution or at

	ftp://ftp.cac.washington.edu/pine/docs/tech-notes.txt

|\ |  |\/|  David L. Miller    [email-masked]  (206) 685-6240
|/ |_ |  |  Software Engineer, Pine Development Team   (206) 685-4045 (FAX)
University of Washington, Networks & Distributed Computing, JE-20
4545 15th Ave NE, Seattle WA 98105, USA

---

From: Mark Crispin <[email-masked]>
Date: Fri, 25 Aug 1995 15:21:59 -0700

On 25 Aug 1995, David L Miller wrote:
> memory.

To elaborate on David's answer, in the standard Unix mailbox format (what
the Pine source code calls "bezerk" for hysterical reasons), various
overhead data (such as message status flags, unique identifiers, etc.) are
stored in additional header lines in the header of the message.  These are
variable-width fields, and aren't present at all when a message is
initially delivered.

In order to update this data, there is a Scylla vs. Charybdis decision:
	Scylla: copy the mail file to a scratch file, then copy back
		with the update
	Charybdis: read the mail file into an in-memory snapshot, then
		dump out updates as necessary
The reason why you can't just overwrite is that messages may grow when
their status changes.

Many mailers choose to fall into the clutches of 6-headed Sylla.  Well,
here at UW we have disk quotas and users who are regularly at the edge of
their quota, so Pine chose the whirlpool of Charybdis instead.  This works
well with ordinary student users, but as you noted it does fall apart with
a 17-megabyte godzilla mailbox.

The Tenex format mentioned by David uses pre-allocated fixed-width fields,
so editing in place with random-access update-mode I/O can work.  Thus, it
only needs to keep an index in memory.  However, even this has hazards;
update mode I/O doesn't work well over NFS.  However, if you use IMAP for
remote access and avoid NFS, you don't have this problem; and Tenex format
does work very well with large mail files.

Apologies to those who are not up on Greek mythology.

-- Mark --

DoD #0105, R90/6 pilot, FAX: (206) 685-4045  ICBM: N 47 39'35" W 122 18'39"
Science does not emerge from voting, party politics, or public debate.