Subject: re: EEE-normous File Attachments

From: "[email-masked]" <[email-masked]>
Date: Fri, 22 Oct 1999 18:00:34 GMT

> There must be adequate virtual memory (and ideally physical memory) to handle
> the attachment.  Depending upon the mailbox format, you may need considerably
> more memory; I would not try to make a 32MB RAM system handle 25MB
> attachments.

This particular machine as 256meg of physical memory, 300MB of swap (don't know
how SGI arrives at that calculation for swap).

> 
> One problem is that c-client needs to know the exact size of the attachment in
> CRLF-normalized form before sending it, as well as present it in that form;

This problem is in retrieval of the message.

> Please clarify what you mean by "bombs"; that's too imprecise to make a
> diagnosis.  What, precisely, happens?  What error message do you get?
> 

I'll give the latest edition as an example.  The mailbox has 4 messages, 1, 2 & 3
are plain/text messages.  Message 4 is multipart, part 1 is plain/text, part two
is an image base64 encoded.

With IPOPD, I can stat the mailbox, retrieve messages 1, 2, and 3 as well as
start retrieval of message 4.  IPOPD coredumps as soon as the headers have been sent,
and the connection is subsequently closed.

Under a handler that I wrote using c-client, I'm able to "stat" the mailbox using
mail_status() as well as get a listing of the messages in the box.  However, when
I want to retrieve message 4, it stops functioning without a core dump (any hint
on telemetry logging would be appreciated).

Do you mean c-client drivers
> into some private mailbox format?  Do you mean your own client and user
> interface?

My own c-client interface.  Generally, my interface has no hope of ever seeing
attachments this large, but I am using it for experimental purposes.

> I can't determine this without knowing what message c-client issued.
> 

I will open up internal.txt and refresh myself on the callback functions, and
then see what c-client has to say.

---

From: Mark Crispin <[email-masked]>
Date: Fri, 22 Oct 1999 11:14:56 -0700 (PDT)

On Fri, 22 Oct 1999 18:00:34 GMT, [email-masked] wrote:
> This particular machine as 256meg of physical memory, 300MB of swap (don't
> know how SGI arrives at that calculation for swap).

Such a machine is certainly quite suitable for a modest attachment of 25MB.

> start retrieval of message 4.  IPOPD coredumps as soon as the headers have
> been sent, and the connection is subsequently closed.

Have you looked at the core dump?  That should be your first step in finding
out what happened.  At least find out what it was doing and get a stack trace.
It's a waste of time to talk about the problem until you do this.

> Under a handler that I wrote using c-client, I'm able to "stat" the mailbox
> using mail_status() as well as get a listing of the messages in the box.

What is a "handler"?  The closest word in c-client's lexicon is "driver", but
it sounds to me like you're referring to an "application", possibly a server.

> However, when I want to retrieve message 4, it stops functioning without a
> core dump (any hint on telemetry logging would be appreciated).

What do you mean by "stops functioning"?  Does it loop, gaining CPU time?  Is
it in an I/O wait?

How to do telemetry logging depends a lot upon what it is you are doing.  I
haven't a clue from your message whether you're reading the file yourself or
via a POP server, whether you are using standard UNIX mailbox format or a
local format, whether the problems are in c-client or your code.

I'm trying to help you, but unfortunately your reports are still too vague for
me to figure out what's going on.  As I said above, your very first step
should be to look into that ipop3d core dump and get a stack trace.  I'm
hoping that it'll be something easy, like "it called abort() through fatal()
with text <Out of memory>".

Next, we need to separate c-client issues from issues in your code.  We need
to settle the question of whether or not you are writing your own drivers;
your use of the word "handler" is raising this question.  If not, the question
turns to whether or not ipop3d is modified by you in any way.

Once we get that out of the way, we can then talk about what your code is
doing, and how that might interfact and cause the problem.

One final note: POP is a wretched protocol to use with messages that have 25MB
attachments.  Why are you using POP instead of IMAP?