Subject: problems with large imap folder/file

From: "Harondel J. Sibble" <[email-masked]>
Date: Thu, 02 Oct 2003 13:27:55 -0700

Have a client running linux with the UW Imapd server, with mbx (mail stored 
in home dirs) using Outlook Express 6 on the client side.  The user having 
problems has approx 900 messages in her "Sent Items" folder/file. The file 
size is ~125mb.

The server has 64mb ram and 128mb swap.

When she attempts to delete and purge messages from the sent items folder, 
she gets an error message that I don't have at hand (I am not at the site, 
troubleshooting over the phone), but basically it says that the server 
reported an error of some sort.

The server mail logs show:

Oct  2 11:51:08 server1 imapd[597]: IMAP toolkit crash: Can't resize memory
Oct  2 12:21:45 server1 imapd[817]: Fatal error user=username host=computer1 
[192.xxx.xxx.xxx] mbx=/home/username/Sent Items: Can't resize memory

and

Oct  2 13:09:46 server1 imapd[1220]: Fatal error user=username host=computer1 
[192.xxx.xxx.xxx] mbx=/home/username/Sent Items: Out of memory
Oct  2 13:09:46 server1 imapd[1220]: IMAP toolkit crash: Out of memory

The obvious error message and googling says it's a memory issue.  Adding 
memory to the server is not a problem, however, is there another way for me 
to purge or move some of the messages from the Sent Items file without adding 
memory. Just so the user can access and delete other messages from the 
folder, until I add more ram next week.

Thanks

-- 
Harondel J. Sibble 
Sibble Computer Consulting
Creating solutions for the small business and home computer user.
[email-masked] (use pgp keyid 0x3AD5C11D) http://www.pdscc.com
(604) 739-3709 (voice/fax)      (604) 686-2253 (pager)

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Thu, 2 Oct 2003 14:22:52 -0700 (Pacific Daylight Time)

Your system should have quite enough memory to handle the mail file that
you describe.  Are you certain that you are using mbx format (files that
have "*mbx*" as the first line)?

Have you tried upgrading to the latest version of UW imapd?
	ftp://ftp.cac.washington.edu/mail/imap.tar.Z

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Thu, 02 Oct 2003 23:40:33 -0700

> have "*mbx*" as the first line)?

in the user's home dir

# head INBOX
*mbx*
3e3df7c800003343

# head "Sent Items"
>From MAILER-DAEMON Thu Oct  2 21:23:00 2003
Date: 02 Oct 2003 21:23:00 -0700
From: Mail System Internal Data <[email-masked]>
Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA
Message-ID: <[email-masked]>
X-IMAP: 1065128750 0000018201
Status: RO

This text is part of the internal format of your mail folder, and is not
a real message.  It is created automatically by the mail system software.

Hmm, is there a way to convert the sent items folder to mbx as it looks like 
it's in mbox format....
 
>  ftp://ftp.cac.washington.edu/mail/imap.tar.Z

No, as that is not something that I can easily do until late next week due to 
time constraints. Idealy I'd like to find a workaround to this problem until 
that time.
-- 
Harondel J. Sibble 
Sibble Computer Consulting
Creating solutions for the small business and home computer user.
[email-masked] (use pgp keyid 0x3AD5C11D) http://www.pdscc.com
(604) 739-3709 (voice/fax)      (604) 686-2253 (pager)

---

From: Mark Crispin <[email-masked]>
Date: Fri, 3 Oct 2003 10:07:59 -0700 (PDT)

> Hmm, is there a way to convert the sent items folder to mbx as it looks like

Yes, the mailutil program (included as part of the IMAP toolkit) has a
mechanisms to copy a mailbox with a different format in the destination
copy.

> time constraints. Idealy I'd like to find a workaround to this problem until

The reason why I suggest upgrading is that the problem may be a bug that
has already been fixed.  I find it very difficult to believe that imapd
was really out of memory.

-- Mark --

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Mon, 06 Oct 2003 09:17:20 -0700

> mechanisms to copy a mailbox with a different format in the destination

Hmm, using the currently installed mailutil fails. I've renamed the "Sent 
Items" to sentold and run following command which just pops up the usage 
info:

# mailutil copy -verbose sentold #driver.mbx/sentnew
usage: mailutil copy [-debug] [-verbose] source destination

> was really out of memory.

My real concern, is what if it doesn't fix the problem :-(  Then I am pretty 
much back at square one.  I am currently running 2002b.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 6 Oct 2003 09:20:14 -0700 (PDT)

> usage: mailutil copy [-debug] [-verbose] source destination

In some shells, "#" is a comment character, so you have to quote any
string which uses it:

mailutil copy -verbose sentold "#driver.mbx/sentnew"

> much back at square one.  I am currently running 2002b.

If it doesn't solve the problem, then I will continue working with you
until it is solved.  I am very concerned about this report, but before
expending resources on it I want to make sure that it isn't something that
is already fixed.

-- Mark --

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Mon, 06 Oct 2003 09:57:19 -0700

> mailutil copy -verbose sentold "#driver.mbx/sentnew"

Hmmm... that also fails

# mailutil copy -verbose sentold "#driver.mbx/sentnew"
Can't open mailbox sentold: no such mailbox

# ls -la sentold
-rw-------   1 username groupname    108783116 Oct  6 09:39 sentold

> expending resources on it I want to make sure that it isn't something that is
> already fixed.

That makes good sense.  Just downloading the latest version now.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 6 Oct 2003 10:01:13 -0700 (PDT)

> -rw-------   1 username groupname    108783116 Oct  6 09:39 sentold

mailutil does not operate on the connected directory; it operates on the
mail home directory (which, unless you've set it elsewhere, would be the
user's UNIX home directory).  Try specifying a fully-qualified path, or at
least a path from the home directory.

If you are running this as root, try becoming the target user instead.
Strange things happen when you run these tools as root.

-- Mark --

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Mon, 06 Oct 2003 11:15:58 -0700

> Strange things happen when you run these tools as root.

That worked fine and I made a copy of the Sent Items into mbx format while 
running mailutil as the user.

Now.. the memory crash still happens with latest version of imapd.

Here are the steps I took and how to recreate problem.

downloaded: ftp://ftp.cac.washington.edu/mail/imap.tar.Z

as root

-extracted: tar xvzf imap.tar.Z
-cd imap-2002e/
-read the README file and looked at unix build instructions on the website
-make lnp SSLTYPE=none
-cd imapd/
-backup current imapd: mv /usr/sbin/imapd /usr/sbin/imapd.bak-oct06-03-1008
-cp imapd /usr/sbin/imapd

Now I startup outlook express on the clients machine (actually ssh tunneled 
connection to my machine, however errors are the same regardless)

I tell OE synchronize folders, it downloads approx 900 messages to the sent 
items folder. I highlight say 5 or 10 messages in sent items and drag them to 
a local (move rather than copy) so I have backups of messages I am working 
with.

Server shows following

Oct  6 10:45:39 server1 imapd[2199]: imap service init from 192.168.xxx.xxx
Oct  6 10:45:39 server1 imapd[2199]: Login user=username host=localhost 
[192.168.xxx.xxx]
Oct  6 10:47:34 server1 imapd[2204]: imap service init from 192.168.xxx.xxx
Oct  6 10:47:34 server1 imapd[2204]: Login user=username host=localhost 
[192.168.xxx.xxx]
Oct  6 10:48:05 server1 imapd[2204]: Command stream end of file, while 
reading line
 user=username host=localhost [192.168.xxx.xxx]

also sometimes see

Oct  6 10:53:43 server1 imapd[2220]: Killed (lost mailbox lock) user=username 
host=localhost [192.168.xxx.xxx]
Oct  6 10:53:45 server1 imapd[2220]: Broken pipe, while reading line 
user=username host=localhost [192.168.xxx.xxx]

OE reports loss of mailbox lock

The messages are now all marked with a line through them and a red X, so I 
then click edit/purge deleted messages and see following at mail server side:

Oct  6 10:55:16 server1 imapd[2222]: Fatal error user=username host=localhost 
[192.168.xxx.xxx] mbx=/home/username/Sent Items: Can't resize memory
Oct  6 10:55:16 server1 imapd[2222]: IMAP toolkit crash: Can't resize memory

OE reports

Your server has unexpecteldy terminated the connection. Possible causes for 
this include server problems, network problems or a long period of 
inactivity,

Details shows:
The server did not respond to this IMAP command before the connection was 
terminated.

Clicking okay results in message:

Your IMAP server wishes to alert you to the following: IMAP4rev1 server 
crashing: Can't resize memory.

Next I'll try using the mbx converted version of the Sent Items folder and 
see what happens. Will report under another email.

Running top shows

 10:43am  up  2:46,  2 users,  load average: 0.20, 0.30, 0.33
46 processes: 43 sleeping, 3 running, 0 zombie, 0 stopped
CPU states:  3.5% user,  2.3% system,  0.0% nice, 94.0% idle
Mem:   63080K av,  61616K used,   1464K free,  19684K shrd,  15552K buff
Swap:      0K av,      0K used,      0K free                 24504K cached

and the line for the user

2181 username     9   0   956  956   784 S       0  0.1  1.5   0:00 imapd

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Mon, 06 Oct 2003 11:50:38 -0700

On 6 Oct 2003 at 11:15, Harondel J. Sibble wrote:

> Next I'll try using the mbx converted version of the Sent Items folder and see
> what happens. Will report under another email.

I select messages, move them to local folder, select edit/purge ...

Oct  6 11:35:52 server1 imapd[2415]: imap service init from 192.168.xxx.xxx
Oct  6 11:35:53 server1 imapd[2415]: Login user=username host=localhost 
[192.168.xxx.xxx]
Oct  6 11:36:09 server1 imapd[2416]: imap service init from 192.168.xxx.xxx
Oct  6 11:36:09 server1 imapd[2416]: Login user=username host=localhost 
[192.168.xxx.xxx]
Oct  6 11:36:11 server1 imapd[2416]: Command stream end of file, while 
reading line user=username host=localhost [192.168.xxx.xxx]

And lo and behold, it appears to purge the messages fine. The user in 
question is currently in a meeting, once she gets out, I'll have her try the 
same thing and report back, meanwhile, I can finally have some breakfast ;-)
-- 
Harondel J. Sibble 
Sibble Computer Consulting
Creating solutions for the small business and home computer user.
[email-masked] (use pgp keyid 0x3AD5C11D) http://www.pdscc.com
(604) 739-3709 (voice/fax)      (604) 686-2253 (pager)

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Mon, 06 Oct 2003 12:35:44 -0700

> same thing and report back, meanwhile, I can finally have some breakfast ;-)

Nope, must have been a lucky break, it still crashes with memory errors when 
accessing the Sent Items folder from the users' end.  Trying to synchronize 
the imap folder also does the same thing after downloading a few messages.

Oct  6 12:22:16 server1 imapd[2739]: Logout user=username host=computer1 
[192.168.xxx.xxx]
Oct  6 12:22:16 server1 imapd[2740]: imap service init from 192.168.xxx.xxx
Oct  6 12:22:16 server1 imapd[2740]: Login user=username host=computer1 
[192.168.xxx.xxx]
Oct  6 12:22:21 server1 imapd[2740]: Fatal error user=username host=computer1 
[192.168.xxx.xxx] mbx=/home/username/Sent Items: Out of memory
Oct  6 12:22:21 server1 imapd[2740]: IMAP toolkit crash: Out of memory
Oct  6 12:22:21 server1 imapd[2741]: imap service init from 192.168.xxx.xxx
Oct  6 12:22:21 server1 imapd[2741]: Login user=username host=computer1 
[192.168.xxx.xxx]
Oct  6 12:22:29 server1 imapd[2741]: Fatal error user=username host=computer1 
[192.168.xxx.xxx] mbx=/home/username/Sent Items: Out of memory
Oct  6 12:22:29 server1 imapd[2741]: IMAP toolkit crash: Out of memory
Oct  6 12:24:30 server1 imapd[2743]: imap service init from 192.168.xxx.xxx
Oct  6 12:24:31 server1 imapd[2743]: Login user=username host=computer1 
[192.168.xxx.xxx]

---

From: Mark Crispin <[email-masked]>
Date: Mon, 6 Oct 2003 12:44:14 -0700 (Pacific Daylight Time)

On Mon, 6 Oct 2003, Harondel J. Sibble wrote:
> accessing the Sent Items folder from the users' end.

This is with an mbx-format "Sent Items" mailbox, and is *after* you purged
it?  So it is empty?

Can you provide me with access to your system?  Something very strange is
going on.

-- Mark --

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Mon, 06 Oct 2003 13:18:24 -0700

On 6 Oct 2003 at 12:44, Mark Crispin wrote:

> This is with an mbx-format "Sent Items" mailbox, and is *after* you purged it?
>  So it is empty?

Yes, converted from mbox to mbx, no it's not empty, there are approx 930 
messages in the file, attempting to purge any of them results in the memory 
errors.
 
> Can you provide me with access to your system?  Something very strange is

It's actually a client's machine and they aren't comfortable about someone 
they don't know being in the machine.
-- 
Harondel J. Sibble 
Sibble Computer Consulting
Creating solutions for the small business and home computer user.
[email-masked] (use pgp keyid 0x3AD5C11D) http://www.pdscc.com
(604) 739-3709 (voice/fax)      (604) 686-2253 (pager)

---

From: Mark Crispin <[email-masked]>
Date: Mon, 6 Oct 2003 13:26:19 -0700 (Pacific Daylight Time)

> messages in the file, attempting to purge any of them results in the memory

There are no calls to fs_resize() in the mbx driver expunge.  Something
else is going on.  I would like to know what that is.

> they don't know being in the machine.

Well, you can tell your client that this "someone they don't know" is the
author of the imapd software that your client is giving root-level access.

-- Mark --

---

From: "Harondel J. Sibble" <[email-masked]>
Date: Mon, 06 Oct 2003 13:39:23 -0700

> else is going on.  I would like to know what that is.

Heh, me too. Something else is going on indeed, just called the client back 
and it's working fine now, while 1 hour ago it wasn't  <shrug> and the hour 
before that it worked fine for me.

Actually that concerns me more than it not working at all.

> author of the imapd software that your client is giving root-level access.

I've explained that to them, we are going to leave it and see if it continues 
to work okay. If not, then I'll contact you about remote access.  In that 
case can you send me a pgp signed/encrypted email with phone contact info? 
That way I can pass you login info directly.

---

From: "Guillaume Vaillant" <[email-masked]>
Date: Thu, 9 Oct 2003 10:06:22 +0200

Hello,

I'm trying to use the c-client library to perform some IMAP requests
My tools needs to establish simultaneously more than one IMAP connections to
the IMAP server.
To do that I'm calling the MAILSTREAM *mail_open (MAILSTREAM *oldstream,char
*name,long options) for each connection.
The result is strange. I have a big time before the connection answer.

If i make some network traces i can see that this function
    - connect to the server
    - request server capabilities
    - And request login

But the request login step is done after a long time.
Time 0 ms : My tools connect to server on port 143
Time + 2 ms : Server reply OK + ready
Time + 2 ms : My tools request capabilities
Time + 3 ms : Server return capabilities
.... During this time other connection to the imap server was done but each
connection are blocked after Server return capabilities
Time + 4000 ms : My tools request login

Why the request login is so long ?

Thanks for your help
Guyom