Subject: two mailutil segfaults with 2002e or 2004a

From: Tim Mooney <[email-masked]>
Date: Tue, 31 Aug 2004 11:45:57 -0500 (CDT)

Hello!

We're using (locally compiled) imapd and the mailutil that comes with it on
several Red Hat Advanced Server 2.1 systems.  These systems serve thousands
of IMAP mail accounts, using mbx format folders.  We have several local
scripts that depend on mailutil.  In particular, we use mailutil when we
shuffle users between imap hosts, to balance the load on our systems.

I recently had mailutil from the imap-2002e distribution segfault when
doing a move for two separate folders, both owned by the same user.  I've
tried the move with the 2002e version on both RH AS 2.1 and on my Tru64
UNIX 5.1b workstation, and I've also tried the move on RH AS 2.1 with
the 2004a mailutil.  In all cases, mailutil coredumps in the same place.

Here's a little example:

$pwd
/local/src/RPM/BUILD/imap-2004a/mailutil
$ls -al ~/folders/
total 838
drwx------    2 mooney   sysadmin     1024 Aug 30 19:28 ./
drwxr-x---   15 mooney   sysadmin     1024 Aug 30 19:26 ../
-rw-------    1 mooney   sysadmin   370479 Aug 27 15:36 MiscSaved
-rw-------    1 mooney   sysadmin   479202 Aug 27 14:50 auser
$file  ~/folders/*
/home/mooney/folders/MiscSaved: MBX mail folder
/home/mooney/folders/auser:     MBX mail folder
$./mailutil check folders/MiscSaved
No new messages, 46 total in folders/MiscSaved
$./mailutil check folders/auser
No new messages, 127 total in folders/auser

$gdb mailutil
GNU gdb Red Hat Linux (5.3.90-0.20030710.41.2.1rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux"...Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) set args move folders/MiscSaved MiscSaved
(gdb) run
Starting program: /usr/local/src/RPM/BUILD/imap-2004a/mailutil/mailutil
move folders/MiscSaved MiscSaved

Program received signal SIGSEGV, Segmentation fault.
0x0804ce2d in mm_append (stream=0x8115790, data=0xbffeaa90,
flags=0xbffe98f0,
      date=0xbffe98f4, message=0xbffe98f8) at mailutil.c:488
488           if ((MAILTMPLEN - ((t += strlen (t)) - tmp)) > (long)
(gdb) where
#0  0x0804ce2d in mm_append (stream=0x8115790, data=0xbffeaa90,
      flags=0xbffe98f0, date=0xbffe98f4, message=0xbffe98f8) at
mailutil.c:488
#1  0x0808db7e in mbx_append (stream=0x0, mailbox=0xbffed8f8 "MiscSaved",
      af=0x804cc7c <mm_append>, data=0xbffeaa90) at mbx.c:1074
#2  0x0805e594 in mail_append_multiple (stream=0x0,
      mailbox=0xbffed8f8 "MiscSaved", af=0x804cc7c <mm_append>,
data=0xbffeaa90)
      at mail.c:2486
#3  0x0804cb66 in mbxcopy (source=0x80f3a48, dest=0x0,
      dst=0xbffed8f8 "MiscSaved", create=1, del=1, mode=0) at mailutil.c:435
#4  0x0804c076 in main (argc=4, argv=0xbffeb7d4) at mailutil.c:245

(gdb) print u
$1 = 0
(gdb) print elt
$2 = (MESSAGECACHE *) 0x8115100
(gdb) print *elt
$3 = {msgno = 32, lockcount = 1, rfc822_size = 2593, private = {uid = 33,
      special = {offset = 328743, text = {data = 0x0, size = 55}}, msg = {
        env = 0x0, body = 0x0, full = {offset = 0, text = {data = 0x0,
            size = 0}}, lines = 0x0, header = {offset = 0, text = {data =
0x0,
            size = 0}}, text = {offset = 0, text = {data = 0x0, size = 0}}},
      sequence = 0, dirty = 0, filter = 0, data = 0}, day = 4, month = 2,
    year = 33, hours = 8, minutes = 14, seconds = 57, zoccident = 1, zhours
= 6,
    zminutes = 0, seen = 1, deleted = 0, flagged = 0, answered = 0, draft =
0,
    recent = 0, valid = 1, searched = 0, sequence = 1, spare = 0, spare2 =
0,
    spare3 = 0, spare4 = 0, spare5 = 0, spare6 = 0, spare7 = 0, spare8 = 0,
    sparep = 0x0, user_flags = 1}
(gdb)
(gdb) print t
$4 = 0xbffe9486 ""
(gdb) print tmp
$5 = " \\Seen", '\0' <repeats 1017 times>

The other core dump, on the `auser' folder, is very very similar:

(gdb) set args move folders/auser auser
(gdb) run
Starting program: /usr/local/src/RPM/BUILD/imap-2004a/mailutil/mailutil
move folders/auser auser

Program received signal SIGSEGV, Segmentation fault.
0x0804ce2d in mm_append (stream=0x8117af8, data=0xbffec710,
flags=0xbffeb570,
      date=0xbffeb574, message=0xbffeb578) at mailutil.c:488
488           if ((MAILTMPLEN - ((t += strlen (t)) - tmp)) > (long)
(gdb) print t
$1 = 0xbffeb110 ""
(gdb) print tmp
$2 = " \\Seen \\Answered", '\0' <repeats 1007 times>

Other than providing a copy of the folders themselves, is there any
additional information I could provide that would be useful in diagnosing
what's going wrong here?

Thanks,

Tim
--
Tim Mooney                              [email-masked]
Information Technology Services         (701) 231-1076 (Voice)
Room 242-J6, IACC Building              (701) 231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164
-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 31 Aug 2004 13:49:26 -0700 (Pacific Daylight Time)

In one of those crashes, what is in ap->stream->user_flags[0] ?

My guess is that it is a null pointer, and that somehow a keyword flag got 
set in the message without any keywords being defined.

If, in fact, that is the case, I'll send you a patch.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Tim Mooney <[email-masked]>
Date: Tue, 31 Aug 2004 17:03:07 -0500 (CDT)

In regard to: Re: two mailutil segfaults with 2002e or 2004a, Mark Crispin...:

> My guess is that it is a null pointer, and that somehow a keyword flag got set 
> in the message without any keywords being defined.
>

That is in fact the case, for both folders:

(gdb) run

Program received signal SIGSEGV, Segmentation fault.
0x402594f1 in strlen () from /lib/i686/libc.so.6
(gdb) where
#0  0x402594f1 in strlen () from /lib/i686/libc.so.6
#1  0x0804d680 in mm_append (stream=0x8132738, data=0xbffeb8c0,
     flags=0xbffeb3b0, date=0xbffeb3ac, message=0xbffea710) at
mailutil.c:488
#2  0x0809f569 in mbx_append (stream=0x0, mailbox=0xbffed8fb "auser",
     af=0x804d464 <mm_append>, data=0xbffeb8c0) at mbx.c:1074
#3  0x080618f2 in mail_append_multiple (stream=0x0,
     mailbox=0xbffed8fb "auser", af=0x804d464 <mm_append>, data=0xbffeb8c0)
     at mail.c:2486
#4  0x0804d324 in mbxcopy (source=0x810e688, dest=0x0, dst=0xbffed8fb
"auser",
     create=1, del=1, mode=0) at mailutil.c:435
#5  0x0804c45e in main (argc=4, argv=0xbffec654) at mailutil.c:233
(gdb) up
#1  0x0804d680 in mm_append (stream=0x8132738, data=0xbffeb8c0,
     flags=0xbffeb3b0, date=0xbffeb3ac, message=0xbffea710) at
mailutil.c:488
488           if ((MAILTMPLEN - ((t += strlen (t)) - tmp)) > (long)
(gdb) print ap->stream->user_flags[0]
$1 = 0x0

Program received signal SIGSEGV, Segmentation fault.
0x402594f1 in strlen () from /lib/i686/libc.so.6
(gdb) up
#1  0x0804d680 in mm_append (stream=0x81303d0, data=0xbffeb1c0,
     flags=0xbffeacb0, date=0xbffeacac, message=0xbffea010) at
mailutil.c:488
488           if ((MAILTMPLEN - ((t += strlen (t)) - tmp)) > (long)
(gdb) print ap->stream->user_flags[0]
$1 = 0x0

Thanks Mark!

Tim

---

From: Mark Crispin <[email-masked]>
Date: Tue, 31 Aug 2004 15:08:52 -0700 (PDT)

> That is in fact the case, for both folders:

OK.  The following patch should remedy the problem:

*** mailutil.c~	Sun Jul 11 12:19:24 2004
--- mailutil.c	Tue Aug 31 15:06:52 2004
***************
*** 485,493 ****
       if (elt->answered) strcat (t," \\Answered");
       if (elt->draft) strcat (t," \\Draft");
       if (u = elt->user_flags) do	/* any user flags? */
!       if ((MAILTMPLEN - ((t += strlen (t)) - tmp)) > (long)
! 	  (2 + strlen
! 	   (t1 = ap->stream->user_flags[find_rightmost_bit (&u)]))) {
   	*t++ = ' ';		/* space delimiter */
   	strcpy (t,t1);		/* copy the user flag */
         }
--- 485,492 ----
       if (elt->answered) strcat (t," \\Answered");
       if (elt->draft) strcat (t," \\Draft");
       if (u = elt->user_flags) do	/* any user flags? */
!       if ((t1 = ap->stream->user_flags[find_rightmost_bit (&u)]) &&
! 	  (MAILTMPLEN - ((t += strlen (t)) - tmp)) > (long) (2 + strlen (t1))){
   	*t++ = ' ';		/* space delimiter */
   	strcpy (t,t1);		/* copy the user flag */
         }

---

From: Shawn Walker <[email-masked]>
Date: Tue, 31 Aug 2004 17:52:20 -0500

How would I go about setting a X-Flag in the header?

The setextendedheaders() apparently been nixed.

Thanks,
Shawn

---

From: Mark Crispin <[email-masked]>
Date: Tue, 31 Aug 2004 15:54:43 -0700 (Pacific Daylight Time)

> The setextendedheaders() apparently been nixed.

I've never heard of "setextendedheaders()".

What do you mean by "setting a X-Flag"?

Do you mean adding a header line in an outgoing (to SMTP or NNTP) message 
that you are composing?

Or, do you mean adding an "X-Flag" header line to a message in the 
mailbox?

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 31 Aug 2004 16:03:51 -0700 (Pacific Daylight Time)

You can't add a header to a message in the mailbox.  The only thing you 
can do is fetch the message, make the alteration to the string, and append 
it back as a new copy of the message (with a new UID).

To write an "X-Flag" header in an outgoing message, write your own 
rfc822out_t function, and set it with the mail_parameters() 
SET_RFC822OUTPUT operation.  smtp_mail() and nntp_mail() will both use it 
when rfc822_output() is called.

Look at rfc822_output() for the default action.  Basically, you need to 
duplicate the action from the line below the comment "encode body as 
necessary".  The important thing is that instead of calling rfc822_header, 
you call your own function that duplicates the action of rfc822_header() 
but also does your own header operations.  While you're at it, you 
probably want don't want to write to a fixed buffer the way 
rfc822_header() does but instead do something that won't risk a buffer 
overflow.

On Tue, 31 Aug 2004, Shawn Walker wrote:

> Well, both actually.  I must have been confused of setextendedheaders() from 
> something else then.
>
> Thanks,
> Shawn
>
> Mark Crispin wrote:
>
>> Or, do you mean adding an "X-Flag" header line to a message in the mailbox?
>> 
>> -- Mark --
>> 

-- Mark --

---

From: Shawn Walker <[email-masked]>
Date: Tue, 31 Aug 2004 17:56:11 -0500

Well, both actually.  I must have been confused of setextendedheaders() 
from something else then.

Thanks,
Shawn

Mark Crispin wrote:

> Do you mean adding a header line in an outgoing (to SMTP or NNTP) 
> message that you are composing?
>

---

From: Shawn Walker <[email-masked]>
Date: Wed, 08 Sep 2004 17:06:52 -0500

How do I get mail_fetch_fast() to get the ENVELOPE along with "UID 
INTERNALDATE RFC822.SIZE FLAGS" for IMAP, so this is what I want 
c-client to send to the server "(UID INTERNALDATE RFC822.SIZE FLAGS 
ENVELOPE)".

Thanks,
Shawn

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Sep 2004 15:12:56 -0700 (Pacific Daylight Time)

On Wed, 8 Sep 2004, Shawn Walker wrote:
> INTERNALDATE RFC822.SIZE FLAGS" for IMAP, so this is what I want c-client to 
> send to the server "(UID INTERNALDATE RFC822.SIZE FLAGS ENVELOPE)".

You don't.  You use mail_fetch_structure() instead.

-- Mark --

---

From: Shawn Walker <[email-masked]>
Date: Wed, 08 Sep 2004 17:17:02 -0500

What I want to do is get the list of messages back with all the info 
with one call and the loop through all the messages that came back.  Not 
make 15,000 requests to server for 15,000 messages.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Sep 2004 15:32:54 -0700 (Pacific Daylight Time)

On Wed, 8 Sep 2004, Shawn Walker wrote:
> What I want to do is get the list of messages back with all the info with one 
> call and the loop through all the messages that came back.  Not make 15,000 
> requests to server for 15,000 messages.

You *must* call mail_fetch_structure() to get the envelope.

If your application is using something like elt->private.msg.env, fix it 
*now* so that it does not do that; otherwise I guarantee that it will 
break in the future.

mail_fetch_structure() does not necessarily send a request to the server. 
If the envelope is in the cache, it will return the cached value.  There 
is a cache lookahead (that defaults to 20) which will fill the cache up to 
that number of messages ahead of the message you requested on a cache 
miss.

If you really want to lookahead for all messages, then do:
   mail_parameters (NIL,SET_LOOKAHEAD,(void *) LONG_MAX);
prior to your mail_fetch_structure() call.

I don't recommend doing that.  Envelopes are not part of the fast data, 
and there is no particular reason to believe that doing
 	tag FETCH 1:15000 ENVELOPE
will complete in anything approaching a reasonable timeframe.

Good quality IMAP applications do not generally do a "download all 
messages" operation.  Instead, they only fetch data as needed.  Most users 
do not have screens that can display 15,000 messages at once, and most 
display toolkits allow for a "draw line method" in a scrolling view (which 
can in turn call mail_fetch_structure()) rather than requiring that the 
entire scrolled text be loaded at once.  Similarly, most program-related 
processing of envelope data can be done via the SEARCH, SORT, and THREAD 
operations.

-- Mark --

---

From: Shawn Walker <[email-masked]>
Date: Wed, 08 Sep 2004 17:43:58 -0500

I'm not using "elt->private.msg.env".  I just want to be able to get the 
  response that IMAP server returned from the fast command.

I have no choice but to download all the message headers but I want to 
get them all at once with one request and many responses which will be 
much faster than 15,000 requests and 15,000 responses.

But what I'm trying to do is get all the message header (no bodies) to 
be able to display the list of messages in a folder.  Blame MS for the 
bad implementation of how Outlook work.  ;)

Shawn

Mark Crispin wrote:

>> with one call and the loop through all the messages that came back.  
>> Not make 15,000 requests to server for 15,000 messages.
> 
> 
> mail_fetch_structure() does not necessarily send a request to the 
> server. If the envelope is in the cache, it will return the cached 
> value.  There is a cache lookahead (that defaults to 20) which will fill 
> the cache up to that number of messages ahead of the message you 
> requested on a cache miss.
> 
> messages" operation.  Instead, they only fetch data as needed.  Most 
> users do not have screens that can display 15,000 messages at once, and 
> most display toolkits allow for a "draw line method" in a scrolling view 
> (which can in turn call mail_fetch_structure()) rather than requiring 
> that the entire scrolled text be loaded at once.  Similarly, most 
> program-related processing of envelope data can be done via the SEARCH, 
> SORT, and THREAD operations.
> 
> -- Mark --
>

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Sep 2004 15:50:42 -0700 (Pacific Daylight Time)

On Wed, 8 Sep 2004, Shawn Walker wrote:
> I have no choice but to download all the message headers

Why?

> but I want to get 
> them all at once with one request and many responses which will be much 
> faster than 15,000 requests and 15,000 responses.

Actually, the cache lookahead means that it will be more like 750 
requests.  If you do the
   mail_parameters (NIL,SET_LOOKAHEAD,(void *) LONG_MAX);
call then you'll get one request.

> But what I'm trying to do is get all the message header (no bodies) to be 
> able to display the list of messages in a folder.  Blame MS for the bad 
> implementation of how Outlook work.  ;)

What does Outlook have to do with this?

-- Mark --

---

From: Shawn Walker <[email-masked]>
Date: Wed, 08 Sep 2004 17:58:33 -0500

>> I have no choice but to download all the message headers

Because we have no way of knowing what messages that the user want to 
see.  Outlook just say "Give me the contents of this folder".  Nothing 
about saying "give me 1000 messages", though that could be a user's 
option but we are talking about users that have used Outlook and are 
used to seeing 15,000 messages in a folder (believe me, there are users 
out there and have dealt with them before) and so that is what they want 
to see.

As for the Contacts, Appointment, Tasks, Todo and etc folders, there is 
simply no way that we could limit how many not to return to the user, we 
have no way of know what is important to a user to see in thier 
appointment, contacts, etc.

> 
>> but I want to get them all at once with one request and many responses 
>> which will be much faster than 15,000 requests and 15,000 responses.
> 

I will try that.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 8 Sep 2004 16:01:00 -0700 (Pacific Daylight Time)

On Wed, 8 Sep 2004, Shawn Walker wrote:
> Because we have no way of knowing what messages that the user want to see. 
> Outlook just say "Give me the contents of this folder".

I still don't understand.  Are you writing replacement IMAP client code 
for the existing code in Outlook?

-- Mark --

---

From: Shawn Walker <[email-masked]>
Date: Wed, 08 Sep 2004 18:03:54 -0500

Mark Crispin wrote:

>> see. Outlook just say "Give me the contents of this folder".
> 
> 

Yes.