Subject: IMAPd( UNIX driver ) performance problem

From: 최성훈(Sung-hoon Choi) <[email-masked]>
Date: Wed, 17 Dec 2003 09:40:00 +0900 (KST)

We use imapd( UNIX driver ) and our client program code is following [Client code example].

If a message consists of 100 bodies and message size is 20M bytes, then read() called 100 times.

Why?

For every mail_fetch_body() call for one message, the imapd calls stream->dtb->text() ( i.e. unix_text() ).

unix_text_work() ( from unix_work() ) calls read() with full message text size.

Totally, 20M*100times = 2GB read() is requested. !!!

I made a temporary patch for fix this problem. ( Please, check [My temporary patch for unix.c & unix.h] )

I think this is not correct solution, but to show my idea. ;-)

Thanks for reading.

------------------- [ Client code example ] ----------------------------

int fetch_body_contents( BODY *body, char *pfx )
{
	mail_fetch_body( body, pfx, ... )
}

int fetch_body( BODY *body, char *pfx, long i )
{
	char tmp[MAILTMPLEN];

        if ( body->type == TYPEMULTIPART )
        {
                if ( pfx ) /* if not first time, extend prefix */
                        sprintf( tmp,"%s%ld.", pfx, ++i );
                else
                        tmp[0] = '\0';

                for ( i = 0, part = body->nested.part ; part ; part = part->next )
                {
                        if ( !fetch_body( &part->body, tmp, i++ ) )
                                return -1;
                }
                return 1;
        }
        else /* non-multipart */
        {
                if ( !pfx ) /* dummy prefix if top level */
                        pfx = "";
                sprintf( tmp, "%s%ld", pfx, ++i );

                if ( fetch_body_contents( body, tmp ) < 0 )
                        return -1;

                /* encapsulated message? */
                if ( (body->type == TYPEMESSAGE) &&
                         (body->subtype[0] == 'R' &&
                          body->subtype[1] == 'F' &&
                          body->subtype[2] == 'C' &&
                          body->subtype[3] == '8' &&
                          body->subtype[4] == '2' &&
                          body->subtype[5] == '2' &&
                          body->subtype[6] == 0 ) &&
                         (body->nested.msg) &&
                         (body = body->nested.msg->body) )
                         (body = body->nested.msg->body) )
                {
                        if ( body->type == TYPEMULTIPART )
                        {
                                return fetch_body( body, pfx, i-1 );
                        }
                        else /* build encapsulation prefix */
                        {
                                sprintf( tmp, "%s%ld.", pfx, i );
                                return fetch_body( body, tmp, (long)0 );
                        }
                }
                return 1;
        }
}

int parse_mail()
{
	int result;

        mail_fetchstructure_full( ... )
	result = fetch_body( body, "", 0 );
}

----------------- [ My temporary patch ] -----------------------

*** unix.c.orig Tue Dec 16 12:33:34 2003
--- unix.c      Tue Dec 16 12:37:47 2003
***************
*** 558,563 ****
--- 558,580 ----
    FDDATA d;
    STRING bs;
    char *s,*t,*tl,tmp[CHUNK];
+   char *_buf_;
+   int   _buflen_;
+ 
+   if ( LOCAL->fullmsg != NIL &&
+          LOCAL->fullmsg_elt == elt &&
+          LOCAL->fullmsg_flags == (flags & FT_INTERNAL) ) {
+         *length = LOCAL->fullmsg_len;
+         return LOCAL->fullmsg;
+   }
+   fs_give( (void **)&LOCAL->fullmsg );
+   LOCAL->fullmsg_len = 0;
+   LOCAL->fullmsg_elt = NIL;
+   _buf_ = LOCAL->buf;
+   _buflen_ = LOCAL->buflen;
+   LOCAL->buflen = (flags & FT_INTERNAL) ? elt->private.msg.text.text.size :
+         elt->rfc822_size;
+   LOCAL->buf = fs_get( LOCAL->buflen+1 );
                                /* go to text position */
    lseek (LOCAL->fd,elt->private.special.offset +
         elt->private.msg.text.offset,L_SET);
***************
*** 602,608 ****
      *s = '\0';                        /* tie off buffer */
      *length = s - LOCAL->buf; /* calculate length */
    }
!   return LOCAL->buf;
  }

  /* UNIX per-message modify flag
--- 619,631 ----
      *s = '\0';                        /* tie off buffer */
      *length = s - LOCAL->buf; /* calculate length */
    }
!   LOCAL->fullmsg = LOCAL->buf;
!   LOCAL->fullmsg_len = *length;
!   LOCAL->fullmsg_elt = elt;
!   LOCAL->fullmsg_flags = flags & FT_INTERNAL;
!   LOCAL->buf = _buf_;
!   LOCAL->buflen = _buflen_;
!   return LOCAL->fullmsg;
  }

  /* UNIX per-message modify flag
***************
*** 1009,1014 ****
--- 1032,1038 ----
                                /* free local text buffers */
      if (LOCAL->buf) fs_give ((void **) &LOCAL->buf);
      if (LOCAL->line) fs_give ((void **) &LOCAL->line);
+     if (LOCAL->fullmsg) fs_give ((void **) &LOCAL->fullmsg);
                                /* nuke the local data */
      fs_give ((void **) &stream->local);
      stream->dtb = NIL;                /* log out the DTB */

*** unix.h.orig Wed Oct 25 08:41:26 2000
--- unix.h      Tue Dec 16 12:36:08 2003
***************
*** 168,173 ****
--- 168,177 ----
    char *buf;                  /* temporary buffer */
    unsigned long buflen;               /* current size of temporary buffer */
    char *line;                 /* returned line */
+   char *fullmsg;
+   unsigned long fullmsg_len;
+   void *fullmsg_elt;
+   long  fullmsg_flags;
  } UNIXLOCAL;

Choi, Sung-hoon 

DreamWiz Inc. - Development Team/Manager 

Major : Mail/Web/Network/System/Security 
Phone : +82-2-3434-3541 
MSN   : [email-masked] 
WWW   : http://my.dreamwiz.com/shoon/ 
E-mail: [email-masked]

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 16 Dec 2003 16:43:54 -0800 (Pacific Standard Time)

Normally, this is not an issue since the buffer cache prevents re-reading.
However, the next snapshot of imap-2003 (the code is being tested now)
will cache the most recently read message body in the driver.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.