Subject: [Imap-uw] help with INTERNALDATE weirdness?

From: "Jim O'Leary" <[email-masked]>
Date: Fri, 26 Aug 2005 12:19:02 -0500

Our Outlook users (few, thank goodness) always complain that the 
timestamps in their inboxes are showing the wrong GMT offset.  We've 
worked around it for a couple of years by telling them to reconfigure 
Outlook to display the "Sent Date" instead of the "Received Date", 
assuming Outlook was misinterpreting some SMTP header or the RFC822 
"Date:" field.

Recently a student employee pointed out that the problem may be that 
our UW-IMAP server is returning incorrect information for the 
INTERNALDATE flag.  I've pasted an example conversation below.  The 
mail was actually sent at "10:56:00 -0500" today, but the 
INTERNALDATE response shows "15:56:46 -0500" which would be GMT if 
not for the bogus   offset.

We admittedly are running imap2002e and will upgrade immediately if 
someone tells us this is a bug that has been fixed.  The only other 
point of interest is that we use qmail/procmail as the delivery 
agent.  Any help is appreciated.

-- cut here --

00 select inbox
* 1 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1125071880] UID validity status
* OK [UIDNEXT 2] Predicted next UID
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] 
Permanent flags
* OK [UNSEEN 1] first unseen message in /var/mail/joleary
00 OK [READ-WRITE] SELECT completed
uid fetch 1 (uid flags rfc822.header)
* 1 FETCH (UID 1 FLAGS (\Recent) RFC822.HEADER {922}
(128.248.155.180)
   by icarus.cc.uic.edu with SMTP; 26 Aug 2005 10:56:46 -0500
Received: from jimo.uic.edu (jimo.cc.uic.edu [128.248.5.52])
         (authenticated bits=0)
         by webmail-1.priv.cc.uic.edu (8.12.10/8.12.10) with ESMTP id 
j7QFujvK008493
         (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
         for <[email-masked]>; Fri, 26 Aug 2005 10:56:45 -0500
Message-Id: <[email-masked]>
X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.0
Date: Fri, 26 Aug 2005 10:56:45 -0500
To: [email-masked]
From: "Jim O'Leary" <[email-masked]>
Subject: test
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed

)
uid OK FETCH completed
00 uid fetch 1 (flags internaldate)
* 1 FETCH (UID 1 FLAGS (\Recent) INTERNALDATE "26-Aug-2005 15:56:46 -0500")
00 OK UID FETCH completed

Jim O'Leary
University of Illinois at Chicago
Academic Computing and Communications Center
(312) 996-4643
www.accc.uic.edu 

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Fri, 26 Aug 2005 10:57:15 -0700 (PDT)

What mailbox format are you using?

One very obvious way that this problem could be caused is if your system 
uses traditional UNIX mailbox format with a delivery program that writes a 
"From " header in UTC without a timezone, e.g.,
 	From fred Fri Aug 26 15:56:46 2005

Since the timezone is missing, UW imapd's parsing code assumes that the 
time is in the local timezone.  On every UNIX system that I've seen so 
far, that assumption would be correct.

Are you delivering directly from procmail?  Or, is procmail calling some 
program such as /bin/mail, /bin/mail.local, etc. for final delivery?

There are three ways that you can get around it.

The first is to replace the delivery program with a program (such as the 
UW IMAP toolkit's tmail or dmail) that writes POSIX-style dates that 
include a timezone, e.g.,
 	From fred Fri Aug 26 10:56:46 2005 -0500

Although this is what I would recommend, a problem with doing this is that 
there are some ancient UNIX programs which choke on POSIX-style dates.

Alternatively, you can change the traditional UNIX mailbox format's 
default handling of timezone by editing imap-????/src/osdep/unix/unix.c to 
change the string "LCL" to be "GMT" in the statement:
 				/* zzz */
 	t = zn ? (t + zn + 1) : (unsigned char *) "LCL";

Finally, you can look into why your delivery program is writing UTC 
date/time in the From without a timestamp, and get it to either write the 
date/time in local time or to write a POSIX-style date with a timestamp.

One last thing...

Since you mentioned qmail; if you use maildir format with a third-party 
maildir driver in UW imapd, then this is probably a bug in that driver. 
I can't say for sure what's wrong (I don't pretend to support maildir), 
but the following may help:

You need to find out how it determines the internaldate.  If it sets the 
internaldate from the file mtime, make sure that it does something like:
       struct tm *tm = gmtime (&sbuf.st_mtime);
       elt->day = tm->tm_mday; elt->month = tm->tm_mon + 1;
       elt->year = tm->tm_year + 1900 - BASEYEAR;
       elt->hours = tm->tm_hour; elt->minutes = tm->tm_min;
       elt->seconds = tm->tm_sec;
       elt->zhours = 0; elt->zminutes = 0;

The setting of zhours and zminutes to zero makes it be UTC.

If the maildir driver uses localtime(), make sure that it calculates the 
timezone offset using a delta from the gmtime() value and *NOT* the 
timezone values in the returned struct tm.  Look at phile.c to see how 
this is done.

The reason that you have to do this is that localtime() depends upon 
environment that isn't always completely set up for daemons; thus you can 
get localtime() returning a struct tm with the correct timezone offset but 
the time being UTC!  Calculating the delta with gmtime() avoids this.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: "Jim O'Leary" <[email-masked]>
Date: Fri, 26 Aug 2005 14:03:16 -0500

>What mailbox format are you using?

mbx

>One very obvious way that this problem could be caused is if your 
>system uses traditional UNIX mailbox format with a delivery program 
>that writes a "From " header in UTC without a timezone, e.g.,

You called it Mark.  That is the case!

>Since the timezone is missing, UW imapd's parsing code assumes that 
>the time is in the local timezone.  On every UNIX system that I've 
>seen so far, that assumption would be correct.
>
>Are you delivering directly from procmail?  Or, is procmail calling 
>some program such as /bin/mail, /bin/mail.local, etc. for final delivery?

qmail hands it off to procmail-3.10, which does the final mbox 
delivery.  This makes me wonder why I can't find any other references 
to such a problem.

>The first is to replace the delivery program with a program (such as 
>the UW IMAP toolkit's tmail or dmail) that writes POSIX-style dates 
>that include a timezone, e.g.,
>Although this is what I would recommend, a problem with doing this 
>is that there are some ancient UNIX programs which choke on POSIX-style dates.
>
>default handling of timezone by editing 
>imap-????/src/osdep/unix/unix.c to change the string "LCL" to be 
>"GMT" in the statement:
>                                 /* zzz */
>date/time in the From without a timestamp, and get it to either 
>write the date/time in local time or to write a POSIX-style date 
>with a timestamp.

Thanks for the choices and the fast response.  We'll discuss these 
and make a choice.

>One last thing...
>
>Since you mentioned qmail; if you use maildir format with a 
>third-party maildir driver in UW imapd, then this is probably a bug 
>in that driver. I can't say for sure what's wrong (I don't pretend 
>to support maildir), but the following may help:

no maildir but thanks again for taking the time to spell it all out 
so quickly.

>You need to find out how it determines the internaldate.  If it sets 
>the internaldate from the file mtime, make sure that it does something like:
>If the maildir driver uses localtime(), make sure that it calculates 
>the timezone offset using a delta from the gmtime() value and *NOT* 
>the timezone values in the returned struct tm.  Look at phile.c to 
>see how this is done.
>
>environment that isn't always completely set up for daemons; thus 
>you can get localtime() returning a struct tm with the correct 
>timezone offset but the time being UTC!  Calculating the delta with 
>gmtime() avoids this.
>
>-- Mark --
>
>http://panda.com/mrc

Jim O'Leary