Subject: Multiple imapd connections and message handling

From: Carl Stehle <[email-masked]>
Date: Thu, 12 Jun 2003 13:09:45 -0700

We are experiencing problems with expunge and message flags with
a certain email client when it performs a specific sequence of
operations with imapd.

The problem with expunge occurs when the email client chooses to 
create a separate connection to imapd in addition to one already
existing and in an IDLE state. The second connection is used for
moving a message (from INBOX to a local mailbox) and marking the
message for deletion. The original connection is then used for 
expunge. 
i.e.
Connection 1: opened, ..., put in IDLE state

Connection 2: opened, message moved from the INBOX to a local
              mailbox, message marked for deletion, closed

Connection 1: taken out of IDLE state, messages expunged, ...

The message expunge has no effect and, so far as we can tell,
the reason is that the file size (of INBOX) does not change when
the message is marked for deletion (setting the deleted flag
increases the file size by 1 byte, but this is compensated by 
a reduction of one blank space in the X-Keywords line due to a
change in padding). Therefore, the connection doing the expunge
does not know about the flag change, and assumes no changes were
made to the (INBOX) file.

The (related) problem with the message flags occurs if message
flags were changed prior to moving the message (e.g. if a 'recent'
message becomes 'seen' during Connection 1). In that case, the 
file will be rewritten after the expunge (by Connection 1),
overwriting the flags set by the message move (Connection 2).

This situation does not occur when a single connection is used
to read, mark for delete, and expunge messages since the internal 
state information of that connection is always used to rewrite
the file.

We are running imapd (2002.332) on Windows, and using traditional
UNIX mailbox format for INBOX and would appreciate any 
recommendations on how to best handle this situation in imapd.
Forcibly re-parsing the mailbox (INBOX) would probably work,
but I imagine there is a very good reason for the file size check
(optimization) and do not want to break it.

TIA

-- 
Carl Stehle
-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Thu, 12 Jun 2003 15:44:07 -0700 (Pacific Daylight Time)

Whoa, pardner!

Towards the end of your message, you said that you were using the
traditional UNIX mailbox format.  Shared access is *NOT* supported in the
traditional UNIX mailbox format.

In your example, Connection 1 *should* terminate the moment Connection 2
opened the mailbox.  There are reasons why this might not happen.  I am
concerned by your report of "seen" status being revoked that this might be
happening.

However, I noticed your comment to the effect that you "are running imapd
on Windows".  Do you really mean that the IMAP server itself is running on
Windows, and you are using some sort of front end such as inetlisn?

If you really are running imapd on Windows, note that imapd is *NOT*
supported on Windows, and usage of traditional UNIX mailbox format (as
opposed to mbx format) is *especially* not supported in imapd on Windows.

I strongly suggest that you give serious consideration to running imapd on
UNIX *and* using the mbx format which supports shared access.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Carl Stehle <[email-masked]>
Date: Tue, 17 Jun 2003 14:59:01 -0700

We were not very clear in describing the IMAP transactions that led
up to the problem with message flags we reported. From your comments,
related problems that have been reported before, and further testing
we found as follows:

1. The situation on non-NT versions of Windows, where flock() is a
no-op, should be expected. In this case, the INBOX (Unix format)
sharing (locking) protections of imapd do not apply and the email
client is given read-write access on both connections. Since INBOX
is non-sharable, data integrity is not guaranteed for writes.

2. On NT versions of Windows (where flock() is implemented), the
end results are similar but for a different reason. While Connection
1 is open and in IDLE state, Connection 2 is used to 'SELECT INBOX'. 
The response includes 'PERMANENTFLAGS ()' and several indications of
READ-ONLY state, but the client ignores these and sets the flags
anyway. Of course, the flags are reset after Connection 2 terminates,
(since PERMANENTFLAGS was null) as per IMAP specification.

Presumably this does not happen on UNIX as most people (wisely)
choose to use mbx format and locking is always implemented, making
the INBOX file fully sharable. Hence a READ-ONLY INBOX never exists.
In fact, we verified that message flags are treated correctly when
moving messages between mbx folders.

As for support, it is ironic that when the 'unsupported' software
is *used* correctly, the so-called 'supported' software is where the
problem lies.

-- 
Carl Stehle

---

From: Mark Crispin <[email-masked]>
Date: Tue, 17 Jun 2003 15:18:47 -0700 (Pacific Daylight Time)

You're running imapd on NT (using something like inetlisn), right?

Is there a reason why you aren't running imapd on UNIX?

Is there a reason why you choose to use the traditional UNIX mailbox
format on NT instead of mbx?  mbx is the default in the NT build.

> is non-sharable, data integrity is not guaranteed for writes.

Why would you be running an IMAP server on a non-NT version of Windows?
Win95/98/Me are dead, and the corpse has started to stink.  Although I can
understand diehards continuing to run it on the desktop, I can't imagine
why people would use it as a server (particularly an IMAP server!)

> (since PERMANENTFLAGS was null) as per IMAP specification.

Yes, there is no "kiss of death" feature in the traditional UNIX mailbox
driver for Windows (both NT and non-NT).  The assumption is that anyone
using traditional UNIX format on the PC is doing so as a desktop and not
as a server

> the INBOX file fully sharable.

On UNIX, the older session having a traditional UNIX format file is
killed.

> is *used* correctly, the so-called 'supported' software is where the

I don't understand what you mean by this.

-- Mark --

---

From: Carl Stehle <[email-masked]>
Date: Tue, 17 Jun 2003 17:03:52 -0700

> You're running imapd on NT (using something like inetlisn), right?

Yes, we are running imapd on NT, and other versions of Windows and have
done so from W95 forward. We are using a highly modified version of
inetlisn as a front-end.

> Is there a reason why you aren't running imapd on UNIX?

Yes, not everyone has UNIX.

> format on NT instead of mbx?  mbx is the default in the NT build.

The reason we chose UNIX format is purely due to format simplicity
(for SMTP mail delivery). I understand that mbx has better share
characteristics and we will likely move to it (or write a DB driver).

> why people would use it as a server (particularly an IMAP server!)

I did not say I recommended this, but imapd does run there (with caveats).

> On UNIX, the older session having a traditional UNIX format file is

Thanks for the clarifications.

> I don't understand what you mean by this.

I had interpreted your statement that imapd was 'unsupported' as a
server on Windows to mean 'it does not work'. And by 'used correctly',
I meant used with NT (not Win9x). Given that, and the fact that
a somewhat obscure problem like this is not a problem with imapd, but
a problem in an extremely popular email client just makes me wonder
about why imapd should not be running as a server on NT; it seems
to work well enough.

-- 
Carl Stehle

---

From: Mark Crispin <[email-masked]>
Date: Tue, 17 Jun 2003 17:28:25 -0700 (Pacific Daylight Time)

> done so from W95 forward. We are using a highly modified version of

Let's just say that I *am* impressed.

> Yes, not everyone has UNIX.

Of course, that isn't a particular barrier; you can get a bare-bones Linux
system from Wal-Mart for $200.

> characteristics and we will likely move to it (or write a DB driver).

I recommend that you take that step.  Besides having read/write sharing,
mbx will also perform *much* better.

> Thanks for the clarifications.

If you search in the UNIX code for "kiss of death" you'll see how it
works.  I guess that with some effort, you can figure out how to send a
signal from one imapd to the other and accomplish the same behavior as on
UNIX.  I never thought that anyone would need such a thing.

-- Mark --