Subject: implementation of UIDs in c-client

From: Mark Crispin <[email-masked]>
Date: Thu, 22 Sep 1994 21:36:40 -0700 (PDT)

I've been thinking about the implementation of UIDs in c-client.  This is
clearly the single largest task in updating c-client to IMAP4, and thus is the
one I am going to undertake first.

I would like comments about the plans I have outlined below for each of the
different formats.

One problem is that UIDs are unavailable in some formats.  I think that it is
going to be necessary to have a default value of zero meaning ``no UID
available'' from c-client.  I'm not sure what the c-client based imapd is
supposed to do.  Should it return an error for FETCH UID (but doesn't this
violate IMAP4?)?  Should it return the zero (and perhaps we need to say that
``zero means unavailable or indeterminate'' to the IMAP4 spec?)?

Another matter has to do with the assignment of UIDs.  There are two basic
possibilities.

One is to assign a UID based upon the current time-of-day, with some nasty bit
of fuzz to make sure that we don't duplicate UIDs or spend an hour assigning
UIDs for a 3600 message mailbox.  The disadvantage to this is that the UIDs
will tend to be very large.

The other is to have a resource that assigns UIDs, essentially a uid = i++;
This can be per-folder, per-user, or systemwide.  The problem here is, where
do you store the resource so it doesn't get stepped on?  Ideally, it should be
stored in the folder, but none of the current folder formats have a provision
for doing so.  If you store it in a file, then it is vulnerable to being lost
or destroyed if the file is stepped on and/or the folders are moved.  Perhaps
the current high UID can be stored as a hidden first message.

In the current gaggle of bundled c-client drivers, here's the score:

1) bezerk/mmdf/mbox format

There are two major strategies that I can think of:
 a) stash the UID in the existing X-Status: or Status: header
 b) write the UID in a new X-UID: header

The main issue here is what happens if an old, UID-clueless c-client (or other
software) gets ahold of a mailbox with UIDs.  Most likely, it will toss out
the UIDs.  This could be a terrible bug or a wonderful feature, depending upon
how you look at it.  If it was stored in X-Status or (especially) Status, the
user can't shoot himself in the foot by accidentally using UID-clueless
software to move messages around in different folders, writing now-bogus UIDs.

2) tenex/mtx format

The quick and dirty method would be to use the 30 high order bits in the flags
field of the header, and perhaps the last unused system bit to indicate
``valid UID'' to work around the fact that the little-used keyflags feature
uses those bits for named flags.

The problems with this are that this would irritate the folks who use keyflags
(yours truly included).  Although the c-client community does not use keyflags
much, it is very extensively entrenched in the mm community, having been
inherited from MM on the DEC-20.

The other possibility is to add a new field to the header.  I'm thinking of a
hyphen followed by the UID.  The current version of c-client will barf if
there is anything after the flags field of the header.  That error test can be
removed now and distributed in Pine 3.91 and IMAP toolkit 3.5, so hopefully by
the time a UID version of c-client and Pine are out, a UID-tolerant older
version would have had time to propagate.  The change to make older versions
of c-client UID-tolerant (assuming UIDs are done here) is trivial, and can be
distributed as a source patch.

A third possibility is to write the data on the next line.  I really don't
like this; although it makes compatibility with the past easier, it's kludgy
and has other bad effects.

I'm not sure how much I should worry about mm.  mm (or at least the copy I
have on my system) trashes the internal dates of c-client written mail.txt
files; it doesn't know about 4-digit years or numeric timezones.  mm ignores
anything after the flags field of the header, and will not write it out when
rewriting the mailbox (hence it will trash UIDs).  We could perhaps nag the
owners of mm to fix this.

The other issue here is adding UIDs to folders that don't have them.  Unlike
bezerk/mbox/mmdf format, it isn't routine to rewrite tenex/mtx folders.  So we
can either bite the bullet and rewrite to add the UID fields, or we can say
that there are no UID fields and you have to rewrite the folder by hand (e.g.
using a tool such as mbxcopy) to get UID capability.  I lean towards the
latter, especially if we do decide that it's alright to have zero for ``no UID
available''.

3) mh format

This should be easy since the format is very much like news.  Unfortunately,
it is not, because the charming mh system has a facility that renumbers the
message files to be consecutive.  The c-client mh driver will never do this,
though.

It isn't clear to me what the right thing is.  My inclination is to go with
what is fast and just say that if the guy renumbers his messages with mh, he
will screw up his UIDs for disconnected clients.  I don't think it is a good
idea to do something (considerably) more expensive just to keep an mh user
from shooting himself in the foot.

4) news/nntp format

This is easy; I will use the news message numbers (as opposed to the c-client
message numbers) since these are immutable.

5) phile format (raw files)

Not clear what's best here, but I've been thinking about using the file
modification time.  This is better than the device/inode number (my initial
idea) because that wouldn't catch appends or overwrites.  I doubt that UIDs
are really going to matter much here.

6) pop3 format

Use UIDL if the server offers it, otherwise return zero for UIDs.

7) imap format

Use FETCH UID if IMAP4, otherwise return zero for UIDs.

---

From: "Steven D. Majewski" <[email-masked]>
Date: Fri, 23 Sep 1994 11:46:27 -0400 (EDT)

What is the scope of a UID in time and space ? 

This point came up when we were discussing imap: URL's. 
John Gardiner Myers stated his point of view for CMU's 
implementation, and what would be required for IMSP, but
I don't think there was a stated consensus on what IMAP
should require.  And my reading of the IMAP4 draft, at
the time, was that it's only *required* to be unique for
that folder, and if you want to get legalistic about it,
for that implementation. ( i.e. an IMAP folder is a logical
entity that maps to a physical store of some sort, and changing
IMAP implementations could be considered as implicitly 
creating new IMAP mailboxes. ) 

The IMAP4 draft merely says that they "persist across sessions."

* Are UID's preserved when mail is moved to another mailbox ? 

- NO: "Unique identifiers are assigned in a strictly ascending fashion 
for each message added to the mailbox." 

Are UID's preserved when changing IMAP implementation ? 

- ??? From memory: I think that John said it was his intention 
  that they should, but we both agreed that some of this problem
  would be punted into the IMSP domain. [ I'll have to dig out 
  that thread. I may be misremembering his statements. ] 

And in fact. looking at the draft again, I see that "unique for 
the lifetime of that folder" was *MY* rewording of what I *thought*
was the meaning of "persist across sessions". In fact, that language
is loose enough to allow intentional reassignment if UID's: IMAP 
doesn't define a way to do that, but IMSP or some other facility 
*COULD*. ( In common CS usage - "persistent" things CAN be intentionally 
deleted or modified - they just don't "evaporate" when you let go 
of them for a while. ) 

I would *like* the scope to be defined as widely as is practical.
[ What's practical? ]

Reading your (Mark's) note, I'm not sure if I still agree 
with that position. But I'm glad you brought it up! 

-- IAC DONT RANDOMLY-LOOSE ;-) 

-- Steve Majewski       (804-982-0831)      <[email-masked]> --
-- UVA Department of Molecular Physiology and Biological Physics --
-- Box 449 Health Science Center        Charlottesville,VA 22908 --
		 [ "Cheese is more macho?" ]

---

From: John Gardiner Myers <[email-masked]>
Date: Fri, 23 Sep 1994 11:59:56 -0400 (EDT)

I don't like the "zero uid" approach, it unnecessarily complicates
clients.  If nothing else works, you can assign time-based UID's,
making it look to the disconnected client like all the previous
messages got expunged and a whole bunch more got inserted.  A
time-based UID would be (time-of-connection*N)+message-number.  N
could be 1, or some arbitrary higher number if you're worried about a
second connection within the same number of seconds as there are
messages in the folder.

>  b) write the UID in a new X-UID: header

I would tend to suggest (b).  To protect against UID-clueless
software, I would put in a check to ensure the UID's are in strictly
ascendending order, reassigning UIDs as necessary.

> 3) mh format
> 
> My inclination is to go with
> will screw up his UIDs for disconnected clients.

This probably isn't too bad a plan.  Can you detect when a mh folder
has been renumbered?  You could keep a file of the mh-number to UID
mapping (or mappings) if the effort turns out to be worth it.

You have to make sure that when you expunge the last message in the
folder, that UID doesn't get reassigned to the next incoming message.

> modification time.

Sounds right.

> Use UIDL if the server offers it, otherwise return zero for UIDs.

Use UIDL if the server offers it and the pop-uids are both numeric and
strictly ascending.  Time-based UIDs otherwise.

-- 
_.John G. Myers		Internet: [email-masked]
			LoseNet:  ...!seismo!ihnp4!wiscvm.wisc.edu!give!up

---

From: Chris Newman <[email-masked]>
Date: Fri, 23 Sep 1994 14:23:49 -0400 (EDT)

Given the wording in the draft that UIDs must be "strictly ascending"
within a mailbox, it is quite clear that returning UID 0 for every
message in the folder is not currently legal.  I also think it would
be a bad idea to make it legal, because it would break disconnected
clients in unacceptable ways.

I think it _is_ legal to say NO to a FETCH UID, SEARCH UID, and the
UID commands if UIDs are unavailable.  A client written to the
specification must be able to accept NO responses to these commands.
(For that matter, an IMAP server which said NO or BAD to every command
other than CAPABILITY, NOOP, and LOGOUT would be a legal
implementation -- just not a useful one).

My suggestion would be that when the specification moves to draft
status, we add a special information token "[TRYSEQUENCE]" indicating
that UIDs aren't available, but message sequence numbers will work.
This would be required in the NO response to UID requests within a
mailstore technology that doesn't support UIDs.  In addition, a server
could optionally send it in an unsolicited OK in response to a SELECT
of such a mailstore.

		- Chris