Subject: Re: message state preservation in COPY and APPEND operations

From: [email-masked] (Michael S Shappe)
Date: Mon, 12 Jul 1993 13:06:29 -0400

At 17.20 930711 -0400, Mark Crispin wrote:
>Should c-client attempt to preserve message status when copying messages to
>other folders?

I think that status flags should be preserved. If I move a message to
another mailbox, that does not mean I'm finished with it; therefore,
knowing for certain whether or not I've already responded (for example) to
a message is useful information that would be best kepts.

It seems to me that user flags could theoretically be preserved in other
formats as well -- for example, as an X- header in bezerkly format...

---

From: John Gardiner Myers <[email-masked]>
Date: Mon, 12 Jul 1993 16:04:03 -0400 (EDT)

We could easily extend the APPEND command to accept an optional

FLAGS flag_list

after the mailbox and string arguments.  Clients would have to do
fallback on BAD, of course.

				_.John

---

From: Adam Treister <[email-masked]>
Date: Mon, 12 Jul 93 16:24:30 -0800

Just to chime in quickly, I think the status of flags needs to be preserved
across transfer of messages.  I can see that its messy to implement, but
its "the right thing" as far as the user is concerned. If the user has set
keywords, they need to be preserved. 

I could take it so far as to say that a client may want to be able to add
status information to the header in the process of doing the move.  Imagine
an agent which moves mail from the inbox to a folder without the user
actually knowing.  It may be beneficial if the agent adds a X-Moved-Because
header in the process to include the rule that caused the action.  (This
may be a bit futuristic, but was the first example that came to mind.  I'm
sure there are more mundane examples.)

It sure seems to me that there is a lot of redundancy between Move, Copy,
and Append, and maybe preservation of attributes could be a differentation
among these commands. (The implication is that Copy is creating a new
message, which may not have attributes set, but Move should not change the
message in the process)

Adam
--------------------------------------------------
Adam Treister  <[email-masked]>
Polya Hall 205, Stanford CA 94305 - (415) 725-9449
--------------------------------------------------

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 12 Jul 1993 16:39:37 -0700 (PDT)

Actually I don't think preserving flags on a COPY is always the right 
thing to do. For example is someone saves a messages with the \DELETE 
flag on, the flag should probably not be copied. The user probably is 
saving the message from the next expunge. 

The best solution would, I believe, enable you to set the flags as you wish
on all the operations without having to actually open the mailbox. That
is, the behavior should be left up to the user interface, and not imposed 
by IMAP.

LL

---

From: Mark Crispin <[email-masked]>
Date: Mon, 12 Jul 1993 21:51:38 -0700 (PDT)

> is, the behavior should be left up to the user interface, and not imposed

This, I think, is the crux of the problem.

Anything that I do in c-client/IMAP is imposed on the user interfaces, even
when that choice may be considered wrong by the user interface.  I think that
is worse than the error of omission, which in this case is simply not to copy
the user flags in any message copying operation and let the user interface
decide which flags, if any, it wishes to be preserved.

However, assuming that it is decided that IMAP should preserve the message
state.  Let's also assume that we have settled on the following:
 1) internal date is NOT copied
 2) user flags (keywords) are NOT copied
 3) system flags are copied

Then that leaves us with the problem of APPEND.  We can extend APPEND to have
a flags argument.  However, that leaves the question of what to do when the
server returns BAD because it's a server written for the previous spec.

I consider it to be of *utmost* importance to have consistant behavior across
all variants of IMAP.  Differences in version should be differences in
functionality, not differences in fundamental behavior.

If we have a case in which APPEND may not preserve the flags in certain cases
but would in others, then the user interface is going to have to have code to
cover both of these cases, and to do something manually to fix things up.  Or
the user will have to be told ``that's just tough, sometimes your flags won't
be preserved, and there's no way of telling when that will happen.''

I don't consider that to be a desirable situation.

Can anyone give a scenario in which consistent behavior is presented to the
user?  I feel that dropping flags is a minor inconvenience compared to having
totally random behavior.

Please, when you answer this, don't try to convince me that saving flags would
be a nice thing to do.  We're all in agreement about this (I think).  Try to
convince me that we can solve the technical problem of inconsistant behavior.

-- Mark --

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 13 Jul 1993 10:57:32 -0700 (PDT)

Well, I don't have any great technical solutions for you, but here's some 
other suggestions.

OK, I understand the problem better now. For APPEND I was thinking that it
was recent enough and not so widely implemented that we might be able to
afford an incompatible change.  Stated another way, I was thinking that the
amount of inconsistent behavior experienced by users would be small when
considering that IMAP (esp APPEND) is at the beginning of it's life now. I
infer that Mark doesn't agree with this.  I'd like to hear what other
folks on this list think. 

On COPY, I believe this was under-specifed in RFC-1176 so it's not clear
whether the correct behavior is to copy flags or not.  If that's the case,
then we may have random behavior amongst the existing servers right now
and nothing to loose.  I suggest that we tighten the spec now as Mark
recently suggested. (Note: A user program can set the flags on the source
message, copy the message, then reset the flags to get the destination
flags set as desired without too much trouble and the overhead of opening
the destination mailbox). 

LL

---

From: Mike Macgirvin <[email-masked]>
Date: Tue, 13 Jul 1993 13:03:49 -0700 (PDT)

>   OK, I understand the problem better now. For APPEND I was
>  thinking that it was recent enough and not so widely
>  implemented that we might be able to afford an incompatible
>  change.  Stated another way, I was thinking that the amount of
>  inconsistent behavior experienced by users would be small when considering
>  that IMAP (esp APPEND) is at the beginning of it's life now. I infer
>  that Mark doesn't agree with this.  I'd like to hear what
>  other folks on this list think.
 
	IMAP has been around for quite a long time. 

	The changes not too long ago regarding CREATE and it's relation to 
MOVE/COPY already broke compatibility on clients, unless you pull some real 
hacks to parse telemetry messages and use that to determine behaviour. This 
makes the exact text of the error messages an (unofficial) part of the 
protocol. If behaviour is changed for APPEND, let's at least use the same back 
door for negotiating differences. I agree, there probably aren't many APPEND 
implementations out there, but there might be a few. If it is to be changed, 
the sooner, the better; while the count of affected clients is still reasonably
low....

mike

---

From: John Gardiner Myers <[email-masked]>
Date: Thu, 15 Jul 1993 22:48:29 -0400 (EDT)

> server returns BAD because it's a server written for the previous spec.

The client then has to fall back to using APPEND without the flags
argument.

The message will be inserted without the flags set, but this is then
the same situation you have when you do a COPY on a server written for
the "don't copy flags" interpretation of the previous spec.

> functionality, not differences in fundamental behavior.

I'm not quite sure why the inability to preserve flags is a difference
in "fundamental behavior" instead of a difference in "functionality."

As it is now, you have inconsistent behavior WITHIN a given version of
c-client.  Whether or not you can even use user-defined flags varies
between mailbox to mailbox, depending on what the underlying storage
format is.

Even if we do decide that flags should be preserved, there are some
cases where servers that support fine-grained access control will want
to fail to preserve them.  If a user is allowed to COPY/APPEND a
message into a folder, yet is not allowed to do a SET FLAGS on that
folder, the server should not preserve the flags on the inserted
message.

-- 
_.John G. Myers		Internet: [email-masked]
			LoseNet:  ...!seismo!ihnp4!wiscvm.wisc.edu!give!up

---

From: Peter Svanberg <[email-masked]>
Date: Mon, 23 Aug 1993 17:41:04 +0200

There are in my opinion two time stamps which could be
interesting for an IMAP client user to use:

* send date - the time and date when the author of the
  message sent it; i.e. the "Date" field in an 822 message

Why: As a search constraint when a send date is known. If a
message B was sent before a message A, B can't be a comment to
A, for example.

* arrival date - the date and time when the message was
  delivered to me, i.e. to the IMAP server

Why: As a search constraint when a read date is known. If a
user knows to have read a certain message at time T (or later),
then that message must have arrived before T.

My view of a typical usage of the mailbox concept in IMAP - for
personal correspondence - is that the user gets all incoming
mail to INBOX, reads them, deletes some and stores others in
different mailboxes (as the folders in MH). This storage is
done occasionally.

RFC 1176 (and the current IMAP2bis draft) specifies:
>      INTERNALDATE    The date and time the message was written to
>                      the mailbox.

Some persons on this list has argued that this must be
interpreted as what could be called the "storage time" in the
above scenario, i.e. at what moment the user happen to take
time to tidy up in her INBOX.

I don't see why this time stamp would ever be interesting to
know, at least not more important than the other two dates.
Perhaps there are such examples, in other scenarios. Could
anyone tell me?

I'm not aware of the history of the INTERNALDATE specification
but in my opinion you _could_ interpret it as meaning arrival
date. (Or at least that this was what the specifier had in
mind, but expressed it vague.)

My suggestion is that INTERNALDATE is clarified in IMAP2bis
to mean arrival date. I don't know how this is best specified,
perhaps:

      INTERNALDATE    The date and time this message was
		      delivered to a mailbox on this IMAP
		      server (via non-IMAP services)

Would this give any compatibility problems? As servers have
done differently over time, maybe not?

My description above also implies that searching on send date
is definitely important. Has this been considered?
---
Peter Svanberg, NADA, KTH		    Email: [email-masked]
Dept of Num Analysis and Comp. Science,
Royal Institute of Technology		    Phone: +46 8 790 71 40
S-100 44  Stockholm, SWEDEN		    Fax:   +46 8 790 09 30

---

From: "David Herron" <[email-masked]>
Date: Mon, 23 Aug 93 9:59:39 PDT

>From: Peter Svanberg <[email-masked]>
>

er... that's assuming the system clocks at A and B are accurate with
respect to one another.

>then that message must have arrived before T.

Again relying on accurate system clocks.  Also mail may get delayed by
random amounts in transmission even if one is well connected to the network.

<- David Herron <[email-masked]> (work) <[email-masked]> (home)
<-
<- All hard work brings a profit, but mere talk leads only to poverty.
<-               Proverbs 14:23

---

From: Mark Crispin <[email-masked]>
Date: Mon, 23 Aug 1993 13:10:03 -0700 (PDT)

Thank you for your comments.

On most UNIX-based systems (and on the TOPS-20 system that preceeded it), the
``date/time the message was written to the mailbox'' is an easily available
piece of information.  On UNIX, this is the date/time in the ``From '' header
in mbox-format mailboxes.  In mh-format, it's the file write time.

This information is generally quite readily available at the time the mailbox
is scanned to find messages, making it ideal for any sort of fast date/time
dependent activities.  One problem which comes up with parsing dates is the
various formats which dates may be.  UNIX mbox-format has 20 (possibly 28)
different formats of ``From '' line, mostly variations in the date/time
format, of which 4 or 5 are in common usage.

But that's the easy part.

The much harder problem is processing the date/times that appear in Date:
header lines.  If everyone would follow the rules in RFC 822 it would be easy;
unfortunately many people do not.  So, on top of having to find the Date:
header line (which involves actually doing an RFC 822 header parse), you have
to worry about doing something reasonable with non-conforming Date headers.

It would be attractive if there was some type of searching based upon the
Date: header line date (what you call the ``send date'').  There are three
issues here:
 1) reliably understanding the Date: as noted above.
 2) getting the Date: quickly as noted above.
 3) modifications to SEARCH

It was the first two problems that caused a punt on the ``send date'' back in
1986.  The third problem is introduced by the necessity of compatibility with
the past.  If you specify a search that uses the ``send date'', what do you do
with a server which has not yet implemented that set of extensions.

This is a Pandora's box, since other excellent suggestions have been made for
extensions to SEARCH.  I am of the opinion that incrementally extending SEARCH
is a terrible idea because it will create a plethora of different levels of
searching capability.  Rather, it would be better to preserve the current
level of searching as a basic level, and then define a new extended level that
gets a complete set of new capabilities.  There is some IMSP interaction here
that should be considered too.

The other two problems pale by comparison, I think.  If someone insists upon
sending bogons like:
	Date: 5/3/93 1:23 BST
the software cannot be blamed for whatever interpretation it may make (May 3
or March 5?  1:23 AM or PM?  British Summer Time or Bering Standard Time?).
The bogon date problem has ameliorated quite a bit since 1986 though.

Searching based upon the Date: header will always, I suspect, be quite a bit
slower than the internal date, but educated users would be aware of this.

---

From: Peter Svanberg <[email-masked]>
Date: Tue, 24 Aug 1993 13:19:05 +0200

Quoting:  Mark Crispin <[email-masked]>
> Thank you for your comments.

Hmm, the presentation part of my message got lost:

    I am a Unix and Mac systems administrator and programmer
    and (as such) a heavy email user, having lots of opinions
    on how email usage could be more effective and nice. I and
    my colleague Olle Jarnefors have discussed some aspects of
    the IMAP2bis protocol and will try to express our opinions
    in a few messages on this list before the WG meeting.

So, hopefully it will come more.

> Rather, it would be better to preserve the current level of
> searching as a basic level, and then define a new extended
> level that gets a complete set of new capabilities.  There is
> some IMSP interaction here that should be considered too.
	:
> Searching based upon the Date: header will always, I suspect,
> be quite a bit slower than the internal date, but educated
> users would be aware of this.

I infer from this that you think searching with "send date",
inspite of its problems, could be put in the extended search
level. Correct?

_If_ the "send date" is considered important for the client
and the user, the server could relieve the client from
the parsing problem (as is already done with other parts)
through for example "RFC-822-ing" it.

But what is your view of my suggestion on clarifying
INTERNALDATE and the implications of that to (at least) the
date preservation of COPY? Is there _any_ motivation for
keeping a "storage date"?
---
Peter Svanberg, NADA, KTH		    Email: [email-masked]
Dept of Num An & CS,
Royal Inst of Tech			    Phone: +46 8 790 71 40
S-100 44  Stockholm, SWEDEN		    Fax:   +46 8 790 09 30

---

From: Mark Crispin <[email-masked]>
Date: Tue, 24 Aug 1993 18:26:21 -0700 (PDT)

Yes, I think that a date criteria based upon the RFC-822 Date: header (that
is, the ``send date'' as you call it) would be something reasonable to include
in a future extended SEARCH.

Dates are not preserved in COPY.  What makes you think they would be?

-- Mark --

---

From: Peter Svanberg <[email-masked]>
Date: Wed, 25 Aug 1993 04:34:59 +0200

Quoting:  Mark Crispin <[email-masked]>

> Dates are not preserved in COPY.  What makes you think they would be?

My thoughts about how it will be used (is what makes me think so).

In the debate about preservation, you said:

> Arguably, this behavior [not preserving] is justified; the
> copy of the message is a separate instance of the message, and
> the state could be thought of as being associated with the
> instance instead of with the message.  For example, you can
> define the internal date as being ``when the message was placed
> in this mailbox'' as opposed to ``when the user received this
> message''.
>
> However, to end users of applications such as Pine, the loss
> of seen status when a message is copied from one folder to
> another is a bug.

In my scenario, the "copy" of the message is in fact _the_
message, moved from INBOX for storage. If then the internal
date is not preserved, it becomes what I called a "storage
date", which I still can't see any use of. Furthermore, as this
is the date used in SEARCH, the search by date functionality
becomes almost useless.

What's wrong? My scenario and/or usage of COPY? Then please
show me how _you_ think it would be used, and how my usage is
made possible.
---
Peter Svanberg, Nada, KTH

---

From: Laurence Lundblade <[email-masked]>
Date: Wed, 25 Aug 1993 11:35:40 -0400 (EDT)

Just thought I'd mention that Pine does parse the Date: field. It uses 
some heuristics and is generally successful. The message date displayed in 
Pine's index is from a full parse of the Date: field. In thousands of 
messages I've seen in Pine I can only think of less than 5 or so that had 
dates so bad you couldn't parse them.  It also uses the parse date field 
for sorting by date, though the lack of consist time zone usage sometimes 
causes problems. 

Laurence Lundblade                             
  [email-masked]        703-552-2537
     Virginia Tech -- Blacksburg, Virginia 

On Mon, 23 Aug 1993, Mark Crispin wrote:

> ....

---

From: "David Herron" <[email-masked]>
Date: Wed, 25 Aug 93 9:59:34 PDT

>is the date used in SEARCH, the search by date functionality

Peter's scenario matches with mine.

(Given accuracy and synchronization of time stamps...)  Copying messages around
should preserve the contents of the message.  It should be a *copy* of the
message, not a new instance.  (I hadn't ever seen this detail in previous
readings of IMAP... I saw `copy' and thought it would only copy, not munge
while copying).

I fail to see any logic in a munging copy.  Care to elaborate?

---

From: William Yeager <[email-masked]>
Date: Wed, 25 Aug 1993 11:14:31 -0700 (PDT)

I think the idea behind recreating for bsd style mail boxes the "From .." line 
during a COPY, from which the "storage" or internal date is determined is that 
a 

SEARCH ON/SINCE/BEFORE date

means since, before or after the message *arrived in the selected mailbox.* 
Thus, the "From .." lines are always in chronological order. If one preserved 
this line from the original INBOX, then this order is essentially random. As 
Mark mentioned, mh like folders avoid this difficulty by using the write date 
of the folder. 

The current approach for bsd and tenex style internal dates has historical 
precedence going back about a couple of decades, and certainly, one can argue 
that the current implementation gives a consistency across all mailboxes. 

I personally prefer this approach. And agree with Mark that one should consider
new date searches in the new extended search command.

Bill

---

From: William Yeager <[email-masked]>
Date: Wed, 25 Aug 1993 12:01:41 -0700 (PDT)

> means since, before or after

Yes, yes, I meant "means on, since or before" here. Sorry :-). I just got back 
from an hour+ of tennis, it was 80F, and my brain is in a slightly marginal 
state.

Bill

---

From: "David Herron" <[email-masked]>
Date: Wed, 25 Aug 93 13:50:48 PDT

To clarify ..

I may have been mistaken about what I was replying to.  (I no longer have
the message handy to check)  I took the quote from Mark as saying that
COPY creates a new instance and modifies the date in Date: not the
one in From<space>.  Since the From<space> line is ridiculous (messages
should be stored in a directory with one message per file) and in any case
is not *part* of the message so I don't care much about what is done with it.
If it is where you wish to store a time stamp for when it was appended
to the folder, be my guest.

Does it also change the return address that's stored on the From<space> line?  
(Doesn't have to but it might be interesting to record the old folder name..)
Does it preserve old From<space> lines?  (Doesn't have to)

Having the Date: field independantly searchable from the storage date is
what should be done.  

	David

---

From: William Yeager <[email-masked]>
Date: Wed, 25 Aug 1993 14:07:22 -0700 (PDT)

> Having the Date: field independantly searchable from the storage

I think there will be a consensus on this one. 

Bill

---

From: Mark Crispin <[email-masked]>
Date: Wed, 25 Aug 1993 14:17:37 -0700 (PDT)

> I think there will be a consensus on this one.

I agree.  The delay is not on whether or not this is a good idea, but in
getting together a full list of searching extensions.  There is enough problem
with multiple levels of support in IMAP software as it is without adding more
complex levels.

---

From: Peter Svanberg <[email-masked]>
Date: Thu, 26 Aug 1993 01:43:33 +0200

William Yeager wrote:

> I think the idea behind recreating for bsd style mail boxes the
> "From .." line during a COPY, from which the "storage" or
> internal date is determined is that a
> 
> means since, before or after the message *arrived in the
> selected mailbox.* Thus, the "From .." lines are always in
> chronological order. If one preserved this line from the
> original INBOX, then this order is essentially random.

But isn't the message number enough for the chronology?

One more time (I _want_ to know how you are thinking):

What you are saying is that you consider it more interesting
to search on (an hence more likely that you know) when a
certain message was _stored_ in the mailbox, than when it
_arrived_ to you?

In my mail usage I often store messages in a very
un-chronological mode - suddenly I can find some old messages
in my "unsorted" folder which I move to another folder, putting
them after more recent messages. Is that so unusual? If I later
search in that folder, with your approach I will have to
remember when those old messages were stored.

> As Mark
> mentioned, mh like folders avoid this difficulty by using the
> write date of the folder.

(I suppose you meant "of the file".)  This is unimportant in
MH, as it uses the Date:-field for sorting and searching.

Quoting:  "David Herron" <[email-masked]>
>
> Since the From<space> line is ridiculous (messages

So you don't care about searching? (This i the date used for that.)
---
Peter Svanberg, Nada, KTH