Subject: re: Some problems with the imapd-3.0 (Pine 3.87) release

From: Steve Hole <[email-masked]>
Date: Fri, 15 Oct 1993 10:03:38 -0600

On Thu, 14 Oct 1993 20:18:26 -0600 Mark Crispin wrote:

> From: Mark Crispin
> Date: Thu, 14 Oct 1993 20:18:26 -0600
> Subject: re: Some problems with the imapd-3.0 (Pine 3.87) release
> To: Steve Hole <[email-masked]>
> Cc: c-client Interest List <[email-masked]>
> 
>      Pine doesn't use subscription lists yet, so it used findall.  This didn't
> work out.  The problem was, checking for mailbox validity via the
> <driver>_valid() routine turned out to be a big performance lose.  Substantial
> amounts of time was expended in that code.

This is true.   It kind of worries me that somebody can select a file as a folder
when it really isn't one, but I guess we can live with that.   It's too bad that
we couldn't provide some sort of feedback to the user on type, but I don't see
how without looking inside.   Even if we were to start writing and looking for
a folder magic number, it would require some sort of conversion on existing 
folders.

>      There are no plans to re-attempt any validation that it is really a
> mailbox object, since the concept of a filesystem object that cannot be
> referenced as a mailsystem object is becoming extinct in c-client.  The fact
> that the validation was horribly slow is just one more nail in the coffin...

Ok.   I think that some of the more clever users will find out that they can
cause problems by writing to files that are not folders.   In general, the 
performance win should be worth it, though.   I guess we'll find out.
 
>      In 3.1 of the c-client/IMAP toolkit, the transition is well under way.
> Among some of the things which have already happened are:
>  1) the special extension of Tenex format is now .TxT
>  2) an MTX driver for UNIX, compatible with the format used on DOS, is in the
>     works and probably will replace the Tenex driver eventually (or the Tenex
>     driver will be used just to read in old cmm files).  It uses the special
>     extension of .MTX
>  3) work is underway to eliminate special extensions entirely; that is, to
>     deal with the zero-length file problem in a different way.
> 

Hmm ... new drivers.   We have some ideas for a new driver as well that I
would like to discuss with you.   I will send an outline of my thoughts in
a separate message.

>      This isn't quite what you've asked for, but I think it's probably an
> improvement over what you're dealing with now with the transitional code.
> Please take a look at the 3.1 toolkit since this is closer to what is in Pine
> 3.87 than the 3.0 toolkit.  In particular, the unfavorable interaction with
> subscriptions will go away.

The key requirement for us is that a mail_find() return a nil list if there
is no subscription database.   Otherwise our subscription interface will simply
not work.   Does the 3.1 server do this now?   If not, can it be modified to do
so?  We actually have already modified the 3.0 release that came with pine to
do this.   Will it cause a problem for the latest release of Pine?
 
>      Not clear yet, but it's likely that a 4.0 toolkit will be sooner than
> later; that's going to be the version that supports the IMAP2bis Proposed
> Standard when that happens.
> 
>      I hope this message is good news for you...

I think it is.   We really need the subscription stuff to work right because
we want to release v2.2 next week, and it supports subscription.   We will 
have to distribute a modified version of imapd I guess in order to support
it.   If that will not adversely affect Pine, then I don't mind doing that
until version 3.1 is snapped off.

Cheers.
--
Steve Hole  		        Director of Research and Communications
ISA Corporation			mail:  [email-masked]
Suite 835, 10040 - 104 St.      phone: (403) 420-8081
Edmonton, Alberta, Canada       fax:   (403) 420-8037
T5J 0Z2

---

From: Terry Gray <[email-masked]>
Date: Fri, 15 Oct 1993 20:22:58 -0700 (PDT)

> until version 3.1 is snapped off.

Steve,
It's a transition problem.  Pine 3.87 and later will not use FIND,
therefore they will happily ignore .mailboxlist files.  I *think* Pine
3.07 and earlier will also happily ignore .mailboxlist.  (Can one of the
Pine team confirm or deny this?)

The problem is with 3.85, which when confronted with a a null .mailboxlist
happily asserts to the user that all of his/her folders are gone...
Fortunately, 3.85 was superceded fairly quickly, so maybe the 
incompatible installed base is fairly small.

I think it makes sense for you to release an IMAPd with the modified FIND 
behavior, and we should accelerate that change in our code, to reduce
compatibility problems with ECSmail 2.2.

You will need to tell your clients who have trouble with Pine to upgrade 
to Pine 3.87 or later, and we'll need to tell our users who have trouble 
with ECSmail to upgrade IMAPd.

Pine guys, does that seem right?  (If I'm wrong about 3.07, the problem 
is much bigger, and we need to consider further.)

Sorry about this... we normally try to warn and/or seek consensus on
incompatible IMAP changes, but this one slipped by.  There is still an
open question of how much cross-client interoperability we really have in
listing folders... 

-teg

---

From: Steve Hole <[email-masked]>
Date: Sat, 16 Oct 1993 11:52:56 -0600

> compatibility problems with ECSmail 2.2.

OK, I will make the necessary changes this weekend.   I will forward the 
patches to you, but the change is so simple, I expect that you won't 
really need them.

> with ECSmail to upgrade IMAPd.

I just put the modified IMAP out at the same time with a TTL notice on it
for imapd-3.1.   That should make it clear to most.  
 
> open question of how much cross-client interoperability we really have in

No problem.   I know that we didn't really tell anybody what we were doing
and, as Mark said, nobody was previously making use of the subscription
mechanisms before now.

Thanks for the help guys.  Cheers.

---

From: Mark Crispin <[email-masked]>
Date: Sat, 16 Oct 1993 12:56:45 -0700 (PDT)

Steve -

     I haven't yet had the chance to digest the most recent messages on this
thread.

     I think we should have a bit of discussion first as to what these changes
are, if you're talking about going beyond the calls to findall from find
(which are interim and will probably be removed from IMAP toolkit 3.1 real
soon now).

     Of course, if that's exactly what the change is, no problem; consider the
poke to remove the interim code ``received, loud and clear''!  ;-)

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Sun, 17 Oct 1993 18:20:42 -0700 (PDT)

On Fri, 15 Oct 1993, Steve Hole wrote:
> when it really isn't one, but I guess we can live with that.

Do you understand that the dichotomy between ``file'' and ``folder'' is
intended to vanish?  A ``file'' now looks like a folder with one
``message'' in it.  Can you suggest why you wouldn't want ECS Mail users 
to have this capability?  [One obvious way of having distributed address 
books is to have a well-known folder name for the address book...]

> cause problems by writing to files that are not folders.

At the present time, files-as-folders can't be written to.  But, that is 
supposed to change in the future (obviously, it has to if that's how 
address books are accessed!).

> would like to discuss with you.   I will send an outline of my thoughts in

By all means, please do!  The current drivers more or less do nothing 
other than deal with extant UNIX-based mailstore technology.  They have 
little or nothing to do with what I would consider a ``good'' mailstore 
technology!  It's for this reason that the new linkage mechanism was 
invented.

> is no subscription database.

This change has *just* appeared in the sources; essentially, elimination 
of the two else clauses in dummy_find() and dummy_find_bboards().  It is 
not in the imapd distributed with Pine 3.87, although Pine 3.87 can be 
used with an imapd with this change.  Only my two machines have this 
change installed right now.

I suggest, as a *temporary* workaround, that you treat results from 
mail_find() that begin with "." and ".." as being equivalent to ``no 
subscription database presently exists.''  This will allow you to 
interact with the currently released set of servers without requiring you 
to supply a modified version.

I'm sorry you have to deal with this wart, but hopefully it'll be easy to 
excise one we're all sure that Pine 3.8[456] are extinct and Pine 3.87 
(or greater) rules... ;-)

> We will

Will my suggested workaround enable you to avoid this?

-- Mark --

---

From: Michael Seibel <[email-masked]>
Date: Mon, 18 Oct 1993 09:23:44 -0700 (PDT)

Word from the pine team is that versions 3.07 and earlier built all folder
lists by hand (actually sniffing local directories).  All pine versions,
including pc-pine, since 3.80 use c-client provided tools (i.e., find,
find_all) for putting together folder lists. 

The conversion strategy sound right from here, as well. 

-mikes