Subject: Some problems with the imapd-3.0 (Pine 3.87) release

From: Steve Hole <[email-masked]>
Date: Thu, 14 Oct 1993 17:05:23 -0600

Hi Mark.   We have found some problems with the imapd-3.0 release that is
distributed with Pine 3.87.   As far as I know, this is the definitive version
of the imapd-3.0 release?

(1)  The behaviour of a mail_find_all (FIND ALL *) has changed for the Bezerk
driver.   It now return a list of every file that is matched by the pattern,
without checking to see if they are valid folders.   

This is causing us some problems with our new folder subscription mechanism
because it will include non-mail folder files in the list.   We have provided
a mechanism for users to access folders in (potentially) any place on the 
server in which they have access.   If they get binaries and such in the folder
list, then the potential problems are obvious.    

The bottom line is that we pretty much like the way it worked before. 
What we would like to see specifically is:

  * bezerk_find_all() should perform bezerk_is_valid() on any potential
    match before including it in the list of matches sent to the client.
  * 0 length files should match *regardless* of whether they have a ".txt"
    suffix or not.

(2)  The behaviour of mail_find is not what we expected for the Bezerk 
driver.   If you do a bezerk_find() and there is no subscription database, then
it does a call to bezerk_find_all() - returning all of the folders that are
returned by (1).

This completely defeats our subscription mechanism where we wish to present
a list of subscribed folders on one side and a list of unsubscribed but
available folders on the other side.   News and Dawz do the right thing.
Also, the Bezerk driver will (correctly) create a subscription database 
if it does not exist on the first subscription.

We would like:

  * bezerk_find to return an empty list if there is no subscription database
    - just like News and Dawz do.

  * bezerk_find to return only the contents of the subscription database 
    if it does exist.   This is the way that it works now, I'm just trying
    to be complete.

I can see how some of the changes have evolved with new the new version 
of Pine.   I suspect that some of the changes that we have seen were 
required by Pine.   If so, perhaps there are some work arounds that we
can do?   We would be perfectly happy to make these changes and send them off 
to you in the form of patches Mark.    What do you think? 

Cheers.
--
Steve Hole  		        Director of Research and Communications
ISA Corporation			mail:  [email-masked]
Suite 835, 10040 - 104 St.      phone: (403) 420-8081
Edmonton, Alberta, Canada       fax:   (403) 420-8037
T5J 0Z2

---

From: Mark Crispin <[email-masked]>
Date: Thu, 14 Oct 1993 19:18:26 -0700 (PDT)

Hi Steve -

     mail_find*() behavior has changed several times in c-client, as we tried
various schemes and found which worked and which didn't work.  It's been
changed several times.  We weren't aware that you (or anyone else outside of
UW) were using it, or we would have made sure you were informed of what was
going on.  We discovered so many problems in find handling (only the really
major ones are discussed here) that we didn't think anyone could use it!  ;-)

     The orignal find handling was:
 find		use subscription list (null results if no subscription list)
 findall	do directory scan with validation

     Pine doesn't use subscription lists yet, so it used findall.  This didn't
work out.  The problem was, checking for mailbox validity via the
<driver>_valid() routine turned out to be a big performance lose.  Substantial
amounts of time was expended in that code.

     The next means of find handling was:
 find		if a subscription list exists, use it, else do directory scan
		without validation (``fast finding'')
 findall	do directory scan with validation

     The idea here was to leave findall as a thorough validating find, but
have something fast for regular find (and let the few people who use
subscription lists win).

     Then came the killer: the new arbitrary file driver (phile), which
essentially meant that there was no such thing as a ``file'' that was not a
mailbox.  That brings us to the current, transitional means of find handling:
 find		if a subscription list exists, use it, else do findall
 findall	do directory scan without validation

     However, that is only until all old versions of Pine (mostly PC-Pine)
that used mail_find() (3.84 and 3.85) become extinct.  Hopefully this will not
be long.  The planned final version is:
 find		use subscription list (null results if no subscription list)
 findall	do directory scan without validation

     There are no plans to re-attempt any validation that it is really a
mailbox object, since the concept of a filesystem object that cannot be
referenced as a mailsystem object is becoming extinct in c-client.  The fact
that the validation was horribly slow is just one more nail in the coffin...

     In 3.1 of the c-client/IMAP toolkit, the transition is well under way.
Among some of the things which have already happened are:
 1) the special extension of Tenex format is now .TxT
 2) an MTX driver for UNIX, compatible with the format used on DOS, is in the
    works and probably will replace the Tenex driver eventually (or the Tenex
    driver will be used just to read in old cmm files).  It uses the special
    extension of .MTX
 3) work is underway to eliminate special extensions entirely; that is, to
    deal with the zero-length file problem in a different way.

     This isn't quite what you've asked for, but I think it's probably an
improvement over what you're dealing with now with the transitional code.
Please take a look at the 3.1 toolkit since this is closer to what is in Pine
3.87 than the 3.0 toolkit.  In particular, the unfavorable interaction with
subscriptions will go away.

     Not clear yet, but it's likely that a 4.0 toolkit will be sooner than
later; that's going to be the version that supports the IMAP2bis Proposed
Standard when that happens.

     I hope this message is good news for you...

-- Mark --