Subject: non-netnews bboard access in c-client/imapd

From: Mark Crispin <[email-masked]>
Date: Sun, 11 Jul 1993 19:04:07 -0700 (PDT)

I expect to soon be implementing non-netnews bboard support in c-client
(and, by extension, imapd).  Essentially, what will happen is that the bezerk
(/usr/spool/mail), tenex (mail.txt), etc. drivers will start responding to
open bboard and find bboard requests (presently, there are stubs).

     There are some issues that need to be straightened out, and I would
appreciate input.

     Tenatively, we have decided that a bboard name that begins with / is not
in the netnews namespace.  Essentially, a well-known bboard directory name
will be prepended to the specification to form a complete filename (or perhaps
some sort of chroot() would be done).

     We are planning on using ~ftp as the directory name, as part of an
embryonic capability of being able to use IMAP as an access mechanism to our
archives.  ~ftp has the advantage of being a well-known place for files that
are exported to the network at large, so anonymous interaction is less of an
issue here.

     There has been some discussion about possibly also using
/usr/spool/bboard (ala SUMEX) but no conclusion has been reached.  Plus,
people currently using that name may have some strong feelings on the
interaction with anonymous (e.g. they don't want their /usr/spool/bboard to be
exported to the entire Internet via IMAP).  Present feeling is to shelve this
until it is better thought out and there is a concensus.

     Another anonymous interaction issue is preserving server state ala
.newsrc if anonymous is not enabled.  Present feeling is to shelve this
question as well.

     So, as currently contemplated, if a user opens the bboard
	*{ftp.cac.washington.edu}/mail/imap_archive
it would open mail/imap_archive on the ~ftp directory as a read-only folder.

     I'm not sure what should be done for non-anonymous access right now; if
it should be treated identically to anonymous access or if it should be an
error pending further definition.  Ideas?

     Any other ideas about what this new capability might look like would be
gratefully appreciated.  I'm hoping to have something running by the end of
this week, and I fully expect that this issue will be re-visited again in the
future, so please think in terms of a Phase I (albeit not precluding future
changes) implementation.

Thanks,

-- Mark --

---

From: Terry Gray <[email-masked]>
Date: Sun, 11 Jul 1993 23:32:05 -0700 (PDT)

> it would open mail/imap_archive on the ~ftp directory as a read-only folder.

Mark,
Is there a "/anonymous" missing from the above example?

(If not, why not?)

-teg

---

From: Mark Crispin <[email-masked]>
Date: Sun, 11 Jul 1993 23:40:17 -0700 (PDT)

> Is there a "/anonymous" missing from the above example?

Yes and no.

> (If not, why not?)

Yes, because we're defining anonymous access behavior.

No, because we haven't defined non-anonymous access behavior (but need to, at
least a stub).

---

From: John Gardiner Myers <[email-masked]>
Date: Mon, 12 Jul 1993 10:48:53 -0400 (EDT)

For non-anonymous access, c-client should do .newsrc-style
preservation of \SEEN state.  You proabably don't want to use .newsrc
itself in order to avoid interaction with netnews systems.

Other flags (except \RECENT) are associated with the bboard and should
only be manipulated if the user has write access to the bboard (and
only if you want to deal with the locking issues)

The "don't want /usr/spool/bboard to be exported to anonymous" issue
is just an instance of the ACL problem.

I hate the "/" naming convention.  What's wrong with scanning the
netnews driver first, followed by the bezerk and tenex drivers?

-- 
_.John G. Myers		Internet: [email-masked]
			LoseNet:  ...!seismo!ihnp4!wiscvm.wisc.edu!give!up

---

From: Terry Gray <[email-masked]>
Date: Mon, 12 Jul 1993 08:16:36 -0700 (PDT)

> netnews driver first, followed by the bezerk and tenex drivers?

I think what's wrong is that it defines the berzerk namespace as (a)
what's left after all possible newsgroup names are subtracted, or worse
(b) what's left after *current* newsgroup names are subtracted.  Since
these two worlds are often administered independently (different places,
different people) it is much worse than in a personal/home directory
situation where the user *may* be able to avoid conflicts.

It's not that I love the "/" convention, I just see the search-path 
approach as leading to unpredictable behavior, and therefore much greater 
confusion.

-teg

---

From: Terry Gray <[email-masked]>
Date: Mon, 12 Jul 1993 08:27:55 -0700 (PDT)

> approach as leading to unpredictable behavior, and therefore much greater 

I just wanted to add that I see great value in this facility if it can be 
dropped into lots of existing ftp archive servers... but I think the odds 
are high that somewhere there already exists an archive name that 
matches a netnews name.  (I know I have plenty of conflicts in my own 
archives.)

-teg

---

From: Mike Macgirvin <[email-masked]>
Date: Mon, 12 Jul 1993 08:04:48 -0700 (PDT)

mrc:
>  /usr/spool/bboard (ala SUMEX) but no conclusion has been
>  reached.  Plus, people currently using that name may have some
>  strong feelings on the interaction with anonymous (e.g. they
>  don't want their /usr/spool/bboard to be exported to the
>  entire Internet via IMAP).  Present feeling is to shelve this

	This is no longer in active use for SUMEX or derivative systems. Unless
there are others still using this name, I'm inclined to let it fade into 
oblivion. 

>  it would open mail/imap_archive on the ~ftp directory as a 

	I wouldn't want to force this on anybody. If not a run-time option, the
directory prefix should be at least a build-time option. The best long-term 
strategy (IMHO) would be to have a generalized alias for this much as the 
"INBOX" spec does for mail locations. I may be old-fashioned, but I consider a 
leading path separator to mean that you want this path taken literally.
How about:
	*{ftp.cac.washington.edu}$ARCHIVE/imap_archive
(with or without the '$' or some other character you could check quickly.).
 
..and as an implementaation concern, you couldn't enter an absolute pathname 
unless logged in.

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 12 Jul 1993 09:42:05 -0700 (PDT)

The only thought I had is that it might be desireable for some situations 
in the future to have the IMAP anon archive set up differently from the 
anon FTP archive.  This is thinking from a NIC operations view.  Perhaps 
some way to configure the path would be good (environment variable?).

Also, is there a problem when the /anonymous gets appended that would 
make it impossible to name the file xxx/anonymous? 

LL

---

From: John Gardiner Myers <[email-masked]>
Date: Mon, 12 Jul 1993 15:04:26 -0400 (EDT)

Conflicts with the netnews namespace can be dealt with by proper
selection of the namespace.  Sites have been defining their own local
hierarchies for quite some time.  For example, there is no netnews
"archive" hierarchy in use, so it is relatively safe to name the
archive for this list "archive.c-client".

I maintain that it is a mistake to expose implementation details
through the interface unless the user is explicitly making
implementation-specific requests.  In particular, it is wrong to
expose the storage format through the default "rubber-room" namespace.
Why should the user have to know or care whether the mailing list
archives are stored in netnews, berkeley, or tenex format?

---

From: Mark Crispin <[email-masked]>
Date: Mon, 12 Jul 1993 12:39:58 -0700 (PDT)

> some way to configure the path would be good (environment variable?).

It looks like something of this nature will need to be done, perhaps in a
Phase II.

> make it impossible to name the file xxx/anonymous?

No, because the /anonymous occurs within the {} set.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 12 Jul 1993 12:40:52 -0700 (PDT)

I am sympathetic to both John's and Terry's points of view, and can go either
way.

In defense of Terry's point of view, the context management layer in Pine
would tend to hide the implementation details; or rather, the implementation
details would only be in a .pinerc file.

However, I believe that John has a point about hiding implementation details
in general, and that the best way to deal with namespace conflicts is by
proper selection of the names within the namespace.

---

From: John Gardiner Myers <[email-masked]>
Date: Mon, 12 Jul 1993 15:48:32 -0400 (EDT)

> details would only be in a .pinerc file.

In this case, wouldn't that be in a .imapdrc file?  It should be the
IMAP server that hides this particular implementation detail, not the
client.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 12 Jul 1993 12:51:11 -0700 (PDT)

> IMAP server that hides this particular implementation detail, not the

No, because the ambiguity would still exist at the IMAP level, and the whole
purpose of the ugly syntax is to eliminate the ambiguity.  Of course, you can
(as you have proposed) declare the ambiguity unimportant, and thus eliminate
the need for the ugly syntax.

---

From: John Gardiner Myers <[email-masked]>
Date: Mon, 12 Jul 1993 16:27:45 -0400 (EDT)

Mark Crispin <[email-masked]> writes:
> purpose of the ugly syntax is to eliminate the ambiguity.

Oh boy, it's the old ambiguous/unambiguous namespace argument again.
I mistakenly thought we had gotten this resolved.

Exporting an ambiguous/"rubber room" name does not prevent you from
also accepting an implementation-dependent unambiguous name, for
clients that care.

In the absense of a user who explicitly gives implementation-specific
information, clients and servers should communicate with
ambiguous/"rubber room" names.  Otherwise, clients have to make
possibly invalid assumptions about the server implementation.

If pine assumes that the IMAP server is a c-client imapd, it's likely
to get very confused when talking to a non-c-client IMAP server.  The
CMU IMAP server, for instance, will tell any client trying to use a
mailbox or bboard name with a "/" in it to go jump in a lake.

---

From: Terry Gray <[email-masked]>
Date: Mon, 12 Jul 1993 13:14:07 -0700 (PDT)

> archives are stored in netnews, berkeley, or tenex format?

The user shouldn't know or care, neither the format, nor the absolute 
location in a filesystem...

Things I believe to be true:

 1. The issue is to define a satisfactory user-visible namespace, not 
    expose implementation details.  But the user-visible namespace 
    *might* use a syntax familiar in *some* filesystem!  Even if the
    mapping to the actual filesystem organization is different.

 2. It is highly desirable for the sysadmin to be able to choose 
    where/how to store archive or other data.

 3. The user-visible namespace *must* support hierarchy, which means
    a canonical path syntax must be defined, even though it may be relative
    to a sysadmin-defined root, or a driver-defined abstract root.

 4. We are really searching for a way to select an IMAP driver via the 
    name string syntax that goes across the wire.  However, whatever means
    is chosen to tell a driver that "this name's for you", the 
    requirement to support a hierarchical path (without using netnews dot 
    notation) is still with us.

-teg

---

From: Terry Gray <[email-masked]>
Date: Mon, 12 Jul 1993 13:33:31 -0700 (PDT)

> I mistakenly thought we had gotten this resolved.

I believe the ambig/nonambig name issues *have* finally been resolved in 
the context of a single driver.  I don't believe they have been resolved 
in the context of multiple drivers, as we are facing here.
 
> also accepting an implementation-dependent unambiguous name, for

I think I agree, though I'm not sure who is doing the exporting.
 
> possibly invalid assumptions about the server implementation.

Agreed.

> mailbox or bboard name with a "/" in it to go jump in a lake.

Pine makes no such assumption.  In fact, we've worked hard to keep *any*
notion of filesystem name syntax out of the Pine code.  (OK, so there are
a couple of exceptions for DOS, but not in c-client functions.) However, a
Pine user can *configure* his/her environment to take advantage of an
IMAPd that will make an effort to map a user-provided path name into the
actual filesystem.  This is especially handy (indeed, essential) in
environments such as ours where an IMAP server may also be a timesharing
host, and it is a requirement to be able to get at the same mailboxes both
locally and remotely. 

-teg

---

From: John Gardiner Myers <[email-masked]>
Date: Mon, 12 Jul 1993 17:11:08 -0400 (EDT)

> in the context of multiple drivers, as we are facing here.

I think you've already solved the problem of ambiguity between the
bezerk and tenex drivers.  Why not apply the same solution to the
bboard/netnews instance of the same problem?

> I think I agree, though I'm not sure who is doing the exporting.

I'm trying to make a statement about interfaces in general, but the
particular interface I'm trying to apply this to is a server exporting
the IMAP protocol.

>     where/how to store archive or other data.

I see this as a driver-parameter/server-configuration issue.  A driver
could presumably be given a table like:

{ "archive.", "~ftp/list-archive/" },
{ "", "/usr/spool/bboard/" },

To tell it where to search for folders.

>From a human-factors standpoint, I would think users are far more used
to the netnews dot notation than the unix pathname notation in the
context of bboards.

---

From: Laurence Lundblade <[email-masked]>
Date: Mon, 12 Jul 1993 14:11:45 -0700 (PDT)

This is interesting. When you say canonical, do you mean as part of the 
IMAP standard so that all implementations should be able to deal with the 
heirarchy or is this for particular drivers?  Seems like doing it for 
particular drivers could lead to interoperbility problems, though I 
certainly see it as desirable.

LL

On 12 Jul 1993, Terry Gray wrote:

.....

---

From: Terry Gray <[email-masked]>
Date: Mon, 12 Jul 1993 23:31:19 -0700 (PDT)

> bboard/netnews instance of the same problem?

Not all of us are convinced that saying you can never have a Berzerk 
folder ending in the string ".txt" is an acceptable solution...
 
<stuff on IMAPd configuration omitted; we agree on the need for server
configuration...>

> to the netnews dot notation than the unix pathname notation in the

The reason I find this unacceptable is that it precludes having file names
with dots in them.  I believe such names must be allowed, or we need not
bother with this exercise at all.  (Remember, I believe a major objective
of this activity is to be able to make *existing* anonymous ftp archives
available via IMAP.)

-teg

---

From: Terry Gray <[email-masked]>
Date: Mon, 12 Jul 1993 23:45:51 -0700 (PDT)

LL,
About the same time you wrote your msg, I was having some second thoughts 
about the canonical path syntax issue...

I still believe that IMAP must not preclude hierarchical naming syntax;
but whether it is both feasible and desirable to define a *canonical*
syntax for hierarchy is less clear to me.  As John and others have pointed
out, we want to keep the protocol (and even its implementation) as free of
OS dependencies as possible. 

As I understand it, IMAP views a hierarchical path name as simply a 
string name with some possibly funny characters in it.  This allows IMAP 
to deal with "/foo/bar" and "\foo\bar" with equal facility.

This seems like a win.  Now we also want to distinguish between a 
namespace recognized by the news driver and a namespace recognized by
a different (in this case, Bky) driver.

So there are really two issues:

 1. Should a canonical path syntax be defined, or leave well-enough alone
    and keep hierarchy in the "eye of the beholder"?  Proably the latter,
    unless we can really convince ourselves that the former is both 
    feasible and provides some important advantages over the latter.

 2. How to differentiate driver namespaces.  (An old, old, problem!)
    We can:
	a. Leave this to each driver, but with the expectation that a
	   globally unique namespace can be contrived by appropriate link
	   order (so if Berzerk wants to look for "/" so be it...)
	b. Leave it to each driver, but don't attempt to identify any
	   globally unique namespace across drivers.  (John's favorite;
	   my least favorite.)
	c. Define in IMAP some characteristic of "typical" path names
	   that can be used to select the appropriate driver (e.g. the
	   presence or absence of a "/" or "\".)
	d. Define some meta-syntax in an IMAP name that can be used to 
	   select drivers.

Options b and c strike me as the least attractive; option d seems doable 
but clumsy, leaving option a as the least of several evils in my book.

Are there any other options?

-teg

---

From: [email-masked]
Date: Tue, 13 Jul 93 11:04:15 MDT

I agree that implementation details should be hidden-
seems like the right way to avoid namespace clashes is to add a way to
"mount" various information sources at arbitrary points in the namespace
a la /etc/fstab in sunos.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 14 Jul 1993 21:44:14 -0700 (PDT)

> OS dependencies as possible.

I think we are all in violent agreement on this point.

> to deal with "/foo/bar" and "\foo\bar" with equal facility.

This is essentially correct.  IMAP has no knowledge of hierarchy.

> a different (in this case, Bky) driver.

This is where we have the disagreement.  One religion says that it is
important to divide the namespace, because it is a problem if two different
drivers both believe that the same name is valid, and that this situation must
be prevented at all cost.  The other religion says that if this situation
arises, it is a user error, and is not worth worrying about.

>     feasible and provides some important advantages over the latter.

Certainly the latter.  I, as Official Devil's Advocate (or perhaps just
Official Devil?), would be very hard to convince on the feasibility issue.  No
need to talk about the advantages, they're fairly obvious.

I remember having to do with TOPS-10/TOPS-20/ITS/UNIX filename mapping issues
many years ago.  It was surprisingly obvious to a human what the ``right''
mapping was.  The only problem, it wasn't possible to express that in any
rational rule set that could be implemented in software.

> but clumsy, leaving option a as the least of several evils in my book.

I don't quite understand option (a).  There are some implicit link order
rules; in particular, any drivers which support the concept of INBOX must come
before bezerk or mmdf, since bezerk/mmdf always accept the name INBOX (I
haven't quite addressed how these two will fight it out yet...).  I think that
it is a terrible idea to change link order on the fly or anything like that;
ideally, all software should use as close to the same order as possible to
keep some semblance of consistant behavior.

I personally favor option (b).

I see option (c) as a kludge to prevent option (d), and find option (d) to be
the least attractive of all.

---

From: John Gardiner Myers <[email-masked]>
Date: Thu, 15 Jul 1993 23:42:40 -0400 (EDT)

> folder ending in the string ".txt" is an acceptable solution...

I have a bezerk mailbox named "foo.txt".  Seems to work fine, though I
had to create it by hand.

The drivers are presumably able to check appropriate magic numbers
before accepting names.  This admittedly might not perform all that
well.

> The reason I find [netnews dot notation] unacceptable is that it
> precludes having file names with dots in them.

Not at all.  Remember, in the IMAP world, hierarchy is in the eyes of
the beholder.  In the mailbox namespace, the bezerk driver can
simulate the netnews dot notation using flat files in a single
directory.  It can do the same in the bboard namespace as long as a
leading "/" is not required.

Consider my previous search table:

In ~ftp/list-archive/, you can have the mailboxes "c-client",
"c-client.1992", "c-client.1991", "imap", "imap.1992", "imap.1991",
etc.

> (Remember, I believe a major objective of this activity is to be
> able to make *existing* anonymous ftp archives available via IMAP.)

To get best results, an archive site is going to have to structure
their archive in an IMAP-friendly way and/or spend time configuring
their IMAP server.  I think this is to be expected.

An archive site can alternatively spend minimal effort and have the
archives exported via IMAP with unix pathname format names.  The
ugliness of the names will be compensated for by their trivial
conversion between the FTP and IMAP namespaces.

---

From: Laurence Lundblade <[email-masked]>
Date: Fri, 16 Jul 1993 10:21:07 -0700 (PDT)

I pretty much agree that keeping the hierarchy in the "eye of the
beholder" is the best thing.  The main advantage of having hierarchy is
being able to navigate it: be at one level, get a list at that level,
change levels (pwd, ls, cd).  Unless we add this functionality to IMAP
there's not much to gain by defining hierarchy.

Well, actually that's not entirely true.  Netnews groups have a hierarchy
and there are no NNTP commands to navigate it, but user agents can
navigate it because they know the format of the name space.  If a hierarchy
were defined for the IMAP name space clients could navigate it with the
existing protocol, though it would be inefficient for large collections. 
They would have to get the entire list of all entries in the name space 
and then navigate it locally.  This doesn't seem very practical 
especially for small clients and large lists.

My biggest concern about leaving the hierarchy in the "eye of the
beholder" without standardizing it is that specific implementations of
client-server pairs will adopt some convention about the name space.  They
then will only be able to operate well with each other and not other
clients or servers. 

LL