Subject: c-client literals in LIST ?

From: Dag Nygren <[email-masked]>
Date: Wed, 09 Aug 2000 00:20:05 +0300

Hi,

I am using the c-client library from
imap-4.7c here and just found out
that the mail-list() call doesn't seem
to accept literal strings from the server.

As I have mailboxes with 8-bit chars in them
this will break my applications.

Is there a patch available for this ?

BRGDS

-- 
Dag Nygren                               email: [email-masked]
Oy NewTech Ab                            phone: +358 9 8024910
Trasktorpet 3                              fax: +358 9 8024916
02360 ESBO                                 GSM: 0400-426312
FINLAND

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 8 Aug 2000 15:10:03 -0700 (PDT)

> to accept literal strings from the server.

I don't think so.  c-client deals with literals in list responses all the
time.

Why do you believe that to be the case?

> this will break my applications.

8-bit characters in mailbox names are reserved for future use as UTF-8.  Any
other use (e.g. ISO-8859-1, SJIS, etc.) is NON-COMPLIANT with the IMAP
specification.  No client is currently obliged to do anything reasonable with
8-bit characters in mailbox names.

You need to use the mechanism described in section 5.1.3 of RFC 2060 instead.

---

From: Dag Nygren <[email-masked]>
Date: Wed, 09 Aug 2000 07:17:17 +0300

> Why do you believe that to be the case?

I will be a bit more specific:

The environment is:
- MH-mailboxes
-UW-IMAP 4.7c serving
- IMP web client, uses:
  - PHP3.0.16, uses c-client for the imap-functions
  - Apache  1.3.12

I stated tracing a problem with 8-bit characters in mailboxes
and ended up in the call to mail_list() (or the parsing of it).

I will now check if it really is the move mail_list() return ->
PHP array, or if the result already is corrupted and get
back.

> specification.

This is ISO-8859-1.

> No client is currently obliged to do anything reasonable with

So nothing for non-Americans then... ;-)

I was afraid of that.... I will have to check with the IMP author.

> You need to use the mechanism described in section 5.1.3 of RFC 2060 instead.

Many thanks for responding.

---

From: Dag Nygren <[email-masked]>
Date: Wed, 09 Aug 2000 08:06:03 +0300

> PHP array, or if the result already is corrupted and get

OK,

just checked it, and very sorry for wasting your time :-(.
It isn't in c-client,  it is up in the php3 layer of IMP.

Thankyou very much for responding anyway, just
proved once again that Open Source support works :-))

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Aug 2000 00:16:16 -0700 (PDT)

> and ended up in the call to mail_list() (or the parsing of it).

8-bit characters are currently not permitted in IMAP mailbox names, and are
reserved for future use in UTF-8.  Refer to section 5.1.3 of RFC 2060 to find
out the correct way to represent non-ASCII characters in IMAP mailbox names.

> PHP array, or if the result already is corrupted and get

Perhaps the problem is that PHP is treating the 8-bit characters as UTF-8 in
anticipation of the time when UTF-8 will be permitted.  You haven't given
enough information for me to diagnose what's going on; in particular, what you
think the IMAP server is sending and what you are seeing.

> This is ISO-8859-1.

ISO-8859-1 is forbidden in IMAP mailbox names, and always will be forbidden.
IETF policy does not permit Internet standards-track protocols to be locked
into supporting only western Europe to the detriment of the rest of the world.
All new protocol internationalization is with Unicode.

> So nothing for non-Americans then... ;-)

That's completely false.  The characters of every language in the world are
representable in IMAP mailbox names, since IMAP mailbox names are Unicode,
encoded in 7 bits through "modified UTF-7" as described in 5.1.3.

The main reason why UTF-8 is not in use in IMAP yet is to exterminate any
current usage of ISO-8859-1 mailbox names.  Otherwise we would have huge
potential interoperability problems.  However, UTF-8 mailbox names in IMAP are
coming, so you need to fix dependencies on ISO-8859-1 now, to prevent your
application from being badly broken in the future.

Good luck!

---

From: Dag Nygren <[email-masked]>
Date: Wed, 09 Aug 2000 19:16:53 +0300

Hi again,

Now I think I know what is going on here:

As MH allows directories with 8-bit characters
the MH driver 'forgets' to UTF-7 encode these when returning
the list of mailboxes.

I will check the MH-driver in my UW imap 4.7c if I can fix this

BRGDS

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Aug 2000 10:44:50 -0700 (PDT)

> the MH driver 'forgets' to UTF-7 encode these when returning

Presumably, you're talking about MH mailboxes created externally from IMAP,
since there is no way that IMAP will create such a mailbox name.

Unfortunately, there's no way to win in that situation.  There's no way for
the software to know what character set the directories are in.  It depends
upon which country you are in!

To make it worse, when UTF-8 happens, there would be 8-bit names in UTF-8, and
thus no way to distinguish between the UTF-8 names and the older names.

> I will check the MH-driver in my UW imap 4.7c if I can fix this

The problem is that you would then have to make the MH driver assume that the
8-bit directory names are ISO-8859-1.  They wouldn't be that in Japan.

In my opinion, the best thing to do is use external software to convert your
existing 8-bit names to the modified UTF-7 convention used by IMAP.
Unfortunately, this requires changing everything else that assumes that
everything is ISO-8859-1, and that is certainly not a pleasant undertaking!
But, sooner or later, it has to be done, and the longer you wait, the more
painful the eventual conversion will be.

---

From: Dag Nygren <[email-masked]>
Date: Wed, 09 Aug 2000 22:06:35 +0300

> since there is no way that IMAP will create such a mailbox name.

Yep, the main mailreader will still be exmh here. IMAP was only meant
for enabling a webmail service to access the same mailboxes.

> 8-bit directory names are ISO-8859-1.  They wouldn't be that in Japan.

But does it have to assume that ?
There is a routine  imap_utf7_encode() in PHP3.0 that will encode  the
string it gets and return a valid utf7 coded string.
Cannot understand why I couldnt use that?

> everything is ISO-8859-1, and that is certainly not a pleasant undertaking!

This is more work than changing the above. It is even a good reason to
re-evaluate our decision to use IMAP...

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Aug 2000 12:07:55 -0700 (PDT)

On Wed, 09 Aug 2000 22:06:35 +0300, Dag Nygren wrote:
> > The problem is that you would then have to make the MH driver assume that
> > the 8-bit directory names are ISO-8859-1.  They wouldn't be that in Japan.

You still have to know what character set the 8-bit data is in, so that you
can do the correct character lookup and conversion.  Unless the software is
told, it has no way to know.  It's just a sequence of octets on the disk,
which could be in any character set; and different regions of the world use
different character sets to interpret those octets.

That's right.  Two completely different file names, in different languages,
may be represented by the same sequence of octets.

That's the problem with using such things as ISO 8859; they only have
applicability in a certain area of the world.  The same sequence of octets
will have completely different interpretations in other regions of the world.

So, even if you kludge using PHP3.0's imap_utf7_encode(), you have to edit the
software in a way that makes it run only in western Europe.  You also have the
problem of having two different representations on the disk for the same file
names, so if the user selects a European string, the software would have to
try both the ISO-8859-1 representation and the Unicode representation.

This problem was understood long ago.  The time has come to pay the price for
the mistaken decision of "just send 8-bit ISO-8859-1".

> > In my opinion, the best thing to do is use external software to convert
> > your existing 8-bit names to the modified UTF-7 convention used by IMAP.

This isn't an IMAP issue; it's an Internet issue.  *ALL* Internet protocols
are converting to Unicode.  It isn't IMAP that is forcing this upon you; it is
the Internet forcing this upon both IMAP and you.

Unless you plan to re-evaluate your decision to use the Internet, you are
going to have to do this conversion, sooner or later.  It is going to be a
difficult transition for all of us; but the longer that you wait, the worse
the situation will become.

Of course, it may be possible to delay it until after you quit the company,
and then it can become your successor's problem... :-)

You should be beating up on your vendors to start providing you with Unicode
capable operating systems and tools.  The Microsoft world is doing a fair job
of this, but most UNIX systems are way behind.

---

From: Dag Nygren <[email-masked]>
Date: Wed, 09 Aug 2000 22:55:21 +0300

> different character sets to interpret those octets.

Why would IMAP have to know the characterset ?
As long as it passes on the names of the mailboxes as they are 
represented on disk/in the database and so on ?
That is the byte sequence.

> may be represented by the same sequence of octets.

So ? Then the name would be given to the client represented
the same way.
There must be some philosofical thing here that I don't get ?!

> try both the ISO-8859-1 representation and the Unicode representation.

In other words: The coding standard should be sent together with the string ?

> and then it can become your successor's problem... :-)

Hope so.... ;-)

> of this, but most UNIX systems are way behind.

Perhaps you are right.

And once again thanks for your time on this.

Dag

---

From: Mark Crispin <[email-masked]>
Date: Wed, 9 Aug 2000 12:58:01 -0700 (PDT)

If you wish to continue this discussion, you should use some mailing list
other than c-client.  Although character set issues are fascinating in their
own way, this has nothing to do with c-client.

If you are looking for pedagogical information about character set issues and
Internet adoption of Unicode as the standard, refer to RFC 2130.  There are
many other RFCs about this issue as well.

> That is the byte sequence.

Human users do not type in a byte sequence.  They type in a sequence of
characters.  Those characters have to be converted into a byte sequence.  Nor
do human users read a byte sequence.  They read a sequence of characters.

The Internet standard character set is Unicode.  A particular byte sequence in
Unicode has the same character sequence interpretation everywhere in the
world.

This is not the case with regional character sets such as ISO 8859, SJIS, etc.
The same byte sequence will mean completely different characters depending
upon whether you are in western Europe, Russia, or Japan.  Your ISO-8859-1
strings look completely different outside of Europe.

It is impossible to convert a byte sequence in a regional character set to a
Unicode byte sequence without knowing the regional character set.  Simpilarly,
it is impossible to display a byte sequence to the user without knowing its
character set.  The Internet standard is UTF-8 (an encoding of Unicode), not
ISO-8859-1.

It is permissible to use regional character sets in cases where the character
set can be identified (an obvious example being email in MIME format), but
there is no such mechanism for file names, domain names, mailbox names, or
URLs.  These all must be Unicode.

---

From: Karsten Ballüder <[email-masked]>
Date: Fri, 11 Aug 2000 10:21:32 +0100 (GMT)

I've followed this discussion with interest, as I noticed months ago that the
c-client library would not allow me to create/access mailbox files with 8bit
characters in it.

Now I see that there are all kind of implications when doing such things over
some network protocol, but where is the problem with accessing files locally?

If the app asks c-client to open a file named "ÄÖÜ?" why not let it open that?
Isn't it the applications responsibility to display the name in the correct
charset? I.e. what for does the library/mailbox driver need to know the charset?

I guess you have a reason for doing this, and I'm curious to know it. Also, what
is the correct way to get around?
Can I pass a UTF-7 (or something else) encoded filenames to the c-client library
to open filenames containing 8bit characters? 

--
 Karsten Ballüder  http://Ballueder.home.dhs.org/  mailto:[email-masked]
 Physics Department, Heriot-Watt University, Edinburgh, EH14 4AS, Scotland
                     Tel. +44-(0)-131-4513068  Fax. +44-(0)-131-4513136
            "In a world without fences, who needs Gates?"

---

From: Mark Crispin <[email-masked]>
Date: Fri, 11 Aug 2000 10:21:12 -0700 (PDT)

> Now I see that there are all kind of implications when doing such things
> over some network protocol, but where is the problem with accessing files
> locally?

What happens when those files get exported via IMAP?

> charset?

How does the application know the correct charset?

What happens when your application is run in Japan and instead of outputting
proper kanji writes some garbage using European characters?  Or what happens
when your application thinks that it's writing a European name, but instead
writes some garbage in kanji?

The current software world is infested with software that does just that -- it
writes garbage.

This is because of the "just send 8-bits" attitude that was in vogue in Europe
(and elsewhere).  People were alerted to the problems of doing this, and
didn't care.  We're paying the price now.

The basic flaw in your thinking is that filenames (or mailbox names) should
have a charset.  They shouldn't.  They are global concepts, and need to have a
representation that is valid world-wide.  That leaves you with two choices:
stick to ASCII, or go to Unicode.

> I.e. what for does the library/mailbox driver need to know the charset?

For example: one day, those "local" mailbox names will get exported via IMAP,
and they then have to be exported correctly.  We're having problems now,
because people are clinging to their old ISO-8859-1 or SJIS filenames instead
of moving into the modern world.

> Can I pass a UTF-7 (or something else) encoded filenames to the c-client
> library to open filenames containing 8bit characters?

You can open filenames containing 8-bit characters.  You just can't create
them.  You have to create UTF-7 names.  When UTF-8 names are allowed, you will
have to use UTF-8 if you use 8-bit characters.  If you pass ISO-8859-1, your
software will fail in bizarre ways.

I understand that this is painful.  Unfortunately, it's something that we have
to go through to achieve international interoperability.  The longer we
procrastinate or kludge around it, the worse it will become.