Subject: some c-client questions

From: Karsten Ballueder <[email-masked]>
Date: Fri, 11 Feb 2000 12:19:56 +0000 (GMT)

I have a few problems with c-client, and hoped that someone
might have an idea how to overcome them. Sorry for this long
mail, but I thought I better bundle my questions rather than
flooding you with lots of individual mails. Fortunately
their not all bug reports.

1. It does not allow me to use 8 bit characters in mailbox
   names. I don't know if this is an IMAP specification, but
   at least for accessing local mailbox files this should be
   possible.
   Is there any way around it?

2. I have had several requests from Mahogany users
   (http://mahogany.home.dhs.org/) whether I couldn't make
   it run as root. So far that's not possible as c-client will
   not work with uid 0.
   Could you give me any hints on how difficult it would be
   for me to change this? I was told that UID 0 is used
   internally, but maybe I could modify it to use something
   like "-1" or "65535" instead (commonly used as UID for
   "nobody" on unix systems anyway).

3. c-client refuses to read one of my mailbox files on which
   the program crashed earlier, complaining about inconsistent
   header sizes.
   Is there any way to get it to recover from such a
   situation, to at least parse the non-corrupted bits of the
   folder without crashing.

4. c-client uses syslog() and abort() in lots of "critical"
   situations. For a user-level program that is not a good
   choice. So I am thinking of replacing syslog() with calls
   to a messagedialog in our program and to do something
   about the abort() calls (although saving everything else
   before returning from the "syslog" call might be safe
   enough). Has anyone tried this before?

5. After upgrading from 4.4 to 4.7a of the toolkit we came
   across some corrupted message contents. Cannot say much more
   as I haven't found out why, but it appears that there might
   be some kind of memory corruption bug in it - but could be
   in our program as well.

   Also, 4.7a occasionally produces SIGPIPE interrupts on
   POP3 connections, something 4.4 never did. Have there
   been any changes in the interrupt handling or networking
   code which could cause this?

After all the questions, something positive as well: :-)
I have applied and slightly modified the "Pine+SSL patch"
that can be found on some web page. And I'm happy to report
that c-client works just perfectly fine with
POP/IMAP/SMTP/NNTP over SSL now. If anyone is interested in
the details, http://mahogany.home.dhs.org/

Any help would be greatly appreciated! 
--
 Karsten Ballüder  http://Ballueder.home.dhs.org/  mailto:[email-masked]
 Physics Department, Heriot-Watt University, Edinburgh, EH14 4AS, Scotland
                     Tel. +44-(0)-131-4513068  Fax. +44-(0)-131-4513136
            "In a world without fences, who needs Gates?"  

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Fri, 11 Feb 2000 09:47:29 -0800 (PST)

On Fri, 11 Feb 2000 12:19:56 +0000 (GMT), Karsten Ballueder wrote:
> Sorry for this long
> flooding you with lots of individual mails.

It's actually better to put separate issues in separate messages.  This
permits prioritizing responses, and getting a faster response out for easier
issues.

>    at least for accessing local mailbox files this should be

8-bit characters in IMAP mailbox names are reserved for future use with UTF-8
(8-bit Unicode).  8-bit is specifically forbidden in mailbox names today, to
exterminate current non-standard use of ISO 8859-1, S-JIS, etc. names.

>    Is there any way around it?

IMAP permits Unicode names today, using a mechanism called "modified UTF-7"
which is a 7-bit encoding.  Refer to section 5.1.3 of RFC 2060.

If you have 8-bit names on your local filesystem in some non-Unicode character
set, there's no way for the IMAP server to know whether this set of octets is
ISO 8859-1, KOI-8, S-JIS, GB2312, BIG5, etc.  All the world is not ISO 8859-1.
Since IMAP mailbox names are Unicode, any 8-bit names have to be converted to
Unicode; but without knowing the character set it's a bit difficult.

>    it run as root. So far that's not possible as c-client will

That's right.  You may be able to get a local-only program to half-work when
running as root (Pine does so), but there's no chance with the daemons.

>    for me to change this?

I'm not going to support it, *ESPECIALLY* not in imapd.  It is a very bad
idea.  The problem with kludging it as you propose is that it opens up HUGE
security holes.

The best answer is "only cretins run non-essential programs as root.  Don't do
it."

>    the program crashed earlier, complaining about inconsistent

Have you tried reading the file in emacs (not vi)?  c-client should have told
you where the problem was located; it should be easy to go to that position in
emacs, figure out what's wrong, and fix it.

>    situations.

Those situations shouldn't ever happen.  If they do, it's either a c-client
bug or a bug in your program.  Rather than sweep the problem under the rug,
it's better to fix the bug.  Since you didn't say what syslog() you were
getting, I can't comment on what the bug may be.

>    across some corrupted message contents.

Haven't had that happen.  Are you using the correct access means to get data,
in particular, all the mail_fetch_body(), etc. calls?  You must *NOT*
reference the "PARTTEXT" members of the various structures -- those are
private to c-client!  Their usage is driver-specific!!

>    POP3 connections, something 4.4 never did.

Do you use /etc/mlock?

Does your SSL code use pipes?

---

From: Karsten Ballueder <[email-masked]>
Date: Fri, 11 Feb 2000 19:58:20 +0000 (GMT)

> private to c-client!  Their usage is driver-specific!!

I will double check that.

 > >    POP3 connections, something 4.4 never did.

It only happens if the server and client run on the same
host, i.e. under extremely fast network conditions. The same
program works perfectly well with a remote POP server.

 > Do you use /etc/mlock?

What is that?

 > Does your SSL code use pipes?

The POP3 problem is there no matter whether I compile in SSL
or not.

---

From: Mark Crispin <[email-masked]>
Date: Fri, 11 Feb 2000 12:01:17 -0800 (PST)

> program works perfectly well with a remote POP server.

Then you probably don't use it, so it doesn't matter.

> The POP3 problem is there no matter whether I compile in SSL

The POP3 client code doesn't use any form of piping, and we've never seen
anything like that.

---

From: Karsten Ballueder <[email-masked]>
Date: Wed, 16 Feb 2000 18:54:31 +0000 (GMT)

Following up on my earlier mail about corrupted unix mailbox
folders after upgrading from 4.4 to 4.7a.

 > I will double check that.
Have done so now and do not find any suspicious code. I use
mail_fetchbody_full() to get the contents of the individual
parts.

The problem seems to be always exactly the same: The message
content (text) starts immediately after the X-UID: XXX
header entry. instead of an empty line to separate the
header and body, I find messages like this:

[some header lines]
X-UID: 1234 beginning of
message text on same line as X-UID
[...]

These messages have been collected from an IMAP server and
stored to a unix mailbox folder, using the following bit of
code:

   STRING str;
   INIT(&str, mail_string, (void *) msg.c_str(), msg.Length());

   bool rc = ( mail_append(
      m_MailStream, (char *)m_MailboxPath.c_str(), &str)
               != 0);

Can you see anything wrong in this bit?

It's a pity that the c-client mailbxo parsing code doesn't
handle corrupted folder contents gracefully. It bombs out
from unx_rewrite_new() by calling fatal() rather than just
reporting an error and skipping that message.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 16 Feb 2000 12:33:20 -0800 (PST)

> message text on same line as X-UID

I don't see how this could happen.  The only code that writes X-UID: is
guaranteed to write two newlines after the UID value.

>    INIT(&str, mail_string, (void *) msg.c_str(), msg.Length());

How did you load msg?  Did you try printing out what was in msg before the
append?  I suspect that the problem is in there, somehow.

> reporting an error and skipping that message.

Are you claiming that this still happens in imap-4.7a?  If so, this is the
first that I have heard of it.  Perhaps you ought to file a bug report with me
with the details of what happened, because otherwise I'm going to keep on
thinking that it doesn't happen any more.

All fatal() calls issued from c-client represent either a bug in c-client or
in the main program.

The fatal() calls in the rewrite routine are NOT about corrupted folder
contents.  They are about corrupted c-client internal data structures.  Now,
it may be that there are people in this world who prefer software that, when
it detects that it has been corrupted internally, nonetheless blithely goes on
and continues irreverable writing of user data based upon the corrupted data.
I'm not one of those people.

In imap-4.6 and imap-4.7, there were certain forms of folder contents
corruption that lead to c-client internal corruption, specifically:
 1) CRs embedded in the middle of a header line (that is, CR but no LF)
 2) bogus headers that used the same field names (Status, X-Status,
    X-Keywords, X-UID) as the internal header
These were bugs, and were fixed as soon as I found out about them.  4.7a has
those fixes.

Of course, c-client can't save itself from main program mangling of the stack
or the heap.  That's what I would look for first.