Subject: 64-bit problems with c-client imap2004a

From: Chris Ross <[email-masked]>
Date: Wed, 22 Sep 2004 03:12:48 -0400

Hi.  I'm having some problems getting imap2004a working on a NetBSD
Sparc64 host.  There are quite a few:

env_unix.c:1633: warning: cast to pointer from integer of different size
env_unix.c:1636: warning: cast from pointer to integer of different size

    ..type of warnings.  The ones listed there I doubt are terribly 
important,
but some of them may very well indicate problems.

   Particularly, tho, I've found a problem that doesn't produce an error.
In tcp_unix.c, getsockname() is called, and the third argument given
to is is a size_t*.  However, on NetBSD, it calls for a socklen_t*, 
which
is a pointer to a 32-bit value, and size_t is a long, and therefore 64 
bits.

   Changing this to a socklen_t (in a few places in tcp_unix.c and 
ip4_unix.c)
fixes the problem, and causes the returned struct to actually contain
values, as getsockbyname() won't now understand it's third arg to be
a zero...

   Is there any other work done on making this safe for a 64-bit machine?

                               - Chris

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Wed, 22 Sep 2004 00:42:35 -0700 (PDT)

On Wed, 22 Sep 2004, Chris Ross wrote:
>   ..type of warnings.  The ones listed there I doubt are terribly important,

You can disregard these warnings entirely.

A few pedantic C compilers get upset at the use of void* as a bucket to 
hold an arbitrary cell, even though this is often the only way to do 
something *and* the programmer indicated his intent with an explict cast 
(which supposedly tells the compiler "I know what I'm doing, damnit!").

> to is is a size_t*.  However, on NetBSD, it calls for a socklen_t*, which
> is a pointer to a 32-bit value, and size_t is a long, and therefore 64 bits.

Sigh.  On some systems, it's an int* (and socklen_t is undefined).  On 
some, it's a socklen_t*.  And on some, it's a size_t* (and socklen_t is 
undefined).  At UW, we have examples of all three; and there's no good way 
for the code to know what it is on the system being built.  That's why 
c-client casts it to a void*.

And on some systems, an int is 16 bits, and if getsockname() expects a 
pointer to a 32-bit cell it won't do the right thing either.

There's simply no way to win on every possible platform.

I suggest that the best thing to do is have the NETBSD osdep files 
redefine getsockname() as a jacket function into the real one.  Take a 
look at the SCO osdep files for an example (e.g. how rename() gets 
redefined).  Of course, that assumes that all NETBSD is 64 bit or at least 
follows the socklen_t* convention; if not then a new port has to be 
spawned off.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Chris Ross <[email-masked]>
Date: Wed, 22 Sep 2004 12:05:49 -0400

> (which supposedly tells the compiler "I know what I'm doing, damnit!").

   Okay.  Tho, in these cases you're casting a 64-bit pointer
to and from a 32-bit unsigned int.  So, it's not wrong to
warn about it.

> undefined).  At UW, we have examples of all three; and there's no good 
> way for the code to know what it is on the system being built.  That's 
> why c-client casts it to a void*.

   Right.  I understand the problem.

> There's simply no way to win on every possible platform.

   Well, without having some sort of "smart" system, or lots
of OS-dependent #def's or typedef's for things...

> redefined).  Of course, that assumes that all NETBSD is 64 bit or at 
> least follows the socklen_t* convention; if not then a new port has to 
> be spawned off.

   Eep.  Well, I'm willing to presume all NetBSD's (at least of
a relatively recent vintage) have a socklen_t, so that will work.
Seems an ugly way to do it tho.

   Obviously, you've chosen not to go down the autoconf path
already.  But, you could have each OS define what it's
socklen type is, like you do with a few other config things
in the Makefile.  Or is that just opening a larger can
of worms than you're comfortable with (despite, IMO, doing
much that already for other things)?

                          - Chris

---

From: Mark Crispin <[email-masked]>
Date: Thu, 23 Sep 2004 13:07:15 -0700 (PDT)

> to and from a 32-bit unsigned int.  So, it's not wrong to

Actually, it is wrong to warn; at least if you believe K&R.  The whole 
point of explicit casting is to tell the compiler not to warn about 
"unsafe" casts; and there's substantial text (and precedent) about how an 
"unsafe" cast is done.  Every compiler which gives these warnings ends up 
generating the correct code anyway.

Unfortunately, some people who make a programming mistake blame the 
compiler for not warning them the way a Pascal compiler would.  Or, there 
are compiler authors who see the function pointer casts extension (K.5.7 
in the old copy of the C specification that I have) as meaning that the 
compiler is obliged to issue a warning when it is used.

> But, you could have each OS define what it's

In general, I follow the path of least resistance.  There's a great many 
ports in c-client which I would like to retire, but dare not even though 
it's been many years since I could actually test c-client on such a 
system.  After all these years, you'd think that ANSI C compilers and 
POSIX compliance would be universal...until you discover that someone's 
using system which still is in the pre-ANSI, pre-POSIX world.

As a result, I'm loathe to do something which requires a new definition 
added in all ports; instead, I do what is necessary to coerce the oddball 
that doesn't fit into fitting.  It's the "don't fix what isn't broken" 
syndrome.

Autoconf helps in some cases but not in others.  I have literally spent 
hours in fighting autoconf when it does the wrong thing for a particular 
platform.  With c-client the way it is now, it's relatively easy to add an 
oddball and get it to work without affecting anything else.  I have had 
little success doing the same with autoconf.

Of course, that's just me.  Lots of people use autoconf and are happy with 
it.  But they generally don't need to worry about oddball systems.

Anyway, if you can suggest something that can be done at the osdep level, 
I'll be very interested.  Currently, the same osdep file is used for all 
versions of BSD/i386, OpenBSD, and NetBSD.  Obviously, it would easiest if 
we could do something that applies to all of these.

-- Mark --

---

From: Chris Ross <[email-masked]>
Date: Fri, 24 Sep 2004 08:21:41 -0400

Mark Crispin wrote:
> "unsafe" casts; and there's substantial text (and precedent) about how 
> an "unsafe" cast is done.  Every compiler which gives these warnings 
> ends up generating the correct code anyway.

   Okay.  I can see that point.  But, when K&R was written,
there was much less variety in hardware implementations and
type-sizes.  At least, that's my belief.  It is, sadly, a
bit before my day.

   IMHO, I think it's not unwise to warn about casting a
64-bit pointer value into a 32-bit value.  But, you may
be right that this is just because programmers too oft tend
to assume "pointers are 32-bits", at least in the last 10 years.

> compiler for not warning them the way a Pascal compiler would.[...]

   Understood.  I don't think I'm trying to do that here, but
people do, I agree.

> Anyway, if you can suggest something that can be done at the osdep 
> level, I'll be very interested.  Currently, the same osdep file is used 
> for all versions of BSD/i386, OpenBSD, and NetBSD.  Obviously, it would 
> easiest if we could do something that applies to all of these.

   Yup.  I hope to take a look at this today.

   What's the approximate timeline for bundling and releasing 2004b?
(Which I'll then need to get updated in the NetBSD pkgsrc tree)

   Thanks!

                               - Chris

---

From: Mark Crispin <[email-masked]>
Date: Fri, 24 Sep 2004 07:54:26 -0700 (PDT)

> type-sizes.  At least, that's my belief.  It is, sadly, a

Historically, there was much more variety in hardware implementations and 
type sizes than today.  16-bit ints were common, as were machines that 
weren't byte-addressed (e.g. machines addressed by 36-bit word).

The current 64-bit problems are nothing compared to the problems in 
getting software to work with both 16-bit and 32-bit ints.

> to assume "pointers are 32-bits", at least in the last 10 years.

The whole point is that when there is an explicit cast, as opposed to an 
implicit cast, the compiler should accept that the programmer knows what 
he is doing.  Otherwise, there is no need for explicit casts.

>  Yup.  I hope to take a look at this today.

Great!  Thanks.

> (Which I'll then need to get updated in the NetBSD pkgsrc tree)

imap-2004b will be released concurrently with Pine 4.62.  I don't think 
that the Pine release is imminent, barring some newly-discovered critical 
bug.

-- Mark --

---

From: Niklas Fondberg <[email-masked]>
Date: Fri, 24 Sep 2004 19:05:08 +0200

Hi,
a silly question perhaps:

I looked at the API but couldn't find a general purpose Subject : header 
decoder for subject lines like:

=?ISO-8859-1?Q?=C5=C4=D6_test?= which is ÅÄÖ test
and =?ISO-8859-2?B?[d]SB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?= which I 
can't remember now.

I was reluctant to ask so I wrote my own BUT now I'm eager to know if 
there is one already?

/Niklas Fondberg

---

From: Mark Crispin <[email-masked]>
Date: Fri, 24 Sep 2004 10:09:09 -0700 (PDT)

On Fri, 24 Sep 2004, Niklas Fondberg wrote:
> =?ISO-8859-1?Q?=C5=C4=D6_test?= which is $BEDV(B test
> and =?ISO-8859-2?B?[d]SB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?= which I can't 
> remember now.

Routine utf8_mime2text() will convert a string with embedded quoted-words 
(such as the above) into UTF-8.

-- Mark --

---

From: Niklas Fondberg <[email-masked]>
Date: Fri, 24 Sep 2004 19:22:35 +0200

Thank You!

Mark Crispin wrote:

>> I looked at the API but couldn't find a general purpose Subject : 
>> header decoder for subject lines like:
> Routine utf8_mime2text() will convert a string with embedded 
> quoted-words (such as the above) into UTF-8.
>
> -- Mark --
>

---

From: Niklas Fondberg <[email-masked]>
Date: Mon, 27 Sep 2004 16:58:36 +0200

I'm using this now and I thought that utf8 and 8859-1 had the same 1byte 
character sets up to 255... but I get multibyte results...
For the character Å (decimal197 in 8859-1) I get the byte sequens (in 
hex) c3 85 which translates to something completely different, if it ha 
the byte as c5 I would understand.

Please inform me what I'm doing wrong...

Some code:

char *decodeSubject(char *subj) {
  SIZEDTEXT src;
  SIZEDTEXT dst;
  long res;
  char *ret;

  src.data = (unsigned char *) subj;
  src.size = strlen (subj);

  if((res = utf8_mime2text(&src, &dst)) == NIL) {
    ret = strdup(subj);
  } else {
    ret = strdup(dst.data);
  }
 
  return ret;
}
mail_fetchsubject(buf,stream,idx,(long) 127);
subj = decodeSubject(buf);
puts(subj);

-- 
Niklas Fondberg
Development Manager, User Interfaces
i3 Micro Technology AB
Finlandsgatan 62
SE-164 74 Kista, Sweden
[email-masked]
Phone:  +46 (0)8 506 388 51
Mobile: +46 (0)708 47 20 25
Fax:    +46 (0)8 506 388 75

---

From: Mark Crispin <[email-masked]>
Date: Mon, 27 Sep 2004 08:14:50 -0700 (PDT)

> character sets up to 255... but I get multibyte results...

Unicode codepoints are 20(+1) bits, whereas ISO 8859-1 codepoints are 8 
bits.

You gave the example of 0xc5 in ISO 8859-1, the character known in Unicode 
as LATIN CAPITAL A WITH RING ABOVE.

In the UCS-2 or UTF-16 encodings of Unicode, this character is 0x00c5. 
To the extent that 0xc5 == 0x00c5, these are the same; but one is an 8-bit 
value (a string is unsigned char*) and the other is a 16-bit value (a 
string is unsigned short*).

UTF-8 is an alternative form of encoding Unicode.  Its principle advantage 
is to allow 7-bit ASCII to be represented as a single byte, at the cost of 
making many other Unicode character consume more bytes than UCS-2 or 
UTF-16.  UTF-8 does this with variable byte lengths.

In your example, 0xc3 indicates the first byte of a two-byte sequence; in 
the resulting Unicode value, the highest order 5 bits are 0, and the next 
5 bits are 0x3.  0x85 is a continuation byte, of which the low-order 6 
bits of the Unicode value are 5.  The binary result is:
 	0000000011000101
 	   0   0   c   5
Read the UTF-8 specification for details on how this actually works.

And, indeed, this is U+00c5 which is LATIN CAPITAL A WITH RING ABOVE.

You should not write a new application to output ISO 8859-1 strings.  ISO 
8859 is completely obsolete.  Instead, your application should support 
Unicode and Unicode strings.

-- Mark --

---

From: Chris Ross <[email-masked]>
Date: Thu, 7 Oct 2004 08:44:32 -0400

Much time has passed.  Oops.  :-)

On Sep 22, 2004, at 03:42, Mark Crispin wrote:
> some, it's a socklen_t*.  And on some, it's a size_t* (and socklen_t 
> is undefined).  At UW, we have examples of all three; and there's no 
> good way for the code to know what it is on the system being built.  
> That's why c-client casts it to a void*.
>

   Okay.  I quick look at recent BSD/OS revisions (4.3 and 5.x) shows 
that
socklen_t is used there.  I assume the same is true of OpenBSD, but will
check that.  I should investigate FreeBSD, in case it's the same there, 
too,
and modify bsf as well as bsi.

   So, do you suggest I do what the SCO port does, and have a different 
.c
file with the wrapper function in it, or should I just put it in 
os_bsi.c?  I guess
if both bsi and bsf use it, that answers the question...

                                      - Chris

---

From: Chris Ross <[email-masked]>
Date: Fri, 05 Nov 2004 15:11:56 -0500

Mark Crispin wrote:
>> [- Stuff about using size_t* as the third argument to getsockname()
>>    breaking on 64-bit BSD systems (which expect a socklen_t*) -]

   Sorry it took me so long to get back to this.  Wedding and honey-
moon took priority.  :-)

   I believe this solves the problem.  I have tested that it doesn't
behave as if the connection is not on the imaps port when it actually
is.  I have not yet tested full functionality, but that requires a few
other things to be done as this work is on my production mail server,
and it'll take me a while to swap this code in.  I have no fear that
this will work, tho, as my test case works now, instead of failing.

   Patch attached, uuencoded to preserve space/tabbing.

> that the Pine release is imminent, barring some newly-discovered 
> critical bug.

   Okay.  Hopefully this can get into 2004b, unless it's been realized
without my noticing.  :-)

   I may try to get the pkgsrc folks to make this a local patch
until you get 2004b released.

   Thanks again for your help.

                               - Chris

---8<---8<---8<---

---

From: Mark Crispin <[email-masked]>
Date: Fri, 5 Nov 2004 14:36:03 -0800 (Pacific Standard Time)

>  Sorry it took me so long to get back to this.  Wedding and honey-

As indeed it should!  All due congratulations!

>  Okay.  Hopefully this can get into 2004b, unless it's been realized

I've done more or less this patch (also doing getpeername()) in the latest
 	ftp://ftp.cac.washington.edu/mail/imap-2004b.DEV.tar.Z

Note that this isn't the release version of imap-2004b; it's a development 
snapshot.  But it's pretty close.

-- Mark --

---

From: Chris Ross <[email-masked]>
Date: Fri, 05 Nov 2004 17:49:19 -0500

Mark Crispin wrote:
> Note that this isn't the release version of imap-2004b; it's a 
> development snapshot.  But it's pretty close.

   Excellent.  Thanks.  Had you done that on your own, or did you
do this today as a result of my patch?  Just curious.  :-)

                             - Chris

---

From: Mark Crispin <[email-masked]>
Date: Fri, 5 Nov 2004 14:49:18 -0800 (Pacific Standard Time)

> do this today as a result of my patch?  Just curious.  :-)

You're acknowledged in the source code for getspnam.c ... :-)

-- Mark --