Subject: blocking read in tcp_socket_open() (tcp_unix.c)

From: Vadim Zeitlin <[email-masked]>
Date: Tue, 28 Aug 2001 18:02:45 +0200 (CET)

Hello,

 There is a bug in tcp_socket_open() function implementation for Unix (in
imap2000b and also the imap2001 snapshot I have): it doesn't respect the
open timeout value. The reason is that it does a blocking read immediately
after (successful) select() return and this read() can, and does, block for
arbitrarily long period of time here (Linux 2.2).

 I don't really understand the purpose of this check but assuming it is
useful, why not make it zero-byte for all platforms except Solaris (where,
according to the comment in the file, read(0) crashes)? The current
situation is hardly acceptable as my program may be blocked for several
_minutes_ in tcp_socket_open() before returning with an error. I'd be
grateful if something could be done about it.

 Thank you,
VZ

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Tue, 28 Aug 2001 20:30:08 -0700 (PDT)

I think that you are confused; what you claim is a "bug" in tcp_socket_open()
is not a bug.  The read() isn't done unless the select() returns either data
available to read or error on the socket.  Furthermore, every protocol has a
startup message from server to client immediately after session open, so there
will be data on the socket promptly in any case.  It would certainly not be
"minutes" in any case.

The purpose of the read() is to make sure that tcp_open() does not return
successful unless the connection is open.  Since a non-blocking connect() was
done in order to implement open timeout via select(), there is no indication
otherwise whether the connection actually opened.

It doesn't matter whether it is read() of 0 bytes or 1 byte; that read will
not return until there is definitely an error or definitely data.  But the
previous select() won't supposed to return either in that case.

As I said above, I think that you are confused.  I'm sure that you are seeing
a problem, but I need more information to diagnose it.  For example, how did
you conclude that that read() is the cause of your "minutes" of delay?  Are
you aware of the rsh-IMAP feature, and have you verified that that isn't the
problem?

---

From: Vadim Zeitlin <[email-masked]>
Date: Wed, 29 Aug 2001 13:04:41 +0200 (MET DST)

> will be data on the socket promptly in any case.  It would certainly not be

 Well, I agree with this and it was my understanding of the code when I
tried to hunt down this problem before. However yesterday I saw that it was
blocking inside this read() and it was sitting there for a really long time
- at least a couple of minutes.
 
> otherwise whether the connection actually opened.

 Sorry, I still don't understand. If select() returns that input is
available, why can't we assume that the connection was actually opened?
 
> not return until there is definitely an error or definitely data.

 Yes, but it can take a long time for it to determine this...

> But the previous select() won't supposed to return either in that case. 

 Except that it will return after a (short) timeout in either case.

> you conclude that that read() is the cause of your "minutes" of delay?

 I've waited for some time (~30 seconds) and then broke into debugger. gdb
showed that the program was inside read(). I've put a breakpoint on
the next line in tcp_socket_open() and continued the program - the
breakpoint was hit after another minute or so.

 It is hard to reproduce this bug but it's not the first time it happens.
However thinking a bit more about it, I think I understand the problem:
the IMAP server I use is rather flaky (it's under my ISP control. not mine,
so I can hardly do anything about it) and it seems that it is just
overloaded, so sometimes there may be long pauses (although normally it's
rather fast). During normal operation it leads to timeouts and "connection
lost" messages but if it happens right after the connection the client just
blocks as the TCP connection is well alive but nothing comes over it. You
will probably argue that it's not a network problem then but I believe that
timeouts should still apply - how can a [single threaded] client cope with
this situation otherwise?

> Are

 Yes, disabling rsh in my client was the first thing I did (although it is
off topic, I really don't think this should be enabled by default - just
how many servers support it nowawdays and how often is it useful?).

 BTW, another OT gripe: the ISP server I use (imap.online.fr) has 3 IP
addresses. Normally this is just fine but if the network is completely
down (i.e. the modem hung up) c-client tries to connect to each of these
addresses in turn although it is clear that it can't succeed. Is there any
way to disable this behaviour (ideally it should be smart enough to
distinguish between various network problems but this would be difficult)?

 Thank you,
VZ

---

From: Mark Crispin <[email-masked]>
Date: Wed, 29 Aug 2001 19:41:46 -0700 (PDT)

> available, why can't we assume that the connection was actually opened?

select() returns that there is either an error on the connection or there is
data to be read.

The reason for using non-blocking connect() and select() is so that open
timeouts would be implemented.  Otherwise, you'd be stuck with the fixed UNIX
default of 75 seconds.  The downside is that if there is an error, there's no
way to find that fact out until you try to read() or write() to the
connection.

Hence the 1-byte read().  It's the only way to know whether or not the
connection is actually open.

> blocks as the TCP connection is well alive but nothing comes over it.

The most likely explanation for what is happening is this:

You have network connectivity problems, and these problems cause ICMP
destination unreachable signals.  ICMP destination unreachable causes a
blocking connect() to fail, but is completely ignored by read() and write()
(since otherwise a transitory router flap would nuke your connection -- this
was a major bug in BSD UNIX on LANs in the late 1980s).

But, since the connect() was non-blocking, it doesn't get the error.  The
error causes the select() (which implements the timeout) to complete, and then
it gets to the read(), which ignores the ICMP destination unreachable, and
thus you have to wait for the read() to time out internally.

I don't know of any way to work around this problem.  This is how UNIX sockets
are specified.  If there's a way to test for ICMP unreachable prior to doing
the blocking read(), I don't know of it.  As far as I know, the choices are
either:
 . use blocking connect(), and live with the 75 second fixed open timeout
 . use non-blocking connect() + select() + blocking read(), which gives you
   variable open timeouts, and live with the delays when your network is
   broken.

> how many servers support it nowawdays and how often is it useful?).

We use it quite extensively.  Your client should not unilaterally disable it.
It should allow the user to decide.

> distinguish between various network problems but this would be difficult)?

You can specify an explict IP address instead of the name.

I think that you've verified what I said.  Your network connectivity is
broken.

---

From: Vadim Zeitlin <[email-masked]>
Date: Thu, 30 Aug 2001 19:56:30 +0200 (CET)

On Wed, 29 Aug 2001 19:41:46 -0700 (PDT) Mark Crispin <[email-masked]> wrote:

MC> Hence the 1-byte read().  It's the only way to know whether or not the
MC> connection is actually open.

 Why not do a non blocking read()? If select() returned that input is
available, it should succeed, shouldn't it? I.e. why wouldn't it work if I
just move the line

      fcntl (sock,F_SETFL,flgs);/* restore blocking status */

after read()?

MC> You have network connectivity problems, and these problems cause ICMP
MC> destination unreachable signals.  ICMP destination unreachable causes a
MC> blocking connect() to fail, but is completely ignored by read() and write()

 How does it explain that it does connect finally? No, I think my
explanation is correct: the server is just overloaded and so sometimes it
accepts the connection but doesn't put the greeting string until much
later.

MC> We use it quite extensively.  Your client should not unilaterally disable it.
MC> It should allow the user to decide.

 It does, but it's off by default.

MC> You can specify an explict IP address instead of the name.

 Sure I can (and did), but what about the users? The programs are supposed
to be user-friendly nowadays...

MC> I think that you've verified what I said.  Your network connectivity is
MC> broken.

 I can run the sniffer to check it, but I really doubt it is (well, my
connectivity is surely fine as I don't get such problems with any other
programs/servers, maybe there is something wrong at the ISP end, of course,
but I don't think so).

 Regards,
VZ

---

From: Mark Crispin <[email-masked]>
Date: Thu, 30 Aug 2001 11:01:09 -0700 (PDT)

>       fcntl (sock,F_SETFL,flgs);/* restore blocking status */

Because when there really is an error on the socket, the non-blocking read
would not return the error.  The whole purpose of the read() is to get the
error.  If I'm lucky, it will return EWOULDBLOCK but not on all platforms.

>  How does it explain that it does connect finally?

ICMP destination unreachable means exactly that; the network can not deliver
to that destination.  Only blocking connect() pays attention to that; read()
does not.

In this one case, the read() does the blocking that the connect() would have
done.  It blocks until either there's some other error (including internal
timeout) or the connection gets established and data comes through.

This is how sockets are designed.  It's a deficient design, but it's widely
implemented and we're stuck with it.

> No, I think my

If that explanation was correct, and I doubt that it is, it would not make one
bit of difference to you.

After opening the TCP connection, c-client would read the greeting banner from
the host.  It will block to do so.  You will not get back control until after
the greeting banner is read.

That's right; mail_open(), nntp_open(), and smtp_open() will NOT return until
they have read the entire greeting.

> to be user-friendly nowadays...

The entire reason why hosts are multi-homed, and advertise that fact, is so
clients automatically try all the offered addresses.  The practice of "try the
first address and give up if it can't connect to that address" went out of
fashion in the mid-1980s.

If a client that I used followed that practice, I would not be able to read my
own mail on one of my servers, since one of the IP addresses on that server is
inaccessible unless you are on the correct side of a firewall.

Please understand.  You have a relatively uncommon problem, caused by your
network connectivity and/or server being broken.

It is too late in the imap-2001 development cycle to make a change which can
potentially impact many people (particularly on an OS variant that I can't
test here) in a bad way.  With any luck tommorrow's FTP tarball will be a
release candidate.

I have received a few technical suggestions on how I can test for error, and I
will consider them during development of the next release.  It just can't be
done now.  There's too many OS variants that would be affected.