Subject: [Imap-uw] Long delays & Connection refused

From: [email-masked]
Date: Mon, 9 Jan 2006 11:02:12 -0600

I have been having a problem for a while now.  Recently, it's been 
narrowing down to either inetd or something related to network queues. I'm 
posting this message to help rule out uw-imapd. 

During high load (about 18 imapd logins/second from a webmail system) we 
start to get long delays when connecting to port 143.  Sometimes, even a 
connection refused when attempted from outside localhost.  Delays possibly 
range from 20 seconds to 2 minutes.  When telneting to port 143 from 
localhost, the delays are evident before the IMAP capabilities logo 
splashes.  After IMAP's banner is shown, everything seems fine.  It's 
before getting the banner that lags.  Running a second copy of uw-imapd on 
port 146 through inetd works fine and shows no delays when the problem is 
evident on port 143.  Once logged into IMAP on port 143, mail activities 
are fine.  It's pre-login that runs extremely slow, sometimes with port 
143 refusing the connection.  I'm leaning towards inetd or the kernel not 
keeping up with the socket activity.  Any suggestions or things to test to 
help rule out the imapd binary?

It's a gso compile with passwdtype=std

Solaris 8 on a v880, version d

Before I go into all of the kernel settings and what not, I'd like to see 
if anyone has any recollection of previously related problems.  Thanks!

-Ken
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jan 2006 09:17:39 -0800 (PST)

On Mon, 9 Jan 2006, [email-masked] wrote:
> connection refused when attempted from outside localhost.

This sounds like your [x]inetd is throttling the service port.  Refer to 
the following FAQ:
 	http://www.washington.edu/imap/IMAP-FAQs/index.html#7.16
for more information.

Defective webmail implementations are known to do this.  The typical 
problem is that they spawn a new IMAP session for each user's mouse click 
instead of keeping a single IMAP session open per user.  That bad behavior 
defeats much of what is useful in IMAP.  However, making a mail access 
session be stateful instead of stateless requires some complex 
programming, and only a few webmail programs (such as UW's WebPine) do it.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Don Hayward <[email-masked]>
Date: Mon, 9 Jan 2006 13:22:51 -0500

Hi Mark,

On Mon, 9 Jan 2006, Mark Crispin wrote:

> instead of stateless requires some complex programming, and only a few 
> webmail programs (such as UW's WebPine) do it.

Is UW's WebPine available? Thanks.

Don

Don Hayward  ISDIV		[email-masked]
Mote Marine Laboratory		Office: 941.388.4441x403  Cell: 941.350.8503
1600 Ken Thompson Parkway	Fax: 941.388.4312
Sarasota, FL 34236		See: http://www.mote.org
Independent, non-profit, marine and estuarine research and education facility.
use "DISCLAIMER"; # We run Debian Linux
Ordinary men and women have a right to peace, material comfort, self-respect,
and a touch of joy in living. Eric F. Goldman,  [1915-1989]
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: [email-masked]
Date: Mon, 9 Jan 2006 12:59:56 -0600

Agreed.  However, I haven't seen any messages in the syslog that explain 
our behavior.  This was one of my initial thoughts but since I didn't see 
any warnings or error messages, I dismissed it.  I thought the behavior of 
a flooded inetd was more like an on/off switch rather than congestion? 

=============================
Ken Koch
Information Systems
314.935.8315
=============================

Mark Crispin <[email-masked]> 
Sent by: [email-masked]
01/09/2006 11:17 AM

To
[email-masked]
cc
[email-masked]
Subject
Re: [Imap-uw] Long delays & Connection refused

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jan 2006 11:06:46 -0800

> a flooded inetd was more like an on/off switch rather than congestion?

I don't know.  Perhaps your syslog level is set so those messages don't 
show up?  Or maybe this is something else in your [x]inetd.  Port 143 
listening problems are [x]inetd problems rather than imapd problems; but 
beyond that I can't be authoritative about what may be wrong.

However, there is one imapd issue that you need to look into; at startup, 
it will do a DNS query to get the local server host name, and one to get 
the client host name.  If your DNS does not reliably do reverse lookups 
for your webmail client expediently, that could cause similar symptoms to 
what you report.

Good luck, and let us know what you find!

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jan 2006 11:14:28 -0800

> Is UW's WebPine available? Thanks.

To the best of my knowledge, there is not yet a general release outside of 
UW; as far as I know, there are still some pending technical and 
adminstrative issues (licensing, etc.) that need to be resolved before any 
decision is made about a general release.

I'm not directly in the loop on this, so your best bet for authoritative 
information would be to ask [email-masked] to get on the list of 
people who are interested in being informed about WebPine distribution.

I personally believe that WebPine is "way cool".  It's been in production 
use at UW for a few years now.

I know that there are other webmail products out there that hold static 
IMAP sessions, but I don't know any names.

-- Mark --

---

From: Mark Sirota <[email-masked]>
Date: Mon, 09 Jan 2006 14:16:01 -0500

--On January 9, 2006 11:02:12 AM -0600 [email-masked] wrote:
> connection refused when attempted from outside localhost.  Delays
> possibly range from 20 seconds to 2 minutes.  When telneting to port 143
> from localhost, the delays are evident before the IMAP capabilities logo
> before getting the banner that lags.

Are you using tcp wrappers (tcpd) for imap?  Might it be configured to do
either ident or reverse DNS lookups?  I'm guessing that either a firewall
is dropping the ident queries rather than sending a TCP reset, or
something's funky about the reverse lookup.

The quickest way to test this theory is to eliminate tcp wrappers from
inetd.conf on the imap line, and see if the problem instantly goes away.
The quickest non-disruptive way to test this theory is to sniff the 
traffic.

Mark

---

From: Mark Sirota <[email-masked]>
Date: Mon, 09 Jan 2006 14:18:01 -0500

> IMAP sessions, but I don't know any names.

The coolest is Prayer Webmail from the University of Cambridge.  It uses
HTTP/1.1 persistent connections from the web client to the web server, and
holds open the IMAP connections on the back end.  It's mind-blowingly fast
for a webmail service, but currently somewhat lacking in functionality
compared to the more mature products.

<http://www-uxsup.csx.cam.ac.uk/~dpc22/prayer/>

Mark

---

From: [email-masked]
Date: Mon, 9 Jan 2006 13:19:48 -0600

We are not using tcp wrappers.  Plain old Solaris inetd. 

We have excellent response time throughout normal load.  It is only during 
high load that port 143 lags.  I can even telnet localhost 143 and have a 
delay upward of 50 seconds before getting any kind of imapd banner.  If it 
were network related, I would think localhost would be exempt.  Any other 
ideas?

Sniffing the traffic coming in during a problem state shows the initial 
SYN reaching the port, but absolutely no response going back.  This type 
of evidence is what's leading me to some kind of kernel related queue that 
is backloged or dropping new attempts.

Mark Sirota <[email-masked]> 
01/09/2006 01:16 PM

To
[email-masked], [email-masked]
cc

Subject

Mark

---

From: Mark Crispin <[email-masked]>
Date: Mon, 9 Jan 2006 12:12:12 -0800

On Mon, 9 Jan 2006, Michael Russell wrote:
> As I started tracing this, I 
> found that an attempt to resolve the private IP address (192.168.10.11) of 
> our mail server was being done by the imap server and this attempt was 
> timing out.  The private IP was NOT in the /etc/hosts file and we do not 
> have our own DNS servers.

Yup, that's the sort of mysterious problem that can cause one to scratch 
one's head for quite a long while.  In general, you should always have an 
/etc/hosts entry for the local site and not use the DNS for local lookups.

-- Mark --

---

From: "Clive McDowell" <[email-masked]>
Date: Wed, 11 Jan 2006 16:57:22 -0000

> <http://www-uxsup.csx.cam.ac.uk/~dpc22/prayer/>

I'd like to second that. We use it here at Queen's University, Belfast. I'd be interested in WebPine for comparison though.

 Clive McDowell

Information Services
The Queen's University of Belfast