Subject: Re: Desperate : ipop timeouts

From: Tom Combs <[email-masked]>
Date: Fri, 17 Dec 2004 08:04:27 -0500 (EST)

Hi,

 It turns out that the culprit with the pop/imap time outs was the
 iptables packet filtering. Turn this off and the intermittent time
 outs go away. Knocked myself out for two days and the fix took all
 of 10 seconds - that's the way it always seems to go.
 
 I don't know if my packet filtering ruleset is flawed or if iptables
 just can't handle the high demand of an email server for 600 people.
 
 Thanks to Mark for his quick, detailed response.
 
 Now I can go on holiday with some peace of mind!
 
 Best wishes,
 
                 Tom Combs

------------- Begin Forwarded Message -------------

Date: Thu, 16 Dec 2004 09:06:31 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: Tom Combs <[email-masked]>
cc: [email-masked]
Subject: Re: Desperate : ipop timeouts
MIME-Version: 1.0
X-NHMFL-MailScanner: Found to be clean
X-MailScanner-MCPCheck: MCP-Clean, MCP-Checker (score=0, required 1)
X-NHMFL-MailScanner-SpamCheck: not spam, SpamAssassin (score=-2.599, required 5, 
autolearn=not spam, BAYES_00 -2.60)
X-MailScanner-From: [email-masked]
X-OSPREY-MailScanner: Found to be clean

The connection timeouts that you are getting have nothing to do with 
ipop3d; they indicate that ipop3d has not even started!  So, you were on 
the right track with increasing the number of connections allows in 
xinetd.

Unfortunately, problems such as these require more investigation before 
any definite solution can be found.  The first thing to determine is what 
your POP users "new mail check interval" is.  Unlike IMAP, POP requires a 
new connection for each check of new mail.  So, if the user is checking 
for new mail every second, a new POP session is opened every second!

Opening a POP session is expensive; SSL/TLS encryption needs to be 
negotiated, login authentication needs to be negotiated, and the mailbox 
has to be processed.  On top of that, [x]inetd limits the number of POP 
sessions that can be spawned each minute.

So, take a look at the mail syslog and see if you can find any obvious 
users who are abusively checking for new mail at an excessive rate.  Quite 
frankly, people who need faster than once/minute checking are badly in 
need of a life; and I personally advocate once every 3 minutes.

Your server should be more than capable of the user load.  I think that 
you are running up against [x]inetd.

Also -- gently -- suggest to your users that they really should be using 
IMAP instead of POP.  IMAP notifies of new mail within the session, so it 
isn't necessary with a good client (such as Pine) to re-open new sessions 
all the time.

I can't speak intelligently about what either Outlook or Eudora may do 
with IMAP; it sufficies for me to know that Pine is better. :-)

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

------------- End Forwarded Message -------------

--
Tom Combs                                      E-mail: [email-masked]
National High Magnetic Field Laboratory        Phone:  (850) 644-1657
1800 E. Paul Dirac Drive                       Tallahassee, FL 32310

---

From: Erik Kangas <[email-masked]>
Date: Fri, 17 Dec 2004 08:32:31 -0500

Hello,

We have found that it is the iptables connection state tracking modules 
that casuse issues like this, due to issues with depth of queue, data 
storage in the kernel, etc.  If you have only iptables rules that do not 
use connection state tracking, then you should be fine (though you may have 
to restart the machine to fully switch from connection state tracking to 
not, as once you start it, some kernel-level flags may get set that keep it 
up when iptables are used).

Also, we have found that the use of "tcpserver" is ligher weight than 
"xinetd" for connection processing.

Best,

-Erik Kangas
Lux Scientiae

-- 

Erik Kangas, Ph.D. --- President of Lux Scientiae, Incorporated
[email-masked]  --- http://luxsci.com

Office Phone:        1-617-507-2162
Cell Phone:          1-617-596-9558        P.O. Box 326
Luxsci Toll Free:    1-800-441-6612        Westwood, Massachusetts
LuxSci FAX:          1-413-332-0598        02090, USA

---

From: Rich Graves <[email-masked]>
Date: Fri, 17 Dec 2004 09:39:11 -0500 (EST)

> up when iptables are used).

You ought to be ok with connection tracking (which is required for things
like incoming and outgoing FTP, unless everyone agrees on active/passive
mode) as long as you have ACCEPT rules without the --syn qualifier before
RedHat's default ESTABLISHED line.

I.e., this is good:

# /etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 993 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 995 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 110 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 143 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

And this can be bad under high load:

-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 993 --syn -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 995 --syn -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 110 --syn -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 143 --syn -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
-- 
Rich Graves <[email-masked]>
UNet Systems Administrator