Subject: more hanging ftpd processes

From: Donald Ball <[email-masked]>
Date: Thu, 6 Mar 1997 12:24:31 -0500 (EST)

We're still seeing hanging ftpd processes on sunsite, solaris 2.5.1, and
they don't seem to be related to uploads. Trussing a typical hung process
(hanging for 2 days now) reveals:

# truss -p 12781
write(12, " ^FD077F f10DCD3 \ QBEE9".., 16384) (sleeping...)

The stack is:

# /usr/proc/bin/pstack 12781
12781:  in.ftpd -a
 df6795f4 write    (c, 5f248, 4000)
 df6795f4 _libc_write (c, 5f248, 4000, df678840, 0, 3ec70) + 8
 00016f64 send_data (4000, 5f248, 3ec00, 3f20a, 3b000, 3ec60) + 208
 00016040 retrieve (0, 4ae50, 14, 0, 38678, 331c264c) + 51c
 0001aba0 yyparse  (3e9e0, 146d0, 3d7f4, 3a710, 3e6c0, 0) + b78
 000146c8 main     (38c00, dfffff24, dfffff38, 3de2c, 1, 0) + 964
 00013ce8 _start   (0, 0, 0, 0, 0, 0) + 5c

and the file descriptor to which it's writing is:

# /usr/proc/bin/pfiles 12781
12781:  in.ftpd -a
  ...
  12: S_IFCHR mode:0000 dev:109,8 ino:9464 uid:0 gid:0 rdev:42,34959
      O_RDWR

The process is attached to 204.17.119.64:

# /usr/ucb/ps -uwwwwww 12781
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
ftp      12781  0.0  0.2 1716 1172 ?        S   Mar 04  0:00 ftpd:
204.17.119.64: anonymous/mozilla@: RETR

And netstat reveals the socket is in CLOSE_WAIT:
# netstat -n -f inet | grep 204.17.119.64
152.2.254.81.21      204.17.119.64.4442    8192      0 10122      0
CLOSE_WAIT
152.2.254.81.56495   204.17.119.64.4443       1      0 10122      0
CLOSE_WAIT

Obviously this is not hanging on upload since it's hung trying to write to
is a tcp stream, not a file. Anyone have any clues on this?

- donald

---

From: Kent Landfield <[email-masked]>
Date: Thu, 6 Mar 1997 11:37:19 -0600 (CST)

What version of the software are you using ?

-- 
Kent Landfield                        Phone: 1-817-545-2502             
The Landfield Group                   FAX:   1-817-545-7650             
Email: [email-masked]             http://www.landfield.com/
Please send comp.sources.misc related mail to [email-masked].
Search the Usenet Hypertext FAQ Archive at http://www.landfield.com/faqs/

---

From: Mark Whetzel <[email-masked]>
Date: Thu, 6 Mar 1997 11:56:20 -0600 (CST)

> (hanging for 2 days now) reveals:

Seems to me I recall there were a flurry of patches to the Academ releases
to correct various hung ftpd's and add timeouts to sections of the code.

I recall that problems were associated with cases where the user did not 
disconnect the session gracefully.

 [snip lots of diag information]

> is a tcp stream, not a file. Anyone have any clues on this?

I bet that it does not matter, if a function was started, then the
user session abnormally terminated (link outage, ect..) then it would hang.

-- 
Mark Whetzel     My comments are my own, not my company's.
Western Geophysical - A division of Western Atlas International Inc.,
A Subsidary of Western Atlas Inc.  addr: [email-masked]
VOICE: (713) 963-2544              FAX: (713) 963-2758

---

From: Donald Ball <[email-masked]>
Date: Thu, 6 Mar 1997 15:22:30 -0500 (EST)

beta12

- donald

---

From: David Greenman <[email-masked]>
Date: Thu, 06 Mar 1997 13:37:04 -0800

...
># netstat -n -f inet | grep 204.17.119.64

   This is probably caused by a bug in the Solaris TCP/IP implementation.
There are a few timer/timeout related bugs in the BSD networking code (which
Solaris uses) that I fixed in FreeBSD about a year ago:

RCS file: /home/ncvs/src/sys/netinet/tcp_timer.c,v
...
----------------------------
revision 1.16
date: 1996/04/15 03:46:33;  author: davidg;  state: Exp;  lines: +2 -2
Two fixes from Rich Stevens:

1) Set the persist timer to help time-out connections in the CLOSING state.
2) Honor the keep-alive timer in the CLOSING state.

   This fixes problems with connections getting "stuck" due to incompletion
of the final connection shutdown which can be a BIG problem on busy WWW
servers.
----------------------------
...
revision 1.8
date: 1995/07/29 18:48:43;  author: davidg;  state: Exp;  lines: +17 -1
Add connection drop capability for persist timeouts.

Reviewed by:    Andras Olah
Obtained from:  4.4BSD-lite2 via W. Richard Stevens
----------------------------

   The hang that you're seeing is likely caused by the bug fixed in rev 1.8
above. The comment that was added in 1.8 explains further:

+               /*
+                * Hack: if the peer is dead/unreachable, we do not
+                * time out if the window is closed.  After a full
+                * backoff, drop the connection if the idle time
+                * (no responses to probes) reaches the maximum
+                * backoff that we would use if retransmitting.
+                */

-DG

David Greenman
Core-team/Principal Architect, The FreeBSD Project

---

From: Jim Davis <[email-masked]>
Date: Thu, 6 Mar 1997 17:10:15 -0700 (MST)

There are a number of bugs in the Sun bugs database about problems with
things hanging in a CLOSE_WAIT state: 

1259260	TCP connections get stuck in CLOSE_WAIT state forever
1188068 TCP connections remaining in CLOSE_WAIT indefinitely
1242876 TCP port hangs in CLOSE_WAIT or BOUND state and had to reboot.
1166730 Connections stay in CLOSE_WAIT state for ever
1220193 large number of connections in CLOSE_WAIT

Curiously all these bug reports have a 'closed' state, but there is no
corresponding patch id.  Or workaround listed. 

None mention (wu-)ftpd explicitly, but they all seem to involve clients
connecting to servers and then suddenly going away. 

Incidently, the promising-sounding ndd variable 'tcp_close_wait_interval'
doesn't have anything to do with the CLOSE_WAIT state.  (It's how long a
connection stays in the TIME_WAIT state.)