Subject: Re: Ftp throuput rates for dial-up PPP modems.

From: [email-masked] (Dan Schnur)
Date: Mon, 1 Jul 1996 13:44:26 -0700

Odd, I get the same behavior.  I thought it was just me :)  If anyone knows
a solution for this problem, I would be quite intrested.  I'm using 2.4 on
a Sparc 20 with Solaris 2.5...

  -D

>  I'm part of an ISP and am currently facing the following problem:
>My clients, using dial-up PPP, can't seem to get more than 1/4's worth
>of their total modem throughput (14400, for example), when transferring
>via ftp. Any special reason why? I suspect the ftp protocol itself, but
>can't be sure.
>  Let me illustrate:
>
>        14400 modem = 1.8 kbytes/sec --> user gets roughly 0.4 kbytes/sec.
>        28800 modem = 3.6 kbytes/sec --> user gets roughly 1.0 kbytes/sec.
>
>  What's odd about all this is that if you use windows 95, and bring up the
>modem monitor, it *shows* throughput that nears the modem's full throughput.
>It's just the ftp-data part that is much less.
>
>  Any clues, lights, insights, hints?

--------------------------------------------------------------
               Dan Schnur, System Administrator
 AMUG Internet Services           Telephone    (602) 553-8966
 4131 North 24th Street #A120     Fax          (602) 553-8771
 Phoenix, Arizona  85016          Net:        [email-masked]
 ftp://ftp.amug.org/                     http://www.amug.org/
--------------------------------------------------------------

---

From: Joseph Backo <[email-masked]>
Date: Mon, 01 Jul 1996 21:42:57 -0400

> >  Any clues, lights, insights, hints?

This is a known problem with the tcp/ip stack on Suns.  Patches should
be *real soon now* to handle these problems.  Here is a complete
description of the problems by the engineers who are building the
patches:

     Subject:  Announcing New TCP Performance Patch
        Date:  7 Jun 1996 23:36:21 GMT
        From:  [email-masked] (Cathe A. Ray)
Organization:  Sun Microsystems, Inc.
  Newsgroups:  comp.unix.solaris

Sun doesn't ordinarily announce patches when they're released. But
we've just finished a series of TCP-related fixes and improvements, and
we want to make sure that the news gets out as quickly as possible to
the many people who can benefit from our work.

This patch announcement will be of interest mostly to folks who use Sun
workstations over "slow" links, like most dial-up lines.  Please note,
though, that you might benefit from the work we'll discuss here even if
you've never used one of our workstations directly.  (Many companies
who provide Internet access use Suns as part of the communication path.
And the patches are for Suns running Solaris 2.4 and up.)

Also note: This message is coming to you directly from the engineers
who did the work. We wanted to get the information out to you right
away, but we really aren't trying to replace all the other Sun sources
of information you might have access to. Please, don't send us lots of
detailed questions--we're not volunteering to answer them (or even
respond to many of the followups here). We just really wanted to make
sure this message got out. Thanks.

Cathe A. Ray
Manager, Internet Engineering

        TCP Performance Improvements For Slow Network Links
        ===================================================

Our Sun team is responsible for basic network communications software.
We've been putting in a lot of work lately on improving the performance
of TCP over slow network links. Now we're finished; testing is
complete; and the patches (for Solaris 2.4 and later) will be available
shortly.

We undertook the work in response to feedback from customers serving
WWW users over asynchronous PPP links. Users of LANs and WANs built on
10base-T and faster media never saw the problem behavior, which
actually affected FTP and other TCP-based applications as well.

With the new patches in place, slow links will operate with roughly the
same efficiency as fast links. Without the patches, efficiency of very
slow links could, under Solaris 2.5, sink to as low as 5 per cent of the
theoretical maximum.

In the following sections we will describe in detail what was wrong and
how we fixed it. If you don't need to know all that, just check the
table below for the patch numbers. They'll be available soon from our
usual patch sources. We're confident that customers who have seen the
problem will now observe a remarkable improvement. Others will see no
change.

        SPARC:

                                                 module
             2.4          2.5        2.5.1      affected
        |-----------|-----------|-----------|-----------------|
        | 101945-xx | 103169-05 | 103582-01 | /kernel/drv/ip  |
        | 101945-xx | 103447-03 | 103630-01 | /kernel/drv/tcp |
        |-----------|-----------|-----------|-----------------|

        X86:
                                                 module
             2.4          2.5        2.5.1      affected
        |-----------|-----------|-----------|-----------------|
        | 101946-xx | 103170-05 | 103581-01 | /kernel/drv/ip  |
        | 101946-xx | 103448-03 | 103631-01 | /kernel/drv/tcp |
        |-----------|-----------|-----------|-----------------|

        PowerPC:
                         module
            2.5.1       affected
        |-----------|-----------------|
        | 103583-01 | /kernel/drv/ip  |
        | 103632-01 | /kernel/drv/tcp |
        |-----------|-----------------|

        Note: Where a revision number has been indicated it you should
ask
              for the patch of at least that revision.  In the case of
the 2.4
              patch revision number it was not available at the time of
this
              posting.  Always try to get "the latest version" of any
patch 
              you go after.

HISTORY

Strangely, the decline in throughput was the result of several
improvements we made over the years to the TCP retransmission
algorithms and parameters. Every change improved performance for
systems with fast links. The cumulative effect for slow links was just
the reverse; but almost all our systems--and our customers'--were
hooked up to fast links, and the drawbacks went largely unnoticed. That
was the state of affairs at the time 2.4 was released.

By the time 2.5 came out, async hookups to the Web had exploded. We had
implemented another relatively minor TCP bug fix. Customers with fast
links were better off. The efficiency of slow links declined. We
quickly learned we had a problem.

We tracked down the inconsistencies and rewrote the code. We've
redesigned the algorithm for good behavior across all supported
configurations. We've added slow links and a wide mix of simulated
platforms to our test beds, and tested the fixes in both high-speed and
slow-speed networks. The problem is resolved.

Excellence is a moving target.

TECHNICAL DETAILS

Here are some technical details. As you'll see, we've made it a pretty
frank discussion. (Please be aware, though, that we do not intend to
spend much time debating our decisions here.)

The throughput troubles on slow lines result from an excessive rate of
retransmissions. The rate, in turn, is caused by a mis-tuning adaptive
algorithm.

TCP packets are retransmitted if no response is received before a
timeout period has expired. Our routines implement a variant of the
familiar Karn and Jacobson adaptive algorithms, which attempts to
predict an efficient timeout value based on the time it took previous
packets to complete a roundtrip. Elapsed values are combined into a
smoothed average roundtrip time ("RTT") and variance.

The key elements in this calculation are the initial RTT value and the
subsequent RTT's factored in. The changes we have made involve both of
these key areas.

INITIAL RTT VALUES

As an unintended result of several cumulative changes, the kernel
parameter "tcp_rexmit_interval_initial" was actually not being used. In
fact, all Internet Routing Entry (IRE) RTT values were being
initialized to 512 milliseconds. TCP was using that as an initial
setting.

For connections which flow through a route with a roundtrip time less
than that (such as a LAN or WAN built on 10base-T) all was well. When
the connection closed, the actual IRE RTT value was updated and the
predictive timeout value successfully adjusted.

For connections with an RTT greater then 512 ms, however, the timeout
would necessarily trip, and retransmissions occur. If the actual time
differed sufficiently from the original estimated value, TCP was never
able to send a segment without one or more retransmissions. A realistic
RTT for the route could never be established.  This scenario is the
beginning of the explanation of what has been happening on several-hop
Internet or asynchronous PPP links.

Our solution is to initialize all IRE RTT's to zero instead of 512 ms.
Any new connection for a route will now, when lookup discloses the zero
value, get the value of the "tcp_rexmit_interval_initial" parameter
instead.  (And it's been increased to 3 seconds.)  So in most cases the
adaptive algorithm will now be able to adjust timeout values
effectively.

RTO (RETRANSMIT TIMEOUT) ALGORITHM INTERACTION

Another factor contributing to packet congestion and retransmission was
a change to the RTO algorithm, introduced in a 2.4 Kernel Patch.  The
intent was to make the behavior more "conservative"--that is, lower the
risk of poor timeout values. The effect on low-speed links was
unexpectedly contrary.

A key (and unintended) effect of the code change was that RTT data from
retransmitted packets was discarded. This behavior, together with the
poor initial RTT values described earlier, meant that the adaptive
algorithm was deprived of the information needed to adjust the RTO.

Our solution is to keep the RTO RTT update still conservative, but now
update the RTO after no more than one receive window's worth of valid
RTT's. Further, when an invalid RTT is seen--an ACK of a retransmitted
segment, for example--any valid RTT information is fed into the RTO
algorithm.

ZERO WINDOW PROBE BUG FIX

The problems described so far affect Solaris 2.4 and 2.5 equally. What
changed with 2.5?

One important fix we included in 2.5 was for the "zero window probe"
bug, a  well-publicized problem affecting just about all versions of
UNIX. As part of that rewrite, we removed a nondescript piece of logic
that implemented a simple "backoff" scheme. The excised code caused the
RTO to be lengthened by one-eighth as a result of certain failures. It
seemed not to be needed; but it had concealed the presence of the other
bugs by providing a means for the RTO to reach a successful value. When
this code was removed the other underlying problems were exposed.

IRE RTT LOGIC

This last part of the problem concerns the interaction between TCP and
the Solaris-specific Internet Routing Entries. The IRE RTT logic caches
RTT values to be re-used when a new connection is made over a familiar
link.

This is a fine approach. The implementation, however, had a flaw: the
IRE RTT was updated regardless of the RTT value supplied by TCP.

As you will have guessed by now, users of high-speed links saw no
effect.  But in highly variable RTT routes, when a connection dominated
by small segments was closed, a problem could result. An RTT too short
for large segments was used to update the IRE RTT, and a subsequent
connection dominated by large segments (like FTP) experienced an
excessive retransmission rate. It was a different path to a familiar
dilemma: too small a timeout value.

Naturally the most highly variable RTT's tend to be seen on async PPP
links, where the RTT of the route is compounded from (1) wire latency,
(2) low bandwidth, and (3) congestion/queuing delays as more than one
segment is transmited by TCP.

Our solution is to add an new ndd variable "tcp_rtt_updates". It allows
tuning or disabling of IRE RTT updates. A value of zero disables IRE
RTT updates. A value greater than zero specifies how many RTT updates
to the RTO are required--that is, how many chances the algorithm has
had to adapt the timeout--before a closing connection will be allowed
to update the RTT in the IRE.

CONCLUSION

We've fished out, fixed, and explained some subtle flaws in our
adaptive retransmission algorithm. We take the responsibility for
introducing them--and the credit, too, for practically every piece was,
by itself, a successful response to our customers' needs. Better and
exhaustive testing would have shown up the flaws earlier, privately,
harmlessly.  That's always our goal, and our customers have a right to
expect the best. Yes.

There's always tomorrow. In the meantime: we killed this one, folks.
Our sincere thanks for your attention--and your business.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Joseph F. Backo                E-Mail:     [email-masked]
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

---

From: Darrell Fuhriman <[email-masked]>
Date: Mon, 1 Jul 1996 16:48:39 -0700 (PDT)

> a Sparc 20 with Solaris 2.5...

Well, the other person didn't say what platform he was using, but I'd bet
money it's Solaris as well.

It has nothing to do with FTP.. Bascially, Solaris' TCP/IP blows in a big
way over anything less than LAN speeds.

We've tested a patch for it, but it's still not there yet, though supposedly
Sun is working on it and though they made an annoucement nearly a month ago
(attached) the patch hasn't been released.  So, in other words, you'll have
to suffer or put SunOS 4 on it (you'll be glad you did.)

Darrell Fuhriman
Teleport System Administration

From: [email-masked] (Cathe A. Ray)
Newsgroups: comp.unix.solaris
Subject: Announcing New TCP Performance Patch
Date: 7 Jun 1996 23:36:21 GMT
Organization: Sun Microsystems, Inc.
Lines: 240
Distribution: world
Message-ID: <[email-masked]>
Reply-To: [email-masked]
NNTP-Posting-Host: beeblebrox.eng.sun.com

Cathe A. Ray

	SPARC:

	                                         module
	     2.4          2.5        2.5.1      affected

	X86:
	                                         module
	     2.4          2.5        2.5.1      affected

	PowerPC:
	                 module
	    2.5.1       affected

        Note: Where a revision number has been indicated it you should ask
	      for the patch of at least that revision.  In the case of the 2.4
	      patch revision number it was not available at the time of this
	      posting.  Always try to get "the latest version" of any patch
	      you go after.

HISTORY

TECHNICAL DETAILS

INITIAL RTT VALUES

adaptive algorithm will now be able to adjust timeout values effectively.

IRE RTT LOGIC

CONCLUSION

--

Cathe A. Ray                    | Love makes the world go `round
SunSoft (415) 786-5178		| but Chocolate makes the trip worthwhile!
[email-masked]    	|

---

From: [email-masked]
Date: Mon, 08 Jul 1996 10:04:47 -0400

+- On Monday (7/1/1996 13:44) [email-masked] (Dan Schnur) Wrote-
| Odd, I get the same behavior.  I thought it was just me :)  If anyone knows
| a solution for this problem, I would be quite intrested.  I'm using 2.4 on
| a Sparc 20 with Solaris 2.5...
| 
|   -D

This is (quite possibly) a bug in the Solaris 2.5 networking code.
Put

        /usr/sbin/ndd -set /dev/tcp tcp_rexmit_interval_min 3000
        /usr/sbin/ndd -set /dev/tcp tcp_rexmit_interval_initial 3000

In one of your startus scripts and see if it helps.

Note that you can alos just run these fomr the command line.  SunOS
5.5 is notorious for crappy performance over slow (read Serial PPP)
TCP/IP links.

Stephen L. Ulmer
Senior Systems Administrator
Mercury Communications