Subject: Re: Bandwidth Throttling

From: [email-masked] (Chris Brown)
Date: Sat, 8 Mar 1997 12:26:12 -0800 (PST)

Okay...  I've prepared a test version of the throttler.  It's currently
running as my ftp server at siggy.iceonline.com, but for anonymous access,
it's packed.  Because the server is so packed, I'll attatch the diffs here
as text...  This is a patch of wu-ftpd-2.4.2-beta-12, which I've dubbed
beta-12.2 (perhaps innapropriately.)  The files listed below can be
found at:

ftp://siggy.iceonline.com/pub/other/wu-ftpd-2.4.2-beta-12.2.diff.tar.gz

If you can get on...

Chris...

Notes:
 - throttle() currently only supports 1 second grainyness.  In my opinion
(partly because I run a site with multi-meg files) this is good enough.
If it isn't, I can put in microsecond grainyness, with little difficulty.
It *will* take more cpu time though.

 - In order for anything to happen, THROTTLE must be defined in or around
the config.h file.

 - In the ftp access file, a bandwidth directive must be supplied for each
class, or that class has unlimitted bandwidth.  Also, a negative bandwidth
is treated as unlimitted.  A bandwidth of 0 will result in an actual
bandwidth of 1.

bandwidth <class> <bytes/second>

 - Actual bandwidth for a user is total_bandwidth/number_of_users_connected.

 - This differs from cistron's server in certain ways.  I wrote this with
the idea that it would be easier to incorperate if it caused as few changes
to files outside the new one (throttle.c).  This makes it *slightly* less
effecient than theirs, but not much.

 - throttle.c includes a large section of #if 0 code which I wrote and think
is more efficient than the access.c counterpart, but I left it out.

Okay, here are the files:

=========
ftpd.diff
=========
159,162d158
< #ifdef THROTTLE
< #include "throttle.h"
< #endif
< 
2368,2372d2363
< 
< #ifdef THROTTLE
<     throttle( -1 );
< #endif
< 
2384,2387d2374
< #ifdef THROTTLE
<             if( !(byte_count % BUFSIZ) )
<                 throttle( byte_count );
< #endif
2415,2417d2401
< #ifdef THROTTLE
<             throttle( byte_count );
< #endif
2422d2405
<             {
2424,2427d2406
< #ifdef THROTTLE
<             throttle( byte_count );
< #endif
<             }
2478,2482d2456
< 
< #ifdef THROTTLE
<     throttle( -1 );
< #endif
< 
2491,2493d2464
< #ifdef THROTTLE
<             throttle( byte_count );
< #endif
2518,2521d2488
< #ifdef THROTTLE
<             if( !(byte_count % BUFSIZ) )
<                 throttle( byte_count );
< #endif
2603,2605d2569
< #ifdef THROTTLE
<     int bandwidth;
< #endif
2618,2624d2581
< #ifdef THROTTLE
<         bandwidth = acl_getbandwidth();
<         if (bandwidth > 0)
<             printf("     Bandwidth set to %.2f Kb/sec\r\n", (float)bandwidth / 1000 );
<         else
<             printf("     Bandwidth unlimited.\r\n");
< #endif

=========
vers.diff
=========
1c1
< char version[] = "Version wu-2.4.2-academ[BETA-12](2) Fri Mar 7 19:03:09 PST 1997";
---
> char version[] = "Version wu-2.4.2-academ[BETA-12](1) Sun Jan 19 23:05:28 CST 1997";

===============
extensions.diff
===============
194c194
<     int limit, bandwidth;
---
>     int limit;
237,248d236
< 
< #ifdef THROTTLE
<             case 'B':
<                 bandwidth = acl_getbandwidth();
<                 sprintf(outptr, bandwidth == -1 ?
<                         "unlimited" : "%.2f", (float)(bandwidth / 1000) );
<                 break;
< 
<             case 'D':
<                 strcpy(outptr, buffer);
<                 break;
< #endif

==========
throttle.h
==========
#ifdef THROTTLE

#ifndef __WU_THROTTLE_H
#define __WU_THROTTLE_H

int throttle ( long bytes );
int acl_getbandwidth( void );

#endif

#endif

==========
throttle.c
==========
#include "config.h"

#ifdef THROTTLE

#include <stdlib.h>
#include <time.h>
#include <syslog.h>

#include "extensions.h"

#define BANDCHKDLY (5)

/* ( begin supplementary functions for access.c */

/*
 * Get the total bandwidth for this class and divide it by the
 * number of ACTIVE users plus ourself.
 * A result of -1 means unlimited.  Any other result is in bytes per second.
 */
int acl_getbandwidth( void )
{
  char class[ 1024 ];
  int busy, tot_bw = 0;
  struct aclmember *entry = NULL;

  /* First, get current number of users in this class. */
  if( !acl_getclass( class ) )
    return( -1 );
  if( (busy = acl_countusers( class )) < 0 )
    return( -1 );

  /* Now see what our raw bandwidth is. */
  while( getaclentry( "bandwidth", &entry ) )
    {
    if( !ARG0  ||  !ARG1  ||  strcmp( ARG0, class ) )
      continue;
    tot_bw = atoi( ARG1 );
    break;
    }

  /* Now calculate our actual bandwidth */
  if( tot_bw > 0 )
    return( tot_bw / busy );
  else
    return( -1 );
}

/*
  Purpose: Cause a delay in the download loop.  throttle() must be 'reset'
    before it is used or it will produce useless results.
  Returns: -1 on error, 1 on received signal, 0 in all other cases.
  Arguments: Bytes sent, if this number is negative or zero throttle is reset.
*/

int throttle ( long bytes )
{
  static time_t nextbandtime;
  static int bandwidth;
  static long prev_bytes = -1;
  time_t now = time( NULL ), delay;
  long usebytes;

  if( bytes <= 0  ||  bytes < prev_bytes  ||  prev_bytes == -1 )
    {
    nextbandtime = 0;
    prev_bytes = 0;
    return( 0 );
    }

  if( nextbandtime < now )
    {
    bandwidth = acl_getbandwidth( );
    nextbandtime = now + BANDCHKDLY;
    if( !bandwidth )
      bandwidth = 1;
    }

  if( bandwidth < 0 )
    return( 0 );

  usebytes = bytes - prev_bytes;
  delay = usebytes / bandwidth;
#if 0
syslog( LOG_DEBUG, "de %lu  by %lu-%lu=%lu  bw %ld",
  (long)delay, (long)bytes, (long)prev_bytes, (long)usebytes, (long)bandwidth );
#endif
  if( delay <= 0 )
    return( 0 );

  prev_bytes += delay * bandwidth;
  delay = sleep( delay );

  if( delay < 0 )
    return( -1 );
  if( delay > 0 );
    return( 1 );
  return( 0 );
}

/* ) end supplementary functions for access.c */
#endif  /* defined(THROTTLE) */

#if 0
/* ( begin low level replacement functions for access.c */

/*
  Purpose: A centralized way to figure out whether the pidfile is up to date
           in memory, and update it if it isn't.
  Returns: 0 on no change, 1 if updated, -1 if error.
  Arguments: Class name string, pidlist and maxpids.
*/

static int _acl_getpidfile ( const char *class, pid_t *pids, int maxpids )
{
  static struct stat prev_st;
  static int pidfd = -1;

#ifndef HAVE_FLOCK
  struct flock arg;
#endif
  struct stat st;
  int res;

  if( pidfd < 0 )
    {
    char pidfile [ MAXPATHLEN ];
    sprintf( pidfile, _PATH_PIDNAMES, class );
    pidfd = open( pidfile, O_RDWR | O_CREAT, 0644 );
    if( pidfd < 0 )
      {
      syslog( LOG_ERR, "cannot open pid file '%s':%m", pidfile );
      return( -1 );
      }
    memset( &prev_st, sizeof( struct stat ) );
    }

  fstat( pidfd, &st );
  if( st.mtime == prev_st.mtime )
    return( 0 );
  prev_st = st;

#ifdef HAVE_FLOCK
  while( flock( pidfd, LOCK_EX ) )
#else
  arg.l_type = F_WRLCK;
  arg.l_whence = arg.l_start = arg.l_len = 0;
  while( -1 == fcntl( pidfd, F_SETLK, &arg ) )
#endif
    {
#ifdef HAVE_FLOCK
    syslog( LOG_ERR, "sleeping: flock of pid file failed:%m" );
#else
    syslog( LOG_ERR, "sleeping: fcntl lock of pid file failed:%m" );
#endif
    sleep( 1 );
    }

  lseek( pidfd, 0, L_SET );
  res = read( pidfd, pids, sizeof( pid_t ) * maxpids );

#ifdef HAVE_FLOCK
  flock( pidfd, LOCK_UN );
#else
  arg.l_type = F_UNLCK;
  arg.l_whence = arg.l_start = arg.l_len = 0;
  fcntl( pidfd, F_SETLK, &arg );
#endif

  if( res != sizeof( pid_t ) * maxpids )
    return( -1 )
  else
    return( 1 );
}

/*
  Purpose: Internal mechanism for acl_countusers and acl_countbusy.  The large
    number of 'kills' going on here is probably an inefficient way of doing
    things, but it works.
  Returns: Number of active users in that class.
  Arguments: Class name string.
*/

static int acl_countusers( const char * class )
{
  static pid_t pids[ MAXUSERS ];

  int which, count = 0;

  if( !class )
    return( count );

  if( _acl_getpidfile( class, pids, MAXUSERS ) < 0 )
    return( 0 );

  for( which = 0; which < MAXUSERS; which++ )
    if( pids[ which ]  &&  !kill( buf[ which ], 0 ) )
      count++;
  return( count );
}

/* ) end low level replacement functions for access.c */
#endif

===========
END OF FILE
===========

---

From: "Karl O. Pinc" <[email-masked]>
Date: Mon, 10 Mar 1997 10:21:18 -0600

>Okay...  I've prepared a test version of the throttler.

>bandwidth <class> <bytes/second>

The configuration directive I'd like to see is bandwidth limits on
files/directory trees.  Don't know how feasible this is, but I thought I'd
ask for it.

bandwidth path <path> <bytes> <seconds>

<path> relative to ftp root
<bytes> bytes can be transferred every <seconds> seconds.

So,

bandwidth path /pub/sleeze 5000000 3600

would allow (anonymous ftp -- as the path supplied is relative to anonymous
ftp root) to transfer ~5MB from the sleeze directory tree per hour.

Murky implimentation suggestion:

The implementation could keep track of the number of bytes transferred
during the 16 preceeding <seconds>/16 time intervals, and use the sum of
these/<seconds> to compute actual bandwidth.   (Implimentation could
actually keep a running sum in each "timebucket".)  Every <seconds>/16
interval, it would clear out the oldest "timebucket" and reuse it for the
newest.  Clear like mud?  The idea is to have a bandwidth resolution of
<seconds>/16.  This would keep you from using up all the bandwidth for the
hour in the first minute of the hour.  The bandwidth used could then set an
actual bytes/second limit to try to even out the usage during the
"timebucket" -- keeping you from using up all the bandwidth for the
"timebucket" at the beginning of the timebucket.

Karl

May the Legos (TM) always be swept from your path in the night.

---

From: [email-masked] (Chris Brown)
Date: Mon, 10 Mar 1997 11:38:08 -0800 (PST)

> ftp root) to transfer ~5MB from the sleeze directory tree per hour.

This is an intersting suggestion.  It would not currently be possible, but
when I originally wrote the throttle (hack job that it is) I had another
idea, a bandwidth resource server in mind.  With that I'm pretty sure that
I could restrict in the way you suggest.  Only problem is that that would
require quite a bit of new code that affects wu-ftpd elsewhere, and I'm not
on the developement 'team'...

Any comments on this, anyone?

Chris...

---

From: "Karl O. Pinc" <[email-masked]>
Date: Mon, 10 Mar 1997 14:34:38 -0600

>> files/directory trees.  Don't know how feasible this is, but I thought I'd

<snip>

>I had another
>idea, a bandwidth resource server in mind.

<snip>

So,

restricts members of <class> to those periods when the servers bandwidth is
below <bytes/second>, right?  When the overall server bandwidth goes above
<bytes/second>, the bandwidth of members of <class> goes to zero?

Karl

---

From: Mark Whetzel <[email-masked]>
Date: Mon, 10 Mar 1997 15:34:34 -0600 (CST)

Karl O. Pinc shaped the eletrons to say:
 
[ Lots of cuts ]

> <bytes/second>, the bandwidth of members of <class> goes to zero?

How is enforcement to be done?   Deny the transfer?

What about existing tranfers for that class?  Will it throttle to 
zero and then users connections timeout?  Should a minimum and maximum
apply in for existing transfers to allow them to continue.

-- 
Mark Whetzel     My comments are my own, not my company's.
Western Geophysical - A division of Western Atlas International Inc.,
A Subsidary of Western Atlas Inc.  addr: [email-masked]
VOICE: (713) 963-2544              FAX: (713) 963-2758

---

From: Iqbal Gandham <[email-masked]>
Date: Thu, 13 Mar 1997 16:30:51 +0000

Hi Chris

Do you know where I can get a already built copy of wu-ftpd2.4....
for IRIX6.2.

The problem I am having is that I do not have a C compiler, I could use
gnu, but I have been told that gnu will not work without some libraries
or something (I am not good in C).

Any Ideas..

Thanks Iqbal

---

From: Karl Hubner <[email-masked]>
Date: Thu, 13 Mar 1997 15:18:00 -0500

I saw your posting to Chris and I know what it's like trying to get
executables for IRIX.
Here are the one's I've been using for a couple of months. They work
except for the time (it shows GMT). I've been too busy to recompile.
When I do I'll drop you a line (if you like).

ftp://ftp.tmsdesign.com/private/irixwu.tar.gz

Good Luck,
karl