Subject: Log Interpretation

From: "Tom Rawson" <[email-masked]>
Date: Mon, 25 May 1998 07:27:54 -5

Folks ...

I am trying to interepret some data extracted from WUFTP log files and having a hard 
time.

Note that I am not running the software myself nor am I familiar with it in any detail.  
It's used on a public ftp site which supports downloads of our software, and we're 
trying to figure out the traffic level.  The ISP has sent us log file records that look like 
this:

  Sun May 3 01:10:46 1998 569     <filename>
  Sun May 3 01:11:27 1998 115     <filename>
  Sun May 3 01:11:33 1998 1094    <filename>
  etc.

They extracted this from the log file, leaving off any identity information to retain the 
anonymity.

The questions we are trying to sort out are (a) what is the number after the date and 
time, and (b) how does WUFTP log incomplete transfers?  Most (80%+) of the 
numeric values are far below the actual file sizes, and some are FAR above. The ISP 
(which generally ignores this log and therefore has no experience in interpreting it) 
says the number is a 1K block count for that download, and includes blocks sent for 
resends.  We don't believe that because it would require that 80%+ of the transfers 
were incomplete, and that a few are via connections so bad that resends would boost 
the transfer count to 8 or 10 times the file size.  It just doesn't make sense.

To try to sort it out I downloaded a copy of wuftp, decompressed the files, and looked 
at the xferlog man page.  It seems to say that the number recorded in the log file is a 
number of seconds for the transfer, not a block count.  While I still find it hard to 
believe that there would be, for example, an occasional download taking an hour or 
more for an 800K file, the numbers make a lot more sense as a nuber of seconds.

In order to sort this out, can somebody tell me (in addition to any thoughts on the 
above):

	- Is the "file size" value normally in the log a transfer count, or the total file
	  size (i.e. would it be different for incomplete transfers)?

	- Are incomplete transfers logged, or only complete ones?

Part of the reason for the second question is to interpret the difference between the 
number of log entries per day for each file vs. the rate at which people link to that file 
off our web site.  Statistically the log entries run at about 40% of the number of 
transfers we'd expect from the link counters we set up on the web site (and then 
there are some folks who go directly via an FTP client, so some of that 40% didn't 
even come off the web site).  However we have no way of knowing how many people 
never start the transfer, or abort it in the middle, and it would help to know what 
WUFTP is doing when the transfer is not completed.

Thanks for any info.

-----
Tom Rawson           [email-masked]
JP Software          http://www.jpsoft.com/

---

From: Gregory A Lundberg <[email-masked]>
Date: Mon, 25 May 1998 11:51:58 -0400 (EDT)

On Mon, 25 May 1998, Tom Rawson wrote:

> I am trying to interepret some data extracted from WUFTP log files and
> having a hard time. 
> 
> Note that I am not running the software myself nor am I familiar with it
> in any detail.  It's used on a public ftp site which supports downloads
> of our software, and we're trying to figure out the traffic level.  The
> ISP has sent us log file records that look like this: 
> 
> They extracted this from the log file, leaving off any identity
> information to retain the anonymity. 

How draconian.  Sorry, you'll need to ask the ISP since they've hacked the
log file.  If they can give you the real log file people can help, but
with what they're giving you, only they can tell you what it means.  IMHO
if they don't know how their own code works and what its output means then
what you need is a better ISP. 

> The questions we are trying to sort out are (a) what is the number after
> the date and time, and (b) how does WUFTP log incomplete transfers? 
> Most (80%+) of the numeric values are far below the actual file sizes,
> and some are FAR above. The ISP (which generally ignores this log and
> therefore has no experience in interpreting it) says the number is a 1K
> block count for that download, and includes blocks sent for resends.  We
> don't believe that because it would require that 80%+ of the transfers
> were incomplete, and that a few are via connections so bad that resends
> would boost the transfer count to 8 or 10 times the file size.  It just
> doesn't make sense. 
> 
> To try to sort it out I downloaded a copy of wuftp, decompressed the
> files, and looked at the xferlog man page.  It seems to say that the
> number recorded in the log file is a number of seconds for the transfer,
> not a block count.  While I still find it hard to believe that there
> would be, for example, an occasional download taking an hour or more for
> an 800K file, the numbers make a lot more sense as a nuber of seconds. 

Actually, that wouldn't surprise me given the way the Internet works and
the fact that you have no idea where the connection was from.  Again, only
the ISP can tell you that.

> In order to sort this out, can somebody tell me (in addition to any
> thoughts on the above): 
> 
> 	- Is the "file size" value normally in the log a transfer count,
> 	  or the total file size (i.e. would it be different for
> 	  incomplete transfers)? 

Bytes actually transferred, but at the TCP level so the ISPs statement
that it includes retransmissions is a load of b******t (or they're not
running wu-ftpd.

> 	- Are incomplete transfers logged, or only complete ones?

All, even incomplete.

> Part of the reason for the second question is to interpret the
> difference between the number of log entries per day for each file vs.
> the rate at which people link to that file off our web site. 
> Statistically the log entries run at about 40% of the number of
> transfers we'd expect from the link counters we set up on the web site
> (and then there are some folks who go directly via an FTP client, so
> some of that 40% didn't even come off the web site).  However we have no
> way of knowing how many people never start the transfer, or abort it in
> the middle, and it would help to know what WUFTP is doing when the
> transfer is not completed. 

Silly to think web accesses have anything to do with downloads.  First
off, a lot of people take a quick look to see if it's the download or more
information and abort the FTP session when it's not another page.  Second,
you're forgetting that there's a lot of web crawlers out there.

----

Gregory A Lundberg		Senior Partner, VRnet Company
1441 Elmdale Drive              [email-masked]
Kettering, OH 45409-1615 USA    1-800-809-2195

---

From: "Tom Rawson" <[email-masked]>
Date: Mon, 25 May 1998 15:40:05 -5

> what you need is a better ISP. 

Sorry, I didn't communicate it clearly.  They don't pay any attention to this log, I think 
the person who installed it is gone, the box is not their primary ftp server, and they 
don't worry about the traffic load on it.  They were quite clear about what info they 
had for us and why they didn't keep it more accurately, and that didn't bother me at 
all -- I might have done the same thing in their shoes.

As for the basic question -- it looks to me like wu-ftpd (which they are indeed 
running, seems like version wu-2.4(24)) logs two numeric values in the file, the 
number of seconds required for the transfer and the size of the file in bytes.  If I have 
a set of numbers, one per transfer, it has to be one of those (unless they massaged 
them, which I doubt -- but I can find out).  None of the numbers I've seen are big 
enough to be the byte count, so they pretty much have to be the number of seconds, 
unless wu-ftpd has changed what is logged and wu-2.4(24) logged something 
different.

> you're forgetting that there's a lot of web crawlers out there.

In this case it seems fair to me to look at the relationship between the two and 
question it if it is dramatically different.  The link on the web page exists only for 
downloads, and says so clearly.  Virtually no one is going to look there for another 
page.  People certainly might look and decide not to download, or abort the 
download, and that would make the ftp count lower than the web link count (thought I 
know of at least one browser -- Opera -- that starts the download in the background 
while it is waiting for you to decide what to do with the file).  Others might come to 
the ftp site directly which would increase the ftp count without increasing the web 
count, but that's not a big group.  But even with all that it's possible to do a reality 
check -- we are confident in the web site counters (our own) and if the ftp counts 
don't make some vague sense relative to the web counts, we need to question that 
before using them in decision-making about what our traffic load is.  If I saw ftp 
counts at 70% of the web counts, or 110%, I'd believe them, but 40% seems awfully 
low -- would 60+% of the users who select that link, which is clearly to download a 
file, and comes after navigating one or two levels down on the site, then abort the 
download?

As for the crawlers, will they generate a hit on a link to a cgi script?  I know what 
they do but little about how they do it.

The reason all this matters is that we are moving things around and the site to which 
we are moving them charges for traffic, as do many.  So we need to know the traffic 
load.  A factor of two is a factor of two different in our traffic cost and might be the 
difference between a dedicated server and a virtual host solution being the most cost-
effective.

Thanks for the help!

-----
Tom Rawson ([email-masked])