Subject: Re: How to get NOOP out of pine?

From: Mark Crispin <[email-masked]>
Date: Wed, 1 Feb 1995 21:19:30 -0800

On 31 Jan 1995, Jack Bryans wrote:
> Is there any way to build pine so that it doesn't send "NOOP" out the nntp
> port?

No, there isn't.

The more interesting question is: why do you want to do this?

Do you or your users enjoy the prospect of having connections dropped due 
to a timeout?  Remember, Pine caches much more thoroughly than most NNTP 
clients, so it is perfectly possible for a very active Pine user to have 
a seemingly "idle" NNTP session.

---

From: Mike Brudenell <[email-masked]>
Date: Thu, 2 Feb 1995 16:09:57 +0000 (GMT)

"Why do I want to do this?"

Our INN 1.4 news server software doesn't *have* a NOOP command!

This means that they get logged and reported the following day by e-mail 
as "host xxx used invalid command yyy".

Yes, I know the solution is to "fix INN to have a NOOP command"!  :-)

Mike Brudenell
------------------------------------------------------------------------------
The Computing Service, University of York, Heslington, York, YO1 5DD, UK
Phone: +44-(0)1904-433811  FAX: +44-(0)1904-432767

---

From: Barry Cornelius <[email-masked]>
Date: Thu, 2 Feb 1995 16:44:04 +0000 (GMT)

> Yes, I know the solution is to "fix INN to have a NOOP command"!  :-)

According to the text of the file, I have done this: I don't remeber doing 
it, but still ... .  Anyway the file .../nnrpd/nnrpd.c now has:

    ...
    {   "next",         CMDnextlast,    FALSE,  1,      1,
        NULL },
    /* BJC 941109A: added the following line */
    {   "noop",         CMD_unimp,      FALSE,  1,      CMDany,
        NULL },
    {   "post",         CMDpost,        TRUE,   1,      1,
        NULL },
    ...

It's at about line 100 of the file.  It would appear from the lack of 
other 941109A comments that that's the only change, but I'm not that certain.

--
Barry Cornelius                      Telephone: (0191 or +44 191) 374 4717
Academic Support Group, IT Service,            IT Service Office: 374 2892   
Science Site, University of Durham,                          Fax: 374 3741
Durham, DH1 3LE, UK                   E-mail: [email-masked]

---

From: [email-masked] (Jack Bryans)
Date: 2 Feb 1995 22:46:49 GMT

> The more interesting question is: why do you want to do this?

Well, getting along w/the world as we find it includes placating bnews,
cnews, & INN, at the least.  INN has never recognized NOOP & logs every one
of them.  While we're ok for now for spool space, we're hurting for / space,
which, for us, includes /usr/local/news & all its logs.  A days supply of
NOOPs takes over 10K of log space.

Sure, it can be argued that INN should be changed to accommodate every new
newsreading client that comes along.  I'd rather have all s'ware that speaks
or adds nntp get along w/INN as it is.

INN's timeouts are generous.  W/all the PC & mac rewsreaders on our net, we
see a few timeouts logged.  But they are trivially minor compared to the
NOOP logging.

Jack

---

From: Mark Crispin <[email-masked]>
Date: Thu, 2 Feb 1995 18:54:47 -0800

>     {   "post",         CMDpost,        TRUE,   1,      1,

What a great idea!  Every Internet protocol should have a NOOP command.
Seriously, this is a bug in the NNTP protocol.

We *did* have timeout problems before this NOOP pinging was put in.  I 
think that I used to have a HELP command, and I got complaints that the 
entire help transcript was being logged (so the NOOP was an improvement).

As far as servers which obsessively log commands sent to them, and the
consequent loss of disk space -- are you aware that you have given every
cracker on the Internet instructions of how to hit you with a denial of
service attack? 

It's important that any logging which you have installed is careful not to
allow randoms to fill up your disk.  In particular, for non-authenticated
connections, you should be *very* restricted in what events you log.  If
you log authentication failures, you need to have a throttle in order to
limit the number of such events that can happen in a particular interval.

---

From: Dan Schlitt <[email-masked]>
Date: Fri, 3 Feb 1995 09:30:29 -0500 (EST)

Mark,

I usually find your response to be reasonable and well thought out.  In 
this case I am in doubt.  It seems to me that there are two issues and I 
am not clear about what you are saying about them.

The first is a protocol issue.  Nntp has a well defined protocol 
specified in an RFC.  To complicate this, there are proposed extensions 
and refinements which are not yet in the form of an RFC.  But Rich Salz 
is one of the folks deeply involved in this revision.  So the question is 
whether the NOOP is a part of the protocol (or is likely to be in the 
extensions).  If the answer is no then Pine should not generate the NOOPs.

This is as broader issue than just nntp.  In other areas the pine crew 
make quite a point of following the standards and reject lots of 
suggestions on that basis. In this I believe they are correct.

The second issue is whether client designers know in the abstract what 
timeout behaviour should be better than the folks who are running the 
servers.  I tend to think that the server operators know better in the 
specific cases and should adjust the timeouts if there are serious problems.

/dan

-- 

Dan Schlitt                           School of Engineering Computer Systems
[email-masked]        City College of New York
(212)650-6760                         New York, NY 10031

---

From: Mark Crispin <[email-masked]>
Date: Sun, 5 Feb 1995 18:29:23 -0800

On 3 Feb 1995, Dan Schlitt wrote:
> Nntp has a well defined protocol 
> and refinements which are not yet in the form of an RFC.

Unfortunately, these two statements conflict with each other.  You can not
build a proper functional NNTP client or server using the specification in
RFC-977.  Consider the following differences from RFC-977:

1) You are supposed to use LIST ACTIVE to get the list of active 
   newsgroups.  RFC-977 only defines LIST with no argument; there are
   reports of NNTP servers which reject this.

2) The only way to find out which message numbers are valid in a sequence
   is by the LISTGROUP command (absent in RFC-977) or, failing that, by
   the "XHDR Date" command (also absent in RFC-977).  RFC-977 requires
   you to have holes in your browser -- something that *infuriated* users
   of earlier versions of Pine who tried to use the news support.

3) Many NNTP servers do not permit posting without your having first given
   the AUTHINFO USER and AUTHINFO PASS commands (absent in RFC-977).  [I
   wonder if the paranoid site administrators who worry about people 
   posting forged news messages realize that this "fix" broadcasts 
   passwords in the clear over the net!].

   Pine does not implement this yet, and we're getting beaten up about 
   it. 

4) Many NNTP clients have been changed to take advantage of the XOVER
   command (absent in RFC-977), which permits fetching of sufficient
   information of a message to build a browser line; and what's more,
   permits it over a range of messages.

   Pine does not implement this yes, and we're getting beaten up about
   it.  Right now, Pine laboriously fetches the headers for each message,
   one at a time, and does an RFC-822 parse to build the browser line.
   This is slow.

5) NNTP is conspicuously the *only* protocol that lacks a no-op command.
   Fortunately, unlike POP2, unknown command errors do not cause the
   connection to die, so a client can issue a NOOP command and ignore
   any error in the server's response.

The only rational conclusion that one must make is that there is no such
thing as a standard NNTP protocol.  Rather, each client and server
implementor is on his own to build something that more or less
interoperates with the maximal number of other implementation. 

Yes, this is a lousy circumstance.  But this is the condition that NNTP 
is in.  It is not possible to develop a viable NNTP implementation using 
the published specification.

Given that other NNTP server and client implementors have had no problem
with the concept of making undocumented commands mandatory, I see no
inconsistancy in doing the same thing in Pine.  More importantly, however,
is the fact that there was no choice.  It wasn't just that people were
beating up on us about the lack of NNTP server pinging -- we were being
screwed by it ourselves! 

> But Rich Salz 
> is one of the folks deeply involved in this revision.

Unfortunately, nothing has come out of Salz's efforts.  Even worse, it
seems that the NNTP community seems to have fractured, and there is no
guarantee that anything Salz eventually produces will be accepted.  Which
is a shame, since in general Salz seems to have the right ideas. 

Consider the recent debate about the function of the Newsgroups: header
line, and the emergence of cretins who insist that headers must be
interpreted in a context-sensitive fashion between news/mail.  I don't 
think there is any hope of seeing a viable NNTP standard specification 
until this matter is resolved (and there is little hope of the result 
being accepted if the cretins win).

> So the question is 

As I pointed out earlier, the NOOPs are harmless.  Or rather, they should
be.  NOOP is an unknown command in a strict interpretation of RFC-977. 
The specification is quite clear about the treatment of unknown commands.
So, in that regard, NOOP -- or rather, the correct server behavior to a 
NOOP command -- is part of the protocol.

I am sorry about the log file garbage that some server maintainers have as
a result.  However, part of the process of gathering logging information
is identifying what information is significant, and filtering out the
unimportant information.  Logs with excessive noise are just as bad as 
inadequate logs.

More importantly, stating that "NOOP log events are filling up my disk" 
is an open broadcast to all crackers about how to commence a denial of
service attack on your system.  The security implications are worse than
just denial of service to users.  A cracker could cause denial of service
to security logging mechanisms.  Just send a few hundred thousand bogus
commands at the NNTP server until the disk fills, then crack away without
fear of having evidence against you being logged. 

Furthermore, it is fairly common in older Internet protocols for clients
to probe for server capabilities by attempting commands which the server
may or may not know.  Newer Internet protocols provide a capability
listing mechanism which allegedly makes this unnecessary (and, as I
expected, we're already seeing skews between advertised capabilities and
implemented ones). 

The bottom line is that client behavior (especially non-authenticated
clients such as NNTP clients) that causes excessive resource consumption
on a server identifies a server problem, not a client problem.

I'm confident that if I changed the NOOP to a HELP (which is in RFC-977), 
I'll get complaints from server maintainers of their logs being filled up 
with help texts.  [I know, because that's what I used to do.]

> suggestions on that basis. In this I believe they are correct.

In the case of NNTP, there simply is no standard that one can use to 
develop a viable implementation.  There are *no* NNTP implementations 
which conform strictly to RFC-977.

NNTP sticks out like a sore thumb in this regard.

> specific cases and should adjust the timeouts if there are serious problems.

The Pine team has historically come rather firmly on the side of the 
client controlling all aspects of the user interface.  How long you are 
permitted to be idle is an aspect of the user interface.

Judging from the number of vehement complaints in this newsgroup from
people who have had IMAP connections dropped due to a failure in Pine's
timeout suppression mechanism (and yes, we *are* working to nail these!),
it seems that the user community is on Pine's side on this issue.

---

From: Dan Schlitt <[email-masked]>
Date: Mon, 6 Feb 1995 09:37:00 -0500 (EST)

Mark,

You have made things a bit clearer.  Please note that I was not among 
those that complaines about what ends up in log files.  I know how to 
manage my logfiles so that isn't a problem.  I do like to understand 
things that show up there.

I knew that the revisions of the nntp protocol document was long delayed. 
I did not realize that things had deteriorated to the state that you 
describe.

The reason that I mentioned Salz at all was that the complaint seemed to 
be one about the interoperation of Pine and INN.  I expect INN to reflect 
his thinking and that of the others who I knew were working on the 
revision of the protocol.

I don't have a lot of sympathy with those folks who go out to write the 
world's latest and greatest news reader and do it without regard to 
interoperability and protocol considerations.  (And I don't include the 
Pine team in that group, so don't take this as a criticism of your work.)

I guess that I would come down on the side of using HELP instead of NOOP 
provided using it would not generate unexpected output for the client to 
deal with.  But it is a matter of taste.

It may be that a large part of the problem is that the transport and 
reading are contained in the same protocol.  Separating the two might get 
us standards for both sooner.

On the matter of the timeout, I just disagree about who is best to judge.
The server is in the best position to manage the use of its resources. 
one should not undermine that management by non-standard behaviour of the 
client.  Perhaps the protocol should allow the negotiation of the timeout 
between client and server (back to the subject of point one).  In the 
case where the client and server are under the same administrative 
control you should be able to deal with this problem by modifying the 
server.  That ought to be the case at U. of Wn. where your most important 
user base is.

/dan

--

---

From: Mark Crispin <[email-masked]>
Date: Tue, 7 Feb 1995 16:02:52 -0800 (PST)

> revision of the protocol.

As I said, I think that Salz has the right ideas.  The problem is that I've
seen certain other individuals in the NNTP community challenge him on other
matters (specifically, the role of message headers in news), and what's worse,
by his own admission the NNTP document update effort has stalled.

It's a shame, but given what I've seen, I can understand the burnout.

> deal with.  But it is a matter of taste.

The other idea is to use DATE -- a nice sort-of no-op in INN.  Unfortunately,
DATE is also absent from RFC-977.  I'd rather explain to people what the
unrecognized NOOPs do than what the unrecognized DATEs do!  ;-)

> client.

It depends upon what you consider the role of timeouts to be.  If you see them
as a safety belt to allow the garbage collection of abandoned collections,
then it is not only reasonable, it is *mandatory*, that the client reassure
the server that it is still there at periodic intervals.

> Perhaps the protocol should allow the negotiation of the timeout
> between client and server (back to the subject of point one).

Perhaps, but technically, timeouts are a violation of the TCP spec!  TCP is
rather emphatic about this.  During a nuclear war, you do not want Clinton's
telnet session to nuker.af.mil to be dropped just because a section of network
in Ohio evaporated and it took a few minutes before another 747 with a cisco
and microwave transceiver could be sent up to replace the lost line.

> In the

As I said, timeouts have a useful purpose in an LAN configuration with
machines such as PCs and Macs which are not completely careful about notifying
the other end (via TCP reset) when their applications go away (worse, they
tend to use the same port numbers, and get screwed if the connection is still
there at the server).  We're not really worried about keeping our NNTP service
available during nuclear war.  ;-)  But, we want to have abandoned servers
from PC/Mac clients go away fairly aggressively (for reasons outlined above)
without impacting live clients.  This leads us naturally to short timeouts and
keepalive pings.