Subject: Bug or Feature: "Unverified Newsgroup Header" message is gone

From: Eduardo Chappa <[email-masked]>
Date: Fri, 1 Aug 2003 18:01:44 -0700 (PDT)

Hello,

  Today I received a personal message with a "Newsgroup:" header. If I had
seen the same message in Pine4.53 I would have seen a warning

"Unverified Newsgroup header -- Message MAY or MAY NOT have been posted"

at the bottom of the screen. Pine4.56, however, did not give this warning.
This contributed significantly to make my reply go to the newsgroup.

In this case, no harm was done. However, I believe this is not a good
thing. this change was not documented in the Release Notes of Pine.

However, observe that the file "imap/docs/RELNOTES" says:

<Quote>
Updated: 28 May 2003

imap-2002d is a minor release, released concurrently with Pine 4.56, and
contains primarily bugfixes.  Programs written for imap-2002 should build
with this version without modification, with one exception.  That
exception is the ngbogus envelope flag, which stopped being used in
imap-2002c and is now gone for good.
</Quote>

The change described above caused the problem.  Could the old behavior be
brought back please? Thanks.

Eduardo
http://www.math.washington.edu/~chappa/pine/

---

From: Nicolas Christin <[email-masked]>
Date: Sat, 2 Aug 2003 10:25:25 -0400

Eduardo,

> "Unverified Newsgroup header -- Message MAY or MAY NOT have been posted"

I looked at the code that generates this message when I wrote my
User-Agent patch for 4.56. The test you're refering to was called as
part of a relatively dirty piece of code (labeled as a kludge in the
comments) in 4.55 and below. They removed the kludge, but got rid of
that test as well, it seems.

-- 
Nicolas

---

From: Eduardo Chappa <[email-masked]>
Date: Sat, 2 Aug 2003 10:57:38 -0700 (PDT)

*** Nicolas Christin ([email-masked]) wrote in the pine-info list...:

:) >   Today I received a personal message with a "Newsgroup:" header. If
:) > I had seen the same message in Pine4.53 I would have seen a warning
:) >
:) > "Unverified Newsgroup header -- Message MAY or MAY NOT have been
:) > posted"
:)
:) I looked at the code that generates this message when I wrote my
:) User-Agent patch for 4.56. The test you're refering to was called as
:) part of a relatively dirty piece of code (labeled as a kludge in the
:) comments) in 4.55 and below. They removed the kludge, but got rid of
:) that test as well, it seems.

Right, take a look at Nancy's web page and look at the link to the
"comp.mail.pine fiasco". Maybe it will change your mind, or maybe not. I
believe that this should not have changed, and since it changed it should
have been added to the Release Notes. It was done in secrecy. Bad idea in
my opinion. I believe the old behavior should be back.

Eduardo

---

From: Nicolas Christin <[email-masked]>
Date: Sat, 2 Aug 2003 14:44:02 -0400

On Sat, 2 Aug 2003, Eduardo Chappa wrote:

> "comp.mail.pine fiasco". Maybe it will change your mind, or maybe not.

I think I haven't been very clear in my original post. I totally agree
with you. I didn't advocate removing this piece of code, I was just
saying that, judging from what I have seen while working on the
User-Agent patch, the developers had gotten rid of that entire segment
of code in 4.56, including the Newsgroup header test.

> I believe that this should not have changed, and since it changed it
> should have been added to the Release Notes. It was done in secrecy.
> Bad idea in my opinion. I believe the old behavior should be back.

I don't buy the conspiracy theory - presumably it was just a mistake.
They had the good idea to clean up some of the code, but unfortunately
ended up discarding something useful. That happens. That being said,
yes, I think it should be back too.

-- 
Nicolas

---

From: Nancy McGough <[email-masked]>
Date: Mon, 4 Aug 2003 07:26:30 +0100 (BST)

> yes, I think it should be back too.

And added to the Pine test suite so it doesn't happen again!

-- 
Nancy McGough            <http://www.ii.com>            Infinite Ink
Writing about procmail, imap, pine, spam-deflexion strategies & more

---

From: Jeff Franklin <[email-masked]>
Date: Mon, 4 Aug 2003 16:26:30 -0700 (PDT)

> This contributed significantly to make my reply go to the newsgroup.

You are correct that we did take out this behavior, and it is
unfortunately something that got left off of the Release Notes.  As
Nicolas Christin mentioned in his response, that piece of code was based
on a kludge, and we decided that that code made some rather arbitrary
assumptions that don't make sense anymore.

We took out the message entirely, as it seemed to us a message with very
little information, and one that probably confused more people than it
helped.  There are still safeguards to prevent messages from accidentally
getting sent out, such as the prompt to Post or Followup, and especially
the "Posted message may go to thousands of readers. Really post?".

Jeff

---

From: Eduardo Chappa <[email-masked]>
Date: Mon, 4 Aug 2003 17:08:37 -0700 (PDT)

*** Jeff Franklin ([email-masked]) wrote today:

:) On Fri, 1 Aug 2003, Eduardo Chappa wrote:
:)
:) >   Today I received a personal message with a "Newsgroup:" header. If
:) > I had seen the same message in Pine4.53 I would have seen a warning
:) >
:) > "Unverified Newsgroup header -- Message MAY or MAY NOT have been
:) > posted"
:) >
:) > at the bottom of the screen. Pine4.56, however, did not give this
:) > warning. This contributed significantly to make my reply go to the
:) > newsgroup.
:)
:) You are correct that we did take out this behavior, and it is
:) unfortunately something that got left off of the Release Notes.  As
:) Nicolas Christin mentioned in his response, that piece of code was
:) based on a kludge, and we decided that that code made some rather
:) arbitrary assumptions that don't make sense anymore.
:)
:) We took out the message entirely, as it seemed to us a message with
:) very little information, and one that probably confused more people
:) than it helped.  There are still safeguards to prevent messages from
:) accidentally getting sent out, such as the prompt to Post or Followup,
:) and especially the "Posted message may go to thousands of readers.
:) Really post?".

Hi Jeff,

  There's one thing that is good about the kludge. It would have made me
check if the message actually went to the newsgroup before I replied. I
would not have replied to it and posted it in the newsgroup, had I known
any better.

  I suppose I can be considered a power user of Pine and none of the work
arounds that you suggest really work for me. I would not mind if I had to
enable this warning, but please do not just eliminate it.

  It's really useful, I mean it.

  Thank you for your consideration.

Eduardo

---

From: Nancy McGough <[email-masked]>
Date: Tue, 5 Aug 2003 07:40:43 +0100 (BST)

> the "Posted message may go to thousands of readers. Really post?".

But what if the message you are replying to was a *personal*
email message that happens to have a Newsgroups header for
informational purposes only (i.e., it was not actually posted to
a Newsgroup)? It seems like we are back to the way things were
during "The 1995 Newsgroups Header Fiasco," which I talk about
here:

 <http://www.ii.com/internet/messaging/pine/#fiasco>

Is the kludge that you took out the one that checked for a Path
header? This (I hope) is not going to be as disasterous as it was
in 1995 because 1) there are not as many people using
newsreaders, such as nn, that include a Newsgroups header in
email-only message and 2) maybe less people are *both* posting
and emailing a reply because the world of email has turned into a
cesspool of spam and fake From addresses.

---

From: Nancy McGough <[email-masked]>
Date: Tue, 5 Aug 2003 07:50:37 +0100 (BST)

> cesspool of spam and fake From addresses.

Oops, what I meant to say for #2 is:

 maybe there are less people now who send a reply to a newsgroup
 message with *only* an email message (as opposed to with only a
 news posting or with both a news posting and a cc'd email
 message) because the world of email has turned into a cesspool of
 spam and fake From addresses.

---

From: Jeff Franklin <[email-masked]>
Date: Tue, 5 Aug 2003 16:15:01 -0700 (PDT)

> would not have replied to it and posted it in the newsgroup, had I known

Hi Eduardo,

I'm still trying to figure out why the "Followup, Reply, or Both" prompt
and the "Posted message may go to thousands of readers. Really post?"
prompt are not sufficient enough to prevent most accidents from happening.
Also, why is the old message more useful at view time?  That the message
shows at view time seems unnecessary and distracting, wouldn't such a
message be more useful at compose time?

"Unverified Newsgroup Header" doesn't tell me much.  Any message not in a
newsgroup with a Newsgroup header is unverified, and it seems to me to be
overly paranoid to warn whenever a user views a such a message.

It's easy enough to add a warning back into the code.  I am soliciting
from you your suggestions as to when such a message should appear (and
under what conditions), and why you don't think the two prompts are
enough.  We'll probably just do whatever you suggest.

Thanks,
Jeff

---

From: Eduardo Chappa <[email-masked]>
Date: Tue, 5 Aug 2003 19:37:01 -0700 (PDT)

*** Jeff Franklin ([email-masked]) wrote in the pine-info list today:

:) On Mon, 4 Aug 2003, Eduardo Chappa wrote:
:)
:) > Hi Jeff,
:) >
:) >   There's one thing that is good about the kludge. It would have made
:) > me check if the message actually went to the newsgroup before I
:) > replied. I would not have replied to it and posted it in the
:) > newsgroup, had I known any better.
:)
:) Hi Eduardo,
:)
:) I'm still trying to figure out why the "Followup, Reply, or Both"
:) prompt and the "Posted message may go to thousands of readers. Really
:) post?" prompt are not sufficient enough to prevent most accidents from
:) happening.

Hi Jeff,

  I think we agree on this point. I wish I could find a way to replace
"most" by "all" in your above quoted sentence.

:) Also, why is the old message more useful at view time?  That the
:) message shows at view time seems unnecessary and distracting, wouldn't
:) such a message be more useful at compose time?

  I see your point, but I am not sure when it would be convenient to see
this message. I find it more of a problem when you actually compose the
reply of the message, because verifying that the message was posted (or
not), ultimately decides where are you going to send your reply. In my
case, this would mean that I would receive the warning twice, for as long
as that message lasted in the screen. It would become annoying very
rapidly.

:) "Unverified Newsgroup Header" doesn't tell me much.  Any message not in
:) a newsgroup with a Newsgroup header is unverified, and it seems to me
:) to be overly paranoid to warn whenever a user views a such a message.

Granted, but the kludge tried to identify clients that used the newsgroup
header in the way that you intended it to be used (as a signal that the
message had been sent to a newsgroup), so for example, messages posted and
mailed by Pine did not get that warning. That's a good compromise I think.

Of course, the ideal would be not to have that kludge, but I can't see a
way to eliminate it either. I got an e-mail which proves to me that that
kludge is still useful.

:) It's easy enough to add a warning back into the code.  I am soliciting
:) from you your suggestions as to when such a message should appear (and
:) under what conditions), and why you don't think the two prompts are
:) enough.  We'll probably just do whatever you suggest.

I've been thinking about this, and I am quite convinced that I want this
warning before I press the 'R' key to reply.

Here there are two ideas:

1) To keep the old behavior exactly as it was (that means to return the
   ngbogus field to the c-client library, and all the old code taken from
   Pine), but I would change the text of the warning. Here are some ideas:

  "Newsgroup header indicates that this message MAY have been posted"

  "This message MAY have been posted (verify before replying)"

2) To add a handle at the bottom of the message that says something like

  [ Note: Message MAY have been posted. Verify it now ]

  following that handle would take you to a screen with the list of all
  the newsgroups in the newsgroup fields, which when selected allow you to
  open the above said newsgroup, where you can check if the message was
  posted by looking at the index. I'm not quite sure that this is a god
  idea but I thought I would comment on it, instead of deleting it totally
  from this message.

   The drawbacks of the first and second method are the same. Even if the
message was posted, there's no reason to believe that it will be in the
newsgroup in, say, 15 days, so actually verifying that the message was
posted may turn out unsuccessful, but what is good, is that there was a
warning, and so you'll think twice when you hit the 'R' key to reply about
what you want to do next with your message.

Eduardo

---

From: Jeff Franklin <[email-masked]>
Date: Wed, 6 Aug 2003 17:16:05 -0700 (PDT)

> mailed by Pine did not get that warning. That's a good compromise I think.

This is the part of the old code that will most likely not be returning.
The new behavior would warn for ALL messages in non-nntp mailboxes that
have a Newsgroup header.  The presence of "Pine" in the Message-id is by
no means an indication that the message was posted.  I'm now picturing a
warning to be shown after a user says to Followup this type of message,
saying something like "Replying to message that may or may not have been
posted to newsgroup".  Unless I hear any objections, consider it added.

Jeff

---

From: Eduardo Chappa <[email-masked]>
Date: Wed, 6 Aug 2003 21:47:07 -0700 (PDT)

:) On Tue, 5 Aug 2003, Eduardo Chappa wrote:
:)
:) > Granted, but the kludge tried to identify clients that used the
:) > newsgroup header in the way that you intended it to be used (as a
:) > signal that the message had been sent to a newsgroup), so for
:) > example, messages posted and mailed by Pine did not get that warning.
:) > That's a good compromise I think.
:)
:) This is the part of the old code that will most likely not be
:) returning. The new behavior would warn for ALL messages in non-nntp
:) mailboxes that have a Newsgroup header.  The presence of "Pine" in the
:) Message-id is by no means an indication that the message was posted.
:) I'm now picturing a warning to be shown after a user says to Followup
:) this type of message, saying something like "Replying to message that
:) may or may not have been posted to newsgroup".  Unless I hear any
:) objections, consider it added.

Hi Jeff,

  Consider my earlier message my objection. What will happen will be that
a user will press 'R' to reply, they'll recive the warning, cancel the
reply to check if the message was posted (which may take a long time),
then go back to the folder, press 'R' again and see the same message
again, for as many seconds as that will take (3 to 4 seconds of inactivity
each time?).

   I find this worse than the old behavior.

Eduardo

---

From: Nicolas Christin <[email-masked]>
Date: Thu, 7 Aug 2003 09:27:54 -0400

On Tue, 5 Aug 2003, Eduardo Chappa wrote:

> Granted, but the kludge tried to identify clients that used the
> newsgroup header in the way that you intended it to be used (as a
> signal that the message had been sent to a newsgroup), so for example,
> messages posted and mailed by Pine did not get that warning. That's a
> good compromise I think.

Eduardo,

I do not think identifying a few clients is the way to go. The kludge
was pretty horrible. It was checking against a few "good" MUAs, with
some ad-hoc tests, such as verifying the Message-ID starts with "Pine.".
Needless to say, a simple patch such as moving the "Pine" field to a
User-Agent header would break the whole thing. (That's why my User-Agent
patch was _not_ a two-liner in 4.55 and below.) In addition, as far
as I remember, the kludge did not check against some more recent
newsgroup readers such as slrn, Mozilla, did not check the presence of a
User-Agent field, etc.

I, for one, am glad this kludge is gone. On the other hand, any other
method of identifying that the Newsgroup header has been properly used
would be very welcome.

If you don't see any way of doing that other than checking for a few
"reasonable" mailers (which again I think is a bad idea), I suggest you
revisit the kludge in question. Move the Pine.ABC.xxx to the User-Agent
field (see the patch for that to save you some effort), and test whether
a User-Agent field has been set. Except for pine, pretty much all
"reasonable" mailers/newsreaders use one.

-- 
Nicolas

---

From: Jeff Franklin <[email-masked]>
Date: Thu, 7 Aug 2003 12:21:05 -0700 (PDT)

> again, for as many seconds as that will take (3 to 4 seconds of inactivity

I think we should look at this warning as a measure to prevent people from
shooting themselves in the foot.  Someone that is concerned about the
legitimacy of the mail they're responding to can take the time to verify
it as they see fit, whether it be by trusting the message-id header or by
opening the newsgroup to check for the message.

What Nicolas Christin says is spot on.  There are so many "trustworthy"
news readers that to compare against a list of them is futile.  Also,
those headers can easily be faked.

Jeff

---

From: Eduardo Chappa <[email-masked]>
Date: Thu, 7 Aug 2003 13:06:07 -0700 (PDT)

:) On Wed, 6 Aug 2003, Eduardo Chappa wrote:
:)
:) >   Consider my earlier message my objection. What will happen will be
:) > that a user will press 'R' to reply, they'll recive the warning,
:) > cancel the reply to check if the message was posted (which may take a
:) > long time), then go back to the folder, press 'R' again and see the
:) > same message again, for as many seconds as that will take (3 to 4
:) > seconds of inactivity each time?).
:)
:) I think we should look at this warning as a measure to prevent people
:) from shooting themselves in the foot.  Someone that is concerned about
:) the legitimacy of the mail they're responding to can take the time to
:) verify it as they see fit, whether it be by trusting the message-id
:) header or by opening the newsgroup to check for the message.
:)
:) What Nicolas Christin says is spot on.  There are so many "trustworthy"
:) news readers that to compare against a list of them is futile.

Hi Jeff,

I was not talking about the way that Pine checks for trustworthiness of
the fact that the message was posted (I'll talk about this later). What I
said was that it is annoying to have a message that will be there for 4
seconds at least twice in the screen. It's disruptive, especially the
second time.

I admit that one would like to have a system of checking for
trustowrthiness of the newsgroup header that does not depend on the
newsreader used by the sender, that may not possible yet and simply
declaring that it can not be done and ignoring the extra information is a
complete disregard of the reality that we live in and where Pine should
fit in. I think Pine needs to try harder, whatever method you decide to
use to do this.

I believe that the old behavior, even though not perfect, and maybe a
kludge, was far better that what you are going to do now. I guess we'll
have to agree to disagree and I will have to hack Pine. So glad I don't
have to use PC-Pine.

:) Also, those headers can easily be faked.

There are so many headers that can and are faked, I haven't seen anyone
protesting about verifying the FROM field in the index, even when it's
wrong. Why?, because every one seems to either believe it 100% (like in a
personal message), or disregard it (like in spam). In fact, this header is
used to construct the To: header in a reply. Maybe you should be doing the
case that Pine should give a warning saying "From field not verified".

When Pine sorts by thread, it has to use the Date field, which is
unverified. In fact, there are many fields that are used and all of them
can be faked. Not because of this, Pine does not do threads. You are free
to use as many fields as you can to determine a thread.

The fact is that Pine does not verify any headers, so saying that you
can't trust a newsgroup header ever is simply stating something that Pine
does with any field, it does not single out the newsgroup header. As with
threads, you can use as many fields as you like to determine if a message
was posted to a newsgroup with a certain degree of certainty. As you know,
not even the algorithm for threads is 100% fool proof, not everything is
correctly threaded, but nobody is going to stop using threads because of
this. People will encourage you to use more fields (or the fields that
you used in a better way) if they know what was missing when threading was
done. I hereby encourage you to use the message-id (or other headers) to
determine certainty that the message was posted, and if you have a good
certainty that the message was posted do not make me wait for 8 seconds
until the message is gone.

Thanks,

Eduardo