Subject: Re: Your message (no subject found) #011227@15:02:32.1096

From: Bruce Cohen <[email-masked]>
Date: Sat, 14 Sep 2002 11:49:44 +0200 (MET DST)

Mark,

A few weeks ago my provider finally updated their pop3 servers, now
running imap4 as well. Unfortunately, this has also caused problems with
pine 4.33 (SunOs 5.7), which was running quite stable most of
the time.

Since the update any session over about 5 minutes runs into major problems
(either pop3 or imap!) - at some point pine and the popper lose connection
(waited 15 seconds for server reply, etc. followed by Mailbox closed due
to access error) - very occasionally this is major: abort and core dump.

Once the connection problem has occured, then the popper is locked for
about 30 minutes and I can't log back on during that time.

The pop3 and imap4 announce v.3.0.0.

Do you have any idea what is going on? Why does the popper (and I'm pretty
sure it is the popper) throw out the pine connection and is there anything
that can be done to solve this? Would an update to a later pine version
maybe help?

Even if a connection seems to be stable and has been so for several
minutes, it will eventually be broken at some point either by sending mail
or if new mail comes in.

Unfortunately, my provider only provides marginal
support as far as getting pine running properly is concerned...they have
been aware of the problem for a few weeks but have not really investigated
the matter - I may be one of the last few pine users they have.

Thanks for any help or advice you can offer.

Bruce Cohen

On Thu, 3 Jan 2002, Mark Crispin wrote:

> On Thu, 3 Jan 2002 17:41:30 +0100 (MET), Bruce Cohen wrote:
> > So why the abort and core dump -any ideas?
> >
> > You seem to imply that it shouldn't have happened...
>
> That's right.  An abort/dump should never happen.
>
> As for "why"...do you have a stack trace or any other form of gdb/dbx analysis
> of the dump?  Many times, problems such as this aren't reproducable here.
>
>
>

---

From: Bruce Cohen <[email-masked]>
Date: Sat, 14 Sep 2002 15:20:53 +0200 (MET DST)

Forgot to mention: one of the error messages I get after it has happened
is "UID sequence not valid" (but it was, I was logged in!)

Bruce Cohen

---

From: Mark Crispin <[email-masked]>
Date: Sat, 14 Sep 2002 08:29:27 -0700 (PDT)

Unfortunately, there isn't enough information in your message for me to know
what is going on.  Here's the best that I can offer you in response now,
pending further clarification from you.

Have you tried the most recent release (Pine 4.44) or the pre-release of Pine
4.50?  If the problem happens in 4.33 but not in 4.44 or 4.50, then it's
likely to be a problem which we have already fixed, and our answer would be
"upgrade to the most recent version."

What was the exact and complete text (no paraphrasing) of the error message
that you got?  There is no such error message as "UID sequence not valid" in
Pine, but perhaps such a message came from the IMAP server.

There is a "UID sequence invalid" error message in Pine, which occurs only
when an attempt is made to use a unique identifier of zero.  Since zero is not
permitted as a value, this indicates some major problem.  But it could also be
a wild goose chase if that wasn't the message.

When you say "popper", do you mean to say "POP3 server", or are you referring
to the popper program, which is one of many implementations of POP3 server?  I
have nothing to do with the popper program or its performance.

You say "any session over about 5 minutes", and imply that the sesion is
disconnected.  Is it possible that the POP3 server has been modified by the
ISP to set a 5 minute inactivity autologout timer?  If so, that POP3 server is
non-compliant with the POP3 standard (RFC 1939) which stipulates that the
minimum inactivity autologout timer is 10 minutes.  There isn't much we can do
if external software violates the standards.

You say "the pop3 and imap4 announce v.3.0.0" but no other identifying
details.  That is not a version number that my server implementations have
used.  I have no idea what POP3 or IMAP server implementation this is.  More
details are needed, such as the full greeting banner when you connect to the
server, or the DNS name of the server system.

In conclusion:

Pine should never crash and core dump.  But if it does, just saying "Pine
crashed and core dumped" isn't very helpful; as I indicated in my 3 Jan 2002
message we need a stack trace or other form of gdb/dbx analysis, or
alternatively a reliable way to reproduce the problem.  This is particularly
important if it happens on a system that we don't use here, such as SunOS.

I think, but do not know, that the POP3 and/or IMAP server implementation that
you are using is not the one provided by us.  I don't know if the server
implementation you are using is standards-compliant or not, since I don't even
know which one it is.  But the suggestion of a 5-minute autologout timer on
the POP3 server suggests a non-compliant server.  It could very well be that
the server is non-compliant in other ways, and that non-compliance confuses
Pine sufficiently that Pine crashes.

Even with a non-compliant POP3/IMAP server, Pine should not crash.  But if
that is what is happening, we need to know what the non-compliant POP3/IMAP
server is doing in order to know why it causes a Pine crash.

To make matters worse, even after we fix the problem that causes Pine to crash
with the non-compliant POP3/IMAP server, that does not mean that it will work
for you.  We can't guarantee any good results with a non-compliant server.

We don't know that the POP3/IMAP server is non-compliant.  It may be perfectly
compliant, and we're on the wrong track because your message misled us.
That's why we need more and better information:
 . stack traces or other crash analysis
 . pine-debug files from a session that crashed, preferably with
    protocol telemetry ("pine -d imap=4")
 . complete and exact text of all error messages, with no paraphrasing
 . full identity of the POP3/IMAP server so we know what we're dealing with
    at the other end.

---

From: Bruce Cohen <[email-masked]>
Date: Sat, 14 Sep 2002 18:37:03 +0200 (MET DST)

Hi, Mark,

Thanks for such a speedy reply.

Here's what I have so far:

if I telnet to io 110 or io 143 I will connect to io.worldonline.nl, and
the server calls itself Tiscali POP server (v.3.0.0) (or Tiscali IMAP4
server) (tiscali took over worldonline some time ago). The DNS is
195.241.48.143

yesterday the dns for pop3.worldonline.nl was 195.241.76.34
                      pop3.tiscali.nl  was    195.241.77.228
                      imap.tiscali.nl  was    195.241.77.229

today these all seem to point to 195.241.76.34
(or anything up to and including 195.241.76.40
pop3-1 is 34, pop3-7 is 40)

You're right, it is UID sequence invalid (I did it from memory, sorry!)

popper: not sure what I mean, really (I was referring to the POP3 server,
I thought). Does pine connect directly to the
POP3 server, or can there be an additional program running (interfering)
between the two?

5 or 10 minute inactivity: I assume you mean that nothing is written to
the POP3 or IMAP in that time (flag change, etc.). Even if the session is
disconnected by the POP3 side, shouldn't pine recover and be able to renew
the connection (at least in IMAP4???). I'll try to start timing it more
closely, but it is difficult - I've been on now for way over 10 minutes
without any error messages whatsoever (POP3-3), but I'm sure that once
this is sent they will appear, probably without abort and core dump is my
guess.

No, I have not tried a later version than 4.33. I would have to ask them
to update, and really would like to avoid doing that unless I am very sure
that it will fix the problem.

Stack trace - believe I could not find gdb/dbx last time around, haven't
looked again this time.

Will collect a couple of debug files and send them on.

Bruce

---

From: Mark Crispin <[email-masked]>
Date: Sat, 14 Sep 2002 10:58:36 -0700 (PDT)

I took a look at those servers.

It appears that the IMAP server is based upon an old version of UW imapd, but
has been substantially hacked.  I don't know what was hacked in it.  I could
not determine anything about the POP3 server.

If you use the /pop3 switch in the mailbox specifier in Pine, Pine directly
connects to the POP3 server.  "popper" is the name of a program which
implements a POP3 server; it does not mean "POP3 server".  Not all POP3
servers are popper.  So, unless you know that the POP3 server is based on
popper, it is better not to use the term "popper" since that just confuses
matters.

Yes, an "inactivity autologout timer" refers to the session being disconnected
after some period of inactivity.  The minimum timer is 10 minutes in POP3, and
30 minutes in IMAP.  Any ISP which hacks the timers to be less than that is in
violation of the standards.  If the ISP refuses to comply with standards, you
should fire them and get one that will comply.  However, before you accuse an
ISP of something as serious as standards non-compliance, it is necessary to
make sure that is really the problem.

Pine typically communicates with a POP3 or IMAP server no less often than
every 3 minutes.  So inactivity autologout timers should not be an issue with
Pine.  However, I have heard of an ISP (now out of business) which set the
timer to 1 minute specifically to block Pine.  Such idiocy defies
comprehension, but that's one of the reasons why that ISP is out of business.

No, Pine does not "renew" the connection.  Nor is "renewing" the connection a
meaningful concept in IMAP.  IMAP sessions have state; tear down the IMAP
connection and the state is lost.  A new IMAP connection has entirely new
state, and any previous state is invalidated.

There is one other possibility which you should investigate.  Is there any
possibility that you might be inadvertantly opening more than one POP3 or IMAP
session to the mailbox simultaneously?  A background "check for new mail"
program may do this, or possibly running another mail reading program while
you are running Pine?  If so, that may explain the disconnects but it would
not explain the Pine crashes.

---

From: Bruce Cohen <[email-masked]>
Date: Sun, 15 Sep 2002 08:22:11 +0200 (MET DST)

Hi Mark,

Here's a debug file on an imap connection that failed typically - is this
OK or do you need a higher level debug?

Bruce

 On Sat, 14 Sep 2002, Mark Crispin wrote:

> Unfortunately, there isn't enough information in your message for me
> to know what is going on.  Here's the best that I can offer you in
> response now, pending further clarification from you.
>

---

From: Bruce Cohen <[email-masked]>
Date: Sun, 15 Sep 2002 09:50:55 +0200 (MET DST)

Mark,

Found a server with different behavior: pop3-1.wolmail.nl/imap
195.241.76.55. Several times now have been able to work for a considerable
period of time and have been cleaning up the INBOX (lots of old msgs that
now try to automove to mbox - I had been configured pop3 for several
months, but obviously the imap read msg flags are set even in pop3 once a
msg has been read).

This cleaning up is basically moving to folders and deleting. At some
point a move fails with the waiting for server reply message, with the
option to break the connection after 60 seconds. Then comes the "message
to save shrank" error message (haven't had that for ages!), and the
MAILBOX closed due to access error msg. (debug file enclosed)

At least I can log right back on immediately - so far with this server
there seems to be only a broken connection, not a lock.

It's more comfortable to work with this server than anything else I've
used since the ISP updated, but I've no guarantees this will continue for
any length of time. pop3.worldonline.nl and pop3.tiscali.nl are the ones
we're supposed to use.

Wonder what will happen when I send this - don't know if I'll be able to
continue or thrown out with or without a lock.

Bruce

---

From: Bruce Cohen <[email-masked]>
Date: Sun, 15 Sep 2002 10:05:37 +0200 (MET DST)

The answer to the question is: was thrown out as usual and the mailbox is
locked.

there seems to be an authentication server of some kind at 172.16.1.64
and 172.16.1.66 -
difficult to capture the entire login in my log, but it looks about like
this when the mailbox is locked:

[Opening "INBOX" <\> ] [{pop3-1.wolmail.nl/imap}INBOX : [REFERRAL
IMAP://cohenb%40tiscali.nl;AUTH=*@17]
   [[REFERRAL IMAP://cohenb%40tiscali.nl;AUTH=[email-masked]/] mailbox locked]
          [Connection failed to 172.16.1.64,143: Connection timed out]
                               [No folder opened]

[{pop3.worldonline.nl/imap}INBOX : [REFERRAL IMAP://cohenb%40tiscali.nl;AUTH=*@]
   [[REFERRAL IMAP://cohenb%40tiscali.nl;AUTH=[email-masked]/] mailbox
locked][rsh to IMAP server timed out]
[Connection failed to
172.16.1.66,143: Connection timed out]

Is there any way to break a lock like this short of waiting for it to
happen?

Bruce

---

From: Mark Crispin <[email-masked]>
Date: Mon, 16 Sep 2002 19:48:57 -0700 (Pacific Daylight Time)

> OK or do you need a higher level debug?

Your debug file indicates that you opened the IMAP session at 07:57:52.
You expunged the mailbox at 07:58:32, and at 08:05:55 a checkpoint
occurred.  At 08:13:59 it was detected that the IMAP session had abruptly
disconnected.

Your mail-check-interval is set to 60 seconds, so the IMAP server would
have been communicated with at least every minute.  In any case, the 16
minutes and 7 seconds between the time you established the connection is
less than the 30 minute minimum inactivity autologout interval, not to
mention the mere 8 minutes and 4 seconds since the last logged interaction
(there probably were other interactions, but weren't logged).

This means that something is improperly breaking your connection between
client and server.  In effect, Pine is complaining that "I got hung up
on."

The problem is, that's all that Pine knows.  We don't know if the server
disconnected, or if the disconnection was due to something elsewhere.

If the server was responsible for the disconnect, it is a bug in the
server.  Barring a LOGOUT command from the client or a critical shutdown,
no server should ever disconnect a session that was active as recently as
8 minutes and 4 seconds ago.  So, you need to check with your server
management to see what the server says.

All too often, if you find anything useful in the server log, it will be a
whine to the effect that "I got hung up on."  In other words, both server
and client are saying "it wasn't my fault."  That means you have to do
some network debugging, possibly even packet sniffing.

Here's some network issues that have caused problems in the past:

Some versions of UNIX TCP have a "feature" in which a "destination
unreachable" ICMP message causes the closure of all TCP connections to the
IP address in that destination.  Needless to say, "destination
unreachable" could mean something as insignificant as a router dropping a
packet due to excessive traffic.  This "feature" should have been
exterminated over 10 years ago, but it might still be in the version of
Solaris that you are running.

A variant of this "feature" has recently come up in NAT boxes; some NAT
boxes garbage collect connections by arbitrarily purging memory of all
active connections and then recreating the connection if the client
initiates some activity.  However, if the server initiates the activity
(which can happen in IMAP), the server gets a TCP reset from the NAT box,
which in turn provokes a TCP reset from the server the next time the
client sends anything.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 16 Sep 2002 19:53:06 -0700 (Pacific Daylight Time)

> [[REFERRAL IMAP://cohenb%40tiscali.nl;AUTH=[email-masked]/] mailbox locked]

You'll need to speak to the management of the tiscali.nl IMAP server.
Although that server seems to be based upon the UW IMAP server, the fact
that it issues referrals indicates that it has been modified.  Also, the
"mailbox locked" message is not in the UW server, indicating another
modification.

The underlying problem may not be in their IMAP server, but they need to
be brought into the conversation.

-- Mark --

---

From: Bruce Cohen <[email-masked]>
Date: Tue, 17 Sep 2002 23:24:51 +0200 (MET DST)

Hi, Mark,

Thanks for your detailed replies. I'll work on trying to find someone
responsible at tiscali and try to get them involved. This may take
awhile considering how bureaucratic they can be, but I will get back to
you.

Once again, many thanks.

Bruce

---

From: Bruce Cohen <[email-masked]>
Date: Wed, 18 Sep 2002 07:04:38 +0200 (MET DST)

Mark,

Just confirmed something I had suspected for a couple of days. If new
msgs come in and the index page that they are on has been opened in pine
(pop3), then they will show up without the N flag the next time the
index of the INBOX is viewed in pine (imap) even if the msgs themselves
have not been opened.

Is this intentional, a bug in pine 4.33, or is this due to their hacked
pop3/imap servers?

Bruce

On Tue, 17 Sep 2002, Bruce

---

From: Mark Crispin <[email-masked]>
Date: Tue, 17 Sep 2002 22:25:08 -0700 (PDT)

> Is this intentional, a bug in pine 4.33, or is this due to their hacked

In POP3, obtaining sufficient information about a message to draw the message
index has the effect of causing the message to be marked as read.
Furthermore, in POP3 there is no way to unmark the "read" status.

IMAP has a means to do this without marking the message as read.

This is a limitation of POP3.

---

From: Bruce Cohen <[email-masked]>
Date: Wed, 18 Sep 2002 07:49:45 +0200 (MET DST)

But if one logs back on in pop3 this doesn't happen...even a msg that
has actually been read will show up as New on the next login.

I realize that one normally doesn't keep switching back and forth
continually between pop3 and imap as I have been doing for the past few
days, but it sure would be nice if there were a way for IMAP to work
around this.

Bruce

---

From: Mark Crispin <[email-masked]>
Date: Wed, 18 Sep 2002 08:30:01 -0700 (PDT)

> has actually been read will show up as New on the next login.

This also is a limitation of POP3.  POP3 does not have "read" status, so all
POP3 messages show up as "unread" in Pine.

> days, but it sure would be nice if there were a way for IMAP to work

It would be nice, but POP3 is too limited.  The standard answer is to "stick
with IMAP".