Subject: POP servers not advertising USER in CAPA reply

From: Vadim Zeitlin <[email-masked]>
Date: Tue, 1 Feb 2005 02:30:42 +0100 (Romance Standard Time)

Hello,

 I've got a bug report from a user about POP server at pop.spheregroup.net.
He couldn't login to it using c-client based client because the server
doesn't mention USER in its CAPA reply. Here is an example session:

    % telnet pop.spheregroup.net 110
    Trying 213.246.197.151...
    Connected to pop.spheregroup.net.
    Escape character is '^]'.
    +OK Sphere Solutions Pop Server ready
    capa
    +OK Capability list follows
    TOP
    UIDL
    .
    user [email-masked]
    +OK [email-masked]... Recipient ok
    pass ******
    +OK [email-masked]'s mailbox has 0 total messages (0 octets).
    quit
    +OK [email-masked] yyy.zz POP Server signing off (mailbox empty)
    Connection closed by foreign host.

Because of the line (952 in my UW_IMAP_2004b_SNAP-0409021845 snapshot)

  else if (!LOCAL->cap.user) mm_log ("Can't login to this server",ERROR);

in pop3.c, the absence of "USER" in CAPA reply results in not being able to
login to this server at all. It seems to me that this is not the ideal
library behaviour and that there is no harm to try to login using USER
(considerations of sending password in clear text don't necessarily apply,
i.e. imagine that this happens over a SSL tunnel...).

 I definitely don't pretend that this POP3 behaves correctly and c-client
does not, I'm well aware that the contrary is true. However from the
user-usefullness point of view, it would be really more helpful if c-client
tried to use USER which is understood by 99.99% of all POP3 servers (and
whatever the proportion of broken POP3 servers is, it's definitely more
than 0.01% in my experience).

 Thanks in advance,
VZ

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Mon, 31 Jan 2005 18:04:04 -0800 (Pacific Standard Time)

c-client's POP3 client code is behaving correctly.

A POP3 server that does not advertise USER in CAPA is specifically stating 
that USER is forbidden; it is the exact equivalent of an IMAP server that 
advertises LOGINDISABLED.  A POP3 client MUST NOT send a USER command to a 
POP3 server that supports CAPA but does not advertise USER.

Refer to RFC 2449.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Vadim Zeitlin <[email-masked]>
Date: Tue, 1 Feb 2005 12:19:45 +0100 (Romance Standard Time)

On Mon, 31 Jan 2005 18:04:04 -0800 (Pacific Standard Time) Mark Crispin <[email-masked]> wrote:

MC> c-client's POP3 client code is behaving correctly.

 Sorry for being unclear: I have absolutely no doubt about this. However
the correct behaviour is not very useful in this case.

MC> A POP3 server that does not advertise USER in CAPA is specifically stating 
MC> that USER is forbidden; it is the exact equivalent of an IMAP server that 
MC> advertises LOGINDISABLED.  A POP3 client MUST NOT send a USER command to a 
MC> POP3 server that supports CAPA but does not advertise USER.

 In theory I totally agree but in practice there is this broken server
which doesn't support any other way to login except by using USER but still
doesn't advertise it. It's clearly is a bug in server implementation and
using USER is the only way to work around it. The alternative is to not be
able to login at all which may be correct (although in fact I don't see
anything specifically forbidding use of USER in RFC 2449, it only states
that its presence in CAPA response means that USER/PASS are supported but
doesn't say anything about its absence!) but is absolutely useless.

 Speaking practically, what problems can I have if I still use USER even if
the server doesn't advertise it? AFAICS in the worst case the server will
reply that command is not supported. This doesn't seem very bad to me.

 Thanks,
VZ

---

From: Mark Crispin <[email-masked]>
Date: Tue, 1 Feb 2005 16:31:25 -0800 (Pacific Standard Time)

On Tue, 1 Feb 2005, Vadim Zeitlin wrote:
> using USER is the only way to work around it.

The server may not be broken.

They may have an administrative policy that clients should use the SSL 
POP3 service (port 995) instead of unencrypted POP3 port 110; but for the 
benefit of old pre-SSL clients (which also would not use CAPA) it allows 
the USER/PASS commands.

> The alternative is to not be

Not at all.  Did you try the SSL POP3 service?

> the server doesn't advertise it?

Doing so violates the specifications, and may very well violate the 
intentions of the POP3 server administrator.

Worse, you may find yourself accused of "behaving just like Microsoft" in 
violating specifications for convenience.  All too often the excuse of "a 
necessary workaround" has been offered as to why Outlook, etc. violates a 
specification.

Still worse, if it's considered to be something that c-client does, *I* 
will be accused of "behaving just like Microsoft."  No thanks.  :-)

> AFAICS in the worst case the server will

No.  If it doesn't reject until the PASS command then the result is that 
passwords are sent in the clear.

-- Mark --

---

From: ml <[email-masked]>
Date: Wed, 2 Feb 2005 10:14:49 +0900 (JST)

>benefit of old pre-SSL clients (which also would not use CAPA) it allows

I have come across at least two pop servers that don't advertise USER and
yet don't use SSL-POP3 nor any other authentication schemes.  Both of them
are at rather large ISPs (here in Japan.)  As the chance of getting their
sys. adm. to change their behavior is less than zero, I had to patch my
c-client-based stuff.

Nonetheless, the c-client distrubtion should NOT be changed in order to
serve mis-configured servers.  It is the responsibility of c-client users,
like myself to make the relevant changes. And yes, the banner offered by
my patched ipopd clearly shows that it has been patched:)

-- 
N.

---

From: Vadim Zeitlin <[email-masked]>
Date: Wed, 2 Feb 2005 11:53:05 +0100 (Romance Standard Time)

On Tue, 1 Feb 2005 16:31:25 -0800 (Pacific Standard Time) Mark Crispin <[email-masked]> wrote:

MC> On Tue, 1 Feb 2005, Vadim Zeitlin wrote:
MC> > In theory I totally agree but in practice there is this broken server
MC> > which doesn't support any other way to login except by using USER but still
MC> > doesn't advertise it. It's clearly is a bug in server implementation and
MC> > using USER is the only way to work around it.
MC> 
MC> The server may not be broken.

 Unfortunately in this case it definitely is. It does not support neither
TLS nor SSL (again, I should have written it from the beginning, sorry for
omitting to say this).

MC> They may have an administrative policy that clients should use the SSL 
MC> POP3 service (port 995) instead of unencrypted POP3 port 110; but for the 
MC> benefit of old pre-SSL clients (which also would not use CAPA) it allows 
MC> the USER/PASS commands.

 Ok, but if they [still] allow it, there mustn't be much harm in using it.
I would understand if they only allowed SSL logins perfectly well, but they
don't. In fact, they don't support them at all.

MC> Not at all.  Did you try the SSL POP3 service?

 Yes.

MC> > Speaking practically, what problems can I have if I still use USER even if
MC> > the server doesn't advertise it?
MC> 
MC> Doing so violates the specifications, and may very well violate the 
MC> intentions of the POP3 server administrator.

 Again, not in this case.

MC> Worse, you may find yourself accused of "behaving just like Microsoft" in 
MC> violating specifications for convenience.  All too often the excuse of "a 
MC> necessary workaround" has been offered as to why Outlook, etc. violates a 
MC> specification.
MC> 
MC> Still worse, if it's considered to be something that c-client does, I 
MC> will be accused of "behaving just like Microsoft."  No thanks.  :-)

 I understand your point of view but you should realize, of course, that I
am going to patch my c-client version (once again) because I can't tell the
user with a straight face that I am not going to fix it when it's a whole
of one line fix. So it's just going to be one more patch in my version of
c-client and one more reason I can't use "official" (although at least in
Debian case there are quite a few patches in it too) version of the library
packaged by Debian, RedHat and so on. Not a big deal for me as there are
other issues which are much more important for me which I had to patch in
my version but I still can't prevent myself from thinking that all this is
a big waste of effort and that having at least an option in c-client to
enable the behaviour which makes sense to the users couldn't really be such
a bad thing.

 Regards,
VZ

---

From: Mark Crispin <[email-masked]>
Date: Wed, 2 Feb 2005 10:25:09 -0800

> Ok, but if they [still] allow it, there mustn't be much harm in using it.

There is a substantial amount of harm if the purpose of allowing USER when 
not advertised is to provide temporary reclama for old clients.  By doing 
the old client behavior, you could stymie the site's migration plans.

Doing so is the type of behavior that Microsoft is often accused of doing: 
taking the expedient approach instead of the correct one.  I find it sadly 
ironic that the open source community would even think of doing this, 
after all the years of Microsoft-bashing over this very issue.

> > > Speaking practically, what problems can I have if I still use USER 

It most certainly does violate the specification.

The minute your software is installed at a site which has such reclama, it 
also violates the intentions of the server administrator and adds to his 
(or her) headaches.  Your software has no way of knowing that this is the 
case.

> of one line fix.

In that case, honesty and morality requires that you also disclose to your 
user that your client is BROKEN and NON-COMPLIANT with the specifications, 
and that as a result it has a SECURITY BUG that will continue even when 
your user upgrades his server.

The entire reason why USER is part of CAPA is to enable the behavior of 
client code (such as in c-client that you advocate disabling.  You are 
breaking an important and valuable security mechanism that many people 
spent a long time developing.

And for what reason?  So your client works with one particular broken 
server!

> I still can't prevent myself from thinking that all this is
> a big waste of effort

Then don't do it!

There is no law that says that you have to support broken servers.  The 
world does not become a better place by adding security bugs in order to 
accomodate broken software.

If a site doesn't want to fix its server to comply with specifications and 
run your client, then your client isn't important to that site -- and that 
site shouldn't be important to you either.  There are many other sites 
which run compliant servers; quite enough to keep you in business.

-- Mark --

---

From: Vadim Zeitlin <[email-masked]>
Date: Wed, 2 Feb 2005 19:52:25 +0100 (Romance Standard Time)

On Wed, 2 Feb 2005 10:25:09 -0800 Mark Crispin <[email-masked]> wrote:

MC> Doing so is the type of behavior that Microsoft is often accused of doing: 
MC> taking the expedient approach instead of the correct one.  I find it sadly 
MC> ironic that the open source community would even think of doing this, 
MC> after all the years of Microsoft-bashing over this very issue.

 Mark,

 I'm exchanging sporadic mails with you for the last 9 years or so and
during all this time I don't cease to be amused by your habit of bringing
Microsoft policy into just about any kind of technical discussion. Could we
please just leave Microsoft alone? I have no relation with Microsoft but I
don't like being accused of Microsoft bashing groundlessly and, while I'm
proud to be part of, I don't represent this mythical (but, judging from
your description, nefarious) "open source community" in any way at all, so
could we please just leave it at that? Thanks.

MC> It most certainly does violate the specification.

 How can you seriously state this in a case of a server which violates it
in such way that it doesn't leave me any other choice but to do this?

MC> The minute your software is installed at a site which has such reclama, it 
MC> also violates the intentions of the server administrator and adds to his 
MC> (or her) headaches.  Your software has no way of knowing that this is the 
MC> case.

 But the user does. This is why I think that ideally there would be an
option, e.g. a cclient callback to the main program which would allow it to
decide -- presumably by asking the user -- whether to continue connecting.
If there was a chance of this being ever integrated into c-client you can be
sure that I'd provide a patch doing exactly this or whatever else you'd
accept. However, when confronted to a total and absolute refuse to make any
modifications at all to c-client from your part, I'm understandably
reluctant to spend any amount of time on the code which will only create me
additional maintenance headaches.

MC> > I understand your point of view but you should realize, of course, that I
MC> > am going to patch my c-client version (once again) because I can't tell the
MC> > user with a straight face that I am not going to fix it when it's a whole
MC> > of one line fix.
MC> 
MC> In that case, honesty and morality requires that you also disclose to your 
MC> user that your client is BROKEN and NON-COMPLIANT with the specifications, 

 I do disclose that the version of c-client I use has been modified.

MC> If a site doesn't want to fix its server to comply with specifications and 
MC> run your client, then your client isn't important to that site -- and that 
MC> site shouldn't be important to you either.

 This is an amazing view of a problem which probably represented quite well
the view of Internet in the early eighties. I may assure you however that
when the server is run by a big ISP (as is the case here), no user,
whatever his determination to use my client, can change their mind. And
while I absolutely don't care about that site, I do care about my users
which is the word which appears to never appear in your vocabulary at all.

 Regards,
VZ

---

From: Pawel Salek <[email-masked]>
Date: Wed, 02 Feb 2005 19:33:18 +0000

On 02/02/2005 07:52:25 PM, Vadim Zeitlin wrote:
> [snip]
>   But the user does. This is why I think that ideally there would be
> an  option, e.g. a cclient callback to the main program which would
> allow  it to  decide -- presumably by asking the user -- whether to
> continue  connecting.  If there was a chance of this being ever
> integrated into c-client you  can be  sure that I'd provide a patch
> doing exactly this or whatever else  you'd  accept. However, when
> confronted to a total and absolute refuse to  make any  modifications
> at all to c-client from your part, I'm understandably  reluctant to
> spend any amount of time on the code which will only  create me

I am bit afraid of jumping right in into this heated discussion but I
am going to give it a shot anyway. Working around bugs in somebody
else's software is a tricky business: it's a judgement call and in case
of a reference implementation (ok, c-client is a imap reference
implementation, not pop3 but I can imagine many people may treat it as
such) may cause an injustified impression that the bugs are not really
bugs but just features. Assuming that you really have to work around
this missing USER capability , I am not sure whether shifting
responsibility of maintaining such modifications from you to the author
of c-client is fair - he seemed to be reluctant to accept it, didn't he?

I am sure that you would succeed convincing the right people (i.e. the
writers of the buggy server) if you really wanted to, if you tried as
hard as you try here. (have you tried?). It would be much better spent
time - it would not only fix interaction with c-client but all the
other standard conforming POP3 implementations.

Pawel

PS. my reading of RFC1939 is that it is ok to try USER but send PASS
only if the server answered with +OK. Since the server in question has
CAPA, it is a RFC2449-compliant server and the CAPA response is the
authorative way of finding out which commands are available. While I do
not see in RFC2449 an explicit statement that client MUST NOT try
unadverstised commands, it is pretty much in the spirit of CAPA: if
CAPA was not authorative it would serve no purpose.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 2 Feb 2005 11:35:37 -0800 (PST)

On Wed, 2 Feb 2005, Vadim Zeitlin wrote:
> the view of Internet in the early eighties.

Did you use the Internet in the early eighties?  If not, please do not 
lecture me about what the Internet was like in the early 1980s.

> I may assure you however that
> whatever his determination to use my client, can change their mind.

In that case, the user can cancel his account at that ISP and find an 
alternative ISP which complies with the standards.

> And

If you really believe that I do not care about my users, maybe you should 
use other library for your application.

What you advocate is not caring for your users.  Caring about your users 
requires that you care about your users' interests.  That, among other 
things, necessitates following specifications and the current security 
requirements of those specifications.

The minute that specifications are violated in the name of expediency, the 
entire user community is hurt.

Security requirements exist for a reason.  They often impact expediency. 
The current tsunami of worms, viruses, and spam came about in large part 
because of expediency.

The only reason for the USER capability in POP3 CAPA is to provide, by the 
absence of USER, a means to block compliant POP3 clients from sending a 
USER command.  If you eliminate that block, you make the USER capability 
meaningless, and undo a substantial amount of work put in by many 
individuals.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Wed, 2 Feb 2005 12:03:43 -0800 (PST)

Pawel, thanks for your comments.  It brings up some good points which I 
may have glossed over or imperfectly stated.

On Wed, 2 Feb 2005, Pawel Salek wrote:
> only if the server answered with +OK.

Here's where interpretation of RFCs leads us into trouble.  There is a 
very great difference between:
  . what is "right"
  . what is "obvious" to you and me
  . what is "obvious" to someone else
  . what the RFCs actually say
Failure to take all four of these into account leads to disaster!  [Sad 
experience here...]

This current thread is a good example.

Unfortunately, RFC 1939 has the following statement in its Security 
Considerations section:
    Servers that answer -ERR to the USER command are giving potential
    attackers clues about which names are valid.

We can argue that this does not apply when -ERR is given to any USER 
command.  However, if you read RFC 2449, you will find that the USER 
capability refers to "USER and PASS".

It may be "obvious" that "USER and PASS" means "any USER command and/or 
any PASS command".  But, it doesn't say that.

This opens the door (wide enough to drive a truck through) for an 
interpretation that the USER capability applies *only* to sending USER 
*and* PASS, and not to sending USER by itself (or sending USER along with 
some token other than a password).

Put another way, it can be argued that the USER capability refers to the 
PASS command, not the USER command.  That argument is completely valid 
with the way that RFC 1939 and RFC 2449 are worded.

Once that argument is accepted, you then run into the unfortunate 
statement above in the Security Considerations of RFC 1939, which seems to 
indicate that a server SHOULD NOT answer -ERR to the USER command.

At this point, a client that has been broken not to respect CAPA (as 
discussed in this thread) will send the password in the clear to a server 
which won't accept it and has a policy of no passwords in the clear.

This all may sound like a stretch.  However, 30 years of experience with 
Internet protocols tells me that it isn't a stretch at all.  The door is 
open, and sooner or later a truck *will* be driven through it.  The only 
question left is whether or not we allow the evil hitchhiker through as 
well.

> Since the server in question has
> authorative way of finding out which commands are available.

Indeed.  RFC 1939 reflects the old way of "just try it and see if it 
works" thinking.  Such things has caused enormous problems, and was a 
major factor in adopting capabilities in the IMAP2 -> IMAP4 transition.

A similar discussion is going on with the NNTP protocol.  The difference 
is that it has a much more entrenched infrastructure of "just try it and 
see" and capabilities listing that isn't authoritative either way.

The only sane way out of the mess is to demand that capabilities listings 
to be authoritative, and in turn to respect that authority.  This means, 
among other things, that clients must demand authoritative capabilities 
listings from servers.  Clients should not try to guess and work around, 
and servers must not assume that clients will guess and work around.

Unfortunately, every client which guesses and works around gives the 
server an excuse not to adhere to its requirements.  Time and time again, 
I hear "our server works with Outlook and Thunderbird, therefore the bug 
is in Pine; so we're not going to change our server or even look any 
further."

-- Mark --

---

From: Vadim Zeitlin <[email-masked]>
Date: Wed, 2 Feb 2005 22:55:49 +0100 (Romance Standard Time)

On Wed, 2 Feb 2005 11:35:37 -0800 (PST) Mark Crispin <[email-masked]> wrote:

MC> On Wed, 2 Feb 2005, Vadim Zeitlin wrote:
MC> > This is an amazing view of a problem which probably represented quite well
MC> > the view of Internet in the early eighties.
MC> 
MC> Did you use the Internet in the early eighties?

 You know perfectly well that I didn't (at least I told you so many times
in the past as all our discussions seem to end with this) and you know that
I do know that you did. But this is not the point. The point is that the
balance has changed since then. Back then users could force such changes.
Now they can't.

MC> In that case, the user can cancel his account at that ISP and find an 
MC> alternative ISP which complies with the standards.

 How many users are going to do this? Or, more importantly, how many ISPs
would do anything even if [s]he did?

MC> If you really believe that I do not care about my users, maybe you should 
MC> use other library for your application.

 Unfortunately I don't have resources to redo 10 years of work using
c-client.

MC> The only reason for the USER capability in POP3 CAPA is to provide, by the 
MC> absence of USER, a means to block compliant POP3 clients from sending a 
MC> USER command.  If you eliminate that block, you make the USER capability 
MC> meaningless, and undo a substantial amount of work put in by many 
MC> individuals.

 The answer to this as well to the security problem (how significant it is
compared with the general idea of sending password in the cleartext could
be discussed some other time...) you mention in the other message in this
thread is the same: I do *not* want to eliminate this block. I want to
provide the user with a way to override it. Just as a warning is shown
before connecting to a server which requires sending password in clear
text, a user could be warned that it's a potential security problem and
that what he does may be contrary to the server administration policy. And
then let each user decide for himself whether he wants to do it or not.

 I'm just trying to give user a choice in the matter and how does it mean
"not caring to my users" or being responsible for "tsunami of worms,
viruses, and spam" is beyond me.

 Anyhow, thanks for explaining your point of view.
VZ

---

From: Frode Nordahl <[email-masked]>
Date: Thu, 3 Feb 2005 00:09:38 +0100

> would do anything even if [s]he did?

Many. I'm working at one of the largest ISPs in Norway, and we take 
pride in allways attempting to do things the right way, following 
standards, and generally being nice internet citizens.

However, with your attitude towards the problem, no one will change.

Is the pop server (or proxy) developed inhouse at the ISP, or is it 
from an external vendor?

If it's an external vendor, go after them, and It'll eventually get in 
to the ISP through a security update.

Either way there is probably someone sitting there that is just like 
your or me that would be very happy to learn about their honest mistake 
and correct it. It's not like ISP's or indeed ISV's are large chunks of 
grey matter, molded, never to change.

(Although it might seem that way if you try to fight your way up from 
first line support or sales contacts).

If everyone was to go about and put in band-aids to support everyone 
elses errors and misfeatures, we'll end up with a large, randomly 
misbehaving mess.

You are free to do whatever you want, but I am glad to see that 
c-client will never grow such defects.

Regards,
Frode

---

From: Vadim Zeitlin <[email-masked]>
Date: Thu, 3 Feb 2005 00:21:52 +0100 (Romance Standard Time)

On Thu, 3 Feb 2005 00:09:38 +0100 Frode Nordahl <[email-masked]> wrote:

FN> Many. I'm working at one of the largest ISPs in Norway, and we take 
FN> pride in allways attempting to do things the right way, following 
FN> standards, and generally being nice internet citizens.

 My sincere congratulations. But you also should know how rare this
behaviour is. And, moreover, you probably don't use server broken in such
bad way anyhow because you know what you're doing. ISPs running the servers
like this one do not. And in my experience they don't care.

FN> However, with your attitude towards the problem, no one will change.

 This attitude comes from experience. It may be lesser than that of MRC but
it is not negligible. I'd rather spend time on writing code and improving
my program (for which I don't have enough time now) instead of wasting time
with that ISP.

FN> (Although it might seem that way if you try to fight your way up from 
FN> first line support or sales contacts).

 It certainly seems this way to me. After trying to discuss with a few of
them in the past I am not about to repeat this (painful) experience. Please
don't forget that this is not my work and there is a limit to the time I
can spend helping a user of my program.

 Quoting Mark Crispin:

MRC> Time and time again, I hear "our server works with Outlook and
MRC> Thunderbird, therefore the bug is in Pine; so we're not going to
MRC> change our server or even look any further."

 If he (knowing who he is and all that) keeps hearing this, how many
chances of success do I have? Let's be reasonable for a change...

FN> You are free to do whatever you want, but I am glad to see that 
FN> c-client will never grow such defects.

 This would be a defect in a server. I still have no idea why allowing the
user to override a badly broken server behaviour from his client is a
defect. And so far I still see no reason for a user (knowing that he is
careless enough to use POP, to use clear text auth, and to use such broken
server) to not do it. And no other workaround, except for switching to
another client.

 Regards,
VZ

---

From: ml <[email-masked]>
Date: Thu, 3 Feb 2005 11:00:56 +0900 (JST)

On Thu, 3 Feb 2005, Vadim Zeitlin wrote:

> This would be a defect in a server. I still have no idea why allowing
>the user to override a badly broken server behaviour from his client is a
>careless enough to use POP, to use clear text auth, and to use such
>broken server) to not do it. And no other workaround, except for
>switching to another client.

Leaving the right or wrong question for a moment, I can just imagine how
confused my John/Jane Doe users would be if I were to offer them a choice
of overriding the broken server.  The support desk will be swarmed by
confused users.

If they CARE about the danger of sending passwords in clear text, they
wouldn't have used those ISPs-with-broken-servers in the first place.
You may have covered your behind when you offer users the choice of "Do
you really want to do this?"  but it'll confuse them more than anything
else.