Subject: [Imap-uw] authentication never performed.

From: BuildSmart <[email-masked]>
Date: Sun, 5 Mar 2006 23:47:04 -0500

Iit seems that /etc/passwd is checked for a valid user before  
authentication but since the users are be stored in MySQL they are  
never authenticated.

I currently have pam (using pam_mysql) validating the user and then  
authenticating but if imapd doesn't pass anything to pam  
authentication always fails.

If the user exists in /etc/passwd then imapd does pass to pam and  
authentication does occur and I have verified this by adding users  
and testing the authentication process.

Is there a method/way to disable this check in imapd so that pam can  
handle the whole process?

_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: Mark Crispin <[email-masked]>
Date: Sun, 5 Mar 2006 20:51:07 -0800 (PST)

On Sun, 5 Mar 2006, BuildSmart wrote:
> Is there a method/way to disable this check in imapd so that pam can handle 
> the whole process?

The short answer is "no".

UW imapd users MUST have /etc/passwd entries, even if password 
authentication is done elsewhere.  They must have unique UNIX UID numbers 
(access control depends upon this) and they must have defined UNIX home 
directories.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: BuildSmart <[email-masked]>
Date: Mon, 6 Mar 2006 00:38:17 -0500

On Mar 5, 2006, at 23:51 , Mark Crispin wrote:

>> Is there a method/way to disable this check in imapd so that pam  
>> can handle the whole process?
>

Accepted, explanation not required.

> authentication is done elsewhere.  They must have unique UNIX UID  
> numbers (access control depends upon this) and they must have  
> defined UNIX home directories.

What I'm trying to achieve is true virtual users, all required  
authentication is currently done with mysql, what I'd like to avoid  
is adding 9263 users to /etc/passwd and creating 9263 homes.

What occurs if they don't have home directories???

I was thinking of replacing the qpopper binary with the ipop3d from  
uw-imap however it doesn't appear to be a drop in replacement, is  
there any additional configurating required???

I couldn't find any inetd or xinetd config files and the  
documentation didn't tell me anything special was required for  
arguments or did I miss something?

> -- Mark --

-- Dale

---

From: Mark Crispin <[email-masked]>
Date: Sun, 5 Mar 2006 21:49:21 -0800 (PST)

On Mon, 6 Mar 2006, BuildSmart wrote:
> What I'm trying to achieve is true virtual users, all required authentication 
> is currently done with mysql, what I'd like to avoid is adding 9263 users to 
> /etc/passwd and creating 9263 homes.

By design, UW imapd uses UNIX users and the UNIX security model.  Some 
other IMAP implementations (e.g., Cyrus from CMU) has its own user scheme 
and security model.  So, Cyrus may be more suitable for your purposes.

However, I don't quite understand the cost of adding 9263 users to 
/etc/passwd -- you have to define the users someplace.

> What occurs if they don't have home directories???

IMAP has secondary (non-INBOX) mailboxes.  So the lack of a home directory 
means this important capability of IMAP isn't usable.

For that reason, I don't quite understand the cost of creating 9263 homes 
either -- you have to have a per-user directory for the users' non-INBOX 
mailboxes anyway.

> I was thinking of replacing the qpopper binary with the ipop3d from uw-imap 
> however it doesn't appear to be a drop in replacement, is there any 
> additional configurating required???

No additional configuration is required.  In POP3, unlike IMAP, there is 
only the INBOX.

People have replaced qpopper with ipop3d successfully.  Just remember that 
ipop3d works just like imapd (internally, it's the same library) so 
requires a UNIX userid just like imapd does.

> I couldn't find any inetd or xinetd config files and the documentation didn't 
> tell me anything special was required for arguments or did I miss something?

There isn't anything special required for arguments.  Arguments are only 
used with the SecureWare package used on some Tru64 UNIX and SCO systems; 
and that is determined by SecureWare, not by imapd/ipop3d.

-- Mark --

http://panda.com/mrc

---

From: BuildSmart <[email-masked]>
Date: Mon, 6 Mar 2006 01:43:01 -0500

On Mar 6, 2006, at 24:49 , Mark Crispin wrote:

>> authentication is currently done with mysql, what I'd like to  
>> avoid is adding 9263 users to /etc/passwd and creating 9263 homes.
>
> By design, UW imapd uses UNIX users and the UNIX security model.   
> Some other IMAP implementations (e.g., Cyrus from CMU) has its own  
> user scheme and security model.  So, Cyrus may be more suitable for  
> your purposes.
>
> However, I don't quite understand the cost of adding 9263 users to / 
> etc/passwd -- you have to define the users someplace.
>
> IMAP has secondary (non-INBOX) mailboxes.  So the lack of a home  
> directory means this important capability of IMAP isn't usable.
>
> For that reason, I don't quite understand the cost of creating 9263  
> homes either -- you have to have a per-user directory for the  
> users' non-INBOX mailboxes anyway.
>
>> I was thinking of replacing the qpopper binary with the ipop3d  
>> from uw-imap however it doesn't appear to be a drop in  
>> replacement, is there any additional configurating required???
>
> No additional configuration is required.  In POP3, unlike IMAP,  
> there is only the INBOX.
>
> People have replaced qpopper with ipop3d successfully.  Just  
> remember that ipop3d works just like imapd (internally, it's the  
> same library) so requires a UNIX userid just like imapd does.

I've set up the mail app to use an existing system user (listed in / 
etc/passwd), authentication is set to use pam_unix and with qpopper I  
can log in and read mail.

I'm assuming password is plain-text or clear but tried md5-challenge  
which also didn't work.

I renamed qpopper to ipop3d, restarted everything, mail still works.

Replaced the working ipop3d (qpopper) with the one from wu-imap,  
can't read mail due to authentication problem so it's not exactly a  
drop in replacement for qpopper.

I grabbed a newer version of qpopper, built it, dropped it in, it  
worked.

If it's a build issue, I'll build it any way you specify, drop it in  
and see if it works cause currently I can't get the ipop3d to work at  
all.

> There isn't anything special required for arguments.  Arguments are  
> only used with the SecureWare package used on some Tru64 UNIX and  
> SCO systems; and that is determined by SecureWare, not by imapd/ 
> ipop3d.
>
> -- Mark --

---

From: Erik Kangas <[email-masked]>
Date: Mon, 06 Mar 2006 19:02:13 -0500

BuildSmart wrote:
>> homes either -- you have to have a per-user directory for the users'
>> non-INBOX mailboxes anyway.
>>
>> No additional configuration is required.  In POP3, unlike IMAP, there
>> is only the INBOX.
>>
>> People have replaced qpopper with ipop3d successfully.  Just remember
>> that ipop3d works just like imapd (internally, it's the same library)
>> so requires a UNIX userid just like imapd does.
>
> I've set up the mail app to use an existing system user (listed in
> /etc/passwd), authentication is set to use pam_unix and with qpopper I
> Replaced the working ipop3d (qpopper) with the one from wu-imap, can't
> read mail due to authentication problem so it's not exactly a drop in
> replacement for qpopper.
>
> I grabbed a newer version of qpopper, built it, dropped it in, it worked.
>
>> only used with the SecureWare package used on some Tru64 UNIX and SCO
>> systems; and that is determined by SecureWare, not by imapd/ipop3d.
>>
>> -- Mark --

One thing you could do is use nss-mysql so that your password and other
related files are stored in MySQL.  This will save you the performance
cost of having 10,000 or 100,000 users in flat file databases.  This
should work with UW-IMAPd, right Mark?

-Erik Kangas

---

From: BuildSmart <[email-masked]>
Date: Mon, 6 Mar 2006 19:23:11 -0500

On Mar 6, 2006, at 19:02 , Erik Kangas wrote:

>>> there
>>> remember
>> can't
>>> SCO

uw-imap requires users to be listed in the /etc/passwd database and  
they also require homes.

I'll program a valid solution in C to check in a mysql db (so  
postfixadmin or similiar can be used) instead of /etc/passwd as a  
building option and give it to mark soon as I can get the ipop3d to  
work.

There's no reason why mysql can't be used to supply any information  
required for e-mail purposes and I'd like to move into a totally  
"true virtual" user environment and mail is my only problem.

---

From: BuildSmart <[email-masked]>
Date: Mon, 6 Mar 2006 23:46:22 -0500

I turned the software into an XCode project so I could work in an  
environment I'm familiar with and built specifically for ppc, lo and  
behold the resulting binary came to life.

My second build was as i386 and testing on my Intel Mac gave me a  
working binary

Puzzled, I went over the previous build sessions looking for a  
problem and while it didn't jump out at me right away it eventually  
did rear it's ugly little head.

Surprisingly enough, the problem isn't with the code or the issued  
make command, it has to do with the build process in an environment  
that builds universal binaries, which I confirmed when I turned the  
XCode project into a universal project and the binary failed to  
authenticate in ppc but did continue to authenticate on the Intel Mac.

Why this would cause a problem is beyond me  but changing my build- 
environmemt flags to produce ppc only or i386 only code seemed to  
generate a working solution in both platforms.

To test further, I built as a cross-compiled project for Linux and  
this binary worked as well so all I can conclude is that the code  
used in the ipop3d doesn't like to be built as a dual architecture  
binary or that ther eis an anomially in Apple's universal build  
environment that affects this particular portion of the project.

I'm going to be continuing off-list with Marc in an attempt to  
discuss some strategies and concepts but I did want to thank those  
who responded and voice that this problem is worth noting.

-- Dale

---

From: Erik Kangas <[email-masked]>
Date: Thu, 09 Mar 2006 17:00:22 -0500

Hello,

Taking into consideration the following requirements/issues:

* The time it takes to open a select a folder in mbx format scales as the 
number of messages ... so folders with 1000s of messages can take some time 
to select (and get the # of messages and uid validity), especially if they 
are not currently in the disk cache.

* I cannot always hold open a connection to the same IMAP daemon from 
connection to connection because 1) sometimes the connection is not 
persistent, and 2) sometimes we need [many] additional concurrent 
connections to be opened to the same folder.

* We would like to cache information that is in a folder so that successive 
accesses can incur as little penalty as possible.

Using IMAP only, we would need to look at # of messages, uid validity, and 
all message flags on every connection to pick up any changes.  This can be 
costly.

We do have a means of accessing the email server's file system outside of 
the use of IMAP.  I wanted to check and see if the email folder's modified 
timestamp is unchanged, then that can be always taken to indicate that 
there are no changes to the folder (i.e. additions, deletions, or flag 
changes). Assume modern RedHat Linux as the OS.

If this is a true statement, then the initial access can cache the data and 
  successive accesses can skip making an IMAP connection altogether in the 
modified time is unchanged.... assuming the data cached is sufficient for 
the tasks at hand.

Any suggestions are welcome.

Thank you,

-Erik Kangas
LuxSci, Inc.

---

From: Mark Crispin <[email-masked]>
Date: Thu, 9 Mar 2006 18:35:10 -0800 (Pacific Standard Time)

On Thu, 9 Mar 2006, Erik Kangas wrote:
> We do have a means of accessing the email server's file system outside of the 
> use of IMAP.  I wanted to check and see if the email folder's modified 
> timestamp is unchanged, then that can be always taken to indicate that there 
> are no changes to the folder (i.e. additions, deletions, or flag changes).

That won't work.  The timestamp will change for reasons unrelated to 
changes to the mailbox.

The reason is detailed in Chapter 23, section 41 in Mark Crispin's 
forthcoming "Big Book of All the Reasons Why UNIX Sucks".  In this 
particular section, entitled "You can't peek at a file without modifying 
*something*", we learn that there is no way to open a file without 
changing the atime; that resetting the atime back changes the ctime; and 
there's no way to reset the ctime.

Then, we consider the mbx format, and the need to watch when flag changes 
happen (so we know in a simultaneous session to rescan flags); and we 
balance that need against the need to be able to recognize when new mail 
has arrived; and then we have to consider what happens if any non-UW tool 
touches the file.

What ultimately ends up is:

You can rely upon a comparison of the atime with the ctime telling you 
whether there is new mail; but you can not rely upon the values 
themselves.

You can rely upon the mtime staying more or less constant except when flag 
changes happen.  However, take another simple a short while later.  If 
it's the same, then use it.  If it goes backward, use the second (earlier) 
sample.  If it goes forward, repeat until you get a pair of samples that 
stay the same or go backward.

Oh yeah, and that assumes that your UNIX system handles file times 
correctly.  Linux does, but some other systems do not.  The mbx driver has 
to assume the worst, and...

Hopefully, by now, I've thoroughly discouraged you from using the file 
times on an mbx format file.  This was all a grave mistake made in the 
design of the mbx format, but the magnitude of the mistake wasn't realized 
until quite some time later.

The mix driver (I just finished writing it, now I'm debugging/testing) has 
learned from this mistake, and does not use file times in this way. 
Instead, it has modseqs associated with metadata, index, overall status, 
and individual messages.

Large mailboxes in mix format open much faster than mbx format.  My 
recommendation to you is to continue doing STATUS, SELECT, and/or EXAMINE 
as you are doing now; and wait for a new release that supports mix to be 
the solution.

mbx had a good run.  It is, after all, 11 years old.  It was a victim of 
its own success.  A traditional UNIX mailbox with much more than 1000 
messages was much too slow to use practically.  mbx expanded that by at 
least an order of magnitude; and people have used it.  Now it's time for a 
format that will handle 6-digit message count mailboxes with aplomb.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
_______________________________________________
Imap-uw mailing list
[email-masked]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw

---

From: "Aaron W. LaFramboise" <[email-masked]>
Date: Fri, 10 Mar 2006 18:24:10 -0600

Mark Crispin wrote:

> The mix driver (I just finished writing it, now I'm debugging/testing) 
> has learned from this mistake, and does not use file times in this way. 

This is excellent news!  The sooner "mix" is availible, the better for 
me, and perhaps many others too.  I'd love to test this mailbox format, 
even if it were in a dangerous alpha form.