Subject: Parsing

From: "Oliver Schulze L." <[email-masked]>
Date: Fri, 08 Nov 2002 14:32:45 -0300

Hi,
I'm looking for a function like mail_criteria() that receives an
IMAP4 search string and returns a (SEARCHPGM *) structure:

SEARCHPGM *mail_criteria(char *criteria)

Is there a mail_criteria() equivalent function for imap4? 

Many thanks
Oliver
-- 
Oliver Schulze L.
[email-masked]

-- 
------------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/c-client-list.html
------------------------------------------------------------------

---

From: Mark Crispin <[email-masked]>
Date: Fri, 8 Nov 2002 10:04:58 -0800 (PST)

> Is there a mail_criteria() equivalent function for imap4?

There isn't.  You should build the SEARCHPGM yourself.

It shouldn't be difficult.  In fact, it should be much easier to build a
SEARCHPGM than an IMAP4 string.

---

From: "Oliver Schulze L." <[email-masked]>
Date: Fri, 08 Nov 2002 15:12:01 -0300

The problem I trying to solve is in the php.net implementation of imap 
searching.
They use mail_criteria() because they just pass to c-client what the 
user wants to
search. They don't parse the user input search string.
The problem can be reproduced in every program that accept an input from
the user.

But if it is the only way to do it, I should keep learning c-client.

I was trying some examples of building a SEARCHPGM and could't find
the way to have more than one criteria. Like when using OR/AND, etc
How do I do this?

Thanks
Oliver

-- 
Oliver Schulze L.
[email-masked]

---

From: Mark Crispin <[email-masked]>
Date: Fri, 8 Nov 2002 10:31:50 -0800 (PST)

> The problem I trying to solve is in the php.net implementation of imap

I understand.  This has been a difficult problem.

Basically, my position has been that IMAP search strings, while "logical", are
a terrible interface from a human's point of view.  Consider
	FLAGGED OR FROM SMITH CC JONES
Most humans would interpret this as "messages which are flagged, or are from
SMITH and cc JONES".  But that is not what IMAP does; in IMAP it means
"messages which are flagged, *and* are from SMITH or cc JONES."

I feel that the correct thing would be for PHP and/or the application to
provide an interface suitable for humans, as opposed to passing IMAP strings.

An additional complication, and what was fatal to mail_criteria(), is in the
handling of strings which require representation as a literal.  Then, too,
mail_criteria() is very thread-unsafe...

One problem that people traditionally had was in generating the shortdates
expected by the date search criteria.  imap-2002 now has a mail_shortdate()
function which should make this a lot easier.

> the way to have more than one criteria. Like when using OR/AND, etc

AND is easy.  Just set multiple criteria in the SEARCHPGM.  For example you
can do FLAGGED UNSEEN with:
	SEARCHPGM *pgm = mail_newsearchpgm();
	pgm->flagged = pgm->unseen = T;

OR is more complicated.  You create a SEARCHPGM for each part of the OR'd
expression, and set them in the SEARCHOR in a higher SEARCHPGM.  For example,
to do OR FLAGGED UNSEEN:
	SEARCHPGM *pgm = mail_newsearchpgm();
	pgm->or = mail_newsearchor();
	pgm->or->first = mail_newsearchpgm();
	pgm->or->first->flagged = T;
	pgm->or->second = mail_newsearchpgm();
	pgm->or->second->unseen = T;
This is actually expressing ALL OR FLAGGED UNSEEN, which is equivalent.

NOT works in a corresponding way; it is a linked list of SEARCHPGMs which are
"AND NOT" with the top level.  For example, to do NOT FLAGGED NOT UNSEEN:
	SEARCHPGM *pgm = mail_newsearchpgm();
	pgm->not = mail_newsearchpgmlist();
	pgm->not->pgm = mail_newsearchpgm();
	pgm->not->pgm->flagged = T;
	pgm->not->next = mail_newsearchpgmlist();
	pgm->not->next->pgm = mail_newsearchpgm();
	pgm->not->next->pgm->unseen = T;

Of course, this was a very silly example, since an equivalent search can be
done with UNFLAGGED SEEN:
	SEARCHPGM *pgm = mail_newsearchpgm();
	pgm->unflagged = pgm->seen = T;
NOT generally only happens with string searches, since most other search terms
already have logical opposites (thus making NOT extraneous).

Yes, I know that this all looks pretty imposing, but you have to think about
what the IMAP strings would look like.  No normal mortal will be able to
comprehend RECENT OR FROM SMITH CC JONES correctly, so you will need your own
interface.  It actually is easier, and less bug-prone, to generate a SEARCHPGM
than it is to try to figure out how IMAP expresses things.

But I understand that there can be a comprehension gap.

---

From: "Oliver Schulze L." <[email-masked]>
Date: Wed, 20 Nov 2002 17:57:30 -0300

Hi Mark,
thanks for the examples and explanations. Sorry about the delay ...

I understand your point, but the parsing I'm talking about is the
parsing of a IMAP4 search string. Not a search string entered
by the user.

I think if there is a function that parses an IMAP4 search string
into an *SEARCHPGM, it will help programers that use the
c-client library.

In the PHP example, this is what should happend:
- the PHP programer parse a human redeable search string
   into an IMAP4 search string
- the PHP developer use this IMAP4 search string and convert it
   to a *SEARCHPGM using a parsing function
- the PHP developer then pass this *SEARCHPGM to the function
   imap_search()
- What happend is that work of the PHP developer is more easier and
   the PHP programer does the human-to-imap4 search string convertion.

This proposed parsing function will make it easier for c-client users,
like PHP, to implement IMAP4 search strings. Right now, PHP does
not support IMAP4 search strings because of the lack of this function.

Let me know what you think about this new IMAP4 search string
to SEARCHPGM parsing function.

Regards
Oliver

-- 
Oliver Schulze L.
<[email-masked]>

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Nov 2002 19:33:21 -0500 (Eastern Standard Time)

> parsing of a IMAP4 search string. Not a search string entered

Yes, but when you use mail_criteria(), what happens is that the string is
parsed into a SEARCHPGM which is subsequently converted into an IMAP
SEARCH command.  It's a lot of extra conversions, but this was done
because of the need to handle literals in IMAP.

> into an *SEARCHPGM, it will help programers that use the

Why can't programmers create SEARCHPGMs directly?  They should be easier
to do than IMAP search strings, especially since IMAP search strings are
very difficult to handle when you do string searches.

mail_criteria() is a compatibility routine for old programs which hadn't
been converted yet.  I'm more likely to remove mail_criteria entirely than
try to update it for IMAP4.  There's a reason why SEARCHPGMs were created
instead of using char* strings.

As a result of the additional complexity of IMAP4, IMAP search strings
don't work well as a char* any more.  They are complex, multiline things
that are broken up depending upon underlying IMAP requirements that have
nothing to do with strings.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

---

From: "Oliver Schulze L." <[email-masked]>
Date: Thu, 21 Nov 2002 11:38:02 -0300

My point is that, if every c-client user must parse a IMAP4 serch string 
into
a *SEARCHPGM, why do not let c-client does the job?

Imagine the this code:
SEARCHPGM *my_search;
my_search = imap4_to_searchpgm("FLAGGED OR FROM SMITH CC JONES");
mail_search_full(imap_stream, NIL, my_search, flags);

Is much more easier for the c-client users than this code:
SEARCHPGM *my_search;
/*
{
// parse the imap4 search string into my_search
}
*/
mail_search_full(imap_stream, NIL, my_search, flags);

Please note that the parsing from a IMAP4 search string to a SEARCHPGM*
will always be done in a language like PHP.

PHP use c-client, but a PHP programer does nothing about c-client.
So, the PHP programer will pass a imap4 search string that was entered
in a GUI by the user. The PHP programer will convert this GUI to an imap4
search string, then PHP will parse this imap4 search string to a SEARCHPGM*
and that is where the extra work is done.

PHP, like other languages, offer to the programer an abstraction layer
on top of c-client. So, the programes only need to know PHP and IMAP4,
and not PHP, IMAP4 and c-client.

(answer continue bellow )

>because of the need to handle literals in IMAP.
I know the mail_criteria() is old and ineficient, but I think we need 
somethink like
mail_criteria(). Somethink new but with the same functionallity that 
mail_criteria()
has.

>Why can't programmers create SEARCHPGMs directly?  They should be 
>
Because programers don't program using the c-client library. For example in
PHP, PHP provides an abstraction layer to the PHP programer. So, the
PHP programer only need to know PHP and IMAP4 protocol, not c-client.

>easier
I'm not asking to update mail_criteria(), what I'm saying is that 
c-client needs a
function like mail_criteria(). A totally new thread safe function.

Regards
Oliver

-- 
Oliver Schulze L.
<[email-masked]>

---

From: Mark Crispin <[email-masked]>
Date: Thu, 21 Nov 2002 11:00:58 -0500 (Eastern Standard Time)

On Thu, 21 Nov 2002, Oliver Schulze L. wrote:
> into a *SEARCHPGM, why do not let c-client does the job?

That's the whole point.  Why should a c-client user deal with IMAP4 search
strings at all?  IMAP4 search strings are for protocols, not for human
users.

> mail_search_full(imap_stream, NIL, my_search, flags);

This is a good example.  You are assuming that SMITH and JONES are atomic
values, but they don't have to be.  If non-atomic, then you have to know
how to apply IMAP quoting and literals.  And with literals, you have
nesting (due to the OR) and a query/response.

So your simple example becomes a recursive call plus the send literal and
response.

Take a look at how imapd parses search commands if you don't believe me.
It may look a lot like mail_criteria(), but note those recursive calls and
callbacks.  You can't do that with a char*.

That's why SEARCHPGM was created.  And even if the tokens are atomic, I
challenge you to find any human who will guess the correct interpretation
of
	FLAGGED OR FROM SMITH CC JONES
the first time.

You have to have a different representation of searching for your human
users.  Why do we need *yet another* non-human representation of the same
thing?

> in a GUI by the user.

Why?

Once again, there is no such thing as an "IMAP4 search string".  At best,
you can have a search string that's a subset of IMAP4.  IMAP4 searches can
not be represented by a char*.

-- Mark --

---

From: "Oliver Schulze L." <[email-masked]>
Date: Thu, 21 Nov 2002 14:43:07 -0300

Hi Mark,
thanks for answering,
I now understand that I'm not expresing myself the right way.
We are using diferent terms. Let me defne the terms I'm using.

- "IMAP4 search string": a char string that a IMAP4 server can understand
   when you send it via telnet (i.e. telnet localhost imap). This is a 
search string
   using char as defined in the RFC 2060 that the imap server can 
understand.
   Example of "IMAP4 search string": "FLAGGED OR FROM SMITH CC JONES"
- "Human search string": a search criteria entered by the user using some
   sort of GUI. This string is easily interpreted by the user and can 
not be interpreted
   by the imap server.

I don't know the official name for these strings I'm talking about, thats
why I called "IMAP4 search string". Sorry about the confusion.

Now, let me make an example of the function I'm talking about:
1. The user search this human search string:
     "messages which are flagged, or are from SMITH and cc JONES"
2. The PHP programer, in the PHP program, somehow translates this human
    search string into an IMAP4 search string. The result string is:
     "FLAGGED OR FROM SMITH CC JONES"
3. The PHP programer use a PHP function to send this IMAP4 search
     string to the server.
     <?php imap_search("FLAGGED OR FROM SMITH CC JONES");?>
4. The folowing C code is executed by PHP(inside the PHP search
    function):
SEARCHPGM *my_search;
my_search = imap4_to_searchpgm("FLAGGED OR FROM SMITH CC JONES");
mail_search_full(imap_stream, NIL, my_search, flags);

5. the IMAP server undestand the IMAP4 search string and returns the 
results to the
    C code in the PHP search function.

How I know this IMAP4 search string works?
Well, I telneted my imap server this way:

$ telnet localhost imap
1 login user passwd
2 select INBOX
3 SEARCH FLAGGED OR FROM SMITH CC JONES
4 logout

Why a new function?
I started this thread because I can use telnet to send this char string 
to the imap server,
but I can not do that using c-client. In c-client I have to parse a char 
string into a
structure called SEARCHPGM and then use this structure to call a 
c-client search
function.
Since every c-client user(programer) must do a custom parsing of a IMAP4 
search
string into a SEARCHPGM, I sugest to have a function for doing this parsing.
I'm not asking to create function because I don't want to create SEARCHPGM
structures or becasue I'm lazy, but because it will help languages like 
PHP and
will make programers develop faster aplications using the c-client library.

About your points.
- I understand that most c-client users(programers) receives a human 
search string
from the user via some sort of GUI and then they build a SEARCHPGM, but 
this
is not always the case. In PHP, for example, the PHP seach function 
receives an
IMAP4 search string, not an human search string.
- mail_creteria(), I know this is a deprecated function. I don't pretend 
to upgrade it
or even use it. I propose the creation of a new funtion for the case 
when you have an
IMAP4 search string and: want to sent it to the imap server in the 
simplest way without
having to do any custom parsing from a char IMAP4 search string into a 
structure of type
SEARCHPGM.

I hope you can understand my point. I try to write as good as I can, but 
sometimes
my english does not work too well for me.

Thanks
Oliver

-- 
Oliver Schulze L.
<[email-masked]>

---

From: Mark Crispin <[email-masked]>
Date: Thu, 21 Nov 2002 13:09:16 -0500 (Eastern Standard Time)

On Thu, 21 Nov 2002, Oliver Schulze L. wrote:
>    when you send it via telnet (i.e. telnet localhost imap).

As I explained earlier, IMAP search commands are much more complex than a
single string.  So, what you are calling an "IMAP search string" is a
char* subset of what can be sent to an IMAP server, and thus a *third*
representation of searching: IMAP search command, SEARCHPGM, and this new
subset string which expresses some but not all IMAP4 searches.

>      "FLAGGED OR FROM SMITH CC JONES"

You just proved one of my points.  That is not the correct translation.

Human "messages which are flagged, or are from SMITH and cc JONES" is
	OR FLAGGED (FROM SMITH CC JONES)
in IMAP.

Conversely, IMAP
	FLAGGED OR FROM SMITH CC JONES
is human "messages which are flagged, and are:
 1) from smith
or
 2) cc jones
or
 3) from smith and cc jones".

Now, if you want human "messages which are flagged, or are either from
SMITH or cc JONES", then you want
	OR FLAGGED OR FROM SMITH CC JONES

Or perhaps you want "messages which are flagged, and are from SMITH or cc
JONES but not from SMITH and cc JONES" then you want
	FLAGGED OR FROM SMITH CC JONES NOT FROM SMITH CC JONES

I think that you're starting to see why IMAP search criteria, while very
logical, are absolutely not the sort of thing that you want to send to
humans.

>      <?php imap_search("FLAGGED OR FROM SMITH CC JONES");?>

Why?  Why not send a SEARCHPGM, or something that more reasonably
translates into a SEARCHPGM?

> 5. the IMAP server undestand the IMAP4 search string

No, the IMAP server understands an IMAP4 SEARCH command, which is sent
from a SEARCHPGM, which is totally different from what you're calling an
"IMAP4 search string" even if there are similarities.

> search string into a SEARCHPGM

I am suggesting, once again, that if a C programmer is dealing with "IMAP4
search strings" he's going about it the wrong way.  Such strings are never
sent to the server.  What is sent to the server is generated by the
c-client library, and can easily be a very interactive process resulting
in multiple transmit/response operations.

Why can't PHP deal with SEARCHPGMs?  That *is* the IMAP4 search criteria.

If I could have used a char* string I wouldn't have created SEARCHPGM.

-- Mark --

---

From: "Oliver Schulze L." <[email-masked]>
Date: Thu, 21 Nov 2002 15:40:11 -0300

Hi Mark,

>translates into a SEARCHPGM?
Ok, this is all I want. I want a function that translates the char string
"FLAGGED OR FROM SMITH CC JONES" into a SEARCHPGM.
Can I do this in c-client?

Thanks
Oliver
P.D.: I keep using the string "FLAGGED OR FROM SMITH CC JONES" because
you send me that string in one of your replys. But it can be any oter 
string.

I will avoid using the term "IMAP4 search string". Now, I will call it a
"SEARCH client command" as described in RFC2060 Point 6.4.4.

-- 
Oliver Schulze L.
<[email-masked]>

---

From: "Oliver Schulze L." <[email-masked]>
Date: Fri, 29 Nov 2002 19:33:43 -0300

Hi Mark,
I'm sending this email again because in the past I already
asked this question and received no answer.

Let me know you position about this function, so people
can start planing working on this.

Thanks
Oliver
-- 
Oliver Schulze L.
<[email-masked]>

---

From: "Shawn Walker" <[email-masked]>
Date: Fri, 6 Dec 2002 14:39:51 -0600

How to create a folder with single quote ' (0x27)?   I have encoded a folder
name according to RFC 2060.  The folder example is "Boîte d'Envoi" which get
encoded to "Bo&AO4-te d'Envoi" but the Cyrus IMAP server reject the a
"Invalid name".

Shawn

---

From: Mark Crispin <[email-masked]>
Date: Fri, 6 Dec 2002 12:51:14 -0800 (PST)

On Fri, 6 Dec 2002, Shawn Walker wrote:
> name according to RFC 2060.  The folder example is "Bo$Bn(Bte d'Envoi" which get

That is a valid mailbox name in IMAP, and UW imapd will create a mailbox
with that name.

If Cyrus does not permit it, that is probably a restriction in Cyrus.  A
server is permitted to reject the creation of any mailbox name for any
reason that it deems fit, including not liking single quotes.

Since you asked the c-client mailing list, I assume that you are writing
an IMAP client application using the c-client library and communicating to
a Cyrus server.  To make absolutely certain that my diagnosis is correct,
you should try your program with a UW server.  If you also have a problem
with the UW server and that name, perhaps there is a bug in your
mail_create() call.

-- Mark --