Subject: Folder Format Problem PC-Pine/UNIX Pine

From: Diane Griffitts <[email-masked]>
Date: Tue, 6 Jul 1999 15:07:24 -0700 (PDT)

I'm looking for more information on how PC-Pine 4.10 handles folder files.
I've consulted the following pages on the Pine web site.  

http://www.washington.edu/pine/faq/sysadmins.html
http://www.washington.edu/pine/tech-notes.4.10/low-level.html

I'm a network administrator and die-hard UNIX Pine user (currently using
4.05 on an HP-UX 10.2X server cluster).  I keep ALL my email for future
reference by archiving folder files on a monthly basis via FTP to my local
hard drive.  In the past, when I needed to browse an archived folder, I
would temporarilty FTP it back to my mail directory on the UNIX host.
Recently I installed PC-Pine 4.10 hoping for easier access to the archived
materials and was pleased with the results.  Note that I am not using
PC-Pine to send and receive mail; I'm only using it to access folder files
which I FTP'd from my UNIX space to my local hard drive.  This has mostly
eliminated the need to FTP a folder back to the UNIX side.

Everything was fine until I wanted to forward an old message.  I FTP'd the
folder file back to UNIX, but when I tried to view it, UNIX Pine says it
contains no messages.  I can cat the file and see the messages, so I'm
confident the file is all there.  Also tried the same process on a second
folder with the same results, so I don't think the file became corrupt.
Looking further, I noticed the date and time stamps on these folder files
were updated according to when I had accessed them earlier using PC-Pine.

Based on FAQ 7.2, I'm thinking PC-Pine might have converted these files to
the mbx format and my UNIX Pine implementation can't read them that way.
So does anybody know:  1) is there any way to keep PC-Pine from doing such
a conversion and 2) if my theory is true, is there any way to convert
these folders back to the UNIX Berkeley mail format so UNIX Pine can read
them?  Any light anyone can shed on this would be much appreciated.

_ ^ ______________________________________________
 / \   Diane Griffitts - - [email-masked]
 / \   System Network Analyst
  |    University of Idaho Boise Center
__________________________________________________

-- 
-----------------------------------------------------------------
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/pine/pine-info/
-----------------------------------------------------------------

---

From: "Daniel Sands" <[email-masked]>
Date: Tue, 6 Jul 1999 16:26:22 -0600

Another possibility:  Are there ^M's at the end of each line?  If so, PC-PINE
converted your mail folder to DOS text.

> http://www.washington.edu/pine/tech-notes.4.10/low-level.html

Why did you FTP back?  Why not just use the original?

---

From: Jessica Rasku <[email-masked]>
Date: Tue, 6 Jul 1999 15:29:40 -0700 (PDT)

> them?  Any light anyone can shed on this would be much appreciated.

	Based on a little experiment, and my personal experience with what
seem like simmilar UNIX<>DOS incompatabilities, my feeling is that chances
are it is as simple as the ``end of line'' marker being diffrent.  DOS
uses a ^M^J end of line marker, while UNIX uses ^J.  If I convert a file
to DOS format (insert the ^Ms in the file), the folder then contains ``0''
messages.  I suspect that you simply have to watch this conversion,
possibly you are transfering them to the PC as ASCII and back to UNIX as
binary?  Or, maybe PC-Pine converts UNIX files to PC files when it reads
them.  The ``file date'' has no effect on Pine as far as I know...

								Jessica

-- 
Jessica Rasku, Box 270, Rossland, B.C., V0G 1Y0, (250) 362-5701, 
LinuxBox: (250) 362-9668.

List manager: [email-masked] 
     send command help ---- To get help with majordomo
               or lists ---- To get a list of all lists on server.

WWW: <http://www.geocities.com/RainForest/Andes/8749>

---

From: Diane Griffitts <[email-masked]>
Date: Tue, 6 Jul 1999 16:19:16 -0700 (PDT)

I can't see any ^M's when I cat the file...

I wanted to FTP the folder back to the UNIX side in order to open it and
forward one of the messages.  The original folder no longer existed on the
UNIX host.  In this case, I worked around the problem by using the Export
command in PC-Pine to store a copy of the one message, then FTP'd that
file to my UNIX home directory where I could attach it to the message.  I
suppose I can continue to do this as needed, but I still would like to
know if there's a way to keep PC-Pine from adulterating the folder files.

---

From: Diane Griffitts <[email-masked]>
Date: Tue, 6 Jul 1999 16:26:35 -0700 (PDT)

I can't see any extra ^M's but maybe I'm not viewing the file in such a
way that they are visible.  If this is the cause of the problem, I'm not
sure how to keep it from happening.  

When I FTP, I normally use the binary setting for everything; in this case
the FTP action was the same as all the other times I have done this.  I
did go back and try the FTP set to ASCII, but it didn't seem to change
anything (and didn't solve the problem).

P.S.  I was only noting the file date as evidence that PC-Pine had updated
the file in some manner.  In the meantime I have also noted that the file
size has increased somewhat, so that fits with the conversion to DOS
theory.  But is there a way to convert back to UNIX text?

---

From: "Daniel Sands" <[email-masked]>
Date: Tue, 6 Jul 1999 17:28:44 -0600

CAT won't show those pesky ^M's.  Try using VI on it.

In any case, make sure you use ASCII to FTP the file to your UNIX station.

---

From: "Ed Greshko" <[email-masked]>
Date: Wed, 7 Jul 1999 07:28:49 +0800

Diane,

> them?  Any light anyone can shed on this would be much appreciated.

You are 100% correct.  PC-Pine does use mbx format.  It is not a simple DOS/Unix
file format difference.

There is no way to keep PC-Pine from using mbx format.  It is hardwired and was
done so for good reason.  The reason being "performance".

The easiest way to convert back to Berkeley format is to use PC-Pine, point it
to a folder collection on a Unix server running imapd, create a folder on the
Unix side and then copy (via imap) the messages from the PC-Pine side to the
Unix side.

Regards,
Ed

---

From: Jessica Rasku <[email-masked]>
Date: Tue, 6 Jul 1999 18:36:27 -0700 (PDT)

> theory.  But is there a way to convert back to UNIX text?

	On the UNIX side something like:

cat dosfile | tr -d "\r" > unixfile

	should strip the ^Ms from the file...  That is probably the
quickest, lowest resource method to use.  You could probably also do
something similar in your favorite editor.

							Jessica

---

From: Jessica Rasku <[email-masked]>
Date: Tue, 6 Jul 1999 18:41:55 -0700 (PDT)

On Wed, 7 Jul 1999, Ed Greshko wrote:

> You are 100% correct.  PC-Pine does use mbx format.  It is not a
> simple DOS/Unix file format difference.

	This seems unlikely as the folder is still vissably the same.  

> There is no way to keep PC-Pine from using mbx format.  It is
> hardwired and was done so for good reason.  The reason being
> "performance".

	I find this hard to believe.  Just as I found the ``binary only''
problem with PC-Pine (it has been confirmed that there is no reason that
there is no source avalable).  Performance is not a good enough reason to
use a format that could cause problems (also, I find it hard to believe
that the UNIX version does not understand mbx format).  

> The easiest way to convert back to Berkeley format is to use PC-Pine,
> point it to a folder collection on a Unix server running imapd, create
> a folder on the Unix side and then copy (via imap) the messages from
> the PC-Pine side to the Unix side.

	Hm, this may not be possible.  I don't know though.  I don't know
enough about the setup on this person's system to be sure, but some of
what she is doing SOUNDS like maybe she doesn't have full ``internet''
access.  And, it is indeed MORE likely there is no IMAP server, than the
mbx format isn't supported I'd think...

							Jessica

---

From: "Ed Greshko" <[email-masked]>
Date: Wed, 7 Jul 1999 10:27:59 +0800

Dear Jessica,

> 	This seems unlikely as the folder is still vissably the same.

Sorry to burst your bubble but....

Looking at the top of the PC-Pine's mbx format folder we see (using vi in Unix
after doing a binary xfer of the file):

*mbx*^M
37828c4c00000000^M
^M
^M
^M

Then, down much further we see....

 3-Jun-1999 14:37:43 +0000,1437;000000000000-00000000^M
Return-Path: <[email-masked]>^M

The line with Jun-1999 is part of the mbx format.

Actually, if you do a "more" on the folder you'd see something like:

@@^PÃà^EÑxà^E)ð @@^G 3-Jun-1999 14:37:43 +0000,1437;000000000000-0000000

Furthermore, if you were to use "od" (octal dump) on the file you'd see null
bytes and quite a few other "characters" which are *not* presented in an editor
such as vi but *DO* have significance.

> that the UNIX version does not understand mbx format).

Believe it, or not, it is your choice.

Also sorry to burst your bubble, but it matters not what mailbox format is in
use.  The translator is inherent in the imap protocol.  We, Control Data
Systems, have an imap product which uses an internal mailbox format unlike any
supported by pine.  Yet, I can freely move messages between PC and Unix versions
of Pine as well as the beast known as MS-Outlook!

Would you care to FTP the .pst files used by Outlook and see what can be done
with those. :-) :-)  Yet, with the procedure I've told you about...it is no
problem for me to take a message from an Outlook folder to PC-Pine or Unix Pine
and have it translated into the mailbox format needed.

> mbx format isn't supported I'd think...

If you don't have the setup you need to perform the function, it would not be
possible.  That is obvious.

However, the procedure that I've described worked quite well since I do always
test my solutions  before presenting them.  It makes more sense then saying "try
this, maybe it will work".

The mbx format probably has a definition in a document.  I've not searched for
it.  However, it should not be difficult to write a perl script to convert
between mbx and Berkeley.  Or, you may even be able to hack the source of the
c-client library.

Regards,
Ed

---

From: Terry Gray <[email-masked]>
Date: Tue, 6 Jul 1999 19:28:11 -0700 (Pacific Daylight Time)

On Tue, 6 Jul 1999, Jessica Rasku wrote:

> there is no source avalable).  

Really?  I'd love to learn more about this so-called "confirmation".
(Especially since some of the *actual* reasons have been posted more
than once...)

Reminds me of a button seen in Haight-Ashbury in the 60's: 
  "Acid Indigestion?  Check your source."

> Performance is not a good enough reason to use a format that could
> cause problems (also, I find it hard to believe that the UNIX version
> does not understand mbx format).

PC-Pine can read Unix mailbox format on a read-only basis, but if it
*creates* the folder/messages, it *will* use an mbx-like format... yes,
for efficiency.  Unix Pine should be able to understand all formats, but
at least some of the mailbox drivers have proven to be picky about newline
conventions.  

Please note: PC-Pine was designed to operate in an IMAP environment.  
That it works in any other context is perhaps fortuitous, but definitely
outside of its intended design center.

-teg

---

From: "Ed Greshko" <[email-masked]>
Date: Wed, 7 Jul 1999 10:37:18 +0800

> something similar in your favorite editor.

This strips out ^M from the file.  It doesn't convert between *mailbox* formats!
If you really look at the file with a HEX editor you will see "null bytes" and
other non-printable bytes which have significance in the mbx format.  Using
"visual" editors like vi will not show you bits and bytes which hold information
needed by applications.

BTW, some systems come with the command "dos2unix" and "unix2dos" which perform
the "tr" functions plus other useful functions.

Ed

---

From: "Daniel Sands" <[email-masked]>
Date: Wed, 7 Jul 1999 08:20:04 -0600

> at least some of the mailbox drivers have proven to be picky about newline

According to FAQ 7.2, PC-PINE 4.x "supports mbx, mtx, tenex, and unix formats,
all read-write."  So what do you mean, "read-only"?

---

From: Terry Gray <[email-masked]>
Date: Wed, 7 Jul 1999 09:04:34 -0700 (Pacific Daylight Time)

> all read-write."  So what do you mean, "read-only"?

My mistake.  What I said was true prior to PC-Pine 4.xx, but I had
forgotten that one the (several) motivations we had for abandoning DOS
compatibility last year was to allow read-write unix mailbox access.

-teg

---

From: Jessica Rasku <[email-masked]>
Date: Wed, 7 Jul 1999 10:28:58 -0700 (PDT)

> (Especially since some of the *actual* reasons have been posted more

	The reasons that I have heard are that there is NO code that can
not be released.  Now, there MAY be code that you WISH not to release, but
there is no code that can NOT be released.  Maybe since this HAS come up
since I was on, and I DID ask about it in the past, explanation of why
there is no source avalable.

>   "Acid Indigestion?  Check your source."

	My source, was from someone on the development team.  Someone who
has WORKED with that very source code.  Maybe my ``interpretation'' is
faulty, but my sounce itself isn't.  Maybe before the development team
starts saying things like ``check your source'' they should be very clear
as to who knows what, and why.  

> at least some of the mailbox drivers have proven to be picky about newline

	On my version of Pine compiled on my Linux box, it DOES complain
about new line conventions.  In reality it shouldn't, but you would have
to handle the file in a low level format in order to ensure that.  You
can't rely on the libraries installed on the system to be able to handle
the three (or more) formats of end of line properly, and invisably.  

> outside of its intended design center.

	What I don't understand, is why PC-Pine is SO diffrent from
``normal'' Pine.  Yes, I realise that PC-Pine is designed to run in a
diffrent envioronment, but why are there things like ``window possition''
stored in the PC-Pine pinerc, (yes it has been explained what FUNCTION it
provides, but it seems an odd requirement.  Maybe it's a problem with the
windowing envioronment?), why does PC-Pine ``assume'' that there will only
be one occurance running at a time?  Why does PC-Pine have mouse control
(maybe in X it works, I haven't tried it) while ``normal'' Pine does not?
Maybe I'm ``picky'' but I think that products which seem to be merely
diffrent ports of the same product should for the most part behave in the
same way.  The diffrences should be in how the operating system, or
opperating envioronment behaves diffrently, not in functionality.

							Jessica

---

From: Jessica Rasku <[email-masked]>
Date: Wed, 7 Jul 1999 10:31:47 -0700 (PDT)

> all read-write."  So what do you mean, "read-only"?

	Oh, you really should check your source on that.  :)  I do love it
when someone who rags on someone about checking sources is shown that if
they had been more careful themself they would not have made the false
claim...

							Jessica

---

From: Jessica Rasku <[email-masked]>
Date: Wed, 7 Jul 1999 10:54:23 -0700 (PDT)

> after doing a binary xfer of the file):

	Hm, this is odd, my UNIX Berkly Style folders don't look like
that.  This was PRECICELY my point, and you have made it beutifully.  mbx
format folders DO look diffrent than non mbx format folders.  SURE I may
not be able to see WHAT everything is if I chose an editor poorly (as far
as I know, vi displays all characters, but some characters are
indistinguisable visualy if you don't have a character set that
distinguises them.  But, they are all ``displayed''.  In fact, you
probably can switch the mode of vi to display undisplayable charcters in a
way that they are distinct.  

> such as vi but *DO* have significance.

	They may not be DISTINGUISHABLE on all machines, but I know that I
used to be able (in DOS) to insert spaces by doing alt-2-5-5 but because
of the character set I now have, that displayes quite distincly now...  

> of Pine as well as the beast known as MS-Outlook!

	HUH?  Burst my bubble?  Excuse me?  Where do you read that I don't
think that the format can be changed through IMAP?  I said later on that
maybe the configuration is such to make it difficult, or possibly
``impossible'' for a particular person to do that.  But I don't believe
that you can point to a SINGLE place that you can rightly interpret that I
said that the conversion wouldn't work through IMAP.

> and have it translated into the mailbox format needed.

	Yes, the procedure you told us about should work.  I NEVER
questioned that.  I have this thing personaly, in not forcing a person to
change how they do things simply because there is something that works.  

> possible.  That is obvious.

	Apparently not enough to you that you would be willing to say
``this may not work for you, but it does for me'' or the equivelent.  Only
obvious to you when it is pointed out to you.

> this, maybe it will work".

	Which is all fine and dandy if you have all the tools handy to do
the testing.  But for some people who can't run systems as fancy as you
do, or for whatever reason find it impossible to test the ``specific
circumstances'' (personaly, I think you are misleading people to say
you've tested it, as you can't say that it WILL work, you at best can say
that it ``does work for me'') that a person has presented.  

> between mbx and Berkeley.  Or, you may even be able to hack the source of the

	I BELIEVE that on most systems there should be a conversion
utility that already exists.  I couldn't locate it on my system as I don't
know what it is called, but it was sugested that someone convert to mbx
format and that there was a simple command to do so.

							Jessica

---

From: Jessica Rasku <[email-masked]>
Date: Wed, 7 Jul 1999 10:57:57 -0700 (PDT)

> "visual" editors like vi will not show you bits and bytes which hold information

	This is what was asked.  Or this is what I interpreted as being
asked.  It has been confirmed that the mbx format should not be a problem
with UNIX Pine.  It may be on SPECIFIC installs, but I suspect that to
disable that would take a reasonable amount of hacking, which probably
isn't that worth while...

> the "tr" functions plus other useful functions.

	Yes, but the solution I gave should be avalable for just about
anyone who has a shell account.  It may not do the whole job, but it also
does not dammage the orriginal file (unless you try to redirect back to
the orriginal file, but I have learned through experience that that is a
sure way to corrupt your files).

							Jessica

---

From: "Ed Greshko" <[email-masked]>
Date: Thu, 8 Jul 1999 06:53:03 +0800

Diane,

You're probably pretty tired of all this by now....  I know I am.  :-) :-)

> them?  Any light anyone can shed on this would be much appreciated.

In a previous email I directly answered your 2 questions.  I did fail, however,
to broaden my investigation.  Other than not getting paid for this there
probably is no excuse.  :-) :-)

Anyway, I did do a binary FTP (you shouldn't do ASCII FTP since it will damage
the file) of a PC-Pine folder to both a Sun running pine 4.10 and an HP running
pine 4.10 (GCC build of pine).  In both cases pine was able to
access/read/manipulate the folder without difficulty.

You've not said much more about the problem.  So, maybe you've fixed it (found a
solution) in spite of us.  :-)

Ed

---

From: Diane Griffitts <[email-masked]>
Date: Mon, 12 Jul 1999 15:13:48 -0700 (PDT)

Thanks to Ed and everyone who contributed to this discussion (so far).  I
appreciate the helpful suggestions and information, and I figure the
heated comments signify people who care a lot about the technology issues
they are working with.

As for solving the problem, I'm waiting for a reply from a UNIX
administrator in our IT group to confirm my suspicion that our UNIX Pine
implementation has been configured to not use the mbx folder format since
we are in an NFS environment.  In the meantime, I plan to try your
suggestion of copying messages between PC-Pine and UNIX Pine folders via
IMAP (we do have it, I just haven't been using it up to now).  I'm curious
how exactly to accomplish the transfer between different folder
collections, but I'll see how it goes.

Given my fairly limited needs for PC-Pine, I'm starting to wonder if I
would be/would have been better off with a 3.9X version where the UNIX
folder capabilities were read-only.  However, since I'm reading them on a
hard drive local to the PC, maybe that wouldn't help.  Anyway, it's been
very educational so far.

---

From: "H. Bork" <[email-masked]>
Date: Thu, 15 Jul 1999 12:07:27 -0200 (GMT+2)

Hello out there,
here's a tiny question:

is there any kind of password encryption 
for POP and/or IMAP clients 
when contacting their mailserver ?

TIA kind regards, hal ;-)

---

From: Scott Leibrand <[email-masked]>
Date: Thu, 15 Jul 1999 07:35:48 -0700 (PDT)

Yes and no.  If you're connecting from a Unix machine running Pine to a
Unix mail server that supports SSH, you can tell Pine to use SSH to
connect by modifying the rsh-path and rsh-command entries in the .pinerc.
Of course, you'll need to change the paths to match the location of the
files on your system.

# Sets the name of the command used to open a UNIX remote shell 
# connection.  The default is tyically /usr/ucb/rsh.
rsh-path=/usr/local/bin/ssh

# Sets the format of the command used to open a UNIX remote
# shell connection.  The default is "%s %s -l %s exec /etc/r%sd"
# NOTE: the 4 (four) "%s" entries MUST exist in the provided command
# where the first is for the command's path, the second is for the
# host to connnect to, the third is for the user to connect as, and the
# fourth is for the connection method (typically "imap")
rsh-command=%s %s -q -l %s exec /etc/r%sd

If you're connecting with PC-Pine, there is currently no publically
available method of connecting securely.  Outlook Express supports SSL
IMAP connections, however, and I'm sure an encrypted version of PC-Pine
will be released eventually.

Of course, you could also use PGP or some such program to encrypt the
actual e-mails before you send them...

-- 
Scott Leibrand
[email-masked]
How to contact me:
http://students.washington.edu/leibrand/howtocontactme.html
* RCW 19.190 notice: This email address is located in Washington State. *
* Unsolicited commercial email may be billed $500 per message.          *

---

From: "H. Bork" <[email-masked]>
Date: Thu, 15 Jul 1999 15:56:58 -0200 (GMT+2)

Thanks a lot, Scott,
fine solution to one of my problems, u2u.

>PC-Pine,...no...available method of connecting securely. ... 
>an encrypted version of PC-Pine will be released eventually.
Any Washington hints when that great event might occur ?
Just password encryption would do for the moment.
TIA+kind regards, hal ;-)

ps 
anyone ever heard of SSH for olde DOS andor Win311 ?
well - just wanna ask :)

---

From: "Daniel Sands" <[email-masked]>
Date: Thu, 15 Jul 1999 09:25:48 -0600

Also, you can compile Kerberos into Pine.  Then if your server supports it,
you can use the /secure switch to use that auth method.

Another possibility is to use the SSL hack that somebody posted a while back.

Still another is that if the server is running rimapd, you don't even need to
enter a password.

---

From: Terry Gray <[email-masked]>
Date: Thu, 15 Jul 1999 09:18:18 -0700 (Pacific Daylight Time)

> Just password encryption would do for the moment.

Hal,
We never know when a new release is actually going to appear (except that
it is always much later than we anticipated)... but there is reason for
some *cautious* optimism that a world-releasable form of SSL for PC-Pine
will make it into the upcoming 4.20 release.  That is certainly our goal,
anyway.

-teg