Subject: Improved key recognition

From: "R.Gering" <[email-masked]>
Date: Thu, 26 Oct 1995 00:08:50 +0100 (MET)

On Mon, 9 Oct 1995, Michael Polak wrote:

> I am trying to run Pine on Wyse-50 terminals.  The only problem I'm 
> having is the arrow keys.  I read the pine manual and realize that pine 
> is not using the termcap information.
> 
> Is there a way to reprogram the arrow keys on the terminal to what pine 
> expects?  I've looked in the Wyse-50 manual and coming up blank.
> 
> Is anyone else using Wyse-50s?

Since no one else is picking up this subject I'll give it a go...

The problem Michael is describing here seems to be one many users of non-ANSI
based terminals run into. The reasons why Pine is not using termcap/terminfo
are described in the tech-info document. An excerpt:

          Arrow keys are  a  special  case.   Pine  has  the
          escape  sequences  for a number of conventions for
          arrow keys hard coded and does not use termcap  to
          discover  them.   This is because termcap is some-
          times incorrect, and because many users have  PC's
          running  terminal  emulators  that  don't  conform
          exactly to what they claim to emulate. Some  arrow
          keys  on old terminals send single control charac-
          ters like ^K (one even  sends  ^\).   These  arrow
          keys  will  not  work with Pine.  The most popular
          escape sequences for arrow keys are:

              Up:     <ESC>[A <ESC>?x <ESC>A  <ESC>OA
              Down:   <ESC>[B <ESC>?r <ESC>B  <ESC>OB
              Right:  <ESC>[C <ESC>?v <ESC>C  <ESC>OC
              Left:   <ESC>[D <ESC>?t <ESC>D  <ESC>OD

It is quite true that not all terminal entries in termcap/terminfo are
beauties and that some of them are incomplete. It is also true that some
terminals send unusual control characters that may "overlap" commonly used
CTRL key combinations. Therefore it seems to make sense to have Pine understand
only the most popular escape sequences and let the users program their
terminals to send these escape sequences.  However...

- Not many users know how to change escape sequences generated for the
  arrow/edit/function keys on their terminals.

- Changing the escape sequenses can seriously affect the proper operation
  of other termcap/terminfo based software. Yes, you can change the sequenses,
  invoke Pine and reset the keys with the old values after leaving Pine, but
  that is NOT user friendly. It's not even system administrator friendly.

- Limited support for "different" terminals is better that no support at all.

Therefore I would suggest to change Pine in such a way that the most popular
escape sequences (as shown above) continue to be recognized, but that termcap/
terminfo is being used wherever possible.

This way, people with the right (ANSI based) terminals but an incomplete
termcap/terminfo entry continue to be happy Pine users. The added bonus comes
from the fact that people with the "wrong" terminal that does have a right
termcap/terminfo entry can now also use Pine right out of the box!

I decided to change Pine to do exactly that, since that was the only way to get
all my employer's terminals to work with Pine. The terminals we have there
(MAI terminals) add a difficulty by having ^A as as the control code lead-in
character. This concept was not supported by the original Pine at all.

Since I think that more people may benefit from the changes I've made, I have
attached a patch file (diff -c) to this message (MIME encoded). The changes
included in this patch file are:

pine/ttyin.c :
- Added the kpinsert() function with the supporting static KBSTREE structure.
  Although based on the original function from Pico, kpinsert() was changed
  to prohibit intrusive changes to the escape sequence tree (overlap, hiding
  or value changing). The reasoning for this was to prevent the generic
  escape sequences (eg. VT100) that get defined AFTER the keys from termcap/
  info from ever changing any sequence defined BY termcap/info. Generic keys
  are great as a supplement to an incomplete termcap/info, but when defined,
  termcap/info should have precedence.

  An important change was the removing of the "cut the escape off the sequence"
  statement from kpinsert(). Although the name might indicate otherwise, escape
  sequences don't always start with an escape (the above mentioned MAI terminals
  for example, use CTRL-A to start their key sequences). All sequences are now
  being stored in the tree without change.

- The function read_char() was rewritten to use KBSTREE instead of the fixed
  sequences, but the functionality of the unparsed function-keys eater remained
  in (that the "ESC ESC -> CTRL" mapping also remained in goes without saying).

  A look-back buffer was created to store an escape sequence being compared
  with KBSTREE. If the match now fails at any point, the function-key eater
  can still recognize the beginning of a sequence and pick up the pieces.

  Every character read is compared with the KBSTREE (no single META character
  testing). Since the top of the tree is usually very small (normally 1 entry,
  2 entries when using MAI like terminals) and the code pretty lean this should
  not result in performance problems.

- The function setup_keystrokes() was added to hang the terminal specific and
  generic keystroke sequences into our KBSTREE. If NO_DEFAULT_KEYS is defined,
  the generic keys will not be added (I've used it for debugging).

pine/status.c :
- Added a simple (c < 256) check before calling isalpha(). At least on Linux,
  isalpha() breaks if the input character is outside the 0-255 range.
  (it uses the character as an index into a 256 entry array!)

pine/strings.c :
- Added the KEYs PGUP, PGDN, HOME, END, DEL and, quite important, INVALID to
  the list in pretty_command(). This truly helps debugging code and replaces
  'Command "4" not defined for this screen' with the better understandable 
  'Command "INVALID KEY" not defined for this screen'.

pico/os_unix.c :
- Changed GetKey() to match the new read_char() function of pine/ttyin.c.
- Added ReadWithTimeout() as a support function for GetKey().
- Removed the now obsolete function kbseq().

pico/os_vms.c :
- Did the same type of changes that were done to pico/os_unix.c, including
  adding a somewhat silly ReadWithTimeout().

  Note: I don't have VMS at hand. I can only verify that the code compiles
        correctly and that the changes seem to make sense. Could some kind
        soul please verify that this actually works?

pico/tcap.c :
- Added support for termcap DELete and Function keys.
- Changed kpinsert() to match the same function now in pine/ttyin.c.
- Added "\033" to all generic sequences, since kpinsert() no longer cuts off
  the leading escape.
- Replaced many of the "external" references to "static" (as in pico/tinfo.c).

pico/tinfo.c :    
- Added support for terminfo DELete and Function keys.
- Changed kpinsert() to match the same function now in pine/ttyin.c.
- Added "\033" to all generic sequences, since kpinsert() no longer cuts off
  the leading escape.

NOTE: Although I thought of having just one universal KBSTREE in libpico.a,
      I decided against it (at least for the time being). For this to work well,
      both Pine and Pico need to have unified function key values. At present,
      that looked like pushing it ;-)

The changed Pine and Pico have been used on a fairly wide range of terminals
(ANSI-based and non ANSI-based) ever since last March. No major problems have
been reported so far (none that I could trace back to my changes anyway :-).

People with terminals having arrow keys that send a single control characters
(e.g. Wyse and friends), should be aware that these control character will
most likely overlap certain control key functions of Pine and Pico.
Since the highest priority is given to the termcap/terminfo matching,
attempting to use such functions will result in cursor movement instead.
Should you be noticing this, know that you can ALWAYS use the following 
workaround to access any Pine or Pico function (overlapped or not):

  Instead of keeping CTRL pressed while selecting the key for your function,
  FIRST press ESC twice and THEN type the key without pressing CTRL.

Even though the developers never intended the "2x ESC = CTRL" trick to be
a workaround for this, it works absolutely great (As a matter of fact, it
comes in quite handy when you're holding the phone with one hand!).

If you use some script to program ANSI sequences into your arrow keys, but you
still use the original termcap/terminfo entry, the changed Pine will recognize
the ANSI keys (because their recognition continues to be hard coded in Pine).
However, pressing a CTRL key combination that is defined in termcap/terminfo
as an arrow/edit/function key will now result in cursor movement instead of
the activation of the expected Pico/Pine function.

For Wyse 60 terminals this problem can be avoided if your UNIX distributor
provided you with the ANSI-key ("ak") termcap/terminfo entries. By selecting
"wy60ak" instead of "wy60", the arrow keys are programmed with ANSI escape
sequences upon terminal initialization and will also be recognized as such.
(Yes, that also means it would have worked with the original Pine). 

Should any of these changes ever make it into the Pine distribution, it may
be wise to optionally deactivate the termcap/terminfo recognition using a
configurable parameter.

One final notice: if the DEL is defined in the termcap/terminfo entry for your
terminal, it will now do a delete instead of a backspace. This is a feature,
not a bug ;-)

I would welcome any questions or remarks!

Best regards,

- Richard Gering

  Work:    [email-masked]           <--- Fastest response during weekdays.
  Private: [email-masked]       <--- Possible response during weekend.

P.S.   To the people of the Pine development team: this patch is not exactly
       the same as the one I've sent you last March. The "& 0x7f" masks that
       Pico's input handler wrongfully inherited from Pine in that patch have
       been removed. Their disadvantage wasn't noticed until I needed to enter
       some special ISO-8859-1 characters.

---

From: "R.Gering" <[email-masked]>
Date: Thu, 26 Oct 1995 00:19:21 +0100 (MET)

Oops...

I forgot to mention that the patch included in the previous message
is intended to be applied to a "clean" Pine 3.91 (perhaps with the
"global option" bug-fix applied).

After decompressing keys.diff, run "patch < keys.diff" in the directory 
that has "pine3.91" as a subdirectory. 

Best regards,

- Richard Gering
  ( [email-masked] )