Chalk

Introduction

Chalk is the name of the Open Slate Project's pen-based user interface. It will be based on an existing window manager. The goal is to allow common Free and Open Source Software (FOSS) applications to work unmodified, but to reveive the full benefit of the handwriting recognition system it may be necessary to create Chalk specific versions.

Build on standards

The Open Slate reference design will use commonly available hardware components that function like a traditional IBM PC. The intent is to have a hardware platform capable of running a variety of FOSS operating systems such as Linux or FreeBSD with little or no modification. Some designers may want to explore alternative designs, perhaps using a chip set optimized for low power consumption. All such experimentation is welcome so long as the user land software operating environment remains compatible.

Staying close to the widely used IBM PC hardware design means that it will be easy to get X running, and once that is done there are a number of window managers to choose from. It should even be possible to run KDE or Gnome, either of which provides a consistent user interface across applications. This may sound like a good thing, but such an approach fails to take into account the Open Slate pen interface.

A pen is not a mouse

A mouse has buttons. Pens, pencils, and erasers do not. Neither do crayons, felt markers, and paint brushes. The Open Slate pen will not have buttons, either. Pressure sensitive input may be an option, but will play no part in the window manager user interface.

A user interface with only a button-less pen for input may sound like wings for pigs, but Apple has done it. The Newton can do everything with just a simple pen. The trick is the Newton's clever user interface design.

In order to provide a useful and satisfying user experience it will be necessary to develop a pen-based window manager and associated libraries. We call our pen-based user interface Chalk. The Chalk window manager will not be built from scratch. It will be based on an existing window manager, such as Blackbox or Metacity. We might even pursue a Chalk version of Gnome or KDE.

Chalk must take into account the conflicting goals of power efficiency and eye candy. Obviously a window manager that moves windows by drawing a ghostly wire frame will consume less power than one that redraws the entire window while filling in the exposed areas. Simple window ornaments will take less work to render than shaded gradients with transparency.

How much of a difference is there and how much difference will it make on overall power consumption? To do that we need to come up with a way to measure power consumption. Sounds like a topic for a research paper. Maybe we can get a grant to study the problem. The thing is, if we take this approach and study everything, we will never get it built. What we really need is common sense design guided by experience and a willingness to fail once in a while.

Pen gestures

The Chalk pen driver will need to filter input to determine what the user is trying to accomplish. A mouse driver does this, but the pen is more complex. Here are a few situations inspired by the Newton that suggest how Chalk will work:

Tap

The pen is pressed against the screen and immediately released.

Tap-and-hold

The pen is pressed against the screen and held there.

Double tap

A tap followed immediately by a tap-and-hold.

Pen actions

Tap in text field

Moves the insertion point to the location of the tap.

Tap on button, scroll bar, etc.

Activates the control.

Tap on selection list entry

Selects entry and closes list box.

Double tapping a word

Brings up the word's recognizer list or a list of similarly spelled words from the dictionary.

Double tapping a selection

Makes a copy of that selection (see copy below).

Tap and hold

Holding the pen still for about one second after touching the screen shifts into selection mode signaled by a sound event and the Ink Text line becoming extremely thick.

Clipboard actions

Selecting text

Text is selected by executing a tap-and-hold, then drawing a diagonal line through the desired text. Individual letters or parts of words cannot be selected; any part of a word that falls within the selection area causes the entire word to be added to the selection.

Working with the selection

Cut

A tap-and-hold is performed at any point inside the selection. The selection is detached from its current location and follows the pen movement. Additionally a text insertion (verticle bar) cursor will move to show where the text will be inserted. When the pen is lifted, the text is pasted at the indicated position. As a special case, when the pen is dragged up to and against the edge of the screen and held there, the selection is shrunk to a tiny chip and stuck against the side of the screen. This action is reinforced by a sound event.

Copy

To make a copy of the selection, perform a double tap, leaving the pen on the screen. From this point on the actions are identicale to cut described above.

Paste

A simple cut or copy is completed with an implied paste when the pen is lifted. A cut or copy in which the selection was parked on a clipboard (along the edge of the screen) is completed by navigating to where the clipboard data should go, then executing a tap-and-hold on the chip, which expands to full size and floats with the pen until the pen is lifted.

Delete

The selection is deleted by drawing a "W" or "M" beginning just outside the selection so that the pen passes into and out of the selection at least twice. Best done backwards, right-to-left, to avoid confusion. Deleted text cannot be pasted, but it can be recovered with Undo.

Input mode

When the pen immediately starts to move after touching the screen the system will shift into input mode. If the object receiving input is a graphic object it will receive the raw input stream, the same as if the input device were a mouse with button #1 down. If the target is a text object the handwriting recognition system will intervene in such a way that the target only receives the result of the recognizer.

Improved text entry

There is one more thing the Newton does that makes it such a joy to use. It retains a list of potential solutions for each of the last ten or so words recognized. The list even includes the Ink Text version of the word.

Double tapping a word brings up this list. The first entry is always the same as the current word. Sounds redundant but sometimes the actual text is drawn in a difficult to read font whereas the list is always in the very clean system font. The second entry is the same word with reversed initial capitalization. These two are followed by any other possibilities that scored high enough. The last entry is the ink text version, an image of exactly what was entered with the pen. At the bottom of the list box are two buttons, one to open a virtual keyboard, the other to open a correction tool.

About ten words later the word's recognizer data is tossed. (If I were writing this on a Newton, the recognizer data for "About" in the previous sentence would be lost around the point I entered "tossed.") After that, double tapping a word always brings up the first two entries, for easy correction of capitalization, as well as any similar words from the dictionary.

To implement this feature in Chalk will require more than splicing a pen interface into the X input device system. Either the recognizer will maintain the data and pass a pointer to the target application, or the target application will accept and manage the data. I suspect the former will work best. Perhaps all it will take is an additional X input event which consists of the text, one word at a time, along with an ID number. Standard applications will understand the traditional text input event and ignore the Chalk version, while Chalk enabled applications will be able to use the ID number to interact with the recognizer. More than likely there are other, better ways to do this.

Conclusion

As much work as there is to be done in other areas, it is vital that Chalk development get underway at once due to the impact it will have on how applications behave.