SHARK shorthand is a high performance text and command input method for handheld mobile devices. Technology is advancing to an exciting future in which mobile devices, such as smart phones, can be pervasively used as the physical handle and coordination hub in the digital information world. It will only be a matter of time before desktop-level processing and storage power are available on such devices. However, the improvised user interfaces currently on mobile devices challenge the computer industry to research and invent novel interaction methods on small devices, so that the vision of a post-PC client computing paradigm shift – information access anywhere, anytime – can be possible.
Mobile devices lack efficient methods for two categories of tasks – text input and command/object selection. SHARK shorthand offers high performance solutions to both. With SHARK shorthand, a user may trace the letters on the keyboard to enter a word. We call the pattern formed by the letter trace of a word on the stylus keyboard a “sokgraph”. Over time, the user can learn the sokgraphs, allowing for higher speeds.
The sokgraphs defined on the ATOMIK layout are very efficient (Fig. 1). SHARK shorthand can work on the less efficient, but more familiar, QWERTY (Fig. 2) or any other layout (Fig. 3).
Fig. 1. SHARK shorthand on the optimized ATOMIK Layout
Fig. 2. SHARK shorthand on the conventional QWERTY layout
Fig. 3. SHARK shorthand on an alphabetical layout
There are three key factors that make SHARK shorthand powerful:
- A person’s ability to recognize, memorize and draw patterns is remarkable. We can remember and write hundreds of letters, numerals and symbols. A literate Chinese person is able to write thousands of characters. SHARK shorthand capitalizes on this remarkable capability. Gesturing patterns with a stylus is a particularly fluid, dexterous and fun action.
- SHARK shorthand uses modern digital processing capability to compute the statistical constraints represented by the legitimate sokgraphs on the stylus keyboard. IBM’s novel pattern recognition technology takes advantage of these constraints and gives the user the maximum flexibility in sokgraph gesturing. An intended word can still be entered even if some of the letters in a word are missed in the stylus trace on the keyboard.
- SHARK shorthand bridges initial ease of use with eventual high performance by embedding learning into the method itself. In psychology terms, for initial ease of use, the user interface needs to be recognition-based. This is what makes the graphical user interface (GUI) a success -- elements in the GUI can be visually searched and selected. To reach high performance, however, the user interface should support recall-based skills. This is why touch typing can be so rapid and the command line interface is still faster and preferred by domain expert users. In SHARK shorthand, these two modes are seamlessly and gradually connected -- one shifts from recognition to recall by use. The graphical stylus keyboard serves as a training wheel from visual tracing towards gesturing a form of rapid shorthand.
Recorded peak speed of SHARK shorthand reaches the range of 60-80 words per minute. A study showed that users can learn and recall about 15 new sokgraphs per 45 minute training session. Due to the Zipf's law effect (a small number of common words account for a very large percentage of writing), the benefit of recall-based sokgraph gesturing is rapidly achieved. Outside the laboratory, users have been able to write lengthy articles, such as this essay, with SHARK shorthand only days after downloading it.
Lacking fast hot keys for common commands such as copy and paste, selecting menus items and issuing commands on mobile and pen-based devices tend to be tedious. SHARK shorthand addresses the command selection difficulty by gesture shortcuts. For example, one can gesture Ctrl-p-r-i to issue a print command (Fig. 4). SHARK shorthand has an expandable set of gesture shortcuts to enable efficient command selection.
Fig. 4. Gesture shortcut for the command "Print"
Related Publications
Per-Ola Kristensson and Shumin Zhai. Relaxing stylus typing precision by geometric pattern matching. Proceedings of ACM Conference on Intelligent User Interfaces (IUI). ACM Press, 2005.
Per-Ola Kristensson. Breaking the laws of action in the user interface. Extended Abstracts of the ACM CHI Conference on Human Factors in Computing Systems (CHI). ACM Press, 2005.
Shumin Zhai and 'Per-Ola Kristensson. Shorthand Writing on Stylus Keyboard. CHI 2003: ACM Conference on Human Factors in Computing Systems. ACM, November 2002.
Shumin Zhai, Per-Ola Kristensson and Barton A Smith. In search of effective text interfaces for off the desktop computing. Interacting with Computers 17(3):229-250, 2005.
Per-Ola Kristensson and Shumin Zhai. SHARK2: A large vocabulary shorthand writing system for pen-based computers. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), CHI Letters 6(2). ACM Press, 2004.
News and information
"WHAT'S NEXT; Trying to Make the Pen as Mighty as the Keyboard." The New York Times, 2004.
Text Entry Epiphany for the Tablet PC- SHARK. jkOntheRun (Top Ten Tech Blog), November 03, 2004.
Trying to make the pen as mighty as the keyboard. ACM Tech News 6, 718, 2004.
The pen that's mightier than the keyboard. Globe and Mail, November 24, 2004.
Making more of handhelds. Dan Gillmor's eJournal (Syndicated to San Jose Mercury News, Arizona Star, Seattle Times, etc.), April 9, 2003.
Shorthand-Aided Rapid Keyboarding. slashdot.org, October 28, 2004.
Shark tale a new word for keyboard. Australian Financial Review, November 16, 2004.
Applications, demos, tools & technology
You can download a free trial version of SHARK shorthand at IBM alphaWorks.
Rate this article
Innovator's corner
Shumin Zhai Researcher
What is the most exciting potential future use for the work you're doing?
The potential use of SHARK shorthand is really significant. We envision millions of people using SHARK shorthand on smart phones, PDAs, pen-based computers and other future client computing devices. We hope our research and development on SHARK shorthand will accelerate the progress of client computing beyond the PC paradigm. Looking more broadly, the advent of new writing technologies has always been part of civilization evolution, from carving symbols on tablets, to writing on paper, to the typewriter of the industrial age. Being able to influence and shape a future writing technology is truly exciting.
What is the most interesting part of your research?
What makes SHARK shorthand interesting is the combination of different sources of knowledge, insight and technology. It involves understanding human perceptual, cognitive and motor behavior and performance, mathematical optimization, the nature of language, and the latest technologies in pattern recognition and software development.
What also makes this work fun is the special kind of team collaboration: Per-Ola Kristensson is a PhD student at Linkoping University in Sweden where I also serve as his thesis advisor. He works back and forth between the university and IBM, making this industry-university collaboration particularly effective.
What inspired you to go into this field?
The lack of and need for high performance solutions for mobile human-computer interaction and the potential to broadly impact the evolution of user computing.
What is your favorite invention of all time?
Language is the most important human invention in the general sense. If I have to pick a more mundane example I would say the bicycle is an amazing invention.
Research team
Related Research
Disciplines: Computer Science
Research Areas: Human Computer Interaction
Research Labs: Almaden Research Center, Watson Research Center