Everywhere Interactive Displays

Combining an overhead projector, a rotating mirror, a video camera and special software, the Everywhere Interactive Displays projector can turn walls, floors, tabletops or any surface into a virtual, interactive touch screen display.

The Everywhere Interactive Displays project aims to develop systems that can transform any surface into a touch screen. The basic idea is to use a projector to create the display and a camera to recognize user interaction with the projected image. However, given the current limitations of image resolution for projectors and cameras, it is extremely difficult to cover a significant portion of a room with a single projector/camera pair. To overcome this difficulty, IBM Research invented the concept of "steerable interfaces", where a pan/tilt motorized system moves the projector and the camera to where information is needed by the user.

In our prototypes, an Everywhere Interactive Displays (EID) projector combines an LCD projector, a pan/tilt mirror and a pan/tilt/zoom camera. The mirror is used to deflect the image of the projector to surfaces, walls or the floor of a room. To avoid oblique projection distortion, the projected image has to be warped to compensate for the perspective distortion. The image of the video camera is processed using computer vision techniques that detect hand/body activity on the projected area, so people can interact with the projected image by simply touching the surface.
A major emphasis in this project is to avoid "wiring" surfaces with monitors or other sensing devices. In IBM’s vision, information should be added to space without requiring users to wear cumbersome special technology such as graphics goggles or mobile devices. Everything happens simply by projection and vision processing: light in, light out.

The EID projector is a generic input/output device that can substitute for monitors and kiosks in many scenarios. The main advantage is flexibility: once installed, the system can be easily reconfigured to adapt to changes in the environment, new tasks or even completely new applications, without requiring re-installation or rewiring of equipment. In addition, the EID projector can be installed in the ceiling, preventing vandalism and theft. Projected images easily blend with the real world, facilitating the creation of augmented reality applications.

IBM has built demonstrations of the EID projector in different scenarios. In the BlueSpace project, the EID projector is used in a smart office to create interactive displays on walls, tables and the floor. In the Next Generation Store project, the EID projector is used to create interactive signage on floors, walls and store fronts. It is also used to augment a retail shelf by detecting user manipulation of a product and automatically displaying interactive information about the product. The EID projector is also being used to create two next-generation augmented reality video games: (1) a TicTacToe game where the board moves to different surfaces, where it must be found by the players and (2) a toddler's game where an animated frog jumps to different surfaces as the child approaches it.

We are developing integrated solutions for IBM clients that combine the EID projector with digital content distribution/management systems. The EID projector can be used to deliver traditional advertisement throughout a retail environment with advantages over other media devices, such as plasma displays, and with much greater flexibility and adaptability. Moreover, we see an opportunity in the retail area for developing a new set of marketing processes that explore the closeness between product and information. With the EID projector, usage information can be overlaid directly on the product. For example, an appliance dealer could provide interactive instructions on how to operate a washing machine. Animated characters and imagery can be easily integrated into the retail environment, creating completely new shopping experiences.

Work on the Everywhere Interactive Display started in 1999. The first prototype was built in early 2000. In 2001, IBM applied for the patent on the idea, which was granted in August of 2002. For two years the project was funded by a special program inside IBM Research called the "Adventurous Research Program," which allowed a team of computer scientists, Human Computer Interaction (HCI) specialists and visual designers to improve the technology and study its applications and human interaction capabilities.

Embedding Interactions in a Retail Store Environment: The Design and Lessons Learned, Noi Sukaviriya, Mark Podlaseck, Rick Kjeldsen, Anthony Levas, Gopal Pingali, Claudio Pinhanez In Proc. of the Ninth IFIP International Conference on Human-Computer Interaction (INTERACT'03). Zurich, Switzerland. September 2003.

Steerable Interfaces for Pervasive Computing Spaces, Gopal Pingali, Claudio Pinhanez, Anthony Levas, Rick Kjeldsen, Mark Podlaseck, Han Chen, Noi Sukaviriya Mark Weiser BEST PAPER AWARD In: IEEE International Conference on Pervasive Computing and Communications - PerCom'03. Dallas-Fort Worth, Texas. March 2003.

The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces, Claudio Pinhanez Proc. of Ubiquitous Computing 2001 (Ubicomp'01), Atlanta, Georgia, September 2001.

Awards and Recognition

Best Paper Award:
R. Kjeldsen, A. Levas, C. Pinhanez. Dynamically Reconfigurable Vision-Based User Interfaces. Best conference paper of the 3rd International Conference on Vision Systems (ICVS'03). Graz, Austria. April 2003.

G. Pingali, C. Pinhanez, A. Levas et al. Steerable Interfaces for Pervasive Computing Spaces. Mark Weiser award at the IEEE International Conference on Pervasive Computing and Communications (PerCom'03). Dallas-Fort Worth, Texas. March 2003.

IBM Recognition:
IBM Outstanding Achievement Award, 2003.
IBM Access Technologies for Persons with Disabilities Special Patent Incentive Award: C. Pinhanez. Multi-Surface Interactive Display Projections (filed as US Patent Application Serial No. 09/779212, February 2001). June 2001.
Personal Awards connected to this work:
HENAAC Most Promising Scientist with a Graduate Degree award: C. Pinhanez. Awarded by the Hispanic Engineers National Achievement Awards Conference (HENAAC). Austin, Texas. October 2003.

News and Information:
“Smart carts, Veggie Vision in Grocery Stores to Come” by Karen Vibert-Kennedy, USA Today, October 2003.

Cover article “New Way to Work”.by Aaron Ricadela, Information Week, January 28, 2002.

“A Vision of the Office of the Future” by Claudia Deustch, The New York Times, Business (USA). January 14, 2002.

“The World is Your Computer Screen” by Fenella Saunders, Discovery Magazine (USA), December 2001.

“On Any Window or Wall, a Portable Touch Screen”, by Yudhijit Bhattacharjee, The New York Times, Circuits (USA), November 22, 2001.

Rate this article

Rate from		low...		to	...high
Usefulness: