The IBM Smart Surveillance Solution is helping retailers, manufacturers, entertainment venues and government analyze -- and respond to -- security threats.
As many as four million surveillance cameras reportedly are posted throughout the U.K. on street corners, underground in subways and in buses. In the event of a bombing, investigators can view thousands of images, as they did after the July 7, 2005 transit bombings in London.
Eyeballing surveillance video, however, is a time-consuming, expensive and ultimately inefficient way to study video intelligence data. The IBM Smart Surveillance System (S3), developed by IBM Research, is a set of computer vision and indexing technologies that can help municipalities -- as well as retailers, manufacturers, casinos and others -- analyze video and automate two key surveillance functions: Real-time intervention and post-incident investigation.
Integrated into existing video cameras and capture systems, S3 consists of several security features:
- Video/sensor analytics, based on computer vision, pattern recognition and learning technologies.
- A framework for integrating event information from multiple cameras and other event sources -- based on Web and database technologies.
- A framework for building customer-specific solutions, drawing upon video and sensor events, and integrating them into the customer's business processes via Web services and Java technologies.
- A software development framework for integrating various partner technologies, such as video-capture, and specialized analytics, such as license plate recognition.
Smart technologies enable the detection, tracking and recognition of objects in video streams. The meta-data generated by these capabilities form the basis for "user-defined alert evaluation," event-based indexing and retrieval of surveillance video.
These technologies, however, pose multiple technical challenges.
Evaluating potential dangers
The ability to use statistical models to distinguish normal variations in a specific environment – changes in light or weather – from actual moving objects – a car driving through a parking lot – is key to object detection. IBM researchers have used appearance-based tracking to follow objects while keeping their identities distinct as they interact with other objects. Recognizing objects, such as human faces or license plates, requires the use of sophisticated learning and structural pattern recognition techniques. S3’s tracking algorithms work to keep people and objects distinct from each other.
How S3 processes video (see Figure 1)
- The Smart Surveillance Engine (SSE) analyzes the video to detect, track, classify and recognize moving objects.
- The video analysis generates meta-data that captures the trajectory, color, shape, size, class (vehicle, person, group, face) and identity (license plate number) of the object. The meta-data is used to evaluate user-defined alerts, such as motion detection, etc., which generate
sadditional meta-data. The meta-data are represented as XML documents. - The SSE uploads the meta-data into DB2 using http uploads.
- Middleware for large-scale surveillance (MILS) servlets receive the XML meta-data and put it into DB2 tables using the XML extender.
- In addition to several system management functions, MILS provides Web services, which facilitate the searching of the event meta-data. The MILS servlets internally use SQL queries to select the data from tables. Query results are returned as XML documents.
- Applications built around the S3 framework use the Web services API’s of MILS to provide users with the “real-time alert” and “event search” functionality for surveillance video.
Software architecture of the IBM Smart Surveillance System (S3). This architecture supports “user-defined alerts” and “searching” based on surveillance camera video.
Creating a living lab to test real-world applications
Two factors contributed to the success of the S3 research effort: Working closely with clients and creating a “living lab” for ongoing research and development. Early on in the project, IBM researchers set up surveillance cameras in various parts of the IBM Hawthorne facility and used these feeds to develop the algorithms and the S3 framework. The S3 system has been running for more than three years – 24x7 on at least 10 cameras – and has processed approximately 24,000 hours of video. Every new research algorithm developed in the S3 project is validated in the living lab, thus ensuring that the research effort is guided by real-world problems.
S3 has a presence in the retail industry
Retail stores traditionally have used surveillance cameras for "loss prevention" -- reducing theft by employees and customers. With the introduction of smart surveillance technology, retailers have started reaping many benefits from store cameras, including:
- Loss prevention and liability mitigation. S3’s real-time alerting functions allow appropriate store personnel to keep an eye on multiple events. S3’s searching capabilities allow for effective investigation of “slip-trip-and-fall” liability claims made by customer and employees.
- Enhanced business intelligence. Cameras can monitor the effectiveness of displays. For example: How many customers passed by the Levi’s jeans display? What percentage of them stopped and picked up merchandise? Correlating the display effectiveness data to the transaction log can support various business decisions.
- Improved operations. Cameras at the store entrance can count the number of people entering the store. Count data can help in planning store staffing levels. Cameras at the checkout register can measure wait-times, alerting store management to wait times that exceed the retailer’s quality-of-service guarantees.
Read more about the IBM Smart Surveillance System
IBM Smart Surveillance System [IBM]
IBM Smart Surveillance Solution [IBM]
IBM and Business Partners deliver customer-focused innovations and insights to retailers at NRF 2007 [IBM]
Rate this article
Innovator's corner
Andrew Senior Researcher
What's the potential for the work you are doing?
We're trying to apply video analytics to a range of problems that can increase public safety (detecting passengers falling on subway tracks), prevent shoplifting or reduce traffic congestion. We're also helping to improve the privacy protection where video surveillance is used.
What is the most interesting part of your research?
Finding creative solutions to interesting real-world problems. Computer vision is a field where the possibilities are opening up rapidly, especially as camera and CPU developments enable algorithmic improvements. I find vision is a particularly interesting area to work in because of its connections with human vision and visual art, as well as its immediacy, seeing being a nearly universal experience.
Who or what inspired you to go into this field?
My parents (a math teacher and engineering lecturer) certainly got me interested in the areas of engineering and computer science, but one of the biggest triggers to me ending up in computing was the 32KB BBC Micro I had as a teenager. I spent a lot of time on all kinds of projects like animation and writing a C compiler for it. After that I just gravitated towards pattern recognition and computer vision.
What is your favorite invention of all time?
That's a tough one. There are so many inventions without which we would be much worse off: electric lights, vaccines, agriculture, language and so on. Perhaps the transistor is the invention that most fundamentally underpins information technology, which is creating so many new and exciting possibilities like email, the Internet and digital photography.
Research team
Lisa Brown, Arun Hampapur, Max Lu, Chiao-Fe Shu, Yingli Tian
Related Research
Disciplines: Computer Science
Research Areas: Multimedia, Security and Privacy
Research Labs: Watson Research Center (Hawthorne)