SixthSense Makes the Entire World your Computer

Pranav Mistry demonstrates the SixthSense device. Courtesy of Pranav Mistry.Imagine you're in a bookstore, mulling over a choice of a Salman Rushdie book or a novel from this year's Booker Prize winner. How would you like it if you could see Amazon.com ratings projected on the cover of the book? What if you could dig more information about them - literally out of thin air?

The SixthSense makes this possible.

The device, now a prototype, helps users interact in the physical world with digital information just by moving their fingers. Creator Pranav Mistry, a doctoral student from the MIT Media Lab, sees this as a way to make the “entire world your computer.”

The miniature electronic device is worn around the neck and recognizes hand gestures and physical objects using an attached camera. It then accesses the Internet to retrieve relevant information and project it on any surface -- such as a wall or even the palm of a hand -- with a built-in miniature projector.

The gadget is made up of standard, off-the-shelf components, that cost as much as a fancy mobile phone in total. Mistry plans to make SixthSense's software platform available for various programmers to develop and write applications for the device.

Here are a few things Mistry says SixthSense can do: making a hand gesture of a camera and taking a photograph, navigating a map on any surface, projecting a calculator on the palm of your hand or playing an online video on a print newspaper.

We spoke to Mistry about his new invention.Examples of what the SixthSense device can do. Courtesy of Pranav Mistry.

American Observer: How did you come up with the idea of the SixthSense, and how much time did it take you to come up with a prototype?

Mistry: Most of my recent research projects share a common theme of integrating information with our real, physical world. Of course, the digital world has brought a lot of convenience to our daily life and has enormous advantages. But it has taken away the joy of interaction with our environment and everyday objects. I miss riding my bike and going for shopping with friends.  I imagine and dream the world where our digital technologies blend in our everyday physical activities and surroundings. That will be the true "ubiquitous computing" [former chief scientist at Xerox, and father of ubiquitous computing] Mark Weiser had proposed.

In some of my previous work, I attempted to make intelligent sticky notes, Quickies, that can be searched, located and can send reminders and messages; removed a pen’s tie with paper by giving it an ability to draw in 3D, in project "inktuitive" and implemented TaPuMa, a tangible public map that can act as Google of physical world.

The idea of SixthSense was with me since last summer vacation. I was thinking of it when I went home (India) during past summer vacation. I started designing, implementing, developing the hardware and software for SixthSense around five months back when I came back from India. It took me four months to complete the fully working prototype of SixthSense that we unveiled at TED2009 for the first time.

Observer: What applications do you see SixthSense put to?

Mistry: I see SixthSense as not a device or a platform, but as a way we interact with the world of information. The core vision of SixthSense is to merge our informational experience with our everyday activities and surroundings, the physical objects we use. This opens up enormous opportunities in all fields, whether it is medical or education, from developing nations to advanced specialized applications. Recently, I am getting many ideas and suggestions from people all over the world that how they would like to use this in their context, their daily life with what they do.

You can obviously imagine this to replace our laptops and mobile phones and ... I am more interested in what it can do that those digital gadgets can't.

Observer: What are your plans for the SixthSense in the future?

Mistry: I am sure that there are lots of things I still need to do if I want this to be something that everyone can use it. I am working on to make SixthSense robust enough. Markerless, bare handed tracking of gestures, robust object recognition and tracking and self-correcting projection are some of the technical things I am working on at the moment. I also want to make the development platform around SixthSense so that thousands of developers who are really excited about this can join me to make it better and can develop applications for SixthSense.