Fujistu develops touchscreen interface for seamless data transfer between the real and virtual worlds

Fujitsu Laboratories has developed a next generation user interface which can accurately detect the users finger and what it is touching, creating an interactive touchscreen-like system, using objects in the real word.




"We think paper and many other objects could be manipulated by touching them, as with a touchscreen. This system doesn't use any special hardware; it consists of just a device like an ordinary webcam, plus a commercial projector. Its capabilities are achieved by image processing technology."

Using this technology, information can be imported from a document as data, by selecting the necessary parts with your finger.

This technology measures the shape of real-world objects, and automatically adjusts the coordinate systems for the camera, projector, and real world. In this way, it can coordinate the display with touching, not only for flat surfaces like tables and paper, but also for the curved surfaces of objects such as books.

"Until now, gesturing has often been used to operate PCs and other devices. But with this interface, we're not operating a PC, but touching actual objects directly, and combining them with ICT equipment."

"The system is designed not to react when you make ordinary motions on a table. It can be operated when you point with one finger. What this means is, the system serves as an interface combining analog operations and digital devices."

To detect touch accurately, the system needs to detect fingertip height accurately. In particular, with the low-resolution camera used here (320 x 180), if fingertip detection is off by a single pixel, the height changes by 1 cm. So, the system requires technology for recognizing fingertips with high precision.

"Using a low-res webcam gives a fuzzy picture, but the system calculates 3D positions with high precision, by compensating through image processing."

This system also includes technology for controlling color and brightness, in line with the ambient light, and correcting for individual differences in hand color. In this way, it can identify fingertips consistently, with little influence from the environment or individual differences.

Also, in situations that don't use touch, the system can be operated by gesturing. In this demo, when you move your fist, you can manipulate the viewpoint for 3D CAD data. So, there could be applications for this touch system by combining it with current gesture systems.

"For example, we think this system could be used to show detailed information at a travel agent's counter, or when you need to fill in forms at City Hall."

"We aim to develop a commercial version of this system by fiscal 2014. It's still at the demonstration level, so it's not been used in actual settings. Next, we'd like to get people to use it for actual tasks, see what issues arise, and evaluate usability. We want to reflect such feedback in this system."

Credit: Diginfo.tv