How to get real Haptic Feedback in AR

Ciao to Bella Italia!

A problem all AR and VR systems suffer is the one of haptics. You just can’t touch the virtual world! While you can touch a screen, a marker or a joystick, you still won’t be able to touch the simulated objects. Force feedback is bulky and not as accurate as the real world’s friction or collision. Also you will have trouble of occlusions: is your hand behind the virtual stuff or in front of? The illusion breaks fast with wrong occlusion rendering – and of course – when missing physical feedback.

At RTT AG, I also run into these AR-scenarios where an augmentation is just not enough. If we simulate something the user wants to hold in his or her hand we will use a chroma keying to overlay a colored 3D print rapid prototype that replicates the virtual object’s shape. Now you can actually grab the object and still see your hand on top of it while enclosing it with your fingers.

Of course, we are limited to have a virtual object similar shaped to the 3D print, but it’s a great workflow with realistic haptic feedback. We also integrated push buttons, where your fingers may trigger virtual keystrokes.

One of our partner universities, the Politecnico Milano in Italy and their KAEMaRT group (Knowledge Aided Engineering, Manufacturing and Related Technologies) by Professor Umberto Cugini have been working on extending our module to support even more feedback. Their engineers work on programmable hardware buttons, sliders and nobs, that can communicate with the virtual world: the rendering of the virtual buttons will be updated accordingly to the real world’s buttons. But more – the most important feature: you can set the behavior of the buttons through the software… how hard is it to pull the slider? How many clicks does the wheel have? So it can influence in both directions between real and virtual.

In the following video you can see their great work on haptics simulations combined with our Augmented Reality module RTT RealView and our real-time chroma keying. We believe, this is the way to go for product prototypes, haptics research and user tests.