Playstation on the AR move for more realism!

Sony demos their research on lighting & fluid simulation for Playstation Augmented Reality! They show two videos: the first one demonstrates a fluid/liquid simulation, where they pour water from one container to another. Each container is stuck to one marker and moved around by the user’s movements of the physical marker. The demo also features some matching splash sounds and a beautiful rubber duck.

While rendering quality is not impressive and especially the water is far from looking like uptodate Playstation game’s water, it is good to see, that Sony is playing with this topic to enhance the feeling of the two worlds being one. It helps so much if the augmented objects actually react to yourself. Typically this is only the camera position and not more. But we sure need way more input data than the camera pose to have a good immersion and merge of CGI content into your augmented mirror! Holding two markers up might be a bit dusty, but shows nicely the 100% direct interaction possible. Obviously this could be something cooler (or no marker at all using depth sensors, etc.), but holding up a physical piece will help the users. Augmenting a box-shaped object matching the marker dimensions will also make it easier for the user to forget the augmentation and to have a matching haptics feedback. Plain, but I like it.

The second video does not approach a physical behaviour of things and interaction through that, but rather the interaction with real world’s light. Also seen before, but – again – good to see that the good folks at Sony are working on it to bring it to the PS. Aiming for the Asian market Hatsune Miku is featured and the always-liked-but-usually-dead Dinosaurs:

I’m wondering how Sony researchers are calculating the cast shadow. Is it just comparing the center position of the 2D image with some exposure values from before and see the offsets to calculate a virtual light source relative to the center? On a quick glance it looks pretty nice, but you can notice some shadow direction glitches, especially when close to the center and the spot gets in the front of the Dinosaur. Then the system just cannot know the actual camera’s position and tricks around it.

Honestly, I’d like to see way more on this from Sony with depth cams and reconstructed room data in 3D for occlusions and all! With a simple RGB image you just don’t have enough input data. Sony should add more hardware for next generation and also be able to throw all their real-time shaders on it, too. It is time for a really visually stunning and undistinguishable AR integration! If some companies know how to do good looking real-time 3D content – it is gaming companies like Sony!

Can’t wait!