What if … your augmented pet felt real?

Augmented Reality is further on the rise. No doubt. Hololens celebrates its anniversary, Magic Leap is (again) rumored to release something with way higher photorealism in 2017, Apple could skip VR and jump directly into the AR space through an augmented window frame (= the new iPhone) approach and more and more cyberpunk, tech sci-fi movies and remakes hit the theatres. Thought, I take some time today and think about what the technology could mean to society, what dangers could get created and how the avoid them (if possible at all). Obviously just scratching the surface in a few lines today.

All this is still in the mid-term, distant future – well, we think today, unless someone proves us wrong sooner. What if all AR looked so real that we couldn’t tell the difference? What would this make to us? What if A.I. and mobile computer systems were to support a perfect AR? What if A.I. on iOS / Windows / Android systems becomes a companion to our life? What it they were fully integrated into our view and our daily and nightly life? I almost wanted to ask what if androids dream of electric sheep… kidding. Let me rather ask: what if your augmented pet felt real?

Denno Coil showing the ultimate Augmented City

A pet is just a small demo of capabilities of a future AR, but one that has big impact and answers a lot of challenges and social questions or at least: asks these questions. Therefore I wanted to mention one of my favorite Anime TV shows again: Denno Coil, but also show the latest idea of an augmented pet, as in the great short “Strange Beasts” that just hit the web the other week.

Before we dive in, I highly recommend the series to watch. It may be a children’s show, but it holds a lot of great details and philosophical thoughts on our beloved AR technology. It has been directed by Mitsuo Iso, who is connected to this sci-fi setting for a long time now. He also animated parts of the Ghost in the Shell Anime back then! Since June, 2016 finally the official English dubbed blu-ray is available (next to fan-made subs).

Denno Coil is a Japanese TV series from 2007, that plays in a near future in 2026 where augmented reality glasses are omnipresent and available for 11 years by then (sic! This would have been 2015, damn we were so close!). Wikipedia somehow translates the name to “Electric Brain Coil or Computer Coil”. Young girl Yasako moves into their kind-of Silicon Valley city where AR is heavily rolled out already and the kid experiences quite some adventures with her friends. While some adults only tend to use AR glasses at work, especially children use it in every moment to play, search the web on virtual screens or just to see their augmented pet next to them (like Yasako who got her glasses from her grandpa working as an AR engineer before).

The story starts twinkle-toed into this children’s daily life, a new city, out to discover the place after the family move, look for lost (augmented) pets with the AR glasses, join some fun P.I. team with her granny. They look for augmented metabugs (like harvesting Pokémon spots), shoot rays from their forehead, hack each other in their different peer groups and escape some lunatic augmented garbage collector Satchi. It all is children`s fun before it gets darker…

Denno Coils AR Technology

But let´s detour on the techy side. Technology is not widely explained, but they’ve reached a level of visual perfection and 100% believability. No sensor tech is shown, it all happens in the background. The glasses with perfect tracking, overlay and spatial audio just work. Perfect network coverage (almost always) guarantees a connection to the digital parallel world. The whole city has a digital double (like the perfect Hololens or Google Tango scan): sometimes we see glitches of the augmented overlay or see not-yet-updated space, that is out of synch.

If a kid creates a virtual wall, the second kid wouldn’t be able to tell the difference to the real wall. Hide and seek at a new level! The virtual pets or objects are even sometimes checked by taking off the glasses – is it still there? What is real? Kids use the tech naturally and interact with virtual objects – like the dog – as if they were real. This leads to certain moments where the illusion breaks: dropping a backpack on top of the virtual dog will result in a cut-off dog’s back as in the featured image above. At the same time kids can grab and hug the dog and type pretty damn fast on air-floaty virtual keyboards.

For this to work, each person has a digital double that is aligned with oneself. A virtual body can also take (virtual) damage and people try to avoid this naturally.

Yasako discovers more and more about the city, finds other children, learns about their problems or anxieties and keeps looking for her virtual pet Densuke. Some dark virtual figures, the illegals, appear and get hunted and it´s gets darker when we learn more about personal loss, effort to rescue someone and the Denno Coil. The interconnection between the physical and virtual world, the connection between the virtual representation of yourself and your (real) body, brain and psyche gets crucial and tangled. Well, I won’t describe the story any further, but it holds a lot of cool ideas and shows the emotional connection between real and virtual actors very nicely. The question arises: what is reality? That parts, that I can touch? Or all parts that have impact on my life? Be it physically or emotionally.

Strange Beasts show it for real

While Denno Coil shows us an animated version of things and nicely blurs the line between real and virtual with a certain distance to the viewer, Strange Beasts ports this idea to real world visuals (with awesome CGI). The “perfect” AR system (with laser eye surgery non-removable now) allows for an augmented life, showing the example of a pet as well. I can’t say any more without spoiling, so watch it first!

Strange Beasts from Magali Barbé on Vimeo.

So, when will we believe the AR?

I sure liked it! Well, spooky fun. Strange Beasts allows for perfect rendering and lighting of augmented objects; we see ideal occlusion and collisions among real and virtual. The protagonist grabs a virtual ball to throw like kids would in Denno Coil. Artificial pets behave convincingly. So, in the end, we have three areas that make us believe the system: the behavior and response of a virtual character, the visuals and the form of interaction plus haptics.

Not thinking too much about sensory tech, the behavior and response is a no-brainer: I’m sure better AI will make us believe in virtual characters as being real one day. Especially if it’s something like a dog or a dragon pet. The Turing test to be passed very soon.

The visuals (including the tracking) need to be convincing, too. But this does not necessarily mean to have photorealism. People got addicted and emotionally attached to Tamagotchis in the 90s, kept their Farmville landscape or Fishdom aquarium in shape and got jealous of the neighbor or cried if the virtual pixel pet died. After all, we cry with good books or stories. A convincing character to whom we connect is part of our life and we will miss him, her or it if it disappears.

The third is tricky. How do we interact with the AR’ed objects? Will the illusion always break when I air-tap a virtual keyboard or throw a virtual ball because the haptics sensation is missing? How can I link to a virtual dog if I can’t stroke it and it purrs? We will never be able to substitute for haptics in pure air – unless the holodeck comes along.

I’m no neuroscientist, but maybe we can live with this missing piece of the puzzle. Maybe augmented natives (our children’s children generation maybe) get this ability as a trained ability – to have a strong imagination to have a make-believe haptic created in the brain. When I draw a bow in HTC Vive’s The Lab demo today, I feel the sensation of the friction of the arrow in the bow, but only simulated by the vibrator of the controller. But it makes me believe the world. On my smartphone I only click virtual buttons, nothing physical. The click sound is enough to make me believe in the press-action. Maybe the augmented natives get the trained ability of believing what’s not there – to have a joint reality – of physical and virtual objects that both play an equal part of their lives. Maybe you get used to interact with virtual fellows and friends this way and accept walking through them like you would today in AltspaceVR or other games, where a clipping can just not be avoided. Maybe the purr sound of your virtual dog plus visuals is enough for you to accept him as real and to connect deeply. You might forget the missing haptics like you do when you are perfectly immersed. You won`t be able to tell the difference between real and augmented, there is no more separation between tangible and digital for your own life and feelings. Virtual objects and characters are accepted. Now… let`s integrate this into our daily life!

Huge potential? Scary? You bet. Let’s take a break and continue next week on it.

What could the impact be?