FMX: Storytelling in VR and The Future of Mixed Reality

Story Telling in VR

I talked about FMX and the VR technology of today and the current limitations in the last posts. Now it’s time to take a look at what other industry pro’s have to say about the content and the future of it!

Typically at FMX the speakers reported on their first VR projects (“still learning”) or had more experience, but the overall mindset was: the initial hype phase will be over soon and what counts are compelling stories that draw us into the created worlds. After the initial “Wow!” or “F**k, I’m Iron Man!” we need to tell a story within the new medium. Learn it now or leave it!

We need to invest in narratively good content and not consider the monetary consequences (if we can for now). Everybody needs to rethink story telling (Don’t freak out, Mr. DOP or director!). The medium is way more direct and less restrictive (that’s the charm) and we must chose wisely when creating content: how can I transport my story best? Does it have to be interactive? What’s the advantage? Would I suffer from crappy renderings (2016) on mobile devices that are unexcusable for my story? Do I really need positional tracking for my story?

The stories we can tell will have these technical constraints, that we should not fear. Constraints are always there and let us be creative. There will never be enough RAM or GPU power! To quote Mark Watney: we just need to creatively science the shit out of the current decade’s technology!

So, collect the best programmers and CG artists from the 90s (only they are well aware of the constraints and will make you hit the 2×90 frames per second) and get going! People that use VR don’t want a copy of reality – but want to experience something new and impossible. Games like the funny job simulator are a great tech demo, but after the initial breaking of social rules (throw the knife through the kitchen) it might get a bit boring to find yourself in a room that could well be in your home… We want to get teleported into different worlds. But what questions do we need to answer to create great content?

What to think about

Let’s go over some technicalities. In any pre-created immersive media (a movie, a book) the passing of time and space can be done easily and must be done! We don’t want to know about the boring parts of a travel or story – we focus on the important and emotional elements. But how about in VR? The key fun factor is that we experience it in real-time and live. But then I am bound to the dimensions of my physical space and my sensation of time. We must find the least annoying and most intuitive ways (it just can’t be 100% intuitive) to deal with this. Movement through teleportation (as described Friday) could do the trick, condensing time will probably need work with fast transitions (wipes? fade-overs?) and tell the jump in the story line – best by some helping NPC.
Speaking about helpers – how can we draw the player’s attention to something? This was often discussed. In the end, typical strategies from 3D game design apply: use spatial sounds, spot lights, special guiding animations or character actions to lead the viewer. But avoid being the know-it-all parent explaining life to a kid. If you take away too much exploring it turns into a chore. If you leave the user completely alone he or she might lose interest. In general one can say that we (as unexperienced VR users in 2016) need more time to get into the VR world and to orient. Don’t include too much action and avoid too much ambient animations or actions on the side. Don’t reveal a whole world, but rather use revealing and occluding effects to draw attention towards new parts of the world and to get the viewer away from stuff that’s done already.
In real-time VR we do have the freedom to better guide the viewer, we could even freeze time until he or she looks the right way. We can use so many action triggers to tell a story. We must combine these coding possibilities with the mentioned visual cues to create great interactive VR experiences.

Another World

So we do want to jump into other worlds. Results show (also test results on myself with many hours in VR) that it is less important how your world looks graphically. As long as the world is consistent and the laws of the world stay understandable and predictable we are happy. I can fly? Fine! No questions asked! I cannot pick up objects? Fine! But if I can destroy a table – why can’t I destroy the chair? There the experience plays with our expectations and kills the immersion – or better: the believable world. Speaking about killing: will you have scruples to shoot at a VR buddy? At a dog? Questions on ethics were touched during FMX panels, although I would have loved to hear more about it. Do we just “need to” get through some emotional blunting like we did with first person shooters – or basically any game or media? Is the further improved realism a problem? Some developers I spoke to remembered their first FPS games and had problems stepping across the social order and break the law (“Can I really shoot at the scientist in Half-Life? No, they won’t let me. Will they? F*ck, I just killed a man!”). But then again we see all GTA and FPS players not wasting a thought on it.
Nevertheless this is a valid discussion for the future. I myself remember VR experiences in a different way than screen-based games. I do remember them as “it happened”. Kids until six years don’t have a filter to distinguish real from virtual! Will the next generation be emotionally attached to virtual characters? What if a virtual representation of a real friend gets so perfect that he is undistinguishable from a virtual NPC? Will we still differentiate emotionally? The line could get blurry here. Let’s be on the outlook for this!

The Near Future of VR

Everybody is excited. But how to monetize VR people asked? No one really knows, but everybody agreed on that it’s better to focus on content first anyway: how can I get people to try my story? We do need great immersive experiences first. Though some studio CEOs said that they need to make money within the next three years to not drop it. Overall the number of VR hardware/platform companies will consolidate to 5-10 big players and the costs for VR productions will go down to 10% or 1% of today’s costs – thanks to better tools & pipelines. We just need to go through 1000 bad VR apps first to reach some initial good ones…

New fields that could arise are definitely live streamings (e.g. concerts) or in general virtual travels (from the comfort of your home). VR could reduce real world objects and possesions (less clutter in the environment). Telepresence will get big for sure. Depending on the circumstances we will quite often chose the virtual meeting. In the end we will only remember talking to our friends or collegues – not how we did that (on the phone? face to face? in virtual reality? I don’t totally recall.). Let’s find out!

The Future of AR in story-telling

But what about AR here for story-telling? I raised the question multiple times, but – boilt down – the answer is: everybody would love to try it out and play with it, but nobody knows. Some don’t really see it as a market. I’d say fully convinced: storytelling needs to be completely re-invented for AR after the re-invention for VR! VR story telling is too easy – in comparison. You teleport to another world and once you’ve understood the technology: go for it and create a story for your living room (instead of using the TV)! But with AR you are always inside the real environment and must integrate the alternate reality convincingly into the player’s space. Besides technical problems (tracking, visual overlay, interaction) we must find out what kind of stories would make sense at all. Do we go for augmented pets first (like in good old Denno Coil)? The great thing about AR could be that in contrast to VR – where a comparably small number of users will use the system – AR will scale up big. Everbody will have an AR device on the nose. Initially for peasant tasks (“where’s the next store, what’s on my calendar”) and business. But once rolled out, we could (and should!) go big with it and deploy great everyday, entertaining AR experiences! Exciting times will come! I’d say: the real discruptive social break-through is not happening at all with VR now but will come with AR within the next 5 years! The revolution takes a bit longer. Now for the detour through VR – on the road to augment our real world!

Regular’s Table Today!

… and to make the day perfect let’s meet up tonight for the next #ARMUC regular’s table in Munich and discuss this! Feel free to join on time. It might get crowded (and they will show the soccer match later on), so be there and be square! (Meetup Link)