Microsoft presents new services to grow AR

On Sunday, during AWE in Barcelona, Microsoft presented in an one hour show multiple news on their commitment to the mixed reality space. Including demo showcases with partners, Azure cloud services, new software, ah, yeah, … and the HoloLens 2! Time to compare my wishlist from last week with it and to give my virtual two cents on the news. Is the HoloLens 2 worth mentioning? Revolution or evolution?

Facts

If your #1 source for AR is augmented.org (thanks!) and you have been hiding behind a rock until now, let me summarize the critical facts on the HoloLens 2 for you:

  • “doubled” field of view
  • “tripled” comfort including a visor flip-up display
  • 3500 USD, preorders open, shipping in “summer”
  • full hand and finger tracking, no more clicker
  • eye tracking and iris scan
  • cross-device shared and persistent experiences with spatial anchors
  • cloud rendering for high quality assets
  • hardware customize program for adapted form factors to fit e.g. hard hats, etc.
  • subscription model possible (starting 125 USD/month) incl. more software services

The full show can still be rewatched on the Microsoft channel:



Form Factor & Comfort

I was kind of hoping for a pocket PC on your belt to avoid heavy weight and heat close to your brain, but seems Microsoft didn’t like the idea of strangle-danger with a cable for the B2B market. Noone will ever know how Alex Kipman calculated the “tripled comfort”, but the solution seems practical: easy-on, easy-off with the same dial on the back of your head as before, but with better weight distribution and a stronger rest on your forehead. It seems as if fitting is easier this way – avoiding the typical problems: “I don’t see anything” – “You have to move it up on your nose way more.” – “Ah, now I see something. But it’s broken. The window is still so small.” – “Yeah, that’s how it is.”. Replaceable cushions is really crucial. Everyone learned it painfully with VR headsets on demo days and sweaty beat saber sessions… It’s good that Microsoft integrates it and also allows for different head shapes this way. A few grams less on your head with a better balance seems like a solid way to go. Being able to wear your prescription glasses below (without the need for custom clip-in lenses like the Magic Leap One) feels like a wise decision for business target group, where you might not have your personal device. A flip-up visor is awesome and matches the Windows Mixed Reality family. So, form factor and comfort: check!

Visual Quality

I wished for a doubled field of view (fov) – and got it! Check! Well, well, well, depending on how you calculate it (52° diagonal instead 34°). It seems like Microsoft didn’t find a way to further increase the fov without sacrificing quality or compact form factor. Unfortunately, they never show a first person view of the experience. The raised resolution is promised to still keep the pixel density from before (47 pixels per degree of sight) and should give a fresh and pretty image. But “holograms” will still be cut off easily as it seems and as first hands-on reviewers report. This is my biggest disappointment – from a personal point of view. For my professional use cases it’s not so crucial. The other advantages (below) are more important and I wouldn’t want to have a much bigger form factor or go for more bulkiness just for a few percent of fov more. Visual Quality? Naaa, kind of half-check!

Interaction

The biggest update that happens on the glasses! To start: I was hoping for eye tracking (check!). Microsoft integrates it to measure your IPD automatically and adapt rendering and stereoscopic effect for your need. This seems like a good idea when leaving out the multi-focal plane approach (as I guessed). It could help to make the stereo view more comfortable, although our children’s eyes still get destroyed and they will only be able to accommodate near-eye screens in the future. Well, you gotta go with the flow! No, seriously, eye tracking is a huge deal for foveated rendering, avatar simulation (with natural eye movement) but also for interaction.

I don’t really like gaze interaction and I’ve tried a lot of Tobii systems, etc. It always feels like the machine is dictating what I have to do. It feels as if the threshold for interaction is too low this way. Julia Schwarz (or was it Julia White?) presented this concept in a browser to scroll automatically when reaching the lower part of the virtual screen. It seems to work, but let me opt-out. Interface designers sure have a lot to test and check with the HoloLens 2!

This is especially true on the new full hand and finger tracking! Finally we have what we were all hoping for! Full gestures and direct manipulation of 3D objects. We can forget about the silly pinch gesture. After all, it was only a lame port of a mouse click into 3D that didn’t make sense (is there really an X in the upper corner to close a window?). It was only an interim solution. Now we finally got rid of it! Yes!

The demos we’ve seen are promising, though only demos at first. Tech demos of UX. Playing a virtual piano (maybe not enough fine control for a concert, but for some fun tunes), pinching virtual sliders, move around virtual image search results of jellyfish, … This is the fun part of the demo on stage and I really enjoyed it! Microsoft marketing speech is still silly, talking of “feeling the touch”, that is of course impossible when refering to a haptics sensation. But they did a good job to use visual and audio cues to give feedback on collisions with your hand, etc. Still, I was hoping for a 6-DOF controller for more (incl. vibration feedback). But I do see the advantage of a hands-free interaction as they want to promote it. If an assembly line worker needs his or her hands free: the clicker is a no-go. Until now, the clicker was a must-have for first-time users. If gestures work as solid as seen: no further need!

The only thing missing here seems to be a nice occlusion with your own hands. The Meta glasses showed us how it could look like. Why didn’t Microsoft include it in their demo? Technically it must be possible with their Kinect Azure. Is the effect still too bad? We’ve seen the hand reconstruction mesh at the beginning of the show. Why not use it as an occlusion mesh to cut out pieces of virtual objects? I’m sure this would help more on immersion than it breaks. But I’m pretty positive that we can activate it as developers…

Smarter Software & Connectivity

Speaking about software! I was hoping for updated real-time meshing and AI-based machine learning for a better understanding of the world. What’s a table, what’s a person? In an interview with The Verge, Alex Kipman talked about an improved meshing and semantic learning of the world and it’s objects in it. I was waiting for the big bang and the demo that shows us object tracking (like we’ve seen during last year’s ISMAR), but somehow there was no time for it on stage. Or did Microsoft didn’t do anything in this field? Why do they promote the feature and mention it on stage, but don’t show it? This would open up so many use cases! Feels like a huge fail and that Microsoft ran out of time on software side. This will probably be delivered afterwards, but this was my second disappointment. But they DID deliver updates regarding the spatial understanding and that’s on the bright side:

New services to grow AR

Microsoft presents a step forward (too early to judge if it’s revolutionary): a step towards a connected and shared mixed reality space. A digital twin of the real world to be superimposed by the magicverse layer holographic layer. Kipman talks about the birth hour of the internet of holograms. They tie in their Azure cloud services heavily and want to offer way more on the software side to connect all experiences and support AR spaces better and easier for the people to use.

The first piece are shared anchors (I was hoping for, check) like we’ve seen in a simpleton version on ARKit and ARCore devices before. Microsoft promised to create persistent world anchors, i.e. space representations, that can be uploaded to the Azure cloud and shared with others – even on other devices (iOS with ARKit, Android with ARCore and of course other HoloLenses) to have a mixed-device AR experience together. A dedicated developer page explains more on it.

Time will tell how easy this is to use, but it was overdue. Every start-up and IT house created their own backend to share experiences with others in real-time (we did, too). Now, an out-of-the-box solution from the operating system makes sense. Overall, Microsoft does a good job in trying to establish themselves as the new OS of MR!

But at the same time Kipman states to support openess and that they want AR to become “cultural relevant” for years and decades to come. The future needs to be open! Tim Sweeney joins the stage later on with his Unreal Engine claim “if you love something, set it free…” on his shirt. Totally support here. Let’s do it right for the “new internet”. The spatial AR layer should work for all – independent from their wallet size or device on their nose. Though, you can have your doubts. Microsoft nevertheless states clear commitments to support this. From open app stores (you can have your own within MS MR space), open web browsing (Mozilla Firefox joining in), open platform support (OpenXR, hardware agnostic approaches).

This would be huge if it came true and industries would join forces to reach this – not only for a cloud (hello my friends over at openARCloud!) but for more ecosystem relevant items around. Maybe Microsoft lives up to this ethics? They could still win commercially if they just offered the best service around…

E.g. they showed their Azure Cloud rendering for HoloLens 2. This was huge! Imagine there is no need to strip down your high poly count and textures anymore. Maybe we even get real-time Global Illumination and Raytracing in general. A very smart thing and would be awesome on the go (if we had 5G with the glasses).

So, speaking about software around: they ship additional services like Dynamics 365 Remote Assist, Guides or Layouts to support everyday MR tasks. This gives headaches to a lot of people (selling their own tools for this), but in the end – makes AR more accessible and easier to use. Smart in combination with the easy multi-user sharing. A great move to try to integrate augmented reality into our lives and current devices as standard tools.

I can sure imagine telepresence sessions with the new HoloLens. We’ve seen demos in the past from different teams. Microsoft’s partner Spatial does a good demo on stage showing (easily created 3D) avatars of the remote person including eye blinks (thanks eye tracking!). If it’s that easy for DEVs and users in the future: way to go, MS!

Micorosoft shows off other partner demos (find them all on their channel). We see mixed business scenarios like from Philips for medical use, but also Epic’s great Tim Sweeney with an unworthy teleprompter praise. Kipman clearly stated that it will stay B2B “for now”. But envisions a “human wave of computing” for the future to come.

Verdict

So. Verdict? Too early with no hands-on, without having tried out the Azure software package around. But we can judge about a few things. First, I love the hand and gestures integration (finally!). Once we have it in our hands, I’m sure everybody will say: “why did they dare to deliver HoloLens 1 without it?!”. This could be huge. Though, I’m not yet convinced that it will allow a fine-enough virtual keyboard to air-type like in Dennou Coil. The updated form factor feels adequate and the flip-up great. As stated the field of view feels lame (without having tried it), eye tracking is awesome, iris scan a fun feature. The spatial anchors, the (cloud) services around and the beginning of Microsoft version of the AR cloud is an important move. Maybe the biggest news. But the missing demo of object recognition and semantic understanding of the world was really odd and sad. (As-it-seems-BFF PTC Vuforia could have shown something on it on stage, maybe? Why didn’t they?)

So, in short. It feels like a solid update, that’s def. worth upgrading for your B2B scenario. Especially for going for a better interaction by gestures that was long overdue. Missing 5G connectivity is lame and leaves out a lot of use cases – just like the missing object recognition. Software-side they move forward wisely, giving a great update for multi-user scenarios. A must-have for developers that want to do their own R&D – with an off-the-shelf product and a more user-friendly service in the back, theoretically. We will find out, if it works “out of the box” and improves use cases in actual working environments. So, overall, a nice evolution that’s down to earth, no overhyped revolution, not for consumers as of today. Not a big bang. But worthy along the way to everyday AR. Let’s get coding and move forward for the next one while learning together!

What are your thoughts? Let me know on twitter!