larsklint.com Microsoft MVP, author, speaker, Azure geek

Most people are aware of virtual reality (VR) and to a lesser extent augmented reality (AR). I the last couple of years a new digital reality, mixed reality (MR), has emerged with Microsoft and their HoloLens as the front runner. I have been absolutely blown away by the capabilities and promise of the technology, hence the activity in the form of blog posts, conference talks, workshops and Pluralsight courses. This article explains the various digital realities and was first published by Pluralsight.

In the last couple of years, digital realities have become a more and more stable part of the software landscape. While these technologies aren’t as mainstream as apps for your iPhone or a website for your favorite pizza place, they certainly are pushing their way forward and popping up their digitally distorted face in all kinds of places. Companies like Oculus, Facebook, Google and Microsoft are all heavily involved in developing digital realities for the future of computing. We are talking billions of dollars invested in these projects, which mean as developers and consumers, we should take these products and technologies seriously. They aren’t going anywhere.

However, they aren’t all created equal. In particular, there are three main areas of digital reality.

The essence of VR has been around since the 80s, where the promise was great and the implementation less so. I tried a game called Dactyl Nightmare in the early 1990s and I was blown away.

Virtual reality is a complete virtual world that someone has designed and built for the user. It encapsulates everything around you and removes the physical world completely. This also means that users are confined to sitting or standing still, as they can’t see where they are going (it’s amusing for the spectators though).

The majority of use cases I have seen for VR are either gaming or simulation, and VR lends itself well to prototyping as you have full control over the environment and reality the user experiences.

Often involving a token, shape or other physical object, AR is the idea of taking your reality and altering it slightly. It could be a kids book coming to life when looking at it through your phone’s camera or a t-shirt showing the latest movie when activated through an app. It is an overlay of data, often activated through a mobile device.

The new type of digital reality that Microsoft has named Mixed reality is where HoloLens comes in. Mixed reality is an overlay of your entire physical reality that encourages natural interaction and integration between the digital and the physical worlds. This means that users use natural interactions to manipulate the digital world and they can be in any room, using any surface.

The most unexpected, disruptive and groundbreaking technology to come out of Microsoft for many years is the HoloLens. As devices are now finally starting to ship to developers, the developer tools have been released. And the good news is that you don’t need a physical device to start creating amazing 3D holograms in a mixed reality world. And even better, the tools for creating apps for HoloLens are all completely free. However, before you jump straight into creating holographic kittens there are a few fundamental building blocks you need to have under control and understand.

The gaze is the first form of input into HoloLens apps and it is how you put focus on holograms. It is the center of the field of view when you look through the HoloLens, and it is essentially your “mouse cursor.” You can design this cursor any way you want, so it could be contextual to what your app does, or it could be your company logo or any other 3D shape.

It is important to bear in mind that HoloLens uses the position and orientation of your user’s head, not their eyes, to determine their gaze vector. It is a bit like a laser pointer that comes straight out from the center of the HoloLens.

Once you have targeted an object with gaze that you wish to interact with, you can use gestures to perform the actual interaction. The most common gesture, and one that comes free, is the “tap.” This is a bit like clicking the left mouse button to invoke some action on the object. It is possible to perform more complex gestures as well, such as “tap and hold” to move objects in 3D space.

The last of the three main input methods, and my favorite, is voice. Users can use their voice to interact with all holograms, as well as the “See it. Say it.” rule (if you can see a word inside the HoloLens world, then you can say it). Developers can also design their own voice commands. Often voice commands can feel like shortcuts to users, as they only have to say a single command. The HoloLens uses advanced speech recognition to let developers choose any commands they choose. It works remarkably well, but it is definitely recommended to follow the speech design guidelines.

The last major building block is sound. As the HoloLens field of view is somewhat limited, you can design sound that indicate direction and proximity to the user, so they will turn towards the sound and interact with the object it is coming from. HoloLens is using Head Related Transfer Function (HRTF), which simulates how human ears hear sound. Four individual speakers create 3D sound that simulates your 3D holographic world. The sound is spatial and as a developer you can attach sounds to objects, events and any interaction you need. The HoloLens SDK takes care of the rest.

The hardest part I have found regarding development for HoloLens is to get decent 3D models. I am not a designer or 3D artist by any means, so, for me, creating models consists mainly of spheres and boxes. Not terribly exciting. To create convincing and high quality apps for HoloLens, you definitely need some form of 3D skills, whether they are yours or someone else’s.

Your main workspace will be as much Unity 3D as it will be Visual Studio. I highly recommend becoming familiar with both tools equally, especially if you are working on your own. Unity comes in many flavors, depending on your target platform, and there are good learning resources for it.

There are also a number of online services you can take advantage of to buy either custom made or already finished 3D models.

Let’s not forget that HoloLens runs Windows 10, which means it supports the Universal Windows Platform. You can write apps that work across all Windows 10 devices (check out my webinar on building Windows apps for Windows 10 here), using the same Core API. Your 2D apps that were built for mobile devices, desktop and all the other Windows 10 form factors, will also run on HoloLens. Obviously a 3D holographic world will not work on your phone, but the core business logic of it will.

Having Windows 10 as the backbone of all your apps makes your life as developer even easier, and allows you to target the full 300 million devices currently running the operating system.

Are you ready to get started building holograms and apps for HoloLens? Great! Download the tools as mentioned above and you can start by following the Microsoft tutorial. It is much simpler than you think.