I have been doing HoloLens for a while now, and I am continually amazed of the ideas and projects the community is creating. A lot of the best ideas evolve from the desire to interact with digital assets in a physical world, which is what mixed reality (MR) is all about. This is enabled through spatial mapping, which is easy to use and difficult to master. This article first published at Pluralsight.
Interest in HoloLens, and digital reality in general, is growing rapidly. And a large part of that interest comes from developers wanting to know how to build software for HoloLens, the mixed reality device from Microsoft.
In a previous post, Holographic programming: a HoloLens how-to in a mixed reality world, I gave an overview of what HoloLens is and where it fits in the world of augmented and virtual reality. I also introduced the concepts of gaze, gestures, voice and audio. In this post, I’ll focus on arguably the most important part of the HoloLens experience: spatial mapping.
Capturing spatial awareness
The HoloLens uses four environmental cameras on the front of the device to map the physical surroundings and build up a 3D model of the real world. As you use the device, it continually creates a spatial mapping model of the space you are in and updates any existing mapping. The really cool thing is that the developer portal that comes with the developer tools allows you to see this 3D spatial mapping in real time, and it works for both physical HoloLens devices as well as the emulator.
Make sure your device is on the same physical network as your computer, then access the developer portal by entering your device’s IP address in any browser. In the developer portal, go to the 3D View menu option on the left, which will bring up an empty plane. Press the Update button and the current state of the 3D mapping model is drawn on the plane as seen below.
3D view of the spatial mapping
The 3D view will also show spatial anchors as they are attached to objects in the digital world. Furthermore, you can save 3D spatial mapping models, and then share them between emulators to have a realistic model for app development and testing scenarios.
Spatial mapping purpose
You might be asking yourself why it’s so important to manage spatial mapping integration. There are several purposes of spatial mapping and each is a critical part of creating a successful mixed reality experience.
Occlusion
One of the simplest uses of spatial mapping, and one of the most important ones to create a natural interaction model, is to simply occlude holograms. By hiding holograms, or obscuring parts of them, you increase the perceived realism dramatically. You also create an expectation that holograms will collide with physical objects when they are solid. All together this means simple occlusion is of great importance to effectively convey a realistic integration of physical and digital realities.
Visualization
At times, it may be necessary to visualize physical surfaces and structures to allow holograms to be placed on them accurately and realistically. You might also want to have holograms casting shadows on surfaces as well.
Placement
To place holograms in a physical world, the HoloLens relies on spatial mapping to determine distance, space available and usable surfaces. One of the concessions of using hand gestures to infer distance is that we normally only communicate direction with gestures. When this needs to be translated into distance as well, spatial mapping can help indicate to the user where holograms can be placed. Combining this with the shape of the hologram, placement adds another natural aspect to mixed reality.
Physics
Applying gravity and other expected physics behaviors to holograms, such as balls that bounce off walls and floors, is about making users believe in the holograms as part of the real world. To have realistic physics behaviors, a complete spatial mapping is important and you might have to get your users to scan the room before you can proceed.
Navigation
Often you want holograms to navigate your physical space, as if they were part of it. This requires accurate spatial mapping to make sure you don’t have people walk through furniture and jump through walls. This also requires being able to identify surfaces, so people only walk on the floor and only sit on chairs and couches.
Implementing spatial mapping
Using spatial mapping in your HoloLens application is more simple than you might think. The first thing you do is enable the SpatialPerception capability for your Unity project. This will make 3D objects adhere to the 3D spatial mapping model once you export the project from Unity to Visual Studio and in turn, a HoloLens device.
Adding SpatialPerception in Unity
One of the best and easiest ways to get started with spatial mapping, and HoloLens in general, is using the HoloToolkit. This is a collection of scripts and components intended to accelerate development of holographic applications targeting Windows Holographic. It also gives you shortcuts to some of the “plumbing” tasks, such as setting up a new project and enabling VR in Unity.
The HoloToolkit comes with several scripts that, among other things, manage the number of polygons you want to render for objects, controls the spatial mapping renderer and much more.
Want to know more about HoloLens development?
If you’re looking to get more in-depth insight on how to build successful mixed reality apps and take advantage of the platform in comparison to virtual and augmented reality, check out this on-demand webinar I did for Pluralsight a couple of months ago, or check out my course on it.