How holograms are visualized in your real life environment

Written by Wouter Spaas

The HoloLens 2 has a lot of amazing features. In this blog post we explore the core principles of one of its unique and powerful tricks: Azure Spatial Anchors.

The core concept

Spatial Anchors allow users to place virtual objects in their real life environment. The position of those objects are stored in the cloud. This enables users, who are granted the required access, to see the objects in the exact same position.

How it works

Obviously there is a lot more to this advanced system. Let’s dive into a little more detail on how it works without getting too technical.

In its very essence, a spatial anchor is a point in a 3D coördinate system. This 3D coördinate system is an invisible 3 dimensional ‘grid’ around the user. In all 3D applications, a Cartesian coordinate system is used, which means each point in space is defined by an x, y and z coordinate. In order to persistently maintain the position of the coordinates of the grid, and therefore the spatial anchors, it needs a point of reference.

If the grid would not have a fixed reference, its only reference is the user itself. In that case the grid, including all spatial anchors, would be floating; the grid would stay in exactly the same position relative to the user, instead of the actual environment. This is called an ‘attached frame of reference’ or ‘body-locked content’. This is not the desired behavior, since we want to place objects in our real life environment (ex: a hologram of a coffee mug on a real life table). In order to achieve this, the HoloLens 2 has a feature called ‘Spatial Mapping’. At this moment, we are not going into detail, but what you need to understand about Spatial Mapping is that it generates a 3D representation of the real life environment. Basically, the HoloLens makes a 3D scan of your environment and aligns the resulting model precisely to the objects in your real life environment.

When the user places a spatial anchor, it can be saved relative to the spatial map, which in its turn is anchored to the real world. There you have it; the spatial anchors stay in place which means any objects placed on these anchors will always appear in exactly the same position. Right? Basically yes, but not entirely…

Previously, we mentioned all spatial anchors are placed in a 3 dimensional grid. This is only partly true. There is not one grid, but many. Using one grid is sufficient if the application takes place in a rather small environment; no more than 5x5m. But if you want to place objects in the world regardless of their distance, one fixed coordinate system is not enough. That is because our real life environment is dynamic; objects move or change position, take for instance a chair that can be moved around the living room.

Spatial Anchors allow users to place virtual objects in their real life environment.

 

This implicates that the spatial map will change as well, since it’s generated based on your (live) environment.  If we would be using only one giant grid for all spatial anchors, this could cause certain anchors to drift away from their original position. Take spatial anchors positioned inside an entire building for example. If one floor changes, let’s say tables are moved, the entire spatial map is regenerated. Some spatial anchors will still be located just fine, but anchors on a different floor could end up having a huge offset. That is where multiple grids come in. When each spatial anchor gets its own reference to the real world, any changes to the environment will only affect the anchors related to that specific change.

Conclusion

We have only covered the very basics of Spatial Anchors. There is still a lot more to explore, but we hope to have inspired you to start dreaming of the immense opportunities Spatial Anchors and the HoloLens can bring.