What is Spatial Computing? A basic explicator

Spatial computing is a technology that allows computers to merge with the physical world in a natural way. Apple isn’t the first company to delve into the space, but it believes this will be the next big thing for computing. And the company is willing to bet on it with its new AR/VR headset, the Apple Vision Pro. Here’s our introduction to spatial computing and our thoughts on how it will change the way we interact with computers.

---

How does spatial computation work?

Spatial calculation brings together the virtual and physical worlds in a fluid way. You can experience this through headsets like the aforementioned Apple Vision Pro, Microsoft HoloLens, Meta Quest Pro, and Magic Leap. These devices show the real world and, at the same time, incorporate real objects into the scene in a way that appears three-dimensional. For example, you could place a virtual piece of furniture in your living room to see what it would look like before you buy it; or workers can overlay technical manuals on the machine they are building.

These devices rely on different technologies to mix the digital and physical worlds. They use computer vision, which processes data from cameras and other sensors and captures visual information about the environment, including the location, orientation, and movement of objects. With sensor fusion(Opens in a new window), can combine data from multiple sensors, such as cameras and LiDARs, to create an accurate and comprehensive view of the environment. And they use spatial mapping(Opens in a new window) to create a 3D model of the environment, which allows for more precise positioning and manipulation of digital content

Apple Vision Pro headset (Credit: Apple)

One of the major benefits of spatial computing is its ability to understand the depth of the environment. This allows for realistic and natural interactions with virtual objects, because they can be positioned and manipulated to match the physical world. For example, a virtual object can be placed on a table, moved, or hidden behind other objects to mirror our actions in the real world.

Spatial computing devices have features that allow you to interact with virtual objects. Eye tracking technology monitors your gaze, and handheld controllers and motion sensors let you manipulate virtual objects. For example, the Apple Vision Pro has advanced eye tracking and hand gesture sensors that let you simply look at objects to activate them, or move them around by pinching your fingers or flicking your wrist.

Devices that use spatial computation might also have speech recognition capabilities to support voice commands. These are particularly useful in practical environments such as manufacturing, where hand gestures or controllers cannot be used. Vision Pro, HoloLens and Magic Leap support voice commands.

Magic Leap 2 headphones (Credit: Magic Leap)

Spatial computing is related to both augmented reality (AR) and virtual reality (VR). AR means overlaying digital content onto the real world, typically using a phone or smart glasses. While AR can enhance the user’s perception of the environment, it does not embed digital content in 3D space or provide a sense of depth. VR, on the other hand, creates a fully immersive digital environment that replaces the user’s physical environment. And mixed reality (MR) uses a combination of AR and VR.

---

How can we use spatial computation?

(Credit: Digital Today 2023)

Spatial computing has the potential to revolutionize various industries by creating experiences and applications that were previously impossible. Here are some of the domains where spatial computation is applicable.

Game and entertainment

Spatial computing creates immersive gaming and entertainment experiences that let you interact with virtual objects and characters in a natural and intuitive way. For example, instead of a keyboard or joystick, you might use hand gestures or a gaze to control avatars and manipulate objects in a game. Spatial computing can also make you experience real-world events in a different way. With Apple Vision Pro, you can watch an NBA game that feels like you’re in a courtside seat. It can also record memories in a spatial format that you can later relive in 3D.

Education and training

Spatial computing can provide an engaging and interactive learning experience that enhances knowledge retention and skill acquisition. For example, medical students can use spatial computation to practice surgical procedures in a virtual environment that simulates real-world conditions. In science and engineering, students can use spatial computing to build and test virtual prototypes of machines or conduct experiments.

Architecture and Design

Spatial computing can allow architects to create, visualize and modify designs in the real world without the need for physical prototypes. This can save time and money and allow for more iterative design processes. Designers can also use spatial computing to create virtual prototypes of products and test their functionality and ergonomics in different physical contexts.

Health care

Spatial computing can provide new ways to diagnose, treat and monitor patients. For example, doctors could use spatial computation to overlay virtual screens and diagnostic information onto the real world, helping them make more informed decisions. Or, a doctor might view a patient’s medical image scans through a headset while they’re operating on that patient. Spatial computing could also help patients with physical or cognitive disabilities regain their independence by providing them with virtual assistants or rehabilitation exercises tailored to their needs.

Beyond these domains, spatial computing could have applications in fields such as retail, manufacturing, and transportation. For example, manufacturers could use spatial computing to augment assembly lines with useful virtual objects and information to optimize manufacturing processes.

---

The future of space computing

Despite its allure, spatial computing has had mixed success so far. One of the biggest challenges facing the field is the cost of the hardware. The best spatial computing devices cost thousands of dollars and are not affordable for everyone. This is why they have yet to grapple with the consumer space and have seen limited success in niche business applications where the increased productivity justifies the cost.

Additionally, the headset’s weight, low resolution, and lack of comfort were major concerns, as they can cause fatigue and motion sickness during extended use. Battery life is also a limiting factor: most device batteries last only a few hours and require frequent recharging. But companies are making progress in many of these areas. And as other industries have shown, the cost of sensors, displays, and hardware will gradually decrease as the industry evolves and becomes more competitive.

(Credit: Apple)

The jury is still out on whether Apple’s entry into space computing will be a bust or a defining moment for space computing. But Apple has a history of setting standards: just consider the mouse, the graphical user interface, the iPod and the iPhone. While extremely expensive for now, Vision Pro could spark momentum that will lead to innovative new applications and devices for spatial computing.