Three ways our AI is powering awe-inspiring XR experiences
Today, smartphones, social media, and the internet facilitate the way we work, play, communicate and learn.
However, these experiences are limited to two-dimensional (2D) screens, which can prevent people from being fully focused and fully immersed.
Spatial computing, commonly known as the metaverse, is a three-dimensional (3D) version of the internet where people can interact, work and play. And augmented and virtual reality (AR and VR) devices make it possible.
In addition to engineering Snapdragon XR platforms — we bring together the ecosystem of manufacturers and app developers to leverage the power of artificial intelligence (AI) and build technologies that enable the next generation of immersive experiences. Here are the three ways how:
1. Perception: Qualcomm AI enables machines to comprehend the world like humans do
Immersive technology lets you realistically experience the world in 3D with minimal use of mediums like a controller, a mouse pad or a touchscreen. For this to happen in AR or VR, your device or glasses need to perceive and understand the world around you so it can interact with the virtual content.
Imagine you’re playing Pokémon Go, but instead of capturing Pokémons through your smartphone, you see their holograms in the street through AR glasses. You’re wearing AR glasses that capture information such as your location, how fast you’re running and where you’re looking at, so that the Pokémon you’re trying to chase is slightly faster than you, can hide behind a real wall or jump over a real fence.
Our perception technologies, including spatial mapping, object recognition and hand tracking, among others, are powered by powerful AI algorithms that process images and data captured by sensors and cameras on your glasses.
Called perception algorithms, they understand two concepts:
- Your environment as you move — using spatial mapping, 3D reconstruction or object recognition technology — and,
- Your actions — using head motion, hand tracking and eye tracking technology.
2. Performance: Our AI — perception algorithms — accelerates tasks that would take too long otherwise
Enabling immersive experiences requires capturing and processing a lot of information, such as location in space, speed, acceleration, identification of surfaces and objects, hand recognition and movement tracking. This information needs to be captured, processed and rendered at least 30 times per second (this is how many times an average camera sends images and information to the AI for processing).
This information is resource-intensive to process. However, running the AI algorithms on our Qualcomm AI Engine allows us to process information faster, smarter, and even anticipate what’s next.
Why does this matter?
Perception algorithms enable AR and VR headsets to utilize less resources to process data and render content. As a result, headsets and glasses feel light and comfortable.
For example, the Snapdragon AR2 processor includes a custom-built engine for visual analytics and Qualcomm Hexagon Processor. Combined with other IP innovations, this helps to accelerate AI-intensive tasks such as image recognition, reclassification and hand tracking by 2.5 times compared to the previous platform, while reducing the processing power by 50%.
In addition, perception algorithms enable data processing fast enough to bring you immersive experiences that replicate the rule of physics your brain is comfortable with, without making you feel sick.
AI algorithms can anticipate where you are going to turn your head. This way, the AI doesn’t have to wait to receive the data and process it, it simply anticipates you will turn your head. Without these capabilities, your AR or VR experience would have glitches, or lags — which would make you feel nauseous quickly.
3. Content: Our AI helps generate content
AI plays a huge role in making virtual objects and environments look photorealistic. For example, let’s say you want to experience a gaming scene in high-resolution VR. In this scenario, normally, you would need to be tethered to a computer or your battery would die quickly. However, with our AI-powered feature called dynamic foveated rendering, you can have the quality you want, while using less power. This is how you can enjoy the experience freely on a standalone wireless XR headset.
How?
AI is enabled to mimic a human’s vision: detailed in the center of your field of vision and blurry on the periphery. By identifying where you’re looking, AI only renders the image in high resolution where it needs to and recreates what your eye would consider a natural peripheral vision. You don’t notice it, but it saves a huge amount of power so AR and VR headsets or glasses are light, stay cool to the touch, and allow you to use them longer.
AI can also be used to generate 3D content from scratch using a text prompt or even an idea you simply express through your voice.
For now, generative AI has shown its success on 2D images or text, but soon it will be possible to create a 3D model right out of your imagination.
Welcome to the XR edge of possible
As you can see, we’re excited about how AI is helping us reimagine reality as we know it. Simply put: AI can make everyday tasks more accessible, easier and more engaging. And our AI unleashes the power of immersive technology to transform day-to-day work, play and learning into extraordinary experiences.
It is fascinating how AI revolutionizes spatial computing and permeates our lives, and it’s why we build technology in the first place: to bring humanity together by engineering human progress.
