Why generative AI is the catalyst that mixed reality needs

The promise of mixed reality fundamentally changing the way we interact and live our lives has always been an alluring one. But the technology has long faced challenges in building the kinds of immersive worlds that would meet that promise. 

That’s where generative artificial intelligence (GenAI) comes in.

With its rise to prominence over the last two years, GenAI is poised to disrupt many industries. But perhaps no area stands to benefit more from how AI enables us to create something out of virtually nothing than mixed reality. From creating elements of a virtual world on the fly or rendering a realistic digital avatar of yourself in real time, AI is the spark that could turn mixed reality into the kind of place you’d want to spend more time in.  

A lot of this potential can’t be realized unless these GenAI capabilities are banked into the devices, whether it’s your smartphone, smart glasses or virtual reality headset. That’s why at Qualcomm Technologies, Inc., we’re continuously improving the power, performance and on-device capabilities of the processors running these devices — all with an eye toward fulfilling the promise of mixed reality. 

That’s why AI and its impact on mixed reality was one of the highlights of Tuesday’s Qualcomm Investor Day presentation.

Here are two areas where GenAI can act as a catalyst to mixed reality experiences.

Creating rich worlds

Mixed reality gaming is an area that’s ripe for the kind of varied content that GenAI can create. 

Multiplayer games like Fortnite need to be meticulously crafted by developers, while games like Minecraft let you create items, buildings and vehicles, but only through dedicating tremendous effort and meticulous attention. 

Filling any kind of immersive virtual reality game world is even more difficult. A lot of early virtual spaces in the past have largely been barren places with a few elements. 

GenAI can liven things up. Imagine creating your own rich 3D environment in virtual or augmented reality (AR) using nothing but a voice command. With multimodal inputs that understand written commands, voice prompts or photo inputs, you have the flexibility to create the world around you and the elements within it. 

You want to create a sword inspired by your favorite fantasy novel? Or a ninja outfit from that hit anime? Or construct a digital home complete with personalized furnishings? Just speak a few words to your digital assistant and let the GenAI system work its magic. We can utilize our core AI research where we have enabled models such as stable diffusion running on our products and producing images from text prompts in less than a second. Now, text-to-3D and text-to-video models are emerging that our unique AI solutions can enable entirely on-device.

A true digital twin

We’re not at the point where we can converse with each other via holograms like your favorite sci-fi show or comic book film, but with AI and mixed reality, we’re closer than you think. 

A realistic scenario involves sitting in an empty conference room and donning a pair of smart glasses that project hyper-realistic avatars of the other participants on a call onto the seats around the table. We already have digital avatars in virtual and augmented spaces, but they’re often cartoonish interpretations. GenAI can help generate hyper-real versions of yourself on the fly.

Or perhaps you want to sit in a completely different virtual, but photo-real setting. You can use techniques like Neural Radiance Fields and increasingly Gaussian splatting, which create 3D views out of 2D pictures, to sit anywhere from a country club in Florida to the penthouse of a skyrise building in Dubai.

The idea of an immersive group call via telepresence has been around for decades, but GenAI and AR will finally bring its promise to life. 

Taking the avatar idea further is the use of GenAI to create a true digital twin that can be sent off to meetings in your stead. That digital twin, utilizing LLMs trained on your emails, documents and other preferences, can interact with others as you — even saying words in your accent and voice. 

It doesn’t have to be in a conference setting. In the medical world, a virtual AI-powered remote “doctor” could converse with a patient utilizing the same kind of knowledge and experience of the actual doctor. It’s an illustration of the potentially broad applications for these capabilities.

Providing the spark

Qualcomm Technologies understands the immense value these kinds of experiences bring, and we’ve been busy setting the foundation for their development. 

It starts with the Snapdragon XR2+ Gen 2 Platform, which brings spatial computing capabilities to devices including features like 4K resolution, the ability to run a dozen or more concurrent cameras, ultra-fast connectivity features like Wi-Fi 7, all while paired with on-device NPU for AI use cases and a powerful GPU and CPU. 

Then there’s Snapdragon Spaces XR Platform, an integrated platform and ecosystem for XR-based device makers and developers that gives them the tools to create spatial computing devices and experiences. 

One example includes the ability to turn your room into an AR museum, with blank canvases and sculptures around you ready to display your creations. You can harness the power of AI to speak your creations to life and when you’re done, they can be shared as an AR experience for someone else with their own pair of smart glasses. 

The mixed reality world is not just generated on the fly but is personalized for each one of us based on our likes and dislikes and contextual information that is available on edge devices such as MR headsets.  This is just the beginning of a revolution that’s slowly building thanks to the advances in AI and mixed reality and the innovations from Qualcomm Technologies. 

If you want to learn more about how AI is going to enhance AR, you can read my blog post on how generative AI will change smart glasses for the better.