Co-written with Aleksandra Krstic, Alex Bourd and Shuaib Arshad.

Suppose you came home from vacation with a few dozen photos from different perspectives of the Eiffel Tower or the Taj Mahal or Michelangelo’s David. What if you wanted to “walk” around them again, if only virtually? That would entail stitching all the 2D images together somehow to create a 3D scene you could view from different, freely placed points, as in the video below.

There are ways to synthesize your photos into a scene you can view from arbitrary perspectives. A new technique called MobileNeRF involves machine learning, and it’s designed for using graphics hardware on mobile devices, but the rendering is implemented in WebGL, an older API. While webGL is easy to use API, it was not designed for high performance applications and does not have fain grained control over GPU pipeline.

That’s why we’ve released a new implementation of MobileNeRF Rendering using Vulkan API and we’ve optimized it for the Qualcomm® Adreno™ GPU. In this post, we’ll explore the background of generating realistic, real-time 3D reconstructions from 2D images. The implementation is instructive for AI scientists who want to better understand GPU capabilities and for GPU architects who work with neural networks.

Inference with NeRF and MobileNeRF

Novel view synthesis is the stitching process of taking input images and synthesizing images of the same object or scene from arbitrary viewpoints. It’s a long-standing problem in graphics and computer vision, and a recent approach involves machine learning with neural radiance fields (NeRFs). A NeRF is a fully connected neural network that can generate novel views of complex 3D scenes or objects, based on a partial set of 2D images.

NeRFs use a multilayer perceptron (MLP) network and is an instance-specific, implicit representation of the scene or object. But performance is a problem. The original NeRF technique uses large MLPs during inference (rendering) and the resulting performance hit is on the order of 30 seconds per frame on high-performance desktop GPUs. Variations of NeRFs have emerged for different use cases, and for improving the training and/or inference process. In the quest to get closer to real-time inference, MobileNeRF has emerged as a technique that uses the existing graphics hardware on mobile devices.

On the Adreno team, we continually look for new use cases to bring to our GPU to see how they perform on it. That’s why we became interested in MobileNeRF. Adopting the original paper, we set out to optimize the technique to render MobileNeRF meshes more efficiently during inference. We also decided to extend MobileNeRF using Vulkan API on Adreno GPU.

Neural networks on GPU – It isn’t like gaming

Whereas NeRFs attempt to use the GPU in a way that they have not been used before, MobileNeRF tries to exploit the GPU more closely to the traditional way. The MobileNeRF training process puts out meshes and feature textures that traditional graphics hardware can process in a relatively straightforward manner.

But they are different from the meshes typical of games, which is another use case commonly run on GPU. We found that meshes and models generated by MobileNeRF tend to have very small triangles and high depth complexities, so they are not well suited for traditional GPU rasterizers. During the course of our work, we found that they cause certain performance bottlenecks on the GPU.

That’s a big lesson for engineers in both machine learning and GPUs. In the context of gaming, for instance, the games and the GPUs evolved to coexist. Game developers created content with GPU limitations in mind, and at the same time, GPU architects improved their designs to accommodate the new demands of games. But the performance bottlenecks we saw with MobileNeRF showed us that that has not yet happened with machine learning, so special care is required when rendering these assets on GPU.

From WebGL to Vulkan on Adreno

The authors of the MobileNeRF paper approached novel view synthesis from the perspectives of machine learning and computer vision. They chose to implement the rendering part in WebGL. We examined their rendering techniques from the perspective of GPU and implemented the rendering part of MobileNeRF in Vulkan, a more modern graphics API. Vulkan API was specifically designed for high performance applications and provides fine-grained control over GPU pipeline. This allows applications to better use GPU hardware.

We worked on the commercially available Adreno GPU on the Snapdragon® 8 Gen 2 mobile platform, which supports Vulkan 1.3. Compared to the original WebGL implementation, Vulkan API gives us more opportunities to optimize the rendering pipeline and shaders for a better fit on a modern GPU. We’ve been able to push performance even higher, which allows for rendering bigger, more complex objects and scenes.

Next steps

NeRFs and MobileNeRF have the potential to generate more detailed and realistic 3D reconstructions and visualizations in real time. That has implications for interactive experiences like augmented reality on mobile devices. We also see potential for integration between MobileNeRF and technologies like 5G, edge computing and AI-driven services; think medical imaging, enhanced navigation systems and virtual try-on in online shopping.

We’ve donated our implementation to Khronos as a Vulkan sample. You can download it and try it yourself, then let us know what you think in the Adreno GPU SDK support forum. Meanwhile, we continue our work on non-traditional use cases on GPU. That includes enabling other neural rendering techniques besides MobileNeRF to run more efficiently on Adreno GPU. Stay tuned, and subscribe to the Qualcomm Developer Network for updates.

Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.