GoPro accelerates mobile AI development with Qualcomm AI Hub

Picture being mid-air, moments away from completing a backflip off a cliff.  The adrenaline is pumping, the view is epic, you are laser-focused on your jump. For creators chasing these high-intensity moments, capturing the action shouldn’t be a concern.

GoPro’s Quik mobile app is designed for fast-paced editing, helping users frame, trim, and transform raw footage after capturing their adventures into shareable stories. All directly from their smartphone.

To support these fast-editing workflows, GoPro turned to Qualcomm AI Hub to integrate real-time on-device object tracking. This feature keeps subjects in frame in a cinematic way without relying on cloud connectivity, resulting in faster edits, smoother performance, and better battery efficiency across various smartphones. With AI models running on-device, GoPro creators can track any object, no matter where the adventure takes them. 

Let’s take a closer look at GoPro’s developer journey using Qualcomm AI Hub to bring this feature to life. 

 

Simplifying Object Tracking for Creators

GoPro’s user base spans from professional videographers to weekend adventurers looking to turn one-of-a-kind moments into action packed videos. For these users, seamless and efficient editing is essential, especially when working with fast, unpredictable footage. Object tracking helps simplify the process by automatically keeping the subject in frame, eliminating the need for manual reframing.

To meet this demand, GoPro integrated an object tracking feature into the Quik app that allows users to select a subject, whether it’s a person, a bike, or a snowboarder, and automatically keep the view centered throughout the clip. This makes it significantly easier and faster to edit dynamic footage. The experience is seamless: users open the Quik app, select their video, and the AI model takes it from there, automatically reframing around the object in a cinematic way, using tracking data and other techniques to smooth out the video, all while tracking the object as it moves. 

At the core of this capability is a transformer-based model, designed for complex visual reasoning. Transformers excel at processing hierarchical information in real-world environments, like tracking a human even if they’re located behind a tree branch or under water. It was important to design a model that was fast enough to process every decoded frame, without missing a beat.

GoPro’s developer workflow with Qualcomm AI Hub

Model conversion and export:

GoPro previously converted models between frameworks manually, which was slow and error prone. Using Qualcomm AI Hub, the team compiled models from PyTorch to TensorFlow Lite (LiteRT), speeding up iteration. This improved inference time, allowing GoPro to deploy the model across more devices, which was otherwise not possible.

Device benchmarking and performance validation:

Using Qualcomm AI Hub’s cloud-based tools, GoPro tested key performance metrics like latency and accuracy, across a wide range of real devices. The team evaluated models on different compute units (CPU, GPU, NPU) to determine the best fit for each target platform.

Speed and usability improvements:

Some models had previously failed to run at acceptable speeds on certain devices. By leveraging AI Hub’s optimization capabilities, GoPro improved model performance across various hardware, achieving faster inference times and enabling broader device support, allowing users to enjoy smooth and responsive tracking features, no matter the device they’re using.

Streamlined integration:

Qualcomm AI Hub streamlined the integration process, allowing GoPro to continuously test and refine models. This led to faster development cycles and more responsive features, enhancing the overall user experience. 

Unlocking performance on Qualcomm Devices

To push performance even further, GoPro used the Qualcomm Neural Processing SDK (QNN SDK) from the Qualcomm AI Stack. By accessing the Qualcomm Hexagon NPU, GoPro went beyond their initial GPU and NNAPI delegate performance on device and were able to unlock additional performance, by running on the NPU.

With both Qualcomm AI Hub and the Qualcomm Neural Processing SDK, tracking features are not only smooth and responsive; but they are also key for real time editing experiences.

Looking ahead

With on-device object tracking successfully deployed, GoPro continues to explore new use cases using Qualcomm AI Hub’s growing model library. AI Hub models showcase state of the art capabilities, quickly validating prototypes and accelerating the development of customized features. This cloud-based tool allows the GoPro team to continue evaluating models for video enhancement and experiment with personalized features that tailor editing experiences for each user. 

“AI Hub gives us the ability to prototype, validate, and optimize models quickly. We can confidently deploy features that work across mobile platforms, all while maintaining a high bar for quality and performance.” – Rawia Mhiri, Senior Engineering Manager at GoPro

Take the next step

Learn more about GoPro and Qualcomm AI Hub

Explore how Qualcomm AI Hub can support your AI development by checking out the latest models, sign up to bring your own model and reference the documentation

Join the Qualcomm AI Hub Slack community with any questions as you get started optimizing your model.