Human expertise and robotic precision: how the University of Applied Sciences and Arts of Southern Switzerland uses Qualcomm AI Hub

The Fluently EU project aims to improve human-robot interaction in industrial sectors like manufacturing. One of its partner entities is The University of Applied Sciences and Arts of Southern Switzerland (SUPSI), whose engineers are using AI to help make industrial robots accessible to any human skill profile.

Robots and machines bring greater precision, strength and endurance to manufacturing tasks than humans do. At the same time, humans bring more expertise, judgement and foresight. For industrial and manufacturing applications, the Fluently team is developing a human-to-machine interface so that robots can better interpret human gestures, speech and behavior. Fluently is designed to foster true collaboration between humans and robots by accounting for the human cognitive and physical load. That allows robots not only to respond to commands but also to adapt their assistance based on the operator's condition. By doing so, Fluently ensures that the collaboration enhances the operator’s capabilities without overwhelming them.

Read on to learn how SUPSI engineers have used Qualcomm AI Hub, Qualcomm Innovators Development Kit, Snapdragon 8 Gen 2 SoC and Qualcomm AI Stack to shorten their development timeline and improve the performance of Fluently.

Ideal use cases: Battery recycling, aerospace assembly and part repair

It’s simple to agree that robots should be taking over repetitive, hazardous and physically demanding tasks, and that humans should retain control over decision making, creativity and expertise-driven processes. But how do engineers take full advantage of AI at the human-robot interface?

SUPSI has successfully begun introducing AI in the Fluently project to help solve automation problems in several sectors:

• Recycling – Fluently focuses on the hazardous or repetitive tasks in dismantling and recycling lithium batteries, to mitigate the risks posed to human operators.
• Aerospace – The project contributes to the assembly of engine nacelles. It introduces collaboration to labor-intensive, highly dexterous, iterative tasks such as positioning, fixing and inserting components.
• Repair and remanufacturing – In preparation for repair of high-value metal parts, Fluently streamlines the intricate processes of inspection, reverse engineering and defect analysis.

In those and other industrial sectors, the goal of Fluently is to increase efficiency and safety, reduce complexity and improve the general well-being of workers through human-robot collaboration.

AI in the mobile device and in the robot

To allow robots to support the human operator and adapt and respond to human input, SUPSI engineers have implemented AI in both human and robotic domains. The technology behind Fluently is a smart interface that facilitates seamless human-robot collaboration.

As shown below, the Fluently AI framework comprises 2 elements:

• H-Fluently is a portable AI device designed to capture and process data (speech, cognitive load, emotional cues) from human operators. H-Fluently enables natural, real-time communication between humans and robots so that workers can give commands, provide feedback and interact with robots effortlessly.
• R-Fluently is an AI-powered hardware and software extension that runs on the robot side. It provides robots with cognitive and behavioral abilities to interpret human commands, analyze context, understand the operator’s biophysical status and adapt to the dynamic nature of manufacturing tasks. Over time, the robots also learn from their interactions with human operators, becoming more responsive and actively supporting the human effort, rather than simply automating tasks.

Implementing AI with Qualcomm Technologies’ products

Emphasizing voice control as a natural, effective method of interaction, SUPSI engineers chose Whisper, an open-source automatic speech recognition (ASR) model from Qualcomm AI Hub.

Snapdragon 8 Gen 2 SoC and Qualcomm Innovators Development Kit 

The team’s search for the most powerful devices capable of accelerating AI models and running them locally led to devices powered by the Snapdragon 8 Gen 2 SoC. Qualcomm Technologies provided SUPSI with the Qualcomm Innovators Development Kit (shown below)—built around the Snapdragon 8 Gen2—which allowed them to experiment with different AI models.

With the Qualcomm Innovators Development Kit, SUPSI engineers had access to the same features, functions and performance enjoyed by world-class device manufacturers. But just as important, the kit offered them a pre-existing hardware platform based on a premier Qualcomm Technologies SoC. That meant they could quickly iterate designs, then immediately begin refining their models and algorithms instead of dealing with the complexities of custom hardware design.

Qualcomm AI Stack and Qualcomm Neural Processing SDK for AI

To find the model with the best fit for H-Fluently, the engineers used Qualcomm AI Stack, a set of software tools designed to speed up optimization and deployment of models. Working with the Qualcomm Neural Processing SDK for AI and its Python wrapper, PySNPE, they set out to quantize and optimize Whisper to take advantage of AI-specific engines in the Snapdragon SoC.

Qualcomm AI Hub

When Qualcomm AI Hub was announced, SUPSI engineers recognized it as their best option for optimizing the Whisper model. Within a couple of days, they had learned how to use the Qualcomm AI Hub to optimize Whisper; within a month, they had integrated the model into the application running locally on H-Fluently.

Within a couple of days, the team had learned how to use Qualcomm AI Hub to optimize Whisper; within a month, they had integrated the model to the application running locally on H-Fluently.

The team shifted their development efforts from the Neural Processing SDK to Qualcomm AI Hub, which allowed them to run the model on the graphics processing unit (GPU) and neural processing unit (NPU) of the Snapdragon SoC.

Picking from the multiple versions of the Whisper model that Qualcomm AI Hub offers, SUPSI selected the Base-En version for the best trade-off between accuracy and speed. Now, with the optimal model in hand, the engineers run tests on the Qualcomm Innovators Development Kit. Once new functions are available, they then deploy the application with model to H-Fluently devices.

In Qualcomm AI Hub, the team found much-needed guidance from both a vibrant Slack community and supportive Qualcomm Technologies’ experts. They were also able to collaborate with peers on the same development path.

Performance improvement

In their work with the Qualcomm Neural Processing SDK for AI, SUPSI engineers ran a quantized, optimized version of Whisper Base on the CPU of the Snapdragon SoC. That configuration could accurately transcribe in 700-800 milliseconds a command pronounced by an operator.

When they switched to the Qualcomm AI Hub, they were able to run a non-quantized version of the model and move the AI workload to run fully on the GPU and NPU. The configuration performs comparably without taxing the CPU. For potentially higher performance, they have the option of quantizing the model available on the Qualcomm AI Hub.

Adaptable to future advancements in AI

Besides the smart interface, the Fluently project extends to the Robo-Gym, where humans and robots undergo mutual training to build natural, personalized, collaborative workflows (see below).

Until recently, the idea of robots interpreting human gestures, emotions and speech in real time seemed far-fetched, requiring huge leaps in AI and, natural language processing. Now, those concepts are the new normal, obliging SUPSI engineers to stay up to date on breakthroughs and incorporate them to Fluently.

SUPSI engineers have built a speech-based interaction application, deployed it on the Qualcomm Innovators Development Kit and rolled it out to commercial mobile phones powered by the same Snapdragon chipset. That has allowed them to take full advantage of the GPU and NPU for best performance. By leveraging Qualcomm AI Hub, Fluently remains adaptable to future advancements in AI and hardware. With minimal effort, its engineers can improve the natural aspect of human-robot interaction by continuously adding more responsiveness.

See how you can use Qualcomm AI Hub, Qualcomm Innovators Development Kit, Snapdragon 8 Gen 2 and Qualcomm AI Stack to shorten the timeline and improve performance in your AI development.

Want to know more? Join our Discord community to engage with Qualcomm Technologies’ experts, connect with fellow developers working with our technology and stay updated on the latest developer-focused news and product updates.