Running GenAI with RAG and ASR on the Qualcomm Dragonwing IQ-9075

GenAI Architecture Overview

Running complex GenAI workloads at the edge requires the right software stack. For the Dragonwing IQ-9075 EVK, we utilized two key Qualcomm AI runtimes:

• Genie software library for LLM inference with RAG.
• Whisper ASR for speech-to-text (STT) conversion.

The Use Cases

To demonstrate the power of running LLM, RAG, and STT on-device, we implemented three distinct scenarios: Predictive Maintenance, Factory Management, and a Demo Mode.

The Models

The solution relies on three specific AI models:

• LLM: Meta Llama 3.2 Instruct 3B
• RAG Embeddings: BAAI BGE-large
• STT: OpenAI Whisper-small

The Hardware Advantage

All three models are optimized specifically for Hexagon Tensor Processor (HTP) accelerators. By distributing the workloads across the two HTP cores of the Dragonwing IQ-9075 EVK, we successfully demonstrated that a single embedded device can smoothly host ASR and LLM with RAG simultaneously.

Figure 1. Chat panel for Factory Management - the on-device LLM responds in real time to operator queries, grounded in live factory telemetry. Suggested actions are shown on the right as one-tap shortcuts. 

Predictive Maintenance with RAG

The Predictive Maintenance use case demonstrates how AI can assist with equipment upkeep using Edge AI. The system leverages static documentation such as:

• Manuals
• Maintenance records
• Inspection reports

These documents are ingested and parsed by AI models to surface relevant information for technicians. This enables:

• Proactive issue identification
• Optimized service scheduling

Importantly, the solution can be deployed directly on devices or on local gateways. This ensures that maintenance insights remain available even in environments with limited connectivity. 

Figure 2. Maintenance-required alert — the digital twin shows an exploded view of the faulty robot joint, with the damaged inner gear highlighted in red

— Build AI-powered maintenance workflows: Use RAG to turn static documentation into actionable insights. Explore RAG implementation on Qualcomm with Qt

 RAG Architecture

The implementation uses a Retrieval-Augmented Generation (RAG) architecture centered on ChromaDB. The flow is as follows:

• Technical documentation is encoded into dense vector representations using the BGE-large embedding model.
• Semantic search retrieves relevant maintenance instructions.
• The retrieved context is passed to Llama 3.2 3B, which synthesizes the final response.

Rather than simple keyword matching, this approach enables precise semantic retrieval of maintenance instructions. Llama 3.2 3B is executed directly on the Dragonwing IQ-9075 Hexagon HTP accelerator, enabling:

• Fully offline inference
• Low-latency responses
• Reduced dependency on cloud infrastructure
• Improved data privacy in industrial environments 

Figure 3. Guided part-replacement flow — the LLM, grounded in maintenance documentation through RAG, walks the technician through each step (screw driver, retaining ring, inner gear, etc.) with a live progress indicator in the top bar. 

Watch how the system guides real-time part replacement using LLM + RAG.

Future expansion possibilities

The current implementation focuses on a single RAG document source. Future iterations can expand to include:

• Historical service logs
• Schematic diagrams
• Multi-document repositories

Such extensions would allow richer contextual understanding, pattern recognition across past failures, and more accurate maintenance predictions. By combining centralized knowledge with on-device AI execution, the system can evolve into a distributed knowledge hub for predictive maintenance. 

Factory Management with On-Device LLM + ASR

The Factory Management use case shows how GenAI assists factory personnel through natural-language interaction. A speaker connected to the device allows operators to:

• Start or stop production lines
• Check factory alerts
• Query machine or line status
• Trigger other demonstration use cases

For this scenario, RAG was not used. 

Speech-to-Text optimization

For the speech-to-text (STT) engine, we deployed the Hugging Face Whisper-small ASR model. To make it run efficiently within the board's HTP/NPU constraints, Qualcomm Technologies fully optimized and quantized the model. The heavy lifting is managed by the Qualcomm AI Runtime SDK, which provides the interface for running inference on the NPU side to enable both real-time transcription and translation.

To match the Whisper model's native 16 kHz audio requirement, we customized Qt Multimedia's QAudioDevice via its private interface. This ensures the NPU-accelerated model receives the exact audio stream it needs for high-performance transcription.

Note: the private interface used here is not part of the stable public API and may change between Qt releases. Production code should avoid relying on private interfaces; a public-API path is under investigation for a future Qt Multimedia release.

This configuration enables real-time, on-device speech interaction without cloud services. 

Demo Mode: LLM as an Embedded Presenter

Demo Mode integrates the LLM directly into the Qt application to showcase its full functionality interactively. Instead of playing a prerecorded video, the LLM activates the actual application to demonstrate features dynamically.

When the Demo Mode switch is enabled:

• The application enters an autonomous walk-through experience.
• An LLM-powered presenter narrates the functionality.
• The solution navigates through each use case - Factory Management, Defect Detection, Worker Safety, and Predictive Maintenance.

The chat interface remains interactive - users can jump between use cases using predefined chat options. To avoid conflicts between manual and automated interactions, the 3D virtual factory environment is disabled during Demo Mode. Once all use cases have been presented, the solution automatically restarts while the toggle remains enabled. 

Goals of Demo Mode

The objectives of Demo Mode were to:

• Create a consistent, hands-free demonstration experience
• Support exhibition and customer-demonstration environments
• Showcase an intelligent, self-guided UI
• Demonstrate AI as an embedded assistant 

Key Takeaways for Developers

The GenAI implementation in FactoryPulse demonstrates:

• On-device LLM + RAG + ASR on a single embedded platform
• Fully offline inference using Hexagon HTP acceleration
• Reduced system complexity by avoiding cloud dependency
• Multimodal UI support (text, speech, video) on Dragonwing IQ-9075 EVK

By combining Vision AI and GenAI workloads, the solution highlights the possibility of building multimodal industrial applications on the Dragonwing IQ-9075 EVK. 

From Demonstration to Developer API

The learnings from this project are being used to formulate a new Qt module and API for Edge AI development. At the time of writing, a proof-of-concept of the Qt module and API is available for developers to try. Try the Qt AI Inference API proof-of-concept.

Run LLM + RAG + ASR on a single device: Build fully offline, intelligent edge applications.

Request your QT evaluation license to run on Dragonwing IQ-9075 EVK

Ready to Get Started?

• Explore Llama 3.2 deployment: Learn how to efficiently run large language models locally on Qualcomm edge platforms.
• Review Hexagon HTP optimization: Discover how to quantize and prepare AI graphs for high-performance execution on the NPU.
• Learn about ChromaDB RAG: See how to build and manage vector databases for Retrieval-Augmented Generation pipelines.
• Try the Qt AI Inference API: Test Qt's proof-of-concept designed to drastically simplify AI model integration in QML and C++ apps.
• Build AI-powered maintenance workflows: Use RAG to turn static documentation into actionable insights. Explore RAG implementation on Qualcomm with Qt

Coming soon: Stay tuned for our upcoming technical deep-dive post on Qt Onboard QBSP