AuraTranslator: RealTime On Device Speech Translation on the Qualcomm® Hexagon™ NPU with Android XR Glasses

AuraTranslator delivers fully on-device, real-time Portuguese-to-English live captioning on Android XR glasses. OpenAI's Whisper-Large-V3-Turbo runs on the Hexagon NPU through precompiled QNN context binaries and the ONNX Runtime QNN Execution Provider, a Marian NMT model translates on the CPU in parallel, and speaker embeddings colour code each voice. Captions appear world-anchored in the glasses about 1.5 seconds after each utterance. No cloud, no network, no per-use cost.

Objective

Our QAIRT session at the AWE XR 2026 Hackathon, plus a personal itch: I watch Portuguese television and I am not a native speaker. The xr-labs-for-snapdragon samples proved that third-party HTP inference was possible on the XREAL Aura. AuraTranslator set out to prove that a production shaped workload could live there too, meaning a multimodel, always on, latency sensitive pipeline running on passively cooled hardware.

Show that current Snapdragon® XR silicon can run large open-weights speech models locally and in real time (the turbo encoder alone is a 1.75 GB compiled context binary), and document the integration path with the sharp edges included, so that the next developer gets there in days rather than weeks.

 

Equipment Required Parts List / Tools

 

Source Code /Source Examples

 

Additional Resources

 

Build / Assembly Instructions

1. Add onnxruntime-android-qnn and qnn-runtime from Maven. Enable useLegacyPackaging for JNI libraries so the QNN HTP libraries extract to the native library directory. If they stay compressed inside the APK, QNN cannot dlopen them and the EP silently loads without an HTP backend.

2. Declare the FastRPC vendor library in AndroidManifest.xml.  This one line is the difference between a working NPU and a day of misdirected debugging:

        <uses-native-library android:name="libcdsprpc.so" android:required="false" />

With targetSdk 31 or newer, Android hides public vendor libraries from the app namespace unless they are declared. Without it, QNN device creation fails with QNN_DEVICE_ERROR_INVALID_CONFIG, which reads like a SoC support problem rather than a library visibility one. What makes it especially misleading is that qnn-net-run executes the same context binary on the same device without complaint, because the shell user is not subject to the restriction.

3. Download the precompiled Whisper-Large-V3-Turbo QNN package from Qualcomm AI Hub (fp16, matching your HTP architecture, v79 here) and push the EPContext ONNX files together with their context binaries to the device. The .onnx file references its .bin by relative path, so the two must sit in the same directory.

4. Set ADSP_LIBRARY_PATH to the application native library directory at runtime, so the HTP stub can find the matching Skel library.

5. Drive inference through the ONNX Runtime C API from JNI. The model uses uint16 and fp16 tensors, and neither type can be expressed through the ORT Java API. Create sessions with the QNN EP, pointing backend_path at the bundled libQnnHtp.so and setting htp_performance_mode to high_performance, with graph optimization disabled so that ORT hands the EPContext node straight to QNN.

Project Walkthrough

Audio front end.

Continuous 16 kHz capture from the UNPROCESSED source, because the device voice pipeline treats far-field television speech as noise to be removed. An energy-based VAD with an adaptive noise floor slices utterances, where 600 ms of silence ends a segment and 10 seconds is the maximum window.

Features.

Whisper’s 128-bin log-mel spectrogram computed natively in C++ across four threads, about 227 ms per 30-second window, packed as fp16 to match the encoder input contract.

Speech recognition on the NPU.

The 1.75 GB turbo encoder context binary processes a full 30-second window in roughly 793 ms on the Hexagon HTP. The decoder then runs a fixed-shape KV-cache loop at about 12 ms per token: a 200-token window, a right-aligned attention mask with one slot opened per step, and self-attention KV tensors copied back verbatim between steps because the input and output quantization parameters are identical. Greedy sampling takes the argmax over raw logits directly, since quantization is monotonic.

A note on performance mode.

Burst was the obvious choice for latency and it was the wrong one. On a passively cooled puck, continuous burst-mode captioning reached thermal emergency after about two and a half hours. Measuring high_performance against it showed the encoder pass was essentially unchanged, so burst was buying heat and nothing else.

Hallucination gating.

Whisper invents fluent text on silence, music and applause, and it does so confidently, so confidence alone is not enough. Six gates run in order: the no-speech probability and the language-ID distribution are both read directly from the start-of-transcript logits; segments the model hears as English are dropped, since that is the hallucination signature, but only when the text also carries no Portuguese diacritics, because genuine Portuguese with a noisy onset frequently lands above the English threshold; then mean token confidence, then token density, since a hallucination is typically three tokens from ten seconds of audio while real speech runs well above one token per second; then a small blocklist of bare non-speech labels; then repetition collapse and consecutive-caption dedupe.

Translation.

Whisper transcribes Portuguese and a separate model translates it. That split is deliberate. Whisper’s own translate task was the quality bottleneck at every model size, while its Portuguese transcription is close to flawless, so Opus-MT ROMANCE-en (Marian) with KV-cache decoding handles PT to EN on the CPU, sentence by sentence, overlapped with the next segment’s NPU work. SentencePiece runs natively. One gotcha: Marian’s raw argmax is the pad token, so generation has to suppress it explicitly or every output is empty.

Speaker colours.

WeSpeaker CAM++ embeddings per segment, clustered online by cosine similarity with a hysteresis band, give each voice a stable caption colour. This model runs on the CPU rather than the NPU, and not by choice: the ORT QNN EP’s layout transformer rejects the graph because an AveragePool is inserted as NHWC and then is not claimed by the EP. I would be glad to supply a minimal reproduction to whoever owns the execution provider.

Spatial UI.

A Jetpack XR spatial panel, world-anchored beside the television via a plane hit-test, with teleprompter-style captions emitted one sentence at a time as each translation finishes. Emitting per sentence rather than per segment is what makes the panel feel live instead of arriving in blocks. Hand-tracked scrollback, per-speaker colour, and panel placement and size that persist across sessions.

Measured end to end.

About 1.5 seconds from the end of an utterance to the caption appearing in the glasses, with the whole pipeline running at roughly 0.2x real time, which leaves both thermal and CPU headroom.

Usage Instructions

1. Install the APK and grant microphone permission. Models load from external storage, about 2.8 GB in total.

2. Look at the television. The caption panel anchors to it, or world-locks in front of you as a fallback.

3. Play Portuguese audio. Captions appear about 1.5 seconds after each phrase.

4. Pinch-scroll back through the history, then tap LIVE to rejoin the live feed.

5. Panel position and size persist across sessions, and each voice keeps its caption colour.