Edge AI Developer Hackathon in Paris: SignBridge and EasyForm

It consists of one pane for input text, which can be text that the user types (as shown above) or pastes in. It can also be a spoken message (through the device’s microphone or through the device’s audio hardware) that the app converts to input text using Whisper. For example, the app can capture and transcribe audio from an online video, which is an especially useful feature for hearing-impaired users.

After the user clicks the Translate button, SignBridge populates the other two panes. The second pane displays the translation (using Sockeye) of the input text into SignWriting. SignWriting is a visual notation system for representing the handshapes, movements, and facial expressions of sign languages. Accessible enough for users ranging from children to researchers, SignWriting is included in the Unicode standard. In the screenshot above, the app translates “Hello World!” from the Input Text pane to the two symbols shown in the SignWriting pane.

Then, based on the SignWriting symbols, the third pane displays a 2D-skeleton animation (pose generation) in sign language. The animation renders the input text as a human signer would interpret it for a hearing-impaired person.

The team developed SignBridge for the Copilot+ PC, powered by a Snapdragon® X Elite processor. The app is cross-platform, running on Windows, macOS and Linux as a modern web app in the browser or as a native desktop app using Tauri.

Software and hardware

Back end:

• Groq API – For online simplification of input text using Llama (Simplify Text button in UI)
• PyTorch signwriting-translation package – For text-to-symbol translation in the SignWriting pane
• Python 3.11 – To support PyTorch
• FastAPI – For generating API endpoints
• Qualcomm AI Engine Direct SDK – used to convert their model to the DLL format supported by Qualcomm SDK
• ffmpeg – System dependency for audio file conversion in Input Text pane

Front end:

• React – For UI components
• Vite – For authoring JavaScript
• TypeScript – Superset of JavaScript
• Tauri – For cross-platform support

Models implemented

SignBridge uses multiple AI models.

The Input Text pane accepts a WAV audio file, applies the Python Whisper library and returns the transcribed text in JSON format. The developers obtained the OpenAI model from the Qualcomm AI Hub.

For translation, the app uses Sockeye, a PyTorch translation model from AWS Labs that is fine-tuned and specialized for SignWriting. It accepts the JSON text string and returns a SignWriting notation string that appears in the second pane.

For the Animation pane, the developers started with the MediaPipe Holistic model. The model detects landmarks on hands, face and body, then returns pose data in the form of endpoint coordinates for the animation. The developers trained the model on an ASL dictionary composed of numerous videos. Each part of the text is assigned one sign fetched from the dictionary of encoded movements.

Then all movements are blended and smoothed together and the animation is generated. (During the hackathon they used a smaller, pre-existing model for pose generation.)

It was a hackathon requirement that every application integrate at least one Llama model through an online call to the Groq API. The developers implemented that call in an optional Simplify Text feature that can shorten a long Input Text string, resulting in a simpler SignWriting notation string and animation.

Finally, credits for using certain code generation tools were made available to everybody during the hackathon. The developers used used commercial Gen AI models to generate the code scaffolding and get more components up and working quickly.

Compute cores used

During the hackathon the team ran all required models locally on CPU, and they subsequently moved the inference workload of the Whisper model to NPU. All core workloads in SignBridge run entirely on the edge device – here, the Copilot+ PC. As noted, the optional Simplify Text feature uses the Groq API and cloud-based Llama models, but no cloud connection is required for real-time translation or animation.

Development flow

The team followed a development plan covering these phases:

1. Speech-to-Text (Whisper)
2. Text Simplification (Groq + LLaMA)
3. Text-to-SignWriting Translation
4. SignWriting Rendering
5. 2D-Skeleton Animation
6. UI Packaging and Polish
7. UI Enhancements
8. Tauri Integration
9. Cross-Platform Build Scripts
10. Snapdragon Optimization
11. System Audio and Realtime
12. Settings and Metrics
13. Offline Pose Generation

They accomplished phases 1 through 9 by the end of the hackathon and have since continued work on phases 10 through 13.

Easy parts and hard parts

The Snapdragon X Elite processor enabled the SignBridge app to run multiple models at a time. Given the amount of translation and rendering work the app performs, that combination of different models running simultaneously proved a big advantage. It also simplified development.

The SignBridge developers created a clear separation between the Python running the back end and the JavaScript running the front end, with the two ends communicating via a RESTful API. This type of compartmentalization between the ends made it easier to iterate on the functions as a team.

They chose the Whisper library from the Qualcomm AI Hub for speech-to-text conversion but initially had trouble using the cloud-compile function for their device. The Qualcomm Technologies team at the hackathon sent them other models to try and suggested different versions of Python.

The developers also faced a learning curve in converting the Whisper model to the ONNX format required to run on the Qualcomm® Hexagon™ NPU. The conversion resulted in a spike in graph computation with opcodes for the NPU, which increased app size by hundreds of megabytes and greatly slowed app initialization.

Although this was their first exposure to development for NPU, the team succeeded in getting the converted ONNX model running on it. However, they advise other developers not to bother resorting to vibe coding in this domain, where “a certain rigor is still needed to make nicely optimized models work on NPU.”

They were pleased that the Qualcomm Technologies team was accessible and helpful during the hackathon, and surprised to have a sponsor interested in post-event follow-up.

EasyForm – Intelligent form fill while keeping sensitive information private

We will always have forms to fill out – not even AI can do away with them entirely. But EasyForm addresses the repetitive, error-prone task of filling out PDF and DOCX forms by applying AI to reuse the same data across sessions.

It saves time by automatically extracting personal data from a directory of context documents, such as CV, residence permit, bank details and passport, to build a reusable data context. It then fills out forms in PDF and DOCX files by accurately detecting and mapping placeholders in PDF/DOCX forms using regex and heuristic AI.

At the same time, EasyForm keeps sensitive information private because users can run it fully offline and enjoy its benefits without an internet connection.

The EasyForm team included Tien Dat Phan, Hieu Le, Hong Quan Tran, An An Tran, Thanh Tung Vu and Minh Phuong Ngo.

Overview

EasyForm builds context, then uses it to automatically fill out new PDF and DOCX forms. It detects placeholder fields and checkboxes using regex patterns and applies heuristic matching to ensure accuracy, privacy and a comfortable UX.

 

The app’s intelligent document processing workflow includes the following:

• Accurate placeholder detection with regex
• Heuristic key matching powered by AI
• Continuous context learning
• Robust fallback and retry mechanisms

The developers believed they could create a useful form-filling application and make it run efficiently in the environment of edge AI. They made a priority of local processing, which eliminates network latency and ensures privacy without sharing data. If the user chooses the local model, all workloads run entirely on the edge with no network connection required. Otherwise, the only possible external communication is for calling online LLM provider APIs such as Groq or OpenAI.

At the moment, EasyForm is built for daily users who want their documents filled automatically on their local machines, with or without internet access. Over time, its use case can extend to business environments.

Working on a Copilot+ PC powered by a Snapdragon X Elite processor, the developers discovered they could efficiently run on-device inference with a quantized Llama model. The inference workload consumed relatively little power, which they considered an essential factor for daily users who need both performance and battery efficiency.

Software and hardware

• Back end – Python with FastAPI
• Front end – Electron
• OCR – EasyOCR and Docling

Models implemented

The team found that the tutorials and instructions in the Qualcomm AI Hub were easy to follow. They selected the Llama 3.1 8B with quantization in a version tailored for edge devices. They received ample support from Qualcomm Technologies engineers and were able to run the model efficiently on the Copilot+ PC.

During the hackathon, their approach to optimization focused on prompt engineering, then stacking and restructuring prompts to take full advantage of the model’s context window. That significantly reduced the number of LLM calls and relieved what would otherwise have been a major performance bottleneck.

In their continuing optimization of EasyForm, the team plans to explore strategies such as fine-tuning smaller models (e.g., LLaMa 3.2 3B). That would help reduce overall model size and speed up inference, particularly for edge AI.

Compute cores used

EasyForm runs across the CPU, GPU and NPU:

• CPU – Handles process orchestration and mapping information
• GPU – Runs the OCR model to extract text and information from documents
• NPU – Executes local inference for the LLM, enabling fully offline workflow

Development flow

1. Built the core backend API
2. Developed the UI and connected it to the API
3. Tested the workflow with large models
4. Integrated a quantized local model, optimizing the workflow to handle its smaller size. This stage required significant help from Qualcomm Technologies experts, particularly for model quantization, configuration and guidance on using the model efficiently.

Easy parts and hard parts

The easiest part turned out to be the UI. With the help of AI tools, the team mocked up screens that came to life much faster than expected. They accomplished in a few hours the amount of work that normally would have taken days.

The hardest part in developing EasyForm – or any edge AI application – was in dealing with the model. Everything worked smoothly with a large, 70B model over the Groq API. But unlike cloud or data-center environments with virtually unlimited resources, edge AI is constrained by the hardware capacity of the edge device. Once the developers moved to a local LLaMA 3.1 (8B) model, their focus changed to improving performance and reducing latency through every possible type of optimization.

For example, the smaller context window meant that they had to deal with hallucinations and messy, unparsed responses. They realized in retrospect that if they had planned in advance for the constraints of edge AI, they wouldn’t have needed to spend time changing their workflow.

Next steps

You can test-drive SignBridge from a browser right now, and a five-minute SignBridge video takes you through the app UI and main features. You can have a look at both the EasyForm GitHub repo and the SignBridge GitHub repo, and study them for use in your own projects.

Take a look through our upcoming events for ways you can see us in person and participate in our hackathons, workshops, conferences and other developer events.

Connect with us on Discord and be sure to sign up for the Qualcomm Developer newsletter to be notified of our upcoming developer events across the Unites States, Asia and Europe.