Nomic GPT4ALL and Embeddings: Fast, local AI inference for devices powered by Snapdragon X Series
Sign up for Developer monthly newsletter
Join thousands of developers around the globe who receive latest news and updates from our monthly curated newsletter.
Sign upMany independent solution vendors (ISVs) want to use AI more, but they also want more privacy, control and performance than they get in the cloud. GPT4All from Nomic runs large language models (LLMs) privately as a local application. Together with Nomic’s embedding models, GPT4All lets you chat with thousands of models, turn your local files into information sources for models (LocalDocs), and browse and download models available online. It’s AI that’s fast, on-device and private.
And now GPT4All is available for devices running Windows on Snapdragon. That means consumers, developers and enterprise users can all run LLMs locally on laptops powered by Snapdragon X Series processors, for retrieval-augmented generation (RAG) and local document chat. GPT4ALL is ready to take full advantage of the Qualcomm Hexagon NPU on the Snapdragon X Series.
Making AI accessible to all, locally and privately
The spectrum of AI users and use cases is as broad as computing, stretching from light entertainment to heads-down analysis. Nevertheless, there are a few things almost all users want their AI to be:
- Private – Users want their data to stay on their device, so that their AI chatbot experience remains private.
- Secure – Sensitive or confidential interactions should not run off site.
- Portable – Users want AI independent of a cloud connection so they can work even when completely offline.
- Local – On the same device where AI is running, users have stored thousands of content-rich documents and files. Those local docs should be securely, privately accessible as a content store for RAG.
- Computationally flexible – Local language models should run on any engine – CPU, GPU or NPU.
- Non-proprietary – Nobody wants vendor lock-in. Open-source architecture and language models are the way to ensure that AI remains accessible and transparent.
Nomic has designed GPT4ALL to embody those characteristics. With 250,000 monthly users and enterprise editions of GPT4ALL, Nomic’s byword is to give people access to local AI for business applications.
“One of our core values is to make AI as accessible as possible,” says Andriy Mulyar, CEO and co-founder of Nomic. “By ‘accessible’ we mean allowing users to run AI innovations and language models on as many devices as possible. We’re always keen to increase the base of hardware we support, and we’ve paved the way for our users to take advantage of local AI on several different platforms. With more of our customers purchasing AI PCs, we decided to support devices with Snapdragon X Series due to the incredible performance and efficiency.”
GPT4ALL on Snapdragon X: Fast, local AI on Windows
In the current market landscape, Nomic believes that it’s important to support Windows on Snapdragon because the platform gives users a viable alternative.
To run fast, local AI, consumers have had to choose between affordability and platform. If they want to stay on Windows, they have to buy a more-expensive gaming laptop with a high-powered GPU. Or, if they want a dedicated AI laptop at consumer prices, they have to switch to and learn a different operating system. Either alternative is a speed bump on the road to using AI.
Few consumers know what to buy if they want to accelerate AI inference or take advantage of RAG. When they find out that it involves either switching to a new OS or moving to high-performance hardware, they are likely to hesitate. That’s why Nomic’s GPT4ALL software running on the Snapdragon X Series devices is a compelling alternative for fast, local AI.
Part cross-compile, part port
From the engineering perspective, some of Nomic’s work was cross-compiling – using a different tool chain to compile for Windows on Snapdragon with Visual Studio – and some was porting.
The engineers assessed the amount of work they would have to invest to run on Snapdragon X Series. That involved discovering the capability of the Windows on Snapdragon architecture and tool chain, then ensuring compatibility with their existing code base. Within a week they had an initial, working version.
The team then spent another week using the application and resolving issues related to running on a new platform, including modifications to the LocalDocs feature. LocalDocs is an important differentiator because it brings local files such as PDFs and Microsoft Word documents into LLM chats. It first uses embedding models to assign vectors to text snippets, then it includes semantically similar snippets in the prompt to the LLM.
The engineers modified the code for ingesting files and computing the embeddings, enabling LocalDocs to take advantage of the Snapdragon X Series processor.
For the work of both cross-compiling and porting, the engineers did not need technical support. They found that the tools and documentation provided by Qualcomm Technologies sufficed for Snapdragon X Series support.
Nomic team is now working to accelerate their AI workloads on the Qualcomm Hexagon NPU. Stay tuned!
Your turn
GPT4ALL is an open-sourced product. Nomic invites developers to contribute to improve GPT4ALL better on Snapdragon X Series devices.
To get Nomic models optimized and ready for deployment on devices with Snapdragon X Series, Nomic has also added their embedding models to Qualcomm AI Hub. You can now download and bundle them into your application. Nomic encourages you to try out the Qualcomm AI Hub platform by bringing your own model (BYOM).
Choose a device, set the specified options and submit jobs to confirm performance and accuracy. Then deploy the optimized model to the device.
Ready for fast, local, private AI? Take GPT4ALL for a test drive. Select the “Download for Windows Arm” installer and run it on a laptop powered by a Snapdragon X Series processor.
The documentation includes a Getting Started and a series of cookbooks for different data sets.
Find out more by joining our Discord community. You can rub elbows with our experts, connect with fellow developers working with our technology and keep your fingers on the pulse of developer-focused news and product updates.
Come for support, stay for the community
Get support from experts, connect with like-minded developers, and access exclusive virtual events.
