Lantronix on integrating with Qualcomm AI Hub for shorter inference

Co-written with Rodrigo Caruso Neves do Amaral

As an IoT developer, you already recognize the transformative potential of integrating artificial intelligence into your projects. But where should you start, or how can you streamline your ongoing AI efforts? This blog will guide you through practical steps and resources to effectively leverage AI in your IoT applications, saving you from reinventing the wheel.

Lantronix, a provider of solutions for verticals like transportation and connected healthcare, studied the AI landscape for IoT developers, particularly in the smart-cities market. Seeing a wide variety in the level of knowledge and skills among developers in AI, they built a smart-city prototype solution  on their Percepxion IoT edge platform. Their goal was to make it easier for their customers’ developer teams to build and run apps for smart-city applications like traffic monitoring, congestion management and public safety.

The team built a prototype solution  with  Lantronix’s Percepxion platform using the Lantronix Open-Q™ 5165RB SOM powered by the Qualcomm QRB5165 processor. To simplify the landscape for developers, the build system in Percepxion uses the Qualcomm AI Hub, a suite of AI models optimized to run on devices powered by Qualcomm chipsets. The models run on the CPU, GPU or NPU using TensorFlow Lite or Qualcomm AI Engine Direct.

The Percepxion platform: Build, deploy and accelerate smart-city applications

The solution using Percepxion is organized around three stages:

1. Build: Containerized apps include both user code and models, right at the edge. Lantronix provides tools, documentation, virtual machine and Getting Started guide to begin development. In the future, Lantronix will make sample docker images available for alpha developers. The build stage also involves the Qualcomm AI Hub. Developers can upload their own trained models (including PyTorch, TensorFlow, and ONNX) to the hub for optimization and performance evaluation. They can also build with the AI models already optimized by Qualcomm Technologies for use with Qualcomm Technologies' chipsets..
2. Deploy: Lantronix’s cloud infrastructure is used for deploying containerized applications to endpoint devices and for monitoring performance. Lantronix also provides troubleshooting guidance for a smooth experience. Models can be deployed to a Linux machine, or stored in Amazon S3 and   referenced by applications.
3. Accelerate: Deployed applications get the best of both worlds – on-device processing and cloud computing – to improve performance and reliability. Locally, they process data directly on the device using Lantronix’s SOM, reducing latency and data charges in time-sensitive applications like city traffic management. Applications for environmental monitoring and public safety enjoy real-time processing of data from built-in sensors and actuators. And developers can access inference data and manage analytics in the cloud.

The Qualcomm AI Hub: Models optimized in minutes

To reduce the complexity of AI, Percepxion uses Qualcomm AI Hub so developers can optimize and profile models that work across devices equipped with chipsets from Qualcomm. It also ensures a smooth transition from development to production, letting developers deploy AI models at an accelerated pace.

Lantronix reviewed models in the Qualcomm AI Hub, vetting them for Percepxion and for IoT customers. They recommend the following models as best suited to smart-city applications:

 

The solution using Percepxion incorporates the Qualcomm AI Hub into the continuous improvement-continuous deployment (CI-CD) pipeline, ensuring the seamless integration of optimized models, even after deployment. That lets developers concentrate on their business logic while Lantronix handles the infrastructure.

At the device edge, Lantronix's SOMs are built on Qualcomm chipsets known for their power efficiency, low latency and features such as the Qualcomm Adreno GPU and Qualcomm Hexagon NPU. The Hexagon NPU, in conjunction with the Qualcomm Neural Processing SDK from the Qualcomm AI Stack, enables efficient inference at the edge using frameworks like TensorFlow Lite.

Designed for real-life applications in the smart city

Ideal smart-city applications include these:

• Traffic management: Sensors integrated into traffic lights and roads can optimize traffic flow and reduce congestion by adjusting signal timings based on real-time traffic data.
• Public safety monitoring: Surveillance cameras and emergency response systems can provide local video feeds to quickly recognize and respond to critical incidents, enhancing public safety with reduced dependency on central servers.
• Environmental monitoring: Sensors deployed city-wide for monitoring air quality, noise levels and weather conditions can provide local, real-time processing of environmental data, offering actionable insights to improve urban living conditions.
• Smart utilities: Smart grid systems can efficiently manage and monitor energy consumption to support the integration of renewable energy sources.
• Infrastructure maintenance: IoT sensors monitoring city infrastructure like bridges and tunnels can facilitate predictive maintenance, preventing failures through timely data analysis.
• Smart parking: Smart parking systems can process sensor data to quickly identify available parking spots, reducing search time and traffic.
  

Easy integration and faster inference

As a provider of intelligent hardware and turnkey IoT solutions, Lantronix enjoys the benefits of integrating models from the Qualcomm AI Hub into its Percepxion platform on the prototype solution. The integration effort was led by Ashish Syal, Lantronix’s senior director of AI/ML, Computer Vision and Generative AI.

“Qualcomm Technologies makes it easy to start deploying optimized models,” says Syal. “Qualcomm Technologies’s Getting Started page is easy to follow, all the APIs are clear and the Slack community they’ve built is responsive. I think the AI hub is a good first step for any developer or company looking to move into AI development.”

It took about a year for Syal’s team to build their prototype using Percepxion platform and build a cloud + device solution. AI Hub made plugging AI models into the platform simple and easy. Lantronix was able to integrate models from the Qualcomm AI Hub with Percepxion in just a few days. While they have more on their to-do list – including additional models, support for external sensors and integrations with third-party plugins – Syal is pleased with the performance boost so far.

“We've seen significant improvements in our platform using Qualcomm Hub AI,” he says. “It has helped us test different models to find the best fit. For example, after testing and analyzing performance, we chose the YOLO series for our needs. Even though it's still early, we were able to evaluate a YOLOv5 uint8 and Yolov8  TensorFlow Lite int8 models against a non-quantized model and decided on the former. For example,  for Yolov5 our inference is 6 to 7 times faster, falling from 150-200 ms before integration to 20-25 ms afterward.”

Next steps

Find out more about Lantronix Percepxion and Qualcomm AI Hub. Gauge the fit for your AI development plans and watch the Qualcomm AI Hub for upcoming models, tools and documentation.