ViT (Vision Transformer) Model Compilation and Integration on Qualcomm® Robotics RB5
In today's digital world, the demand for smart applications capable of analysing and understanding images is rapidly growing. At the forefront of this transformation is the Vision Transformer (ViT) model, an innovative approach to image classification. ViT models utilize a self-attention mechanism that allows them to capture global context and intricate patterns across an entire image.
Objective
Deploy the Vision Transformer (ViT) model on Qualcomm Robotics RB5 capable of classifying pictures from the ImageNet dataset.
| Equipment | Description |
| Qualcomm Robotics RB5 | Qualcomm Robotics RB5 Development Kit |
| USB to Micro USB cable | For serial console interface , to view logs |
| USB to USB Type C cable | For connecting the USB3.0 Type C port to the board and flashing images, adb and fastboot |
| Power adapter | 12 V with 2500 mA required by the 96Boards specification |
Source Code: GitHub Link for project source code
https://github.com/globaledgesoft/ViT-Model-Compilation-and-Integration-on-RB5
Qualcomm Robotics RB5 Development Kit bring up
Introduction:
ViT models have demonstrated exceptional performance in a variety of domains. In autonomous driving, ViT models enhance the ability of vehicles to recognize and react to their surroundings with greater accuracy and speed. In the field of medical imaging, ViT models assist in the early detection and diagnosis of diseases by analysing complex medical images more effectively than traditional methods. Moreover, in security and surveillance, ViT models improve the accuracy of identifying and tracking objects or individuals in real-time.
- Model Type: Image classification
- Checkpoint: Trained on ImageNet
- Input Resolution: 224x224 pixels
- Number of Parameters: 86.6 million
- Model Size: 330 MB
Vision Transformer (ViT) Qualcomm® Neural Network Model Inference on Qualcomm Robotics RB5
Prerequisites:
- Ubuntu 20.04
- Conda Environment
- Python3.8
- ONNX
- Qualcomm® AI Engine Direct SDK – Qualcomm Neural Network_SDK 2.19.2.240210
- Qualcomm Robotics RB5
Steps to Execute:
Step 1: Install Conda on Ubuntu and create py38 environment. Install the necessary packages.
Step 2: Install the required dependencies for Qualcomm Neural Network SDK. Check this link for Qualcomm Neural Network setup.
Step 3: Download ViT ONNX model from Qualcomm AI Hub website for Qualcomm Neural Network model conversion.
Model Conversion:
Step 1: Qualcomm Neural Network Op Package Generation.
This model has been trained with Gelu activation function, but Gelu Op support was not provided in QNN_SDK-2.19.2.240210. So we created custom op package for Gelu. Refer this link to create Gelu op package for CPU Runtime.
1. Custom Op XML Config Creation
$ qnn-op-package-generator -p
<QNN_SDK_ROOT>/examples/QNN/OpPackageGenerator/GeluOpPackageCpu.xml -o
<QNN_SDK_ROOT>/examples/QNN/OpPackageGenerator/Gelu
#THIS CAN BE ANY PATH
2. The above command will create GeluOpPackage.
3. We need to write Gelu.cpp script & save it in src/ops folder.
Note:
a. Gelu.cpp script has been provided in the examples section for reference.
b. Below lines highlight the key additional components in the GELU package that are not present in the ReLU package.
a. REGISTER_PACKAGE_OP(Gelu)
b. extern "C" QNN_API Qnn_ErrorHandle_t
GeluOpPackageInterfaceProvider(QnnOpPackage_Interface_t* interface)
d. Compiling for CPU on aarch64-ubuntu, run the following commands
$ export CXX=<path-to-clang++>/clang++
$ export ANDROID_NDK_ROOT=<path-to-ndk-build>
$ cd ${QNN_SDK_ROOT}/examples/Gelu/GeluOpPackage && make aarch64-ubuntuNote:
a. Write the script Makefile.aarch64-ubuntu if it is not provided with Qualcomm Neural Network-SDK for aarch64-ubuntu. Script has been provided in the examples section for reference.
b. Save it in ./OpPackageGenerator/Gelu/GeluOpPackage/makefiles/Makefile.aarch64-ubuntu. For more information refer this link
Step 2: Qualcomm Neural Network model conversion.
$ qnn-onnx-converter --input_network ./vit.onnx --output_path ./vit.cpp --op_package_config /{QNN_SDK_ROOT}/examples/Gelu/GeluOpPackage/config/GeluOpPackageCpu.xmlNote:
a. Since Gelu Op support was not provided in Qualcomm Neural Network_SDK-2.19.2.240210, we need to create GeluOpPackageCpu.xml file by taking Reference from ReluOpPackageCpu.xml file provided in Qualcomm Neural Network_SDK-2.19.2.240210.
b. Reference script is provided in Examples section.
Step 3: Once the model is converted, we need to build with qnn-model-lib-generator for aarch64-ubuntu-gcc7.5 target.
$ qnn-model-lib-generator -c <MODEL_PATH>vit.cpp -b <MODEL_PATH>vit.bin -t aarch64-ubuntu-gcc7.5 -o <OUTPUT_DIR>/model_libs/
Note: After executing above script, libvit.so file will be saved in <OUTPUT_DIR>/model_libs/libvit.so
Deploying the ViT model on Qualcomm Robotics for CPU runtime.
Step 1: To connect with Qualcomm Robotics RB5, use either adb tool or ssh. Create directory in the home directory of Qualcomm Robotics RB5 board.
$ ssh root@<ip_address>
$ mkdir <dir_name>
$ exitStep 2: Copy the AI model and Qualcomm Neural Network dependency file to the Qualcomm Robotics RB5 board from host system.
$ scp ${QNN_SDK_ROOT}/lib/aarch64-ubuntu-gcc7.5 root@<ip_address>:/<dir_name>
$ scp ${QNN_SDK_ROOT}/bin/x86-64-linux-clang/envsetup.sh root@<ip_address>:/<dir_name>
$ scp ${QNN_SDK_ROOT}/bin/aarch64-ubuntu-gcc7.5/qnn-net-run root@<ip_address>:/<dir_name>
$ scp ${QNN_SDK_ROOT}/lib/hexagon/unsigned/ root@<ip_address>:/<dir_name>
$ scp vit/model_libs/aarch64-ubuntu-gcc7.5/*.so root@<ip_address>:/<dir_name>
$ scp vit/images/*.raw root@<ip_address>:/<dir_name>
$ scp vit/image_list.txt root@<ip_address>:/<dir_name>
$ scp {QNN_SDK_ROOT}/examples/opt/QNN/OpPackageGenerator/Gelu/CPU/GeluOpPackage/libs/ aarch64-ubuntu-gcc7.5/libGeluOpPackage.so root@<ip_address>:/<dir_name/CPU/>
$ scp {QNN_SDK_ROOT}/examples/opt/QNN/OpPackageGenerator/Gelu/DSP/GeluOpPackage/build/libGeluOpPackage.so root@<ip_address>:/<dir_name/DSP/>Step 3: Connecting to Qualcomm Robotics RB5.
$ ssh root@<ip_address>
$ cd <dir>- For CPU runtime specify the libQnnCpu.so as a backend to get output. Run the following commands.
$ qnn-net-run --backend ./libQnnCpu.so --model libMobileNet-v3-Large.so --input_list image.txt --op_packages ./libGeluOpPackage.so:GeluPackageInterfaceProvider --output_dir output/
Step 4: Checking Output on Qualcomm Robotics RB5.
- To view results, run the following command.
$ python3 show_vit_classifications.py -i image_list.txt -o output/
-l imagenet_slim_labels.txt
Sign up for the Developer Newsletter.
Get software and hardware tool resources to help optimize your development delivered to your inbox weekly.
