This project aims to build and deploy a depth estimation Android application that generates depth maps from RGB camera frames without using a depth sensor. The Android depth estimation API returns depth maps from live video feeds using a deep learning model - PackNet-SfM, eliminating the need for a dedicated depth sensor. The Android application is also integrated with Qualcomm® Neural Processing SDK. The application runs on the Qualcomm QCM2290 platform present on Thundercomm TurboX CM2290 Development Kit that is running Linux Android, which facilitates the deployment of AI camera applications on edge devices.
Objective
This project demonstrates a monocular depth estimation use case, where an AI model estimates the depth value (distance relative to the camera) of each pixel from a single RGB image. The implementation runs on the Thundercomm TurboX CM2290 Development Kit with Qualcomm Neural Processing SDK on Android, utilizing the PackNet-SfM model. This application is a crucial prerequisite for scene understanding use cases on Qualcomm QCM2290 platforms, such as 3D scene reconstruction, autonomous driving, and augmented reality (AR).
Thundercomm TurboX CM2290 development kit
Depth estimation on TurboX CM2290 development kit without depth sensor
| Equipment | Description | |
|---|---|---|
| Thundercomm TurboX CM2290 Development Kit |
Based on the Qualcomm QCM2290 platform | |
| Power adapter Camera interface |
12 V with 2A minimum required specification 2x MIPI-CSI, 4 + 4 or 4 +2+1, D-PHY 1.2 at 2.5 Gbps per lane |
|
| USB to Micro USB cable USB to USB Type C cable |
For serial console interface For connecting the USB3.0 Type C port to the development kit and flashing images, ADB, Fast boot commands |
|
| OS | Android 11, Linux |
Depth Estimation on CM2290 with SNPE on android Project Source Code
https://github.com/globaledgesoft/Depth-Estimation-on-CM2290-with-SNPE-on-Android
Thundercomm TurboX CM2290 Development Kit Documentation
https://www.thundercomm.com/product/cm2290-c2290-development-kit/
About the Project:
This project showcases how to build an Android application that estimates the depth value (distance relative to the camera) of every pixel from a single (monocular) RGB image using the Thundercomm TurboX CM2290 Development Kit. The application leverages Qualcomm Neural Processing SDK and employs the PackNet-SfM model, trained on the KITTI Dataset. This app/feature is crucial for scene understanding in applications such as 3D scene reconstruction and autonomous driving. It serves as a guide for developers on how to utilize this platform for autonomous vehicle use cases.
Upon starting the Android application, the DLC model and required library files are loaded from the development kit. The application then captures camera input and feeds the frame into the inference engine to obtain predictions. After post-processing, the depth-estimated image and video are saved onto the development kit . Once this process is complete, the application can be stopped, and the post processed image/video should be downloaded from the development kit to the host PC for interpretation.
Prerequisites:
On the Host PC –
The instructions below use a Linux workstation as host PC
- Download the Android Studio version suitable for your host PC.
- Install Android Platform tools (ADB, Fastboot).
- Download OpenCV Android SDK, Go to https://opencv.org/releases/ , for any OpenCV version, select ‘Android’ to download.
Download and set up the Qualcomm Neural Processing SDK for AI
1. Steps to build and deploy application on host PC
1. Clone the project
$ git clone <source repository>
$ cd Depth Estimation on CM2290 with SNPE on Android/2. Launch Android studio by executing ‘studio.sh’ file.
<android_studio_directory>/bin $ ./studio.sh
3. In Android Studio, from the file menu open the folder of cloned Android application source. Once in the folder ‘Depth Estimation on CM2290 with SNPE on Android’, select ‘SDK Manager’ icon, select ‘SDK Tools’ and check ‘Show Package Details’.
4. In the ‘NDK’ section select ‘17.2 version’ and ‘Cmake’ of ‘3.18.1’ and click on apply. This will download the Android NDK and CMake required by Qualcomm Neural Processing SDK.
5. Go to ‘CMakeLists.txt’ file and make the following changes for setting up OPENCV_DIR and include directories of Qualcomm Neural Processing SDK
set(OpenCV_DIR "<opencv_directory>/opencv-4.8.0-android-sdk/OpenCV-android-sdk/sdk/native/jni").
include_directories(<SNPE_ROOT>/include/zdl)
2. Steps to run the Depth Estimation Android application on the development kit-
Before running the application, set the development environment on Thundercomm TurboX CM2290 development kit -
1. Connect the development kit to host PC via a USB C cable.
2. In the host PC terminal, enter adb devices to see the dev kit listed as an adb device.
3. In the host PC terminal, continue with the commands below -
$ adb root
$ adb disable-verity
$ adb reboot
$ adb root
$ adb remount
4. Turn on Wi-Fi in android kit. Select the available Wi-Fi by entering password.
5. Set the latest UTC time and date in setting->Date and time setting.
6. Copy the Qualcomm Neural Processing SDK Library files ( snpe-1.68.0.3932) from host PC to the development kit -
$ adb push <SNPE_ROOT>/lib//<...-android-clang6.0/ /system/lib/7. Copy the DSP files of Qualcomm Neural Processing SDK (snpe-1.68.0.3932) ) from host PC to the development kit
$ adb push <SNPE_ROOT>/lib/dsp/ /system/vendor/lib/rfsa/adsp/8. Copy the libcdsprpc.so file from “/vendor/lib” folder to “/system/lib” folder -
$ adb shell
/# cp /vendor/lib/libcdsprpc.so ./system/lib/9. Push the required database files to internal storage of device.
/# cd storage/emulated/0/
/# mkdir appData
/# exit
$ adb shell
$ adb push Depth Estimation on CM2290 with SNPE on Andriod/models/depth_model.dlc /storage/emulated/0/appData/models/ Steps to run the main application are as follows:
1. In Android Studio, generate APK file for the source ‘Depth Estimation on CM2290 with SNPE on Android’ -
a. Go to Build menu and select Build Bundle(s)/APK(s) from the dropdown then select Build APK(s).
b. Above step will download APK file in the following directory -Depth Estimation on CM2290 with SNPE on Andriod /app/build/outputs/apk/debug/app-debug.apk
2. Install APK file on Thundercomm TurboX CM2290 development kit. In the host PC terminal, enter -
$ adb install app-debug.apk3. Open the application on Thundercomm TurboX CM2290 development kit.
Note: Alternate way to run application on device.
a. Make sure that device is properly connected to host system.
b. Go to Android Studio on host system, from ‘menu’ click on ‘run’. It will launch application on Thundercomm TurboX CM2290 development kit.
4. Once the application opens allow the camera permissions. It will open the live camera and keeps taking the camera frames.
5. Depth estimated image and video saved on the development kit in ‘models’ folder.
6. Download depth estimated image/video from development kit to host PC. In host PC terminal, enter -
$ adb pull /storage/emulated/0/models/depth_image.jpg /home/
$ adb pull /storage/emulated/0/models/depth_video.avi /home/
Depth Estimation image out of live video from Thundercomm TurboX CM2290 is shown below.
