Depth Mapping with Qualcomm® Robotics RB3 Dev Kit

A wide variety of robotics use cases need Depth Mapping for the robots to navigate. This project is designed to help you get started with depth mapping on the Qualcomm Robotics RB3 Development Kit. The project will show how you can capture a raw image with depth map and use it for future use cases such as background blurring etc.

Objective

To create depth mapping ecosystem using Qualcomm Robotics RB3 platform for use-case development.

Materials Required / Parts List / Tools

• Robotics DragonBoard™ 845c development board - Based on the Qualcomm® SDA845 processor
• Power Adapter - 12 V with 2500 mA required by the 96Boards specification
• USB to Micro USB cable - For serial console interface and ADB, Fast boot commands
• USB to USB Type C cable - For connecting the USB3.0 Type C port and flashing images
• Ubuntu 14.04 LTS Linux distribution for 64-bit architecture - Host PC OS for Qualcomm Robotics RB3 SDK build process

Source code: Camera Application

camera_test.tar

(found on the Thundercomm Qualcomm Robotics RB3 Development Kit web page, under the Technical Documents tab in the “Sample-apps-codes” section)

Build / Assembly Instructions

Process for Creating Ecosystem for Depth Mapping in Qualcomm Robotics RB3 platform:

1. Download the Camera application source code in host machine for camera sample codes from Thundercomm website under ‘Sample_apps_codes’ (camera_test.tar)

2. Build and Transfer the application from host machine to Qualcomm Robotics RB3 as mentioned in the Qualcomm Robotics RB3 Linux User Guide

3. In order to capture a raw image with depth properties, operate the cameras in "Three camera concurrency modes" as discussed below.

4. Follow the instructions provided below to understand the commands in the program and then set the device’s four cameras in three different concurrency modes

a) Build and execute the downloaded "camera_test" application on the host machine. This will generate “hal3_test” binary. Run the application ‘hal3_test’ with option ‘h’ to get the detailed instructions.“ The main purpose of this binary is to capture image streams provided by the cameras.

b) This lists down the instructions to be prompted in a particular order to set the camera modes and also to trigger snapshots and videos

c) As shown in the commands above, option ‘A’ will add a camera, which can be specified by mentioning ID. For ex:

- ID 2 is specific to camera 2 to receive instructions

d) Similarly, one can specify preview image size and format, also snapshot size and format through the options given against prompts

e) Let us understand how to set the three camera concurrency mode to capture preview images with depth map

• i. Run the camera application "hal3_test" on the device. Add CAM1(ToF), CAM2(Main Camera) and CAM3(Tracking), which is one the three concurrency combination with ToF being common to capture the depth.
• ii. Give options to the prompts to add cameras and capture the preview with specified format and size and given in the below commands

f) Add other two cameras from the current camera prompt as below and set the preview size and format one after the other

g) Follow the below instructions to quit the application after deleting the cameras (D: Delete camera device, Q:Quit)

5. Similarly, we can capture the depth images with other combinations with ToF, being common with the set of instructions in the same order as given below

• a) CAM1(ToF), CAM2(Main), CAM0(Stereo):

b) CAM1(ToF), CAM3(Tracking), CAM0(Stereo)

The output folder for the images captured is "/data/misc/camera/".

6. The hardware for ToF has certain specifications and parameters for which the images captured are observed to have the following outcome

Camera can switch between two modes namely “Mode 0” and “Mode 1”

Following are the screenshots of sample depth map images captured using 3 camera concurrency modes

(images cannot be viewed using image viewers, a special tool called imageJ as mentioned below, is needed.)

These images can be viewed using “imageJ” tool. Download the tool from https://imagej.nih.gov/ij/ on the host machine.

With the limitations of depth sensing range varying from 0.2m to 4.5m, a depth image is captured in which the objects nearer are darker than those which are far.

Note:

1. Qualcomm Robotics RB3 ToF V4T module provides about 1% depth error and 2% depth variation (standard deviation).

2. The depth sensing ranges from 0.2 m to 4.5 m with '90-degree x 70-degree' wide FOV (Field of View), multiple signal output of depth, IR and background

3. Light scattering occurs due to unwanted reflections within the optics. This cannot be eliminated completely but can be avoided as much as possible by placing the device at a greatest distance possible from disturbing objects

4. Corners of depth map has large depth error due to small mismatch between lens FOV (Field of View) and diffuser FOI (Field of Illumination) of light source in the current TOF module