Real-Time Object Detection and Navigation with Qualcomm® Robotics RB5 and TurtleBot3
The integration of advanced robotics and artificial intelligence is paving the way for increasingly capable and intuitive systems. This project showcases an innovative application of these technologies using the TurtleBot3 robot and Qualcomm® Robotics RB5 platform. By combining real-time object detection, voice command processing, and dynamic navigation, the system offers a comprehensive solution for autonomous robots. The TurtleBot3 leverages the SSD Mobile Net model for efficient object detection and the Whisper model for voice command interpretation, enabling it to identify and navigate target objects with precision.
Objective
The project aims to create a user-friendly robotic system that interacts with voice commands and autonomously navigates complex environments, enhancing the Qualcomm Robotics RB5 and TurtleBot3's capabilities with real-time object detection and dynamic navigation.
Desired Outcome
The goal is to develop a robust robotic system that accurately detects and locates objects in real time based on voice commands, enabling the TurtleBot3 to navigate safely and autonomously, avoiding obstacles and reaching targets efficiently.
| Equipment | Description |
| Qualcomm Robotics RB5 | Qualcomm Robotics RB5 Development Kit |
| Power adapter | 12 V with 2500 mA required by the 96Boards specification |
| USB to Micro USB cable | For serial console interface , to view logs |
| USB to USB Type C cable | For connecting the USB3.0 Type C port to the board and flashing images, adb and fastboot |
| Turtlebot Burger (TurtleBot3) | Off the shelf robot from Robotis |
Source Code: GitHub Link for project source code
https://github.com/globaledgesoft/Object-Detection-and-Navigation-with-TurtleBot3-and-Qualcomm-RB5
Qualcomm Robotics RB5 Development Kit bring up
Prerequisites:
- Install Python 3 on Qualcomm Robotics RB5 and Host Machine (Ubuntu 18.04)
- Install ROS Melodic on Qualcomm Robotics RB5 & Host Machine
- Install TurtleBot3 Packages on Qualcomm Robotics RB5 & Host Machine
- Install python libraries included in requirements.txt on Qualcomm Robotics RB5.
- Install ROS cv_bridge with Python3 on Qualcomm Robotics RB5
- Install Node.js & npm on Host Machine
- Set up Qualcomm® Neural Processing SDK Libraries on Qualcomm Robotics RB5
i) Download Qualcomm® Neural Processing SDK on host machine. Copy the header files & runtime libraries for aarch64-ubuntu-gcc7.5 from host system to Qualcomm Robotics using ADB
ubuntu@ubuntu: adb push <SNPE_ROOT>/include/ /data/snpe/include/
ubuntu@ubuntu: adb push <SNPE_ROOT>/lib/aarch64-ubuntu-gcc7.5/* /data/snpe/
ubuntu@ubuntu: adb push <SNPE_ROOT>/lib/dsp/* /data/snpe/ii) Open a terminal in Qualcomm Robotics RB5 and append the lines given below at the end of ~/.bashrc file.
sh4.4 # export PATH=$PATH:/data/snpe/
sh4.4 # export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/snpe/
sh4.4 # export ADSP_LIBRARY_PATH="/data/snpe;/system/lib/efsa/adsp;system/vendor/lb/rfsa/adsp;/dsp"iii) Run the command given below to reinitialize the terminal session.
root@ # source ~/.bashrcNote:
If device is connected via SSH, please use scp tool for copying the runtime libraries to /data/snpe folder on Qualcomm Robotics RB5.
Steps to set up LIDAR
- Connect LIDAR Scanner to Qualcomm Robotics RB5 board using microUSB cable.
- Once connected, make sure /dev/ttyUSB0 port is accessible.
Setting up the Camera
- Make sure that USB Camera is connected to Qualcomm Robotics RB5.
- Attach the camera to 1st layer of Turtlebot Burger facing front.
- Keep camera angled at 20-30 Degree facing down.
Install ROS Bridge Server on Host Machine
$ sudo apt install ros-melodic-rosbridge-serverNOTE:
Add below lines to the end of ~/.bashrc file, to avoid the environment variables exports every time when opening terminal. Make sure that you reopen the shell once the file is edited.
1. Add below lines in ~/.bashrc of Qualcomm Robotics RB5
# APPEND AT THE END OF ~/.bashrc
export ROS_MASTER_URI=http://<HOST_MACHINE_IP>:11311
export ROS_HOSTNAME=<RB5_IP>
export TURTLEBOT3_MODEL=burger
source /opt/ros/melodic/setup.bash
2. Add below lines in ~/.bashrc of Host Machine
# APPEND AT THE END OF ~/.bashrc
export ROS_MASTER_URI=http://<HOST_MACHINE_IP>:11311
export ROS_HOSTNAME=<HOST_MACHINE_IP>
export TURTLEBOT3_MODEL=burger
source /opt/ros/melodic/setup.bash
Prepare ROS workspace in Host Machine
First, initialize your ROS workspace according to the instructions here. Then, clone the source code Git repository into your workspace.
Set up workspace in Host Machine
To set up the ROS workspace on the host machine, follow these steps:
- Initialize the ROS Workspace: Create a directory for your ROS workspace, initialize it, and set up the necessary environment variables.
$ mkdir -p ~/OBJ_WS/src
$ cd ~/OBJ_WS/src2. Copy the packages from the assets folder to the ~/OBJ_WS/src directory. While the spiral exploration and TurtleBot3 movement after detecting the desired object can be built on the Qualcomm Robotics RB5, it is recommended to perform these steps on the host machine. In this setup, TurtleBot3 is treated as the client and the host system as the server.
$ cp -r assets/frontier_exploration_turtlebot
assets/move_robot
assets/riotu_robot_pose_publisher
~/OBJ_WS/src/3. Install all the dependencies through rosdep
$ cd ~/OBJ_WS
$ rosdep install --from-paths src --ignore-src --rosdistro melodic -yr4. Source ROS 1 and build
$ cd ~/OBJ_WS
$catkin_make -DPYTHON_EXECUTABLE=/usr/bin/python3
$ source ~/OBJ_WS/devel/setup.bash
5. Run the command given below to update the ~/.bashrc
$ echo source ~/OBJ_WS/devel/setup.bash >>
~/.bashrcSet up workspace in Qualcomm Robotics RB5
To set up the ROS workspace on Qualcomm Robotics RB5, follow these steps:
- Initialize the ROS Workspace: Create a directory for your ROS workspace, initialize it, and set up the necessary environment variables.
$ mkdir -p ~/OBJ_WS/src
$ cd ~/OBJ_WS/srcWe are utilizing the SSD MobileNet DLC model on the DSP of Qualcomm Robotics RB5 for real-time inference on each frame. This model publishes data to respective ROS topics, including Xmin, Ymin, Xmax, Ymax, xOffset, and area, which are essential for navigation packages.
2. Copy the package in the assets folder to the ~/OBJ_WS/src directory.
$ cp -r assets/object_detection ~/OBJ_WS/src/3. Install all the dependencies through rosdep
$ cd ~/OBJ_WS
$ rosdep install --from-paths src --ignore-src --rosdistro melodic -yr4. Source ROS 1 and build
$ cd ~/OBJ_WS
$ catkin_make -DPYTHON_EXECUTABLE=/usr/bin/python3
$ source ~/OBJ_WS/devel/setup.bash5. We have created two separate workspaces: one for object_detection and another for whisper_pkg, as they require different Python versions for building.
$ mkdir -p ~/WHISPER_WS/src
$ cd ~/WHISPER_WS/srci) Additionally, we are using the Whisper model for voice commands, which are processed and published to the robot to specify the target object it needs to find.
a) Refer this link to get the whisper-based tflite_model Execute the cell and download the whisper-base.tflite model.
b) Place the downloaded model(whisper-base.tflite) in assets/whisper_pkg/scripts
$ cp -r assets/whisper_pkg
~/WHISPER_WS/src/ii) Install all the dependencies through rosdep
$ cd ~/WHISPER_WS
$ rosdep install --from-paths src --ignore-src --rosdistro melodic -yriii) Source ROS 1 and build
$ cd ~/WHISPER_WS
$ catkin_make -DPYTHON_EXECUTABLE=/usr/bin/python3.8
$ source ~/WHISPER_WS/devel/setup.bashiv) Run the command given below to update the ~/.bashrc
$ echo source ~/OBJ_WS/devel/setup.bash >> ~/.bashrc
$ echo source ~/WHISPER_WS/devel/setup.bash >> ~/.bashrcExecution instructions
- On Host System, Source the melodic and run roscore.
$ roscore2. On Qualcomm Robotics RB5 shell, Source the workspace and launch bring up command.
sh4.4
roslaunch turtlebot3_bringup turtlebot3_robot.launch3. On host system run navigation
$ roslaunch turtlebot3_navigation turtlebot3_navigation.launch map_file:=map.yaml4. On host system run rosbridge for web visualization
$ roslaunch rosbridge_server rosbridge_websocket.launch5. On the host system, run the React application. We developed a web application using React for teleoperating and monitoring the robot through the Robot Operating System (ROS). This application also provides a camera view of the detected objects with bounding boxes.
$ cp -r assets/UI ~/UI && cd UI
$ npm install
$ npm start6. On host system for robot visualization on the web
$ rosrun riotu_robot_pose_publisher riotu_robot_pose_publisher7. On Qualcomm Robotics RB5, start the Object Detection Package
$ roslaunch object_detection rb5_obj_detection.launch8. On Qualcomm Robotics RB5, start the Speech Recognition Package
$ roslaunch ros_speech_recognization speech_recognize.launch9. Start the Exploration Package on Host System. You can specify '0' for linear searching or '1' for spiral pattern searching. This package also includes collision avoidance capabilities.
$ roslaunch frontier_exploration_turtlebot demo.launch10. on Host System, run the demo
$ rosrun move_robot Test.pyNote:
The Turtlebot3 initially searches for the object specified through voice commands. Once the object is detected, it becomes ready to listen for new commands and starts searching for new objects.
Demo:
The Turtlebot3 initially searches for the object specified through voice commands. Once the object is detected, it becomes ready to listen for new commands and starts searching for new objects.
Sign up for the Developer Newsletter.
Get software and hardware tool resources to help optimize your development delivered to your inbox weekly.
