Elder Care with Qualcomm® Robotics RB3

Skill Level	Area of Focus	Operating System	Platform/Hardware	Cloud Services/Platform
Intermediate	Computer Vision, Healthcare, Robotics	Linux	RB3 Robotics Dev Kit	Amazon AWS IoT

This project is a proof of concept that demonstrates real-time object detection and pose estimation using an application from alwaysAI running on the Qualcomm Robotics RB3 platform, and an object detection ML model trained with AWS Sagemaker.

Objective

The project showcases the powerful combination of the Qualcomm Robotics RB3 platform, alwaysAI's deep learning computer vision developer platform, and the training capabilities of AWS Sagemaker. The alwaysAI computer vision development platform is designed to speed creation and deployment of machine learning applications on edge devices, such as Qualcomm Robotics RB3. The developer platform works with many AI model framework technologies and endpoint environments, and comes pre-loaded with a growing model catalog, starter apps, and core computer vision APIs (including object detection, tracking and counting; image classification, pose estimation, and semantic segmentation) enabling the development of a wide range of computer vision applications.

 

Materials Required / Parts List / Tools

• Qualcomm Robotics RB3 Development Kit
• TurtleBot 3

Additional Resources

Using Pose Estimation and Object Detection to Rescue the Elderly

Usage Instructions

In this project, the human is represented by a toy figure and the robot is detecting whether they are fine (standing) or in need of assistance (fallen down). The robot moves closer to the fallen person and could be extended to incorporate Alexa to be able to “speak” with the fallen person and take action accordingly.

• The Robot Operating System (ROS) application, which encompasses alwaysAI computer vision application, running on Qualcomm Robotics RB3 Development Kit uses a local model, trained in AWS Sagemaker, to identify humans in video frames, develop situational awareness about the human, and offer assistance as needed.
• The person detection model and the pose estimation models are pre-trained in AWS Sagemaker and downloaded to the Qualcomm Robotics RB3 platform.
• The alwaysAI application is designed to run object detection and pose estimation on frames received from the camera. The results are used to decide any action needed and are displayed on a streamer service. ROS application reads the decision to move the robot and operates the wheels accordingly.
• The robot detects human activity by analyzing the sequence of poses and classifies the human’s situation as safe or in need of help. In near future, the application will be extended to send a message to AWS IoT MQTT topic. The ROS application will integrate with Amazon Kinesis Video Streams ROS extension to capture live video feeds. We plan to use AWS Greengrass for deploying models on the robots. We also plan to integrate Alexa to initiate a conversation with the human under distress, call 911, etc.

Qualcomm Robotics RB3 is a product of Qualcomm Technologies, Inc. and/or its subsidiaries.