Large Language Model (LLM) training and integration on Qualcomm® Robotics RB5 Development Kit
This project is a voice-activated chatbot application built with Flask, leveraging advanced AI models to transcribe, understand, and respond to user queries. It uses Google Cloud's Speech-to-Text for audio transcription and integrates the NeMo Large Language Model (LLM) API for response generation. Alternatively, it employs a PEFT-tuned sequence-to-sequence model for natural language processing tasks, ensuring flexible and accurate conversational interactions. The system supports real-time audio recording, transcription, and response delivery.
Objective
The inspiration for this project stemmed from the potential of integrating advanced AI-driven natural language processing (NLP) and speech recognition technologies into robotics. Working with the Qualcomm Robotics RB5 Development Kit, we aimed to explore how these technologies could enhance human-robot interactions. The project was motivated by the desire to create more intuitive and responsive communication systems within robotics, making interactions with robots more natural and efficient.
The desired outcome is to develop a highly accurate and flexible conversational AI system that can be seamlessly integrated into the Qualcomm Robotics RB5 Development Kit. This system will enable real-time audio transcription and intelligent response generation, improving the robot's ability to understand and interact with users. By leveraging advanced AI models like the NeMo Large Language Model (LLM) API and PEFT-tuned sequence-to-sequence models, we aim to create a versatile tool that enhances user experiences in various applications, from customer service robots to educational and assistive technologies.
| Equipment | Description |
|---|---|
| Qualcomm Robotics RB5 | Qualcomm Robotics RB5 Development Kit |
| Power adapter |
12 V with 2500 mA required by the 96Boards specification |
| USB to Micro USB cable USB to USB Type C cable |
For serial console interface , to view logs For connecting the USB3.0 Type C port to the board and flashing images, adb and fastboot |
| Turtlebot Burger (TurtleBot3) | Off the shelf robot from Robotis |
Source Code: GitHub Link for Project Code
https://github.com/globaledgesoft/LLM-Training-And-Integration-Using-Nemo-on-RB5.git
Qualcomm Robotics RB5 Development Kit bring up
https://docs.qualcomm.com/bundle/publicresource/topics/80-88500-6/Getting-started.html?product=1601111740013082
Prerequisites:
- A Linux host system with Ubuntu 18.04.
- Install Android Platform tools (ADB)
- Install SSH on host and Qualcomm Robotics RB5 for secure shell communication.
- Install python3 on host and Qualcomm Robotics RB5
Setting up the project
1. Steps to Create a Custom Model on NVIDIA NGC:
- Visit the NVIDIA NGC website. (https://ngc.nvidia.com/)
- Log in with your credentials.
2. Navigate to the NeMo LLM Service:
- From the main dashboard, navigate to NeMo LLM Service.
- In the sidebar, click on Customizations.
3. Create a Custom Model:
- Click on "Create Custom Model".
This will open a new setup page to configure your custom model.
4. Select Base Model:
- Choose a base model, i.e GPT-43B-002. This will be the starting point for your customization.
5. Configure Training Settings:
Training Type: Choose the appropriate training method, such as p-tuning.
Batch Size: Set the batch size. For instance, a batch size of 8.
Learning Rate: Set the learning rate. can set it as 0.0001.
Epochs: Define the number of epochs for training. In the example, it is set to 50 epochs.
Virtual Tokens: set the number of virtual tokens.
6. Upload Your Dataset:
Training Dataset: Upload your training dataset file, such as data.jsonl. Ensure the dataset is formatted correctly for the model.
Validation Dataset: Optionally, upload a validation dataset to monitor training performance.
7. Start Training:
Click Start Training to begin the customization process. You can monitor the progress on the dashboard.
8. Nemo LLM API Key:
Obtain an API key and organization ID for accessing the Nemo LLM service.
9. Steps to Generate Nemo API Key:
- Sign Up / Log In:
a. Visit the NVIDIA Nemo Cloud Service website (https://ngc.nvidia.com/).
b. If you do not have an account, sign up for one. If you already have an account, log in using your credentials.
- Access the NGC Dashboard:
a. After logging in, you will be directed to your NGC dashboard.
- Navigate to API Keys:
a. On the dashboard, click on your profile icon or name in the top-right corner.
b. From the dropdown menu, select "API Key" or "Setup" to access the API key management page.
- Generate a New API Key:
a. On the API key management page, click on the "Generate API Key" button.
b. You may be prompted to name your API key or specify its usage to help you remember its purpose.
- Copy and Store Your API Key:
a. Once the API key is generated, it will be displayed on the screen. Copy the key immediately and store it in a secure location.
b. Note that for security reasons, this will be the only time the full API key is displayed. If you lose it, you will need to generate a new one.
Steps to use the chatbot:
1. Install Conda on WSL2:
a. Follow the step-by-step guide to install Conda on Ubuntu Linux using WSL2.
b. Link: Install Conda on Ubuntu: Quick & Easy Guide (thesecmaster.com)
c. After installation, create a Conda environment by using the following commands.
$ conda create --name <env_name> python=3.8d. To check the conda environment list
$ conda env liste. To activate conda environment
$ conda activate <env_name>2. Install dependencies:
a. Run the following command to install all the required Python libraries listed in the requirements.txt file:
$ pip install -r requirements.txt3. LLM Model Inference on Qualcomm Robotics RB5:
a. push the Model to the Qualcomm Robotics RB5 through ssh root<RB5_IP> or adb push.
$ scp -r Inferencing root@<RB5_IP>:/home/b. Run the inference script to query the model using voice commands. The model will process the input and display the generated text in the user interface.
$ python3 app.py --nemo
