SLM Model Compilation and Integration on Qualcomm® Robotics RB5

The rapid advancement of Artificial Intelligence and Machine Learning has paved the way for sophisticated conversational agents capable of understanding and responding to human language. This project focuses on leveraging the Google T5-Small model, a powerful transformer-based model, for inference tasks on the Qualcomm® Robotics RB5 platform. The primary objective is to create a voice-enabled chatbot tailored to handle queries related to Qualcomm® technologies and services.

Objective

Deploy the Google T5-Small model on Qualcomm Robotics RB5 to develop a voice-enabled chatbot specialized in responding to queries about Qualcomm technologies and services, utilizing its advanced AI capabilities for efficient inference tasks. This model avoids the complexities associated with running larger language models (LLMs) on embedded platforms.

Source Code: GitHub Link for project source code

https://github.com/globaledgesoft/SLM-Model-Training-Compilation-Integration-on-RB5

Qualcomm Robotics RB5 Development Kit bring up

https://docs.qualcomm.com/bundle/publicresource/topics/80-88500-6/Getting-started.html?product=1601111740013082

Model Training:

The Google T5-Small model will be fine-tuned using a dataset specific to Qualcomm Technologies. This dataset will include various queries and responses pertinent to Qualcomm® technologies, products, and services. The fine-tuning process will enhance the model’s ability to generate more accurate and contextually relevant responses.

Voice Command Integration:

To facilitate superior interaction, the chatbot will be equipped with voice recognition capabilities. Users will be able to input queries through voice commands, which will be converted to text for processing by the T5-Small model.

Inference and Response Generation:

The T5-Small model will perform inference on the input text, generating appropriate responses. This step utilizes the model's pre-trained language understanding and generation capabilities, fine-tuned for the specific use case.

Prerequisites:

1. A Linux host system with Ubuntu 18.04.
2. Install Android Platform tools (ADB)
3. Install SSH on host and Qualcomm Robotics RB5 for secure shell communication.
4. Install python3 on host and Qualcomm Robotics RB5

Steps to fine-tune the Google T5-small model

1. Install conda by using this link.
2. After Installation, create the conda environment by using the given commands

3. To check the conda environment list & activate

4. Run the requirements for fine-tunning the model

Data Example

Collect data relevant to your use case, which can be found in the SLM/Training directory. This directory contains both the training and test datasets necessary for model development and evaluation

Collect Data: Gather your dataset and save it in CSV format.

Organize Data: Place the collected data into the data folder.

File Naming: Ensure the CSV files are named as follows:

Training data: train.csv

Testing data: test.csv

Make sure the data folder contains these two files before proceeding with the next steps.

Training Information:

The model was trained for 50 epochs with a learning rate of 1e-3 to ensure efficient and effective learning. Here are the key details of the training configuration:

Epochs: 50 epochs, representing complete passes through the training dataset to achieve optimal performance.

Learning Rate: Set to 1e-3 to accelerate the training process while maintaining stability.

Batch Size: Dynamically determined using auto_find_batch_size for efficient GPU memory utilization.

Model: google/flan-t5-small, designed for tasks like text summarization and question answering.

LoRA Configuration:

r=16

lora_alpha=32

Target modules: ["q", "v"]

lora_dropout=0.05

Task type: SEQ_2_SEQ_LM

LoRA fine-tunes a subset of model parameters, reducing computational resources required.

Data Collator: DataCollatorForSeq2Seq used to handle padding, with label_pad_token_id set to -100 to ignore padding tokens in loss computation.

Logging: Training progress logged every 500 steps, reported to TensorBoard.

Int8 Training: Model prepared for int8 (8-bit integer) training to optimize memory usage and speed without significant accuracy loss.

By combining these techniques, the model efficiently learned and generalized from the training data, ensuring enhanced performance and effective fine-tuning with LoRA.

Fine-tune the Google T5 Model

Execute the fine-tuning script located in the SLM/Training directory.

Upon successful fine-tuning, it will generate the rb5_model, customized for your specific use case.

SLM Model Inference on Qualcomm Robotics RB5

To deploy the custom model and simple UI to the RB5, follow these steps:

Prepare the Files:
       Place the custom model and the simple UI in a folder.

Transfer the Files:

Using SSH:

Using ADB:

Replace <ip_rb5> with the IP address of your Qualcomm Robotics RB5, <path_to_your_folder> with the local path to your folder, and <destination_path_on_rb5> with the desired destination path on the Qualcomm Robotics RB5.

Start the Flask Application: Run the inference script to launch the Flask application.

This command will start the Flask server on rb5_ip:5000.

Use the Web Interface: Open a web browser and navigate to the Flask application's address (http://rb5_ip:5000). The user interface will be displayed.

Record and Process: Click the "Record and Process" button on the UI. You can now ask your question using voice commands.

Voice to Text Translation: The system will capture your voice input, translate it to text, and send it to the fine-tuned model for processing.

Display Generated Output: The model will generate a response based on the input question, and the output will be displayed on the screen in the user interface.