This dataset contains 33 unique robotics tasks ranging from pick, place, throw to sorting, stacking, toppling, and balancing scales.

• Pick
• Place
• Throw
• Touch
• Topple
• Push
• Rotate
• Match Pose
• Trace

• This dataset consists of 3 zip files, each containing data for 1 level of tasks. The zip files are split into ~10GB splits. You might require merging them before unzipping using "zip -s 0 simple_tasks.zip --out simple_tasks_merge.zip"
• Download size is ~111GB
• The archive contains directories named traj_{seed} where seed denotes the seed used to generate the episode in ClevrSkills environment
• Each directory corresponds to one episode and contains,
  • Videos (from multiple cameras – these are used as inputs for models)
  • Actions (Nx7 matrix in .npy format where N denotes the length of the episode)
  • Action labels (.npy file including text label for each action step)
  • Camera parameters (.npy files for camera parameters of each camera)
  • Ep_info.json (meta data of the episode)
  • Info.json (task specification / prompts and textures used in the episode)
  • Keysteps (images of keysteps of the task)
  • Prompt_assets.npy (images used in the prompts)
  • Rewards.npy (reward for each step in the episode)
  • Succes.npy (a label denoting if the task was successful or not at each timestep).

• The data was generated using our ClevrSkills environment suite.
  The environment suite is built on top of ManiSkill2 simulator. We make use of a model of UFACTORY xArm6 robot as the main robot. We also use object models from the YCB dataset and textures from Vima to make the objects more diverse.
• ClevrSkills include 33 different environments and corresponding oracle policies that can solve the task.
• The tasks are randomly initialized based on a random seed. We use different seeds to initialize a new task and then use the oracle policy to solve the task. We record the videos of the actions taken by the oracle policy in the environment along with other meta data including text annotations, rewards, etc.

Citation: 

@article{haresh2024clevrskills,
title={ClevrSkills: Compositional Language And Visual Understanding in Robotics},
author={Haresh, Sanjay and Dijkman, Daniel Bhattacharyya, Apratim and Memisevic, Roland},
journal={Advances in Neural Information Processing Systems},
year={2024}
}

This dataset is intended for research purposes only and to support and contribute to the graph research community. The quality of the configuration space design and the collected execution times may be suboptimal and should not be considered as reference performances of the target device but rather as representative of the problem at hand for research purposes.

Data License Agreement - Research Use

At Qualcomm AI Research, we are advancing AI to make its core capabilities – perception, reasoning, and action – ubiquitous across devices. Our mission is to make breakthroughs in fundamental AI research and scale them across industries. By bringing together some of the best minds in the field, we’re pushing the boundaries of what’s possible and shaping the future of AI.

Find out more about Qualcomm AI Research.

For any questions or technical support, please contact us at [email protected]

Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.