Our dataset comprises 2900 data items, where each item contains:

• A video with audio of a user asking a question about the video.
• A transcription of a human-generated question asked during the recording, and a human-generated answer to that question.
• A ground-truth timestamp that marks when it's appropriate to answer the question.

Each item is assigned to one of 13 predefined semantic categories representing distinct visual reasoning capabilities:

The dataset is distributed in two files:

• videos.zip: 2,900 video files in .mp4 format.
• annotations.zip: Single .json file with all video annotations. This file is structured as a list of 2,900 dictionary entries, where each corresponds to one video and includes the following metadata and labels:
  • video: Video filename (e.g., "00000000.mp4").
  • question: Question asked in the video (e.g., "What am I holding in my left hand?").
  • answer: Complete answer to the question (e.g., "You are holding a Rubik's cube in your left hand.").
  • short_answer: Concise version of the answer with only the key information (e.g., "A Rubik's cube").
  • timestamp: Optimal time in the video to answer the question (e.g., "00:04.4").
  • category: Category of the question (e.g., "object referencing").

The dataset is intended for research purposes only. Please cite our paper if you use this dataset in your research.

@misc{pourreza2025visionlanguagemodelsanswerface,
    title={Can Vision-Language Models Answer Face to Face Questions in the Real-World?},
    author={Reza Pourreza and Rishit Dagli and Apratim Bhattacharyya and Sunny Panchal and Guillaume Berger and Roland Memisevic},
    year={2025},
    eprint={2503.19356},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2503.19356},
}

This dataset is intended for research purposes only and to support and contribute to the graph research community. The quality of the configuration space design and the collected execution times may be suboptimal and should not be considered as reference performances of the target device but rather as representative of the problem at hand for research purposes.

Data License Agreement - Research Use

At Qualcomm AI Research, we are advancing AI to make its core capabilities – perception, reasoning, and action – ubiquitous across devices. Our mission is to make breakthroughs in fundamental AI research and scale them across industries. By bringing together some of the best minds in the field, we’re pushing the boundaries of what’s possible and shaping the future of AI.
 

Qualcomm AI Research continues to invest in and support deep-learning research in computer vision. The publication of the AirLetters dataset for use by the AI research community is one of our many initiatives.

Find out more about Qualcomm AI Research.

For any questions or technical support, please contact us at [email protected]
 

Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.