---
markdown_generated_at: "2026-10-14"
title: "MSViT: Dynamic Mixed-scale Tokenization for Vision Transformers"
url: "https://www.qualcomm.com/research/artificial-intelligence/papers-with-code/paper-12"
description: "Explore MSViT Paper 12. Discover dynamic tokenization strategies for efficient vision transformers. Download the paper now to learn more."
canonicalUrl: "https://www.qualcomm.com/research/artificial-intelligence/papers-with-code/paper-12"
---

PAPER 12

# MSViT: Dynamic Mixed-scale Tokenization for Vision Transformers

Jakob Drachmann Havtorn (DTU)
Amelie Royer (Qualcomm AI Research)
Tijmen Blankevoort (formerly Qualcomm AI Research)
Babak Ehteshami Bejnordi (Qualcomm AI Research)

ICCV Workshops 2023

![](https://s7d1.scene7.com/is/image/dmqualcommprod/paper-12_MSViT_Dynamic%20Mixed-scale%20Tokenization%20for%20Vision%20Transformers?$QC_Responsive$&fmt=png-alpha)

## Summary

The input tokens to vision transformers carry little semantic meaning as they are defined as regular equal-sized patches of the input image, regardless of its content. However, processing uniform background areas of an image should not necessitate as much compute as dense, cluttered areas. To address this issue, we propose a new dynamic mixed-tokenization scheme. Our method introduces a conditional gating mechanism that selects the optimal token scale for every image region, such that the number of tokens is dynamically determined per input. In addition, to enhance the conditional behavior of the gate during training, we introduce a novel generalization of the batch-shaping loss.

[![](https://s7d1.scene7.com/is/image/dmqualcommprod/document-icon-v2_2?$QC_Responsive$&fmt=png-alpha) Paper](https://arxiv.org/abs/2307.02321)

[![](https://s7d1.scene7.com/is/image/dmqualcommprod/code-icon_2?$QC_Responsive$&fmt=png-alpha) Code](https://github.com/qualcomm-ai-research/batchshaping)

## Citation

*@inproceedings{Bejnordi2019BatchshapingFL, title={Batch-{S}haping for learning conditional channel gated networks}, author={Babak Ehteshami Bejnordi and Tijmen Blankevoort and Max Welling}, booktitle={ICLR}, year={2020} }*

*@inproceedings{havtorn2023msvit, title={MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers}, author={Jakob Drachmann Havtorn and Amelie Royer and Tijmen Blankevoort and Babak Ehteshami Bejnordi}, year={2023}, booktitle={ICCV Workshop on New Ideas in Vision Transformers}, }*

## Results

Our experiments on image classification and semantic segmentation show that the proposed dynamic tokenization enhances computational efficiency by reducing the number of input tokens, with minimal impact on performance.

Looking for more papers with code?

[View all papers](https://www.qualcomm.com/research/artificial-intelligence/papers-with-code)

\* Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.