PAPER 12
MSViT: Dynamic Mixed-scale Tokenization for Vision Transformers
Jakob Drachmann Havtorn (DTU)
Amelie Royer (Qualcomm AI Research)
Tijmen Blankevoort (formerly Qualcomm AI Research)
Babak Ehteshami Bejnordi (Qualcomm AI Research)
ICCV Workshops 2023
Summary
The input tokens to vision transformers carry little semantic meaning as they are defined as regular equal-sized patches of the input image, regardless of its content. However, processing uniform background areas of an image should not necessitate as much compute as dense, cluttered areas. To address this issue, we propose a new dynamic mixed-tokenization scheme. Our method introduces a conditional gating mechanism that selects the optimal token scale for every image region, such that the number of tokens is dynamically determined per input. In addition, to enhance the conditional behavior of the gate during training, we introduce a novel generalization of the batch-shaping loss.
Citation
@inproceedings{Bejnordi2019BatchshapingFL, title={Batch-{S}haping for learning conditional channel gated networks}, author={Babak Ehteshami Bejnordi and Tijmen Blankevoort and Max Welling}, booktitle={ICLR}, year={2020} }
@inproceedings{havtorn2023msvit, title={MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers}, author={Jakob Drachmann Havtorn and Amelie Royer and Tijmen Blankevoort and Babak Ehteshami Bejnordi}, year={2023}, booktitle={ICCV Workshop on New Ideas in Vision Transformers}, }
Results
Our experiments on image classification and semantic segmentation show that the proposed dynamic tokenization enhances computational efficiency by reducing the number of input tokens, with minimal impact on performance.
Looking for more papers with code?
* Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
