PAPER 11
FP8 Quantization: The Power of the Exponent
Andrey Kuzmin (Qualcomm AI Research)
Mart Van Baalen (Qualcomm AI Research)
Yuwei Ren (Qualcomm AI Research)
Markus Nagel (Qualcomm AI Research)
Jorn Peters (Qualcomm AI Research)
Tijmen Blankevoort (formerly Qualcomm AI Research)
NeurIPS 2022
Summary
Recently, a new 8-bit floating-point format (FP8) has been suggested for efficient deep-learning network training. As some layers in neural networks can be trained in FP8 as opposed to the incumbent FP16 and FP32 networks, this format would improve efficiency for training tremendously. However, the integer formats such as INT4 and INT8 have traditionally been used for inference, producing an optimal trade-off between network accuracy and efficiency. This paper in-depth investigates this benefit of the floating-point format for neural network inference. We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa and exponent, and show analytically in which settings these choices give better performance.
Citation
@article{kuzmin2022fp8, title={FP8 Quantization: The Power of the Exponent}, author={Kuzmin, Andrey and Van Baalen, Mart and Ren, Yuwei and Nagel, Markus and Peters, Jorn and Blankevoort, Tijmen}, journal={arXiv preprint arXiv:2208.09225}, year={2022} }
Results
We validated the FP8 format for many networks in a post-training quantization setting, showing that for neural networks the 5M2E and 4M3E FP8 format works the best, and that for networks with more outliers like transformers increasing the number of exponent bits works best. We have also shown that when doing quantization-aware training, many of these benefits of the format disappear, as the network learns to perform well for the INT8 quantization grid as well.
Looking for more papers with code?
* Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
