OnQ Blog

New research on quantization could revolutionize power-efficient AI

One of the most relevant challenges for deep neural networks is quantization, which is moving from 32-bit floating point models resulting from cloud-based training toward 16-bit or even 8-bit integer models, while minimizing the loss of accuracy (Figure 1). The big benefit of moving to 8-bit is higher power efficiency and smaller storage needs. This will make it possible to successfully run machine learning tasks such as voice recognition, navigation, and medical image analysis on smartphones and other edge devices with specialized 8-bit hardware. Based on work by Qualcomm AI Research, we authored “Data-Free Quantization through Weight Equalization and Bias Correction,” which has been accepted for an oral presentation at ICCV 2019. Our DFQ technique is a significant step toward automating the process of creating 8-bit models that are as accurate as the 32-bit starting point, without the need to fine-tune with additional data.

Figure 1: Quantization for AI provides many benefits.

Figure 1: Quantization for AI provides many benefits.

To achieve best-in-class power efficiency and performance on mobile devices, Qualcomm Technologies designs heterogeneous computing platforms that include 8-bit processing engines optimized for AI compute. Existing approaches to convert from 32-bit to 8-bit models usually lead to a significant loss of accuracy. In order to improve the accuracy for 8-bit models, current methods such as retraining, fine tuning, and hyperparameter setting are required, which can make the process quite arduous and time-consuming. In some cases, engineers must rework the architecture of the network itself.

Our data-free quantization method (DFQ) saves engineers time and effort. It does not rely on training data in the quantization phase, fine tuning, or hyperparameter setting. Instead, it proposes an automated way for converting deep learning models trained in 32-bit to run on 8-bit hardware. This is achieved by cross-layer equalization and correcting for biases. This method exploits the re-scaling and re-parameterization of the model to make it more robust to quantization. As an example of accuracy achieved for a machine learning task, the DFQ method applied to object detection networks results in less than 0.9% loss in accuracy all the way down to 8-bit quantization for models such as MobileNet V1, MobileNet V2, and Resnet 18. This is important to understand since we are achieving accuracy comparable to the original 32-bit model in an automated process without any training data (Figure 1).

Figure 2: Our DFQ quantization technique maintains close to FP32 accuracy up to 6-bit quantization, while the baseline quantization technique has a significant drop in accuracy starting from 12-bit quantization.

Figure 2: Our DFQ quantization technique maintains close to FP32 accuracy up to 6-bit quantization, while the baseline quantization technique has a significant drop in accuracy starting from 12-bit quantization.

DFQ can be easily applied to many common AI models for computer vision architectures with a straight-forward API call. The algorithm is contained in a compact library that is easy to call and integrate into a user’s workflow. In the product implementation of DFQ, a user will need to provide data, however, not at training time. The user provides a trained model and calls, for instance, run_model_quantized_on_device(trained_model). DFQ is then applied and outputs a quantized model in a format that is understood by the hardware. We are excited about this development and are looking forward to extending it to recurrent neural networks (RNNs). We’re also planning to incorporate DFQ research into the Qualcomm Neural Processing SDK so that the entire developer community can make use of these techniques. We are confident that with the DFQ method, our AI-powered edge devices can support a wide range of machine learning tasks that are set to change our lives for the better.

 

Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

 

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries ("Qualcomm"). Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.