During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight uses half the raw storage of an FP16 or BF16 weight.
Quantization can also make computation faster. Many AI GPUs can perform low-precision matrix operations faster than FP16 operations. The actual speedup depends on the hardware, kernels, and model shapes.
Weight-only quantization stores selected weights as integers while keeping activations in FP16 or BF16. This post begins with the asymmetric affine mapping, then shows how to quantize and run models with PyTorch and other inference tools.
Note: The code examples in this post use TorchAO and require a CUDA-enabled PyTorch installation. You can install the dependencies with:
Source: Machine Learning Mastery — Read the full article on the original site ↗