A model is a collection of numbers. The format used to store and compute with those numbers affects how much memory the model occupies, how quickly the hardware can move the numbers, and which arithmetic units can process them. Changing precision can therefore make inference faster without changing the model architecture.
Lower precision is not automatically better. A format may save memory but run slowly when the target hardware lacks a suitable kernel. It may represent large values but lose small differences, or preserve fine differences near zero but overflow on an activation outlier. The useful question is not which format has the fewest bits. It is which format meets the quality requirement and is implemented efficiently on the deployment hardware.
Floating-Point Precision and Its Effects to Model Inference.Photo by Jean-Luc Benazet. Some rights reserved.
A floating-point number contains a sign, an exponent, and a significand (often called the mantissa). The exponent determines the range of magnitudes that can be represented. The significand determines how precisely nearby values can be distinguished. Reducing either field creates a trade-off.
Source: Machine Learning Mastery — Read the full article on the original site ↗