Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$ and $D$.
This predictability makes scaling laws highly valuable in practice. A common workflow is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models.
The predictability of generalization error with scale had already been investigated before scaling laws became a mainstream concept.
Amari et al. (1992) derived four types of learning curves using a Bayesian approach and the annealed approximation.
Source: Lilian Weng - Lil'Log — Read the full article on the original site ↗