Concept record · edition 1.0.0
Scaling laws
What the term means here
Empirical relations between training compute, parameters, data, and loss.
Why it matters
Scaling laws are the basis of most quantitative capability forecasting, so their contested regions propagate directly into forecast uncertainty.
What this does not establish
That a loss curve extrapolates to a capability claim. Scaling laws are fitted on loss, not on task competence, and the mapping between them is not established.
Related claim records · 3
- si-003
Language-model loss follows empirical power-law relationships with training compute, parameter count, and dataset size over the studied ranges.
- si-004
The compute-optimal allocation between parameters and training tokens has been revised in the literature, and deployed practice diverges from compute-optimal training.
- si-007
Allocating additional computation at inference time can, in studied settings, improve results more than spending the equivalent compute on additional parameters.
Sources used here · 3
Scaling Laws for Neural Language Models
Kaplan et al. · 2020
The original power-law scaling result, and one side of the compute-allocation disagreement.
Source record →Training Compute-Optimal Large Language Models
Hoffmann et al. · 2022
The Chinchilla revision of compute-optimal allocation, and the other side of that disagreement.
Source record →PaLM: Scaling Language Modeling with Pathways
Chowdhery et al. · 2022
A large-scale training report cited for scaling practice, not for leaderboard position.
Source record →