Concept record · edition 1.0.0
Pre-training
What the term means here
The large-scale self-supervised phase in which a model learns from an unlabelled corpus.
Why it matters
Determines what is in the model before any alignment, and is where benchmark contamination enters.
Related claim records · 4
- si-002
Sufficiently large autoregressive language models perform tasks from instructions and examples supplied in context, without gradient updates.
- si-003
Language-model loss follows empirical power-law relationships with training compute, parameter count, and dataset size over the studied ranges.
- si-004
The compute-optimal allocation between parameters and training tokens has been revised in the literature, and deployed practice diverges from compute-optimal training.
- si-014
Contamination of evaluation data into training corpora is a measured problem that must be assessed per benchmark rather than assumed absent.
Sources used here · 5
Language Models are Few-Shot Learners
Brown et al. · 2020
Introduced in-context few-shot learning as an emergent property of scale.
Source record →Scaling Laws for Neural Language Models
Kaplan et al. · 2020
The original power-law scaling result, and one side of the compute-allocation disagreement.
Source record →Training Compute-Optimal Large Language Models
Hoffmann et al. · 2022
The Chinchilla revision of compute-optimal allocation, and the other side of that disagreement.
Source record →The Llama 3 Herd of Models
Grattafiori et al. (Meta) · 2024
A comparatively detailed provider training report; still a self-report.
Source record →Solving Quantitative Reasoning Problems with Language Models
Lewkowycz et al. · 2022
Domain-targeted training on quantitative reasoning; cited for method, not for score comparisons.
Source record →