Concept record · edition 1.0.0
Large language models
What the term means here
Neural networks trained on large text corpora to predict tokens, then adapted to follow instructions and produce text or actions.
Why it matters
The object every other concept in this map is about. Nearly all capability discussion reduces to statements about these systems under particular scaffolding.
What this does not establish
That these systems possess a general capability corresponding to human intelligence. No source in this map supports a single scalar capability score.
Related claim records · 3
- si-001
The transformer architecture replaces recurrence and convolution with attention as the primary sequence-modelling mechanism.
- si-002
Sufficiently large autoregressive language models perform tasks from instructions and examples supplied in context, without gradient updates.
- si-005
Fine-tuning with human feedback substantially changes how closely a model follows instructions relative to its pre-trained base.
Sources used here · 3
Attention Is All You Need
Vaswani et al. · 2017
The architecture paper underlying the transformer models this atlas is about.
Source record →Language Models are Few-Shot Learners
Brown et al. · 2020
Introduced in-context few-shot learning as an emergent property of scale.
Source record →On the Opportunities and Risks of Foundation Models
Bommasani et al. · 2021
Framing source for governance and accountability of general-purpose models.
Source record →