Public claim ledger · v1.1.0 · reviewed 2026-07-28
This index reorganizes existing public records only. Every card retains its status and limitation; no new substantive claim is introduced here.
Architecture, scaling, pre-training, post-training, and inference-time computation.
This is an architectural fact stated by the originating paper and universally adopted since.
Limits of this claim
Architecture alone determines no capability. Nothing about attention implies any particular task competence.
In-context few-shot behaviour was the central reported finding of the GPT-3 paper and is reproduced widely.
Limits of this claim
Reported on the task distributions studied. It is not a claim that in-context learning generalizes to arbitrary tasks, and it says nothing about reliability.
The originating scaling-law work fitted these relations across several orders of magnitude.
Limits of this claim
These are relations on loss within a studied range, not on task capability, and not guaranteed to hold outside it. Treating a loss extrapolation as a capability forecast is not supported by the source.
The Chinchilla work revised the earlier parameter-heavy allocation toward a more balanced ratio, and large deployed models are trained well past compute-optimal token counts to reduce inference cost.
Limits of this claim
The direction of the revision is sourced; the specific exponents and ratios are not restated here, and no claim is made about what any particular current model used.
The InstructGPT work established this as a distinct post-training stage rather than a property of pre-training.
Limits of this claim
Instruction-following is not correctness, safety, or capability. Improvements on preference judgements do not transfer automatically to task accuracy.
Chain-of-thought prompting demonstrated that the inference procedure, not only the weights, determines measured performance.
Limits of this claim
A produced reasoning trace is not evidence of the process that generated the answer. The source measures outputs, not internal reasoning.
This trade-off is the basis of reasoning-style systems and is reported with verifier-based and search-based methods.
Limits of this claim
Demonstrated within specific task families and compute budgets. It is not established as a general substitution, and the studied regimes do not license extrapolation.
How retrieval and external tools change the system being evaluated.
The originating formulation combined a retriever with a generator for knowledge-intensive tasks.
Limits of this claim
Retrieval changes the source of knowledge, not the reliability of reasoning over it. It does not eliminate fabrication.
Both learned tool invocation and interleaved reason-act prompting are documented mechanisms.
Limits of this claim
Tool access is a property of the deployed system, not the model. Comparisons that vary tool access are not model comparisons.
This is the explicit finding of the SWE-agent work: interface affordances, not only model quality, determine outcomes on repository tasks.
Limits of this claim
It follows that an agentic benchmark score measures a model-plus-scaffold system. Attributing such a score to the model alone is unsupported, and this map never does so.
What benchmarks measure, how measurements drift, and why reliability changes long-horizon performance.
That is the benchmark definition given by its authors.
Limits of this claim
Passing tests is not correct engineering. The benchmark does not measure design quality, maintainability, review, or deployment safety, and its authors do not claim it does.
Both are defined by their originating papers, and GPQA reports expert baselines with and without web access.
Limits of this claim
Multiple-choice accuracy over a fixed distribution is not general knowledge or reasoning. Specific baseline percentages are deliberately not restated here; consult the sources.
MMLU-Pro was introduced as a more robust successor to MMLU for precisely these reasons.
Limits of this claim
A score change across a benchmark revision is a measurement change, not necessarily a capability change. Trends must not be spliced across versions.
Multiple studies investigate contamination across widely used benchmarks and argue for per-benchmark measurement.
Limits of this claim
The extent of contamination for any specific model is generally not knowable from outside, because training corpora are not disclosed.
This critique argues that benchmarks index performance on a constructed task distribution and cannot underwrite claims about general ability, and that generalization must be measured differently from skill.
Limits of this claim
This is a methodological argument, not a demonstration that any specific score is wrong. Both critics and benchmark authors are represented in the source set.
Holistic evaluation frameworks measure many metrics across many scenarios rather than reporting one number.
Limits of this claim
Reporting more metrics does not resolve construct validity. It makes the trade-offs visible; it does not make any single metric mean more.
Longitudinal measurement of a hosted service found behaviour changes across dates on identical inputs.
Limits of this claim
A measurement of a hosted endpoint is dated. Reproducibility claims about API-based evaluations require the date and, where available, the model version.
These figures are taken from the abstract of the METR paper, read during this pass. They are the only time-horizon numbers in this map, and they replace the draft report’s unverifiable figures.
Limits of this claim
The authors state limitations including external validity, and their five-year statement is explicitly conditional on the trend generalizing. The measurement is over a specific task suite, not real-world software work. Figures later than this paper are not carried here: the draft report’s "~2 hours as of late 2025" was not independently verified during this pass and is excluded.
The METR paper attributes the horizon increase primarily to greater reliability and ability to adapt to mistakes, combined with logical reasoning and tool use.
Limits of this claim
This is an attribution offered by one research group over one task suite. The decomposition between reliability, reasoning, and scaffolding is not independently established.
This follows from the benchmark definitions, the scaffolding result, the contamination literature, and the drift measurement taken together. It is the organizing rule of this map.
Limits of this claim
This is a methodological position, not a finding that any score is incorrect. It constrains interpretation, not measurement.
Self-reports, forecast instability, and the boundary between a conditional scenario and a prediction.
The GPT-4 technical report states that it withholds such details; model cards are a reporting convention, not an audit mechanism.
Limits of this claim
Self-report status is not an accusation of inaccuracy. It means the figures were not produced by an independent party and cannot be treated as independent measurements.
Large surveys of published AI researchers record materially different aggregate timelines across successive rounds.
Limits of this claim
Instability in aggregate forecasts is evidence about forecasters, not about capability. Specific dates and shift magnitudes are not restated here; consult the source.