Evidence & decision layer · v1.1.0 · updated 2026-07-28
Benchmark performance vs. real-world reliability
What can a benchmark score establish about dependable work?
The bounded distinction
A benchmark score measures a system on a defined task distribution under a stated scaffolding configuration and date. Reliable real-world performance additionally depends on distribution shift, error recovery, interfaces, tool permissions, and the cost of failures.
Why it matters for a decision
Use benchmark results as scoped measurements. Before deployment, require task-relevant evaluation, repeated trials, error analysis, drift monitoring, and a plan for human oversight or recovery.
Where this is context-dependent
Transfer depends on whether the benchmark, environment, workflow, and reliability threshold match the decision at hand. A strong score is not evidence of general capability or dependable autonomy.
Existing claim records
SWE-bench measures whether a system produces a patch that passes a repository’s existing test suite for a real issue.
That is the benchmark definition given by its authors.
Limits of this claim
Passing tests is not correct engineering. The benchmark does not measure design quality, maintainability, review, or deployment safety, and its authors do not claim it does.
MMLU and GPQA are multiple-choice benchmarks over defined subject distributions, with human baselines reported by their authors.
Both are defined by their originating papers, and GPQA reports expert baselines with and without web access.
Limits of this claim
Multiple-choice accuracy over a fixed distribution is not general knowledge or reasoning. Specific baseline percentages are deliberately not restated here; consult the sources.
Benchmarks are revised by their owners in response to saturation and robustness problems, producing successor versions that are not directly comparable to the originals.
MMLU-Pro was introduced as a more robust successor to MMLU for precisely these reasons.
Limits of this claim
A score change across a benchmark revision is a measurement change, not necessarily a capability change. Trends must not be spliced across versions.
Contamination of evaluation data into training corpora is a measured problem that must be assessed per benchmark rather than assumed absent.
Multiple studies investigate contamination across widely used benchmarks and argue for per-benchmark measurement.
Limits of this claim
The extent of contamination for any specific model is generally not knowable from outside, because training corpora are not disclosed.
The use of general-purpose benchmarks as measures of general capability is contested in the literature on construct-validity grounds.
This critique argues that benchmarks index performance on a constructed task distribution and cannot underwrite claims about general ability, and that generalization must be measured differently from skill.
Limits of this claim
This is a methodological argument, not a demonstration that any specific score is wrong. Both critics and benchmark authors are represented in the source set.
Multi-metric evaluation across scenarios has been proposed and implemented as an alternative to single headline scores.
Holistic evaluation frameworks measure many metrics across many scenarios rather than reporting one number.
Limits of this claim
Reporting more metrics does not resolve construct validity. It makes the trade-offs visible; it does not make any single metric mean more.
The behaviour of a served model accessed through an API can change over time on fixed prompts.
Longitudinal measurement of a hosted service found behaviour changes across dates on identical inputs.
Limits of this claim
A measurement of a hosted endpoint is dated. Reproducibility claims about API-based evaluations require the date and, where available, the model version.
METR defines a 50%-task-completion time horizon: the time humans typically take on tasks a model completes with 50% success. Its published measurement reports a horizon of around 50 minutes for one frontier model of that period, and a doubling of the horizon roughly every seven months since 2019, with a possible acceleration in 2024.
These figures are taken from the abstract of the METR paper, read during this pass. They are the only time-horizon numbers in this map, and they replace the draft report’s unverifiable figures.
Limits of this claim
The authors state limitations including external validity, and their five-year statement is explicitly conditional on the trend generalizing. The measurement is over a specific task suite, not real-world software work. Figures later than this paper are not carried here: the draft report’s "~2 hours as of late 2025" was not independently verified during this pass and is excluded.
Increases in measured long-horizon performance are attributed in part to improved reliability and error recovery rather than to reasoning improvements alone.
The METR paper attributes the horizon increase primarily to greater reliability and ability to adapt to mistakes, combined with logical reasoning and tool use.
Limits of this claim
This is an attribution offered by one research group over one task suite. The decomposition between reliability, reasoning, and scaffolding is not independently established.
A benchmark score is a measurement of a system on a task distribution under a scaffolding configuration at a date. It is not a measurement of general capability.
This follows from the benchmark definitions, the scaffolding result, the contamination literature, and the drift measurement taken together. It is the organizing rule of this map.
Limits of this claim
This is a methodological position, not a finding that any score is incorrect. It constrains interpretation, not measurement.
Supporting source records
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al. · 2023
The benchmark definition for test-verified resolution of real repository issues.
Source record →Measuring Massive Multitask Language Understanding
Hendrycks et al. · 2020
The MMLU benchmark definition.
Source record →GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein et al. · 2023
The GPQA definition and its stated human baselines.
Source record →MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Wang et al. · 2024
A benchmark revision motivated by saturation and robustness problems in the original.
Source record →What Will it Take to Fix Benchmarking in Natural Language Understanding?
Bowman and Dahl · 2021
Names the structural failure modes of benchmark-driven progress measurement.
Source record →NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz et al. · 2023
Contamination as a per-benchmark measurement problem rather than an anecdote.
Source record →Investigating Data Contamination in Modern Benchmarks for Large Language Models
Deng et al. · 2023
Empirical contamination investigation across widely used benchmarks.
Source record →AI and the Everything in the Whole Wide World Benchmark
Raji et al. · 2021
The construct-validity critique of general-purpose benchmarks.
Source record →On the Measure of Intelligence
Chollet · 2019
The skill-versus-generalization argument and the ARC formulation, central to construct validity here.
Source record →Holistic Evaluation of Language Models
Liang et al. · 2022
The argument for multi-metric evaluation instead of a single headline number.
Source record →Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava et al. · 2022
A large collaborative benchmark explicitly concerned with extrapolating capability.
Source record →How is ChatGPT's behavior changing over time?
Chen, Zaharia, Zou · 2023
Documents that a fixed prompt against a served model is not a stable measurement over time.
Source record →Measuring AI Ability to Complete Long Software Tasks
Kwa et al. (METR) · 2025
The primary definition of the 50%-task-completion time horizon. Abstract read during this pass; it is the only verified source for any time-horizon figure in this map.
Source record →SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang et al. · 2024
The load-bearing source for separating scaffolding capability from model capability.
Source record →