Evidence & decision layer · v1.1.0 · updated 2026-07-28
LLMs vs. agentic systems
What is being assessed: a model, or a system that uses a model?
The bounded distinction
An LLM produces outputs from its weights and supplied context. An agentic system adds a loop, tools, an environment, stopping rules, and often evaluation-specific scaffolding. Measured system performance therefore cannot be assigned to the underlying model alone.
Why it matters for a decision
For procurement or research comparison, specify the model, interface, tools, permissions, environment, and recovery loop—not only the model name.
Where this is context-dependent
The model/system boundary depends on the task. In a short text-generation task, scaffolding may be minimal; in coding or operations tasks, interface and tool design can dominate outcomes.
Existing claim records
Language models can be trained or prompted to invoke external tools and incorporate returned results.
Both learned tool invocation and interleaved reason-act prompting are documented mechanisms.
Limits of this claim
Tool access is a property of the deployed system, not the model. Comparisons that vary tool access are not model comparisons.
The design of the agent-computer interface materially changes measured coding-agent performance holding the underlying model fixed.
This is the explicit finding of the SWE-agent work: interface affordances, not only model quality, determine outcomes on repository tasks.
Limits of this claim
It follows that an agentic benchmark score measures a model-plus-scaffold system. Attributing such a score to the model alone is unsupported, and this map never does so.
SWE-bench measures whether a system produces a patch that passes a repository’s existing test suite for a real issue.
That is the benchmark definition given by its authors.
Limits of this claim
Passing tests is not correct engineering. The benchmark does not measure design quality, maintainability, review, or deployment safety, and its authors do not claim it does.
Increases in measured long-horizon performance are attributed in part to improved reliability and error recovery rather than to reasoning improvements alone.
The METR paper attributes the horizon increase primarily to greater reliability and ability to adapt to mistakes, combined with logical reasoning and tool use.
Limits of this claim
This is an attribution offered by one research group over one task suite. The decomposition between reliability, reasoning, and scaffolding is not independently established.
A benchmark score is a measurement of a system on a task distribution under a scaffolding configuration at a date. It is not a measurement of general capability.
This follows from the benchmark definitions, the scaffolding result, the contamination literature, and the drift measurement taken together. It is the organizing rule of this map.
Limits of this claim
This is a methodological position, not a finding that any score is incorrect. It constrains interpretation, not measurement.
Supporting source records
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick et al. · 2023
Learned external tool invocation from the model side.
Source record →ReAct: Synergizing Reasoning and Acting in Language Models
Yao et al. · 2022
The interleaved reason-and-act pattern underlying most agent loops.
Source record →SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang et al. · 2024
The load-bearing source for separating scaffolding capability from model capability.
Source record →SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al. · 2023
The benchmark definition for test-verified resolution of real repository issues.
Source record →Measuring AI Ability to Complete Long Software Tasks
Kwa et al. (METR) · 2025
The primary definition of the 50%-task-completion time horizon. Abstract read during this pass; it is the only verified source for any time-horizon figure in this map.
Source record →NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz et al. · 2023
Contamination as a per-benchmark measurement problem rather than an anecdote.
Source record →How is ChatGPT's behavior changing over time?
Chen, Zaharia, Zou · 2023
Documents that a fixed prompt against a served model is not a stable measurement over time.
Source record →AI and the Everything in the Whole Wide World Benchmark
Raji et al. · 2021
The construct-validity critique of general-purpose benchmarks.
Source record →