Evidence & decision layer · v1.1.0 · updated 2026-07-28
Tool use vs. autonomous execution
Does access to a tool establish autonomous task completion?
The bounded distinction
Tool use means a model can invoke external functions and incorporate returned results. Autonomous execution additionally requires an iterated system to select actions, handle failures, observe outcomes, and terminate appropriately across a task.
Why it matters for a decision
Treat tool access as one capability of a system. For higher-stakes workflows, ask for measured success, failure recovery, supervision, permissions, and rollback—not a tool-call demonstration.
Where this is context-dependent
Autonomy is not a binary property. The required evidence grows with task length, consequence, environmental variability, and the ability to reverse errors.
Existing claim records
Language models can be trained or prompted to invoke external tools and incorporate returned results.
Both learned tool invocation and interleaved reason-act prompting are documented mechanisms.
Limits of this claim
Tool access is a property of the deployed system, not the model. Comparisons that vary tool access are not model comparisons.
The design of the agent-computer interface materially changes measured coding-agent performance holding the underlying model fixed.
This is the explicit finding of the SWE-agent work: interface affordances, not only model quality, determine outcomes on repository tasks.
Limits of this claim
It follows that an agentic benchmark score measures a model-plus-scaffold system. Attributing such a score to the model alone is unsupported, and this map never does so.
Increases in measured long-horizon performance are attributed in part to improved reliability and error recovery rather than to reasoning improvements alone.
The METR paper attributes the horizon increase primarily to greater reliability and ability to adapt to mistakes, combined with logical reasoning and tool use.
Limits of this claim
This is an attribution offered by one research group over one task suite. The decomposition between reliability, reasoning, and scaffolding is not independently established.
A benchmark score is a measurement of a system on a task distribution under a scaffolding configuration at a date. It is not a measurement of general capability.
This follows from the benchmark definitions, the scaffolding result, the contamination literature, and the drift measurement taken together. It is the organizing rule of this map.
Limits of this claim
This is a methodological position, not a finding that any score is incorrect. It constrains interpretation, not measurement.
Supporting source records
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick et al. · 2023
Learned external tool invocation from the model side.
Source record →ReAct: Synergizing Reasoning and Acting in Language Models
Yao et al. · 2022
The interleaved reason-and-act pattern underlying most agent loops.
Source record →SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang et al. · 2024
The load-bearing source for separating scaffolding capability from model capability.
Source record →SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al. · 2023
The benchmark definition for test-verified resolution of real repository issues.
Source record →Measuring AI Ability to Complete Long Software Tasks
Kwa et al. (METR) · 2025
The primary definition of the 50%-task-completion time horizon. Abstract read during this pass; it is the only verified source for any time-horizon figure in this map.
Source record →NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz et al. · 2023
Contamination as a per-benchmark measurement problem rather than an anecdote.
Source record →How is ChatGPT's behavior changing over time?
Chen, Zaharia, Zou · 2023
Documents that a fixed prompt against a served model is not a stable measurement over time.
Source record →AI and the Everything in the Whole Wide World Benchmark
Raji et al. · 2021
The construct-validity critique of general-purpose benchmarks.
Source record →