Evidence & decision layer · v1.1.0 · updated 2026-07-28
Local inference vs. cloud inference
How does the delivery mode change what can be evaluated and governed?
The bounded distinction
Local inference runs on user-controlled hardware; cloud inference relies on a served endpoint. A hosted endpoint can change over time on fixed prompts, while a local deployment can make a fixed artifact and environment more reproducible.
Why it matters for a decision
For reproducibility, privacy, and auditability, record the deployment mode, model artifact or API version, date, configuration, and surrounding system—not just the prompt and output.
Where this is context-dependent
Local and cloud deployments differ in hardware, model availability, operations, security, and update mechanisms. The Atlas does not claim a current capability gap between them.
Existing claim records
The behaviour of a served model accessed through an API can change over time on fixed prompts.
Longitudinal measurement of a hosted service found behaviour changes across dates on identical inputs.
Limits of this claim
A measurement of a hosted endpoint is dated. Reproducibility claims about API-based evaluations require the date and, where available, the model version.
Provider technical reports and model cards are self-reports by the organization that built the system, and may withhold architecture, data, and training details.
The GPT-4 technical report states that it withholds such details; model cards are a reporting convention, not an audit mechanism.
Limits of this claim
Self-report status is not an accusation of inaccuracy. It means the figures were not produced by an independent party and cannot be treated as independent measurements.
A benchmark score is a measurement of a system on a task distribution under a scaffolding configuration at a date. It is not a measurement of general capability.
This follows from the benchmark definitions, the scaffolding result, the contamination literature, and the drift measurement taken together. It is the organizing rule of this map.
Limits of this claim
This is a methodological position, not a finding that any score is incorrect. It constrains interpretation, not measurement.
Supporting source records
How is ChatGPT's behavior changing over time?
Chen, Zaharia, Zou · 2023
Documents that a fixed prompt against a served model is not a stable measurement over time.
Source record →GPT-4 Technical Report
OpenAI · 2023
Cited as an example of a provider technical report, including its own disclosure that architecture and training details are withheld. Not an independent measurement.
Source record →Model Cards for Model Reporting
Mitchell et al. · 2019
The reporting convention whose existence is why provider self-reports are structured the way they are.
Source record →The Llama 3 Herd of Models
Grattafiori et al. (Meta) · 2024
A comparatively detailed provider training report; still a self-report.
Source record →SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al. · 2023
The benchmark definition for test-verified resolution of real repository issues.
Source record →SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang et al. · 2024
The load-bearing source for separating scaffolding capability from model capability.
Source record →NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz et al. · 2023
Contamination as a per-benchmark measurement problem rather than an anecdote.
Source record →AI and the Everything in the Whole Wide World Benchmark
Raji et al. · 2021
The construct-validity critique of general-purpose benchmarks.
Source record →