Evidence & decision layer · v1.1.0 · updated 2026-07-28
Retrieval vs. fine-tuning
Should a system receive information at inference time, or have its behaviour adapted through training?
The bounded distinction
Retrieval-augmented generation conditions an output on documents fetched at inference time. Fine-tuning changes behaviour after pre-training. They are different interventions and can be combined; neither is a generic substitute for the other.
Why it matters for a decision
Choose based on evidence requirements: retrieval can make the immediate source context inspectable, while fine-tuning targets behaviour or task adaptation. Evaluate both against the same task, corpus, and update assumptions.
Where this is context-dependent
Retrieval quality, corpus governance, prompt construction, and the fine-tuning objective all shape outcomes. The current source set does not support a universal performance ranking.
Existing claim records
Fine-tuning with human feedback substantially changes how closely a model follows instructions relative to its pre-trained base.
The InstructGPT work established this as a distinct post-training stage rather than a property of pre-training.
Limits of this claim
Instruction-following is not correctness, safety, or capability. Improvements on preference judgements do not transfer automatically to task accuracy.
Retrieval-augmented generation conditions outputs on documents retrieved at inference time rather than on parametric memory alone.
The originating formulation combined a retriever with a generator for knowledge-intensive tasks.
Limits of this claim
Retrieval changes the source of knowledge, not the reliability of reasoning over it. It does not eliminate fabrication.
A benchmark score is a measurement of a system on a task distribution under a scaffolding configuration at a date. It is not a measurement of general capability.
This follows from the benchmark definitions, the scaffolding result, the contamination literature, and the drift measurement taken together. It is the organizing rule of this map.
Limits of this claim
This is a methodological position, not a finding that any score is incorrect. It constrains interpretation, not measurement.
Supporting source records
Training language models to follow instructions with human feedback
Ouyang et al. · 2022
The post-training method that separated instruction-following from raw pre-training.
Source record →Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al. · 2020
The originating formulation of retrieval-augmented generation.
Source record →SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al. · 2023
The benchmark definition for test-verified resolution of real repository issues.
Source record →SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang et al. · 2024
The load-bearing source for separating scaffolding capability from model capability.
Source record →NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz et al. · 2023
Contamination as a per-benchmark measurement problem rather than an anecdote.
Source record →How is ChatGPT's behavior changing over time?
Chen, Zaharia, Zou · 2023
Documents that a fixed prompt against a served model is not a stable measurement over time.
Source record →AI and the Everything in the Whole Wide World Benchmark
Raji et al. · 2021
The construct-validity critique of general-purpose benchmarks.
Source record →