Concept record · edition 1.0.0
Benchmark contamination
What the term means here
Presence of evaluation data, or close variants, in training corpora.
Why it matters
Contamination inflates scores without capability change, and is the main reason benchmark trends cannot be read as capability trends.
Related claim records · 3
- si-013
Benchmarks are revised by their owners in response to saturation and robustness problems, producing successor versions that are not directly comparable to the originals.
- si-014
Contamination of evaluation data into training corpora is a measured problem that must be assessed per benchmark rather than assumed absent.
- si-023
A benchmark score is a measurement of a system on a task distribution under a scaffolding configuration at a date. It is not a measurement of general capability.
Sources used here · 2
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz et al. · 2023
Contamination as a per-benchmark measurement problem rather than an anecdote.
Source record →Investigating Data Contamination in Modern Benchmarks for Large Language Models
Deng et al. · 2023
Empirical contamination investigation across widely used benchmarks.
Source record →