Source record · content verified
What Will it Take to Fix Benchmarking in Natural Language Understanding?
primary paper2021
Why this source is here
Names the structural failure modes of benchmark-driven progress measurement.
- Verification
- Content-verified on 2026-07-27: the canonical source and its title were resolved during the Atlas review. This is not an endorsement of the source’s argument.
- Authors
- Bowman and Dahl
- Identifier
- arXiv:2104.02145
Claims citing this source · 2
- si-013
Benchmarks are revised by their owners in response to saturation and robustness problems, producing successor versions that are not directly comparable to the originals.
- si-015
The use of general-purpose benchmarks as measures of general capability is contested in the literature on construct-validity grounds.