Evaluation science for AI is the argument that the way we currently measure AI systems is too informal and fragmented to be trusted, and that the field needs a mature measurement discipline instead. The position was articulated in the 2025 paper “Toward an Evaluation Science for Generative AI Systems” by Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell, and several co-authors.
The core critique targets validity. Static benchmarks, the paper argues, face validity challenges: a high score on a benchmark may not mean the model is actually good at the real-world task the benchmark is supposed to stand in for. At the same time, the alternative of ad hoc, case-by-case audits rarely scales. The result is an evaluation ecosystem that is both unreliable and hard to extend.
To fix this, the authors draw on how safety is evaluated in transportation, aerospace, and pharmaceuticals. They propose three principles: metrics must reflect actual deployment contexts, metrics must be iteratively refined rather than fixed once and reused forever, and the field must build formal evaluation institutions and norms. In short, AI evaluation should grow from a collection of leaderboards into a scientific discipline.
For a general reader, this matters because nearly every claim about AI progress and AI safety rests on evaluations, and if those evaluations lack validity, the conclusions drawn from them, by companies, regulators, and the public alike, may be misleading.