Benchmark contamination is what happens when the questions used to test a model have already appeared in the data the model was trained on. Modern models are trained on enormous amounts of text scraped from the internet, and popular benchmarks - with their questions and answers - are published on the internet too. When test material ends up in the training set, the model can score well by having effectively memorized the answers rather than by genuinely reasoning, so the benchmark no longer measures what it claims to.
The survey “Benchmark Data Contamination of Large Language Models: A Survey” by Cheng Xu and colleagues (2024) defines the problem directly: contamination occurs “when language models inadvertently incorporate evaluation benchmark information from their training data, leading to inaccurate or unreliable performance during the evaluation phase.” The survey reviews how contamination arises and catalogs the methods researchers use to detect it and to build evaluations that resist it - for instance, by continually refreshing test sets or rewriting questions so memorized answers no longer apply.
Contamination is closely related to benchmark saturation: as a benchmark ages, models climb toward the maximum score, partly through genuine progress and partly because the test has leaked into training data and the field has optimized hard against it. Either way, an old benchmark stops distinguishing strong models from weaker ones, which is why the community keeps introducing harder, fresher tests.
Why business readers should care: a headline benchmark number can overstate how good a model really is, especially for well-known public benchmarks the model may have seen during training. When evaluating models for your own use, the most trustworthy signal is performance on your own private data and tasks - material the model could not have memorized - rather than leaderboard scores alone.