On September 1, 2026, the Allen Institute for AI (Ai2) published BenchMIRT, a method for auditing large language model benchmarks at the level of individual questions using multidimensional item response theory (MIRT). The underlying model was trained on benchmarking results from 100 LLMs across 16 benchmarks and more than 34,000 questions, treating each question as a probe of latent capability rather than trusting the benchmark’s overall label.
Rather than accepting a benchmark’s stated purpose - “safety,” “general reasoning,” “instruction following” - BenchMIRT lets the underlying capability dimensions emerge directly from the response data. Run without being told what each benchmark was designed to measure, it independently converged on two dominant, stable dimensions across repeated analyses: general reasoning and safety.
The audit surfaced concrete mismatches between a benchmark’s label and what it actually measures. BBQ, a social-bias benchmark, aligned more strongly with general reasoning than with safety. WMDP, a benchmark meant to test suppression of dangerous knowledge, showed an inverse correlation with reasoning ability, since refusing to answer is the correct behavior for a capable model. HarmBench’s copyright-infringement questions behaved more like a reasoning probe than its harmful-request questions did. Separately, the team found that keeping just 10 percent of a benchmark’s questions, chosen well, preserved nearly the same picture of model capability as running the full set.
For teams that build eval suites or make purchasing and deployment decisions off leaderboard scores, this is evidence that headline benchmark numbers conflate multiple abilities under a single label, and that most of a benchmark’s actual signal often lives in a small fraction of its questions. BenchMIRT is a tool for auditing existing benchmarks rather than a new capability benchmark itself - its value is in exposing what current evaluations actually reward, not in producing a new score to chase.