Ai2 finds LLM judges of research novelty swing from 93.5 to 40.9 percent accuracy on a prompt change

On October 27, 2026, Ai2 published “Old Ideas, Novel Problems,” a study of how reliably language models judge whether a research idea is novel. The team built benchmarks from ICLR 2026 peer reviews by mining reviewers’ comments on originality rather than using acceptance as a proxy: a Human-Only set of 154 high-novelty versus 145 lower-novelty ideas, and a Human+Generated set pairing the 154 human ideas with 154 from a simple LLM generator. Six judges were tested, from GPT-5.1 to GPT-5.4 and Claude Sonnet 4.5 to Opus 4.6.

Judges proved fragile. For GPT-5.4 on Human+Generated pairs, switching from a prompt that said one idea was novel to a plain “which is novel” prompt dropped accuracy from 93.5 percent to 40.9 percent, below chance. On human-only ideas the same change moved accuracy about 7 points, and 21 points for Claude Opus 4.6. When comparing two AI idea generators, roughly a quarter to a third of verdicts flipped across prompt changes.

More search did not help: an agentic GPT-5.6 Sol judge with arXiv search scored 0.78 macro-F1 versus 0.89 for Claude Opus 4.6, at about 10 times the per-decision cost. A simple Claude Opus 4.6 prompt scored 0.854 macro-F1 for $2.59 total, beating Idea Novelty Checker (0.642, $30.66) and Sakana’s AI Scientist evaluator (0.772, $82.05).

Why it matters: automated research systems increasingly use LLMs to decide which ideas are worth pursuing, and this shows those judgments can hinge on prompt wording, so rankings of AI idea generators built on them may be unstable.

What it does not show: labels on generated ideas are weak rather than verified per idea; the expert check covered only 30 pairs weighted toward the strongest judge’s errors; review mining was checked on 50 cases (47 agreed); and the authors call the benchmark a starting point.

Sources

Last verified November 9, 2026