HalluLens is a hallucination benchmark from Meta researchers, introduced in an April 2025 paper accepted to ACL 2025. Its central argument is that the field has used the word “hallucination” loosely, lumping together different failure modes, so it first proposes a clear taxonomy. It separates intrinsic hallucination, where the generated text contradicts the input or prompt it was given, from extrinsic hallucination, where the output departs from the model’s training data and invents content. The paper also distinguishes hallucination from factuality, treating them as related but separate problems.
A practical feature of HalluLens is that its extrinsic-hallucination tasks can be dynamically regenerated. Because static benchmarks tend to leak into training data over time and become easier to game, regenerating fresh test items keeps the benchmark from saturating and helps ensure that high scores reflect real ability rather than memorization. The benchmark combines these new extrinsic tasks with existing intrinsic tasks and provides a broad comparison across models, with code released publicly.
For organizations choosing models, HalluLens is useful because it forces precision about what kind of error is being measured. “Reduces hallucination” means different things for a summarizer that must stay faithful to a source versus a chatbot that must avoid inventing facts, and a taxonomy makes that distinction measurable.