New Benchmark Reveals How AI Models Hallucinate Citations in Scientific Papers

Large language models trained to cite their sources still struggle to do so accurately. A new research benchmark aims to measure exactly how often they fail.

The authors of the REASONS benchmark—drawn from arXiv papers across 12 subject areas—argue that citation hallucination isn't just about factual errors; it's a complex reliability problem that demands rethinking how we evaluate AI-generated citations.

The Problem with AI Citations#

When LLMs generate responses that reference sources, they often invent citations that sound plausible but don't actually support the claim. This undermines trust in scientific communication. Existing evaluation methods focus mainly on correctness, but the researchers show that's only half the story: systems also need to know when to refuse to answer rather than guess.

Introducing REASONS#

The benchmark contains 12,723 sentence-level citation instances spanning diverse research areas. It evaluates how well models can attribute claims to authors or paper titles under a range of conditions:

• Zero-context prompting — models work from memory alone
• Metadata-augmented — given paper titles or author information
• Cascaded metadata — a multi-step prompting approach
• Retrieval-augmented — systems that search for relevant papers first
• Adversarial settings — deliberately misleading metadata

Key Trade-offs Emerge#

Rather than a single accuracy score, the researchers introduced a dual-metric framework: Abstention Rate (AR, when the model refuses to answer) and Hallucination Rate (HR, false citations). They found a sharp tension between the two.

Advanced retrieval-augmented generation (RAG) reduced hallucination from 87.6% to 65.4%—but at the cost of zero abstention, meaning the system always provided an answer. Simpler RAG approaches often maintained near-zero refusals while achieving high hallucination rates. Under adversarial conditions, some systems exceeded 85% hallucination rates.

Human Judgment Validates the Scale#

Manual review of 1,000 model outputs (κ=0.78) found a striking ratio: of factual hallucinations versus acceptable paraphrases. This suggests most errors are genuine failures, not borderline cases.