Legal AI System Tackles Phantom Citations, Lifts Citation Accuracy in Bar Exam Test

Legal AI systems are being put to work on high-stakes questions, yet traditional benchmarks measure only whether answers land on the right multiple choice. A new paper reveals a deeper problem: answers often cite sources that either don't exist or fail to back up the claimed legal rule.

Researchers introduced L-MARS, an open-source multi-agent system designed to audit and improve citation trustworthiness in legal question-answering. The system pairs document retrieval with multi-turn verification loops where different AI models fact-check each other's claims, and introduces a method called Faith-Search that re-verifies citations after an initial draft is complete.

The Citation Problem#

Current evaluation metrics for legal AI overlook what researchers call citation faithfulness—whether sources actually support the rules attributed to them. The team tested their system on 100 questions from a Bar Exam, classifying citation failures into six categories. They found that naive retrieval-augmented generation (RAG)—the standard approach of fetching documents and drafting an answer—left 34% of answers without citations and achieved a strict citation F1 score of just 0.13.

Multi-Agent Verification Lifts Performance#

The multi-turn judge loop proved effective. By having independent AI models from different families verify each other's work, the system improved citation F1 from 0.13 to 0.25 and reduced the no-citation rate to 13%. Importantly, adding more retrieval steps did not meaningfully improve accuracy on its own, suggesting that the quality of verification matters more than document volume.

Faith-Search: Targeted Reachability, Not a Cure-All#

The post-draft Faith-Search intervention re-checks and repairs citations that cannot be reached, dropping the unreachable citation rate below 1%. However, it did not improve overall citation F1 beyond the multi-turn loop, leading the researchers to frame it as a targeted fix for a specific failure mode rather than a general faithfulness breakthrough.

A follow-up audit on 50 questions from LegalSearchQA confirmed the pattern: retrieve-then-draft pipelines plateau around 0.75 citation F1, while a basic web-search baseline collapsed to 0.22 when audited by external judges. The findings underscore that citation-aware evaluation is essential for deploying legal AI responsibly.