AI
New framework detects AI hallucinations in retrieval systems with fine-grained claim verification
Researchers introduce RT4CHART, a hierarchical testing method that breaks down AI-generated answers into verifiable claims and flags unsupported statements against source material.
A team of researchers has developed a new approach to catch false or unsupported statements in retrieval-augmented generation (RAG) systems—a common way large language models pull from external sources to answer questions. The system, called RT4CHART, offers a more nuanced way to spot when AI models produce information that contradicts or isn't grounded in retrieved documents.
The hallucination problem in RAG#
Large language models built to retrieve and use external context still struggle with accuracy. Despite having access to relevant source material, they can confidently generate claims that either conflict with that material or aren't mentioned in it at all. Current detection methods typically assign a single score to an entire answer, making it hard to pinpoint exactly where errors occur or what evidence contradicts them.
How RT4CHART works#
Rather than evaluating answers wholesale, RT4CHART breaks them down into independent, individually testable claims. The framework then works through verification in layers—starting with local checks of individual claims against the retrieved context, then performing broader consistency checks. Each claim receives one of three verdicts: entailed (supported by the source), contradicted (conflicts with the source), or baseless (not addressed by the source).
The system maps these claim-level judgments back to specific passages in the original answer and explicitly surfaces supporting or conflicting evidence from the retrieved material. This transparency lets auditors see exactly why a statement was flagged rather than relying on opaque numerical scores.
Performance results#
In testing on two datasets—RAGTruth++ (408 samples) and a newly re-annotated version of RAGTruth-Enhance (2,675 samples)—RT4CHART outperformed competing methods. On RAGTruth++, it achieved 84.5% precision, 71.8% recall, and an F1 score of 77.6%, representing an 83% improvement over the best existing approach. The system also reached 47.5% F1 on span-level detection in the larger dataset.
Ablation tests revealed that breaking answers into claims drove most of the gains, while the hierarchical global verification step offered supplementary benefits depending on the dataset.
