New Benchmark Tests AI Fact-Checkers on Real-World Social Impact

Automated fact-checking powered by large language models (LLMs) has become a key tool for combating online misinformation, but a new research study reveals a significant gap: these systems often struggle to prioritize which claims actually need the most scrutiny based on social relevance.

Researchers from arXiv have introduced TrendFact, a new benchmark dataset designed to test whether AI-driven fact-checkers can perceive and allocate resources toward claims that have the greatest public impact. The work addresses what the team calls a "risk asymmetry challenge"—a situation where fact-checking systems waste computational resources on low-impact claims while potentially missing high-visibility misinformation.

The Core Problem#

Current fact-checking benchmarks lack the social metadata necessary to evaluate what researchers call Hotspot Perception Ability (HPA)—the capacity to dynamically channel reasoning effort toward the claims that matter most to communities and audiences. Real-world deployment under resource constraints demands exactly this kind of intelligent triage, yet existing evaluation frameworks don't measure it.

What TrendFact Offers#

The benchmark comprises 7,643 carefully curated samples drawn from trending social platforms and established fact-checking datasets, supported by an evidence library of 366,634 entries. To assess hotspot perception, the research team introduced two new evaluation metrics:

• Explanation Consistency Score (ECS): measures the reliability and coherence of the reasoning given by fact-checkers
• Hotspot Claim Perception Index (HCPI): quantifies overall hotspot perception performance across an entire system

Performance and Solutions#

Early testing reveals that existing AFC systems show limited effectiveness on TrendFact, suggesting the benchmark successfully identifies real weaknesses in current approaches. To address this, the researchers propose FactISR, a framework that enhances both hotspot perception and computational efficiency for reasoning-based LLMs used in fact-checking applications.

The work opens a new research direction: building fact-checkers that are not only accurate but also strategically aware of which claims deserve intensive verification effort.