ReportLogic: A New Benchmark for Evaluating Logical Quality in AI-Generated Research Reports

As large language models increasingly power "deep research" tasks—synthesizing diverse sources into structured reports—a critical gap has emerged: current evaluation methods focus on fluency and apparent informativeness, but largely ignore whether a report's claims are actually supported by evidence and can be trusted for downstream decision-making.

A new paper from arXiv addresses this gap with ReportLogic, a benchmark designed to measure report-level logical quality through an "auditability" lens. Rather than assessing whether a report reads well, ReportLogic asks whether readers can verify its logic step-by-step.

A Three-Layer Framework#

The benchmark adopts a hierarchical taxonomy with three levels:

• Macro-Logic: Can readers trace a coherent, on-topic structure with a unified analytical arc?
• Expositional-Logic: Is the argument progression clear, with necessary context provided?
• Structural-Logic: Can conclusions be verified through explicit, documented claim–support relationships?

Building and Testing the Evaluator#

The researchers constructed a human-annotated dataset guided by detailed rubrics and trained LogicJudge, an open-source evaluation model. They then tested its robustness using adversarial attacks—a revealing exercise that exposed a troubling vulnerability in off-the-shelf LLM judges: these models are frequently swayed by superficial cues like verbosity rather than genuine logical structure. Reasoning modes, while useful, can sometimes mask broken support chains.

Implications#

The work provides practical guidance for developers building more robust logic evaluators and for improving the logical reliability of LLM-generated reports themselves. As research synthesis becomes a core AI capability, ensuring that these systems produce verifiable, auditable reasoning—not just plausible-sounding text—emerges as a critical frontier in AI evaluation.