AI peer-review system produces longer, more granular reports than human reviewers

Researchers have created and evaluated xPeer, a system designed to simulate scholarly peer review using AI, revealing significant contrasts between how machines and humans assess academic manuscripts. The comparison offers a window into both the capabilities and limitations of AI-assisted publishing infrastructure.

What xPeer does#

xPeer operates as a peer-review simulation engine, accessible through a web interface. The system takes manuscripts and generates detailed critical reports comparable to those produced by human reviewers in academic publishing. The team assembled a dual-layer benchmark: one operational component analyzed 352 simulation records across multiple disciplines, and a separate human-reference component drew on 1,108 published manuscripts from F1000Research with paired human reviews and AI assessments.

How the evaluation worked#

The researchers followed a strict protocol to avoid biasing the AI system toward human critiques. Human review text and recommendations were deliberately withheld from the AI model during generation, with data linkage occurring only after xPeer had completed its assessments. This workflow-level separation ensures the AI wasn't trained on or primed by the human reviews it would later be compared against.

Of 802 manuscripts with exactly two human reports, 271 had usable paired xPeer assessments, yielding a 33.8% complete-pair availability rate.

The output divergence#

The systems produced markedly different review profiles. xPeer reports ran to a median of 1,889 words per manuscript versus 763 for human reviewers—more than twice as long. Concern density also differed sharply: the AI identified a median of 41 concerns per review against 13 for humans. xPeer's reports excelled at targeting specific issues, covering diverse categories, and presenting actionable recommendations.

Humans, by contrast, deployed more explicit reasoning language, made greater use of direct manuscript references, applied structured scientific taxonomy more systematically, and showed less within-report repetition.

Critical caveats#

The researchers emphasize that their analysis does not establish which approach is superior. Questions about the scientific accuracy of individual AI-identified concerns, the system's readiness for independent editorial deployment, and cross-system performance comparisons all require expert human judgment and standardized testing protocols. The distinct profiles suggest that AI and human review may be optimized for different aspects of the editorial process, rather than one simply outperforming the other.