AI agents now evaluate research code with 71% accuracy, easing peer review bottleneck

Peer review of research software is becoming a logistical challenge. As submission volumes climb, traditional human-led artifact evaluation—the process of verifying that papers' supporting code actually works—is increasingly strained. A new study from arXiv suggests that AI agents could help close this gap.

The Problem: Growing Evaluation Demand#

Artifact evaluation has become standard practice at software engineering conferences, linking published papers to the tools and code they describe. The process traditionally requires human experts to manually test submissions against vague criteria. Current ACM badge policies describe what deserves recognition (reproducibility, availability, generality) but offer no concrete, executable rules for verification—forcing both reviewers and machines to rely on judgment calls.

A Structured Approach: ArtifactGuide and ArtifactCopilot#

Researchers at arXiv developed two complementary tools. ArtifactGuide is a detailed scoring rubric derived from official ACM policies, calibrated by domain experts, and validated against real artifacts. ArtifactCopilot is an AI agent that follows a systematic investigation sequence, gathering evidence step-by-step and rendering a final badge decision based on accumulated findings.

The framework was tested on 60 actual research artifacts from recent software engineering conferences, using human-reviewed badges as ground truth.

Key Results#

ArtifactCopilot achieved 70.56% exact agreement with human-assigned badges—the highest performance among all tested systems. More tellingly, it was the only approach that successfully completed every evaluation run without crashing while producing a formal review report each time.

Compared to baseline approaches using only ACM policy language as prompts, ArtifactGuide boosted agent accuracy by 10.55 to 28.34 percentage points across three evaluation runs.

Practical Value for Researchers#

A controlled study involving eight experienced researchers showed that ArtifactCopilot's reports made reviewers more confident in their decisions, helped them identify supporting evidence more easily, and clarified what was being assessed. The analysis also revealed practical lessons for artifact authors: clearer mapping between claims and outputs, more explicit reuse instructions, and better documentation substantially improve evaluation outcomes.