AI agents can automate research artifact quality checks, study finds

Ensuring reproducibility in empirical software engineering requires researchers to share complete, well-documented, and usable artifacts—yet evaluating replication packages remains largely a manual, labor-intensive process. A new paper from the arXiv cs.SE community proposes an agentic approach to automate this assessment at scale.

Distilling guidelines into machine-verifiable criteria#

The team consolidated 380 reproducibility requirements from 34 different sources into a streamlined set of 51 criteria. Of those, 31 were operationalized—that is, translated into checks that an automated system could perform. The criteria span structural elements like code availability and environment specifications, as well as qualitative aspects harder to verify computationally.

The multi-agent prototype#

The researchers built a prototype system comprising multiple agents that inspect replication packages and generate evidence-based reports recommending improvements. In a preliminary evaluation on five test packages, the system demonstrated 91.4% consistency across repeated runs. When compared to human reviews, it achieved 75.4% correctness on average using a micro-averaged metric.

Where the approach excels—and where it struggles#

The automated system performed well on structural checks: verifying whether code exists, environment files are present, and artifacts are accessible. It was less reliable with qualitative or mixed-method research, where human judgment about completeness and clarity remains essential.

Researcher feedback on adoption#

A pilot survey with seven software engineering researchers indicated that the system was perceived as useful and adoption-ready. However, responses also highlighted a potential friction point: the human-in-the-loop planning phase created notable cognitive load for users.

Implications for scaling artifact review#

Within the constraints of these small samples, the findings suggest that agentic evaluation could help both authors and reviewers by automating routine, mechanical checks—freeing human effort for more nuanced assessments. The work points toward a hybrid model where machines handle standardized verifications while experts focus on deeper reproducibility concerns.