AI and research methodology
Research proposes guardrails for using AI to conduct academic literature reviews
New guidelines urge caution and human oversight when deploying generative AI for systematic research surveys in software engineering and beyond.
Researchers have developed preliminary recommendations for how to safely use and evaluate generative AI tools when conducting systematic literature reviews—comprehensive academic surveys that synthesize evidence on a particular research question.
The gap between capability and rigour#
Large language models excel at text summarization and rapid information synthesis, yet their use in academic research raises fundamental concerns. Systematic literature reviews demand exacting standards: rigorous methodology, verifiable reliability, and transparent reporting of how conclusions were reached. Current generative AI systems offer no guarantee they can meet these benchmarks, even if they appear to work quickly and competently.
A framework for responsible AI use#
To address this gap, researchers conducted a rapid review of existing studies proposing guidelines for AI-assisted academic work, then synthesized their findings with established review best practices and their own expertise. The result is GUEST (GenAI Use and Evaluation in SLR Tasks)—a set of process recommendations covering three phases:
- Planning: How to design SLR projects that responsibly incorporate AI assistance
- Conducting: Practical considerations when using GenAI for review tasks
- Reporting: How to transparently document AI's role and performance
What AI can and cannot do#
The guidance concludes that generative AI is not ready to work independently on systematic reviews. Instead, the tools show promise in narrower roles: automating repetitive, low-stakes tasks (such as filtering large batches of papers) and providing secondary validation for complex assessments where human reviewers can verify AI conclusions. These applications could reduce the time and cost of conducting reviews without sacrificing their academic integrity.
Next steps#
The GUEST recommendations are designed to help software engineering researchers—and by extension, researchers across other fields—both carry out trustworthy AI-assisted reviews and rigorously evaluate how well the tools perform in these contexts. As AI becomes more central to academic workflows, explicit guardrails and honest assessment of AI's limitations remain essential.
