GenAI Tools Won't Solve Qualitative Research in Software Engineering, New Study Warns

Generative AI has sparked optimism among researchers that it could streamline qualitative analysis in software engineering—but a new arXiv study urges caution, arguing that GenAI is far from a universal solution.

The Promise vs. Reality#

Claims that large language models can automate qualitative research have grown increasingly popular. However, researchers argue these assertions rest on a narrow band of successes that don't generalize across the full spectrum of qualitative methodologies used in the field. Qualitative research in software engineering encompasses everything from in-situ field studies rooted in interpretive theory to structured coding of data extracted from mining projects—each with its own demands.

Key Limitations of GenAI-Assisted Analysis#

The paper identifies critical gaps in how GenAI can serve qualitative work. GenAI support cannot be deployed uniformly; it must be thoughtfully tailored to match both the nature of the data being studied and the specific research approach in use. Applying a one-size-fits-all GenAI tool to diverse qualitative methods risks introducing systematic biases and overlooking the nuanced human dimensions that make qualitative research valuable.

What Comes Next#

The researchers lay out both the potential benefits and drawbacks of using generative AI in qualitative studies, then reexamine how to maintain research quality when AI plays a role in analysis. Their goal is to help scholars navigate the emerging landscape of AI-assisted qualitative methods with clear eyes about what GenAI can and cannot do.

The findings suggest the software engineering research community needs a more measured, method-specific approach to integrating generative AI into qualitative work.