New framework helps researchers design AI prompts for systematic content analysis

Researchers at arXiv have introduced HALC, a systematic framework designed to improve how large language models are used for automated content analysis in social science research. The approach addresses a longstanding challenge: while LLMs show promise for automating coding tasks, their performance often depends on trial-and-error prompt design that varies unpredictably across different models and datasets.

The Problem with Current Approaches#

Using LLMs for content coding—a foundational research method where text is systematically categorized—remains inconsistent. Researchers typically rely on manual tweaking and testing to find effective prompts, rather than following transparent, replicable procedures. This makes it difficult to assess reliability or apply lessons learned from one study to another.

How HALC Works#

The HALC pipeline provides a structured methodology for constructing prompts tailored to any coding task and LLM. Drawing on prior literature and preliminary research into what drives consistency in LLM outputs, the framework allows researchers to systematically test and refine prompts before deployment.

The researchers validated their approach on multiple datasets spanning different languages, document types, subject matter, and coding units—conducting over three million LLM requests in the process. Their results indicate that the pipeline successfully identifies reliable prompts across diverse settings.

Implications for Research#

By standardizing the prompt design process, HALC aims to make AI-assisted content analysis more transparent and reproducible. This could help social scientists integrate LLMs into their workflows with greater confidence, while also establishing shared best practices across the field. The authors acknowledge ongoing limitations and areas where the framework could evolve further.