Study Compares AI Counseling Models in Japanese: Structured Prompts Outperform Minimal Instructions

Researchers examining artificial intelligence's potential as a counseling training tool have tested how different prompt structures affect the quality of simulated therapy conversations in Japanese. The findings highlight both the promise and limitations of using large language models to generate training dialogues for counseling professionals.

Study Design and Conditions#

A team of researchers evaluated 18 synthetic counseling transcripts created through AI-to-AI interactions using three different setups. One approach used GPT-4-turbo with only basic role instructions (termed "GPT-minimal"), while the other two employed a structured methodology called the Structured Multi-step Dialogue Prompt (SMDP): GPT-4-turbo with SMDP, and Claude-3-Opus with SMDP. The SMDP framework provided detailed, multi-step guidance to shape how the counseling interactions unfolded.

Expert Versus AI Ratings#

Fifteen trained counseling experts assessed the transcripts using four adapted scales from the Motivational Interviewing Treatment Integrity coding manual, plus an overall quality rating. Simultaneously, three newer large language models independently scored the same dialogues across three separate rounds. The results showed a clear difference: conversations generated with the SMDP approach scored higher on expert evaluations for fostering change talk, establishing partnership, demonstrating empathy, and overall quality compared to GPT-minimal transcripts. The two SMDP-based models performed comparably to each other.

However, the LLM ratings diverged meaningfully from expert judgments. While the AI ratings proved internally consistent across iterations, they tended to be more lenient than human expert assessments—a gap particularly pronounced when evaluating how well conversations softened resistance and in overall quality judgments.

Key Limitations and Takeaways#

Simulated clients in the dialogues scored below the midpoint on naturalness scales, suggesting room for improvement in realism. Critically, the researchers emphasize that reproducible machine ratings should not be mistaken for validated counseling quality without human expert verification. The work provides a benchmark for Japanese-language AI counseling simulations but does not measure actual clinical effectiveness or client outcomes.