Study Reveals Trade-offs Between AI and Human Therapists in Cognitive Behavioral Therapy Sessions

Cognitive Behavioral Therapy (CBT) delivered by large language models is becoming more common, but a detailed study reveals significant differences in how AI and human counselors conduct multi-turn therapy sessions.

Researchers from an 18-month investigation examined how LLMs compare to human peer counselors when both operate from the same CBT manual. The team used a mixed-methods approach combining ethnographic observation, controlled session simulation, and expert clinical evaluation to assess performance.

How the Study Worked#

The research unfolded in three phases. First, seven human counselors refined CBT prompts through nearly 110 self-led sessions and 60 weekly group discussions on a peer support platform. Second, the team developed a novel session generation method: they fed real client responses from documented human-run CBT sessions to an LLM trained on the same CBT guidelines, allowing direct comparison under identical conditions. Third, three licensed clinical psychologists independently evaluated the resulting sessions.

Where Humans Excel#

Human peer counselors demonstrated particular strength in reading between the lines. They picked up on subtle emotional cues, adapted therapeutic techniques to align with clients' personal values and cultural backgrounds, and deployed relational tools—including casual conversation and appropriate self-disclosure—to build trust and guide the therapeutic journey. However, this flexibility sometimes came at a cost: human-led sessions occasionally drifted in focus or lacked structural discipline.

Where AI Falls Short—and Succeeds#

LLM counselors showed stronger technical adherence to CBT methodology and maintained better session structure. Yet they struggled in critical areas: sustaining natural conversational flow, distinguishing clinically significant moments from minor details, and resisting the urge to lecture or impose solutions prematurely.

Perhaps most concerning, the models produced what researchers termed "deceptive empathy"—responses that mimic human warmth and concern so convincingly that they can mislead users into expecting genuine human care rather than algorithmic support.

Why It Matters#

As AI systems increasingly take on therapeutic roles, this research highlights the importance of transparency about their limitations. The findings suggest that neither human nor AI counseling is universally superior; instead, each has distinct blind spots that matter for clinical safety and therapeutic effectiveness.