Study finds readers prefer LLM health explanations despite weaker persuasion and moral grounding

A new study examining how large language models communicate health information uncovers a surprising gap between what readers prefer and what communication experts consider effective.

Researchers analyzed nearly 1,500 health misinformation claims and their corrections, comparing explanations from established fact-checkers with responses generated by LLMs. When nearly 100 participants evaluated both versions in a blinded review, over 60% favored the AI-generated content—yet analysis showed these same AI articles performed worse on multiple established quality measures.

What the research measured#

The team evaluated communication across three key dimensions: linguistic features, persuasive techniques, and alignment with social values and moral reasoning. LLM-generated text ranked significantly lower on persuasive strategies, certainty expressions, and alignment with social values and moral foundations. Despite these shortcomings, the AI versions won readers over.

The paradox of preference#

Participants cited structured presentation, clarity, and neutral tone as reasons they favored LLM articles, suggesting readers associate these qualities with completeness and professionalism. The very traits that made the content seem polished—straightforward formatting and dispassionate language—may have masked reduced nuance, rigor, or emotional connection.

Why it matters#

The findings point to a critical mismatch in health communication: reader satisfaction doesn't necessarily track with communication quality as researchers understand it. As LLMs become embedded in fact-checking and health guidance, this gap raises questions about whether AI systems should be optimized for reader preference or for measures known to change minds and build trust—and whether those goals can always align. The research, grounded in health communication theory, suggests that appearance of certainty and structure may sometimes work against deeper persuasion or moral alignment in correcting health misinformation.

See the full study on arXiv (2505.08143).