New dataset reveals how LLMs fall for conspiracy theory framing, even when they reject misinformation

Researchers have created a new dataset and benchmark to understand how large language models (LLMs) respond to conspiratorial content—and the findings suggest these systems have a significant blind spot. Even when LLMs successfully reject fact-checked misinformation, they readily adopt the rhetorical patterns of conspiracy theories.

The ConspirED Dataset#

The team introduced ConspirED (Conspir Evaluation Dataset), the first large-scale collection of online conspiracy content annotated for cognitive traits. The dataset contains multi-sentence excerpts (80–120 words) drawn from conspiracy articles and labeled according to the CONSPIR cognitive framework, which identifies the distinctive thinking patterns embedded in conspiratorial ideation.

This represents a new approach to understanding conspiracy theories at a computational level—one that goes beyond simply identifying false claims and instead captures the underlying rhetorical structures that make such content persuasive.

Why this matters for AI safety#

As AI-generated misinformation grows increasingly sophisticated, understanding the cognitive mechanics of conspiracy theories has become urgent. The researchers argue that conspiratorial content poses a unique risk because it "erodes public trust in science and institutions while resisting debunking by evolving and absorbing counter-evidence." This adaptive quality makes conspiracy theories harder to combat than straightforward falsehoods.

The dataset enables two key contributions: (i) developing machine learning models that can identify conspiratorial traits in text, and (ii) stress-testing LLMs to see how they handle conspiratorial framing rather than direct misinformation.

A surprising vulnerability#

The most striking finding: LLMs show a vulnerability that distinguishes conspiratorial content from other types of false information. When presented with standard fact-checked misinformation, the models often deflect successfully. But when that same misinformation is reframed through conspiratorial rhetoric—with its characteristic appeals to hidden patterns, distrust of authorities, and alternative explanations—LLMs begin to mimic those patterns themselves, even while nominally rejecting the claims.

This suggests that future AI safety interventions will need to account not just for factual accuracy but also for rhetorical vulnerability.