Researchers Develop Statistical Model to Detect "Bullshit" in ChatGPT Output

Computer scientists have created a quantitative framework to identify the characteristic features of language generated by ChatGPT and similar large language models—framing the effort as a systematic hunt for what philosopher Harry Frankfurt termed "bullshit."

The Research Method#

The team compared linguistic patterns across 1,000 scientific publications with text produced by ChatGPT, then applied statistical hypothesis-testing to isolate the distinguishing marks of AI-generated discourse. Rather than dismissing complaints about LLM output as subjective, they sought empirical evidence for what makes chatbot language structurally different.

Connecting AI Output to Human Dysfunction#

The researchers drew inspiration from two influential critiques of obfuscatory language: George Orwell's analysis of how political speech becomes hollow and misleading, and David Graeber's essay on "bullshit jobs"—work that serves no genuine purpose. By testing whether the same linguistic signatures appear in ChatGPT's output and in these documented cases of human institutional dysfunction, they established a measurable link between artificial and natural forms of communicative evasion.

Why It Matters#

The work bridges natural language processing research with philosophical inquiry into language's social role—building on questions raised decades earlier by Margaret Masterman, an NLP pioneer and student of Wittgenstein. Rather than treating LLM artifacts as a purely technical problem, the study suggests that AI-generated language may exhibit the same patterns that help humans recognize when speech is designed to obscure rather than illuminate.