Rephrasing Tweets Into Formal Language Boosts Topic Modeling Accuracy

Computer scientists studying social media discourse face a persistent problem: tweets and other short posts are too informal and scattered for conventional topic modeling tools to handle well. A new approach tackles this head-on by having language models standardize raw posts before running them through topic analysis.

The Challenge With Twitter Analysis#

Platforms like Twitter (now X) capture authentic public conversation during critical moments—the COVID-19 pandemic being a prominent example. Yet the very features that make this data valuable also undermine analysis: messages are brief, full of slang and abbreviations, and often noisy. When researchers fed these raw texts into standard topic modeling algorithms, the results tended to be messy and hard to understand, with topics appearing redundant or incoherent.

How TM-Rephrase Works#

Researchers developed TM-Rephrase, a technique that sits between raw social media text and existing topic modeling methods. The framework uses large language models to transform casual posts into more structured, formal language before analysis begins. The team tested two distinct rephrasing approaches: one that generalizes the content broadly, and another that specifically converts colloquial speech patterns into standard English.

Testing proceeded on a dataset of approximately 25,000 COVID-19-related tweets. The researchers measured three key performance indicators—how coherent topics were, how unique they remained from each other, and how diverse the overall topic set became—while also tracking redundancy across results.

Measurable Improvements#

The framework delivered meaningful gains across nearly all tested topic modeling algorithms. Specifically, TM-Rephrase improved topic coherence, uniqueness, and diversity while simultaneously reducing how many overlapping topics the models produced. The colloquial-to-formal rephrasing strategy outperformed the general rephrasing method, with particularly dramatic improvements when applied to Latent Dirichlet Allocation (LDA), a widely used traditional algorithm.

Broader Applications#

While this study focused on pandemic-related discourse, the researchers emphasize that their approach is model-agnostic—meaning it works with different topic modeling methods without requiring customization. This flexibility points toward wider use across public health monitoring, crisis response analysis, and any domain where understanding public sentiment through social media matters.