Topic Models Capture Two Distinct Forms of Meaning—and Task Performance Depends on Which

Recent integration of large pre-trained language models into topic modeling has transformed the field, but a fundamental question remains: are these newer systems capturing the same kinds of meaning as their classical predecessors? New research from arXiv demonstrates they are not—and that the difference matters.

Two Ways Words Relate to Each Other#

The study, titled "Disentangling Similarity and Relatedness in Topic Models", identifies a crucial distinction rooted in psycholinguistics. When words appear in a topic, they can connect in two fundamentally different ways:

• Taxonomic similarity: Words that belong to the same category (dog/wolf, car/truck)
• Thematic relatedness: Words that naturally co-occur or associate within a scenario (dog/bone, car/road)

Classical co-occurrence models like Latent Dirichlet Allocation (LDA) tend to emphasize thematic relationships—the contextual bonds that glue words together in real text. By contrast, models augmented with modern language models shift toward taxonomic similarity, clustering words by their categorical kinship.

Measuring Both Dimensions#

To quantify this distinction across multiple topic models, the researchers built a large synthetic benchmark of word pairs using LLM-based annotation and trained a neural scorer to evaluate both similarity and relatedness. Testing this scorer across multiple corpora and model families revealed a clear pattern: different topic-model families occupy distinct positions in the joint similarity-relatedness space, forming a kind of semantic map.

Why It Matters for Real Applications#

The research shows that downstream task performance depends critically on alignment between a model's semantic structure and the task at hand. Applications requiring categorical knowledge benefit from similarity-rich topics, while those relying on contextual understanding gain from relatedness-focused approaches. When a model over-emphasizes either axis, performance degrades on tasks tuned to the opposing semantic structure.

The findings underscore an important insight: neither axis is universally superior. The semantic structure that makes a topic model excellent for one job may undermine it for another. Measuring both therefore provides a practical, model-agnostic diagnostic tool for evaluating what kind of meaning a topic model truly captures—and whether it matches the demands of downstream work.