Study reveals stopwords follow different statistical pattern than regular words

A new analysis of word frequency distributions in language challenges the assumption that all words follow the same statistical pattern. The research reveals that function words—such as prepositions, conjunctions, and pronouns—exhibit fundamentally different distributional properties than content-rich vocabulary.

Breaking from Zipf's Law#

The classical Zipf's law, which describes how word frequency relates to word rank in language corpora, appears to hold well for most vocabulary. However, when researchers examined stopwords and function words separately, they found these linguistic building blocks deviate markedly from this pattern. Instead of following Zipf's familiar power-law curve, function words are better described by the Beta Rank Function (BRF). Non-function words, meanwhile, show a quadratic relationship between log-frequency and log-rank—still a departure from strict Zipfian behavior.

A New Selection Model#

To explain these observations, researchers propose a probabilistic model based on word rank. The model treats stopword and function word selection as following a decreasing Hill's function, while non-selection follows a standard Hill's function. This framework captures how structural words become concentrated among the most frequent items while still maintaining a distinct distribution shape.

Validation and Implications#

The selection probability model was tested on independent text collections and shown to reliably predict observed frequency patterns. The analysis also has unexpected implications for language development: the model predicts that Zipf's law should not hold for telegraphic speech in young children or patients with agrammatism—conditions characterized by heavy reliance on content words while function words are sparse.

The findings suggest that understanding word frequency distributions requires accounting for grammatical role, not just treating all vocabulary uniformly.