Researchers refine foundational laws of language with hapax-based corrections

Computational linguists have proposed refined versions of two long-standing principles that govern word frequency patterns in natural language. The work, submitted to arXiv by Łukasz Dębowski, builds improved models of Zipf's law and Heaps' law by systematically accounting for hapaxes—words appearing exactly once in a text.

How the refinement works#

The corrections rest on two key ideas. First, the researchers apply the standard urn model, which assumes that shorter text samples behave as if words were drawn randomly from a larger corpus. Second, they introduce a mathematical relationship between text length and the proportion of hapaxes. This dual approach allows them to derive more accurate predictions about word frequency distributions than the classical formulas alone.

Four approaches tested#

The team evaluated four different models for how hapax rates change with text length:

• Constant model: assumes the proportion stays fixed
• Cancelation model: accounts for word saturation effects
• Linear model: predicts a straight-line relationship
• Logistic model: uses an S-shaped curve to capture saturation dynamics

Empirical validation#

Testing across a sample of 14 English texts, the logistic model delivered the strongest fit to actual data. This curve-based approach appears better suited to real language than simpler alternatives, suggesting that hapax rates follow a saturation pattern similar to other natural phenomena.

Looking ahead#

The authors note that larger corpora may require more sophisticated mixture models to capture vocabularies that operate under two distinct regimes—a direction they identify for future refinement of these foundational linguistic laws.