New Tokenizer Slashes Token Counts by Up to 25% While Keeping LLM Performance Intact

How language models break down text into tokens has a direct impact on both their speed and computational cost during inference. When tokenization fragments text too finely, sequences grow longer and generation becomes more expensive. Thunder-Tok, a newly developed tokenizer, tackles this efficiency challenge by cutting token counts without compromising downstream performance.

The Problem with Naive Token Expansion#

While longer, multi-word tokens can theoretically reduce the total number of tokens (what researchers call "fertility"), simply adding them to a tokenizer often backfires—weakening the language model's performance. This trade-off has long posed a challenge: how do you reap efficiency gains without paying a quality penalty?

How Thunder-Tok Works#

Thunder-Tok addresses this through a three-step approach:

• Building a seed vocabulary: The system extracts a large collection of candidate tokens from the training corpus as substrings
• Structural filtering: Invalid Unicode byte fragments and tokens that violate word boundaries are discarded, keeping only well-formed candidates
• Smart pruning: The remaining tokens are ranked using a likelihood-based score grounded in a Jensen-inequality lower bound of training-data probability, allowing the system to keep the most useful tokens while removing less valuable ones

Performance Gains#

Experiments demonstrate meaningful improvements across languages. Thunder-Tok reduces token fertility by approximately 25% in English and 9% in Korean compared with standard BPE tokenization, all while maintaining competitive performance on downstream tasks. These gains translate directly to faster inference and lower computational demands.

For a more detailed discussion, see the full paper on arXiv.