Withdrawn Research Proposes Hybrid Method to Swap Language Model Tokenizers Efficiently

A research paper submitted to arXiv proposed a solution to a persistent limitation in language model deployment: the difficulty of switching to different tokenization schemes without massive computational costs. The work, now withdrawn, outlined approaches for making tokenizers more flexible and adaptive.

The Core Problem#

Large language models lock in a tokenization strategy during pretraining, creating friction when developers want to use different tokenizers—particularly when working across multiple languages or specialized domains. Swapping tokenizers normally demands extensive retraining, making the process prohibitively expensive for most practitioners.

Proposed Solutions#

The research introduced two main contributions:

• TokenAdapt: A method for transplanting tokenizers that doesn't depend on a specific model architecture. It initializes new token embeddings using a hybrid approach: combining local estimates derived from breaking down words using the old tokenizer with global estimates that pull information from the top semantically similar tokens in the original vocabulary. This dual strategy aims to preserve meaning while cutting down retraining overhead.

• Supertokens: A pre-tokenization learning technique for multi-word token sequences designed to improve compression efficiency and reduce how often text fragments across token boundaries.

Experimental Results#

According to the authors, zero-shot testing showed that TokenAdapt's initialization strategy consistently outperformed existing methods like ReTok and TransTokenizer across multiple base models and new tokenizers. The team reported at least a twofold improvement in perplexity ratios compared to ReTok baselines.

Current Status#

The paper, initially submitted in May 2025 and last revised in September 2026, has been withdrawn from arXiv and is no longer available for download.