New technique compresses LLM reasoning by 8% using learned "supertokens"

A team of researchers has developed a method to make large language model reasoning more efficient by identifying and compressing repetitive patterns in how models work through problems. The approach reduces reasoning trace length by over 8% on average across multiple benchmarks while preserving accuracy.

How the compression works#

The method starts with an observation: when language models reason through complex problems, their token sequences fall into two distinct categories. Structural tokens repeat predictably—scaffolding phrases that appear across many problems—while organic tokens vary with the problem itself and contain the core problem-solving content.

The researchers applied byte-pair encoding (BPE) merges directly to a model's own reasoning traces to identify frequent structural patterns. These patterns are then bundled into single supertokens via supervised fine-tuning. The approach is model-agnostic and requires no architectural changes.

Results across benchmarks#

Testing on three model families across five mathematical reasoning benchmarks showed mixed but encouraging outcomes. Under statistical equivalence testing (TOST at a ±2 percentage-point margin), accuracy was equivalent or inconclusive in 13 out of 15 model–benchmark combinations. The method maintained performance on most tests; only DeepSeek-R1-Distill-Llama-70B on MATH-500 and OlympiadBench showed degradation.

Interpretability gains#

Beyond compression, the learned supertokens reveal interpretable reasoning patterns. Traces of correct solutions tend to show more recovery and verification steps, while incorrect traces display more repetitive hedging and unresolved counterarguments. This structural analysis offers a new window into what separates successful reasoning from flawed reasoning paths.

Open release#

The researchers have released their full pipeline as open-source code, making the approach available for further development and adoption.