New method transforms low-quality data into high-quality training material for language models

A research team has introduced a novel approach to addressing a persistent challenge in training large language models: most high-quality instruction data is scarce, while low-quality data is plentiful but typically discarded. Rather than waste that abundant imperfect material, they've developed a system to transform it into useful training content.

The problem with instruction data#

Instruction tuning is essential for aligning LLMs to follow user requests effectively. However, the quality of such training data varies dramatically. Organizations typically keep only premium datasets and throw out mediocre material—even though that discarded data contains valuable information. Existing augmentation techniques struggle to salvage low-quality inputs, and there's been little standardized evaluation of how well they work.

A new distillation approach#

The researchers formally defined a new task called Instruction Distillation: combining multiple flawed or redundant instruction inputs into single, refined instruction-output pairs. To develop and test their method, they created MIXTURE, a dataset of 144,000 examples pairing clusters of low-quality or semantically repetitive instructions with their high-quality distilled versions.

Building on this foundation, they introduced LM-Mixup, which consists of two stages. First, a language model undergoes supervised fine-tuning on the MIXTURE dataset. Then it's refined using reinforcement learning with three complementary reward signals: data quality, semantic consistency, and format correctness—optimized through Group Relative Policy Optimization (GRPO).

Remarkable efficiency gains#

The results demonstrate substantial practical value. When LLMs were trained exclusively on the distilled output—representing just about 3% of the complete dataset—they not only outperformed models trained on the full original dataset but also matched or surpassed specialized high-quality data selection techniques across multiple benchmarks.

This finding challenges conventional wisdom about data quantity versus quality, suggesting that thoughtfully processed imperfect data can be more valuable than raw comprehensive datasets.