New Dataset Lets Models Learn From User Prompts Instead of Raw Text

Most large language models start their training on vast quantities of internet text, learning to predict the next word in an unsupervised way. Only later do they get fine-tuned on human-written instructions—a process held back by the scarcity of that curated data. A new approach flips this workflow by generating instruction-response pairs at pre-training scale.

The FineInstructions Framework#

Researchers introduced a method to convert the knowledge baked into pre-training documents into billions of synthetic instruction-answer pairs. The dataset, called FineInstructions, combines roughly 18 million instruction templates drawn from actual user queries and real-world prompts with human-authored source material from unstructured pre-training corpora. Each template is paired with relevant passages from the pre-training data to create supervised training examples.

A More User-Aligned Training Objective#

The key insight is that models trained on instruction-response pairs may learn patterns closer to their intended downstream use. Rather than the traditional pipeline of unsupervised pre-training followed by supervised fine-tuning on a tiny labeled dataset, this approach allows a language model to be pre-trained entirely on instruction-tuning data—making the entire process more aligned with how users will actually interact with the finished system.

Benchmark Performance#

In controlled experiments comparing equal numbers of training tokens, pre-training on FineInstructions outperformed both standard self-supervised pre-training and other proposed synthetic pre-training approaches on standard benchmarks that evaluate the quality of free-form responses. The results suggest that synthetic instruction data, when generated at sufficient scale and quality, can be an effective alternative to traditional large-corpus pre-training.