Structured formats beat generic text for creating synthetic training data, study finds

A large-scale empirical study has revealed practical principles for generating high-quality synthetic data to pretrain large language models. Researchers ran controlled experiments producing over one trillion tokens to isolate the factors that matter most when converting web text into machine-learning training material.

Key findings on data structuring#

The research demonstrates that reformatting source material into structured layouts yields dramatically better results than simply rephrasing raw web content. Data transformed into tables, mathematical problems, frequently asked questions, and step-by-step tutorials consistently outperformed both the original curated web baselines and earlier synthetic data approaches. This suggests that the way information is organized matters as much as its linguistic content.

Generator model size shows diminishing returns#

Counterintuitively, expanding the generator model beyond 1 billion parameters offered little measurable improvement to final model performance. This finding has important implications for the cost-benefit analysis of synthetic data generation, suggesting that practitioners don't need cutting-edge models to produce effective training material.

FinePhrase dataset and framework#

The team applied these insights to create FinePhrase, an open dataset of 486 billion tokens of rephrased web text. The resulting synthetic data outperformed all comparable baselines while reducing generation costs by up to 30 times compared to alternatives. The authors have released the dataset, their complete set of prompts, and the generation framework to enable further research.

Methodological contribution#

The work addresses a gap in the literature by providing the first systematic comparison across multiple design dimensions—rephrasing strategy, choice of generator model, and source data selection. The researchers found that the original data sources chosen for mixing had substantial influence on downstream model performance, emphasizing that data quality and curation remain critical decisions.