New Open-License Dataset Offers Two Trillion Tokens for Ethical LLM Training

A new research initiative tackles a fundamental tension in large language model development: the need for massive training data versus the legal risks of using copyrighted material. Researchers have introduced Common Corpus, described as the largest openly-licensed dataset assembled specifically for LLM pre-training, containing approximately two trillion tokens.

The Challenge of Legal Training Data#

Current LLMs typically train on enormous datasets spanning multiple sectors—media, code, academic text, and more. Many of these datasets contain substantial amounts of copyrighted or proprietary material, raising concerns about intellectual property violations and regulatory compliance. This legal ambiguity has motivated the search for alternative approaches that rely exclusively on material with clear permission for reuse.

What Common Corpus Offers#

Common Corpus addresses these concerns by aggregating data that is either uncopyrighted (such as public domain works) or released under open licenses. The resulting collection spans an exceptionally wide range of languages—from well-resourced European languages down to low-resource languages that have historically been underrepresented in pre-training datasets. The dataset also incorporates substantial programming code, diversifying its domain coverage across different knowledge areas and time periods.

This breadth of sources opens potential applications for both academic research and commercial development across numerous specialized fields.

Performance Validation#

To demonstrate the dataset's viability, the researchers trained two small-scale language models on Common Corpus and benchmarked their performance against comparable models. The results showed comparable performance, indicating that the dataset successfully supports effective multilingual pre-training despite the constraints of using only openly-licensed material.

Significance for Open Research#

The release of Common Corpus represents a significant contribution to the ecosystem of reproducible AI research, enabling practitioners to build LLMs without navigating complex intellectual property concerns or violating data protection regulations.