AI
New industrial code dataset with 832M lines enables software engineering AI research
Researchers release CIDR, a large-scale repository of real-world code from enterprise partners, to advance machine learning models for code intelligence and software quality.
Researchers have unveiled CIDR (Curated Industrial Developer Repository), a massive dataset designed to accelerate progress in machine learning-based software engineering. The collection brings together source code from real industrial partners, offering an unprecedented resource for training and evaluating AI systems that work with code.
Dataset scope and composition#
The dataset aggregates 4,225 repositories spanning 75 programming languages, containing 832 million raw lines of code (approximately 581 million logical lines). Beyond raw code, CIDR includes structured metadata about each repository, the full version control history documenting how code evolved, and information about engineering practices—such as whether continuous integration is enabled and whether automated tests are present. All data went through a careful anonymization process to protect sensitive information.
Development process and applications#
The team developed a multi-stage pipeline to collect, filter, and de-identify the repositories, ensuring data quality and privacy compliance. This infrastructure was built specifically to handle the scale and complexity of industrial codebases.
The researchers also conducted a pilot study fine-tuning a 3-billion-parameter code language model on CIDR data, measuring how well the adapted model performed on held-out enterprise code. Results demonstrate the dataset's practical value for improving code models.
Research directions enabled#
The dataset supports work across multiple domains: code intelligence systems that understand code semantics, tools for analyzing software quality and detecting defects, and developer-facing applications like code completion and documentation generation. The dataset is released under a restricted license with specific eligibility requirements to balance openness with responsible data stewardship. Researchers can find details on access terms at the project's official repository.
