Researchers create vast training dataset for AI to tackle scientific discovery tasks

Researchers have developed a systematic approach to address a fundamental barrier in building AI systems that collaborate effectively on scientific work: the scarcity of high-quality training examples. The solution, called AutoSDT, automatically assembles practical coding tasks drawn from real-world research workflows.

The data collection challenge#

Training AI models to assist with scientific discovery requires substantial amounts of genuine, well-documented tasks that reflect how researchers actually work. However, such datasets have been scarce and difficult to obtain, limiting the ability to develop and evaluate AI co-scientists. AutoSDT sidesteps this bottleneck by using large language models' own capabilities to systematically search for, validate, and document tasks at scale.

How the system works#

The pipeline leverages the coding proficiency and embedded knowledge of LLMs to accomplish three key steps: identifying diverse sources of data-driven discovery workflows, filtering for tasks that reflect authentic scientific practice, and generating clear instructions alongside working code solutions. The result is AutoSDT-5K, a dataset containing 5,404 coding tasks spanning four scientific domains and 756 different Python libraries. To date, this is the largest openly available dataset constructed this way for scientific discovery work.

Validation and performance gains#

Quality assurance was built in from the start. When domain experts reviewed a sample of 256 tasks, they confirmed that 93% were genuinely useful for scientific work and 92.2% of the generated programs functioned correctly—a strong signal of data integrity.

The research team fine-tuned versions of the Qwen2.5-Coder model on AutoSDT-5K, creating what they call AutoSDT-Coder. On two competitive benchmarks for data-driven discovery:

• ScienceAgentBench: The 32-billion-parameter variant matched GPT-4o's success rate of 7.8%, doubling its baseline performance.
• DiscoveryBench: It achieved a hypothesis matching score of 8.1, a 17.4% relative improvement and a meaningful step toward closing the gap with proprietary frontier models.

The work demonstrates that open-weight models, when trained on curated, domain-specific datasets, can substantially narrow the performance gap with leading commercial systems on specialized scientific tasks.