New Dataset Tackles the Challenge of Finding Scientific Code Across Millions of Repositories

Finding the right software tools for scientific research has become increasingly difficult as open-source code repositories number in the hundreds of millions. Existing search systems are built around general software engineering practices and lack the specialized vocabulary and context that researchers in fields like astronomy, earth science, and planetary science require.

A new research effort addresses this gap by creating comprehensive benchmarks for scientific code discovery. The work combines a curated collection of research-focused repositories with standardized test sets to measure how well search systems perform across different scientific domains.

The Dataset and Collections#

The research team assembled over 5,000 high-quality scientific repositories pulled from code across five major divisions of NASA science — Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological & Physical Sciences. Each repository was carefully classified and enriched with cleaned documentation, extracted topic labels, and supplementary information gathered from project links.

This foundational corpus provides a structured foundation for training and evaluating search systems tailored to scientific computing.

Two Search Benchmarks#

The researchers introduced parallel evaluation frameworks to test different aspects of code discovery:

• Repository-level search: Scientists curated 219 real-world queries reflecting actual research discovery needs. This benchmark tests how well systems can identify entire software projects relevant to specific scientific questions.

• Code snippet retrieval: A larger-scale test set containing nearly 120,000 code snippets paired with over 119,000 targeted queries across seven programming languages. This benchmark evaluates systems' ability to pinpoint specific functions or algorithms within codebases.

Key Findings and Challenges#

Performance evaluations revealed substantial differences in search effectiveness depending on the scientific domain being queried. These gaps appear to stem from variations in how different research communities document their code, establish coding conventions, and structure their projects.

The diversity across programming languages further complicates the challenge — what works well for Python-heavy repositories may struggle with Fortran or C++ codebases common in computational physics.