New Benchmark Exposes How Coding Agents Struggle with Scientific Software Repairs

Software has become integral to scientific research, yet failures in scientific code can undermine both program functionality and the validity of research findings themselves. While evaluations of coding agents typically focus on raw task completion rates, they often obscure the deeper reasons behind failures when agents attempt to repair scientific applications.

Researchers have now created SWE-bench Science, a new benchmark designed to assess how coding agents handle real-world scientific software engineering challenges. The dataset spans 119 distinct repair tasks drawn from 98 open-source repositories across 20 different scientific domains.

Scope and Structure#

The benchmark organizes tasks into three distinct categories: Issue-driven repairs (addressing reported problems), Expert-exploratory work (tackling ambiguous or exploratory tasks), and Engineering-integration challenges (ensuring fixes work within larger systems). This multi-faceted approach provides a more nuanced picture of agent performance across different types of engineering work.

Performance and Limitations#

Even Claude Code with Opus-5—the highest-performing agent tested—achieved a pass rate below 50%, underscoring the real complexity of scientific software repair. This finding suggests that coding agents, despite their general competence, face substantial obstacles when working with scientific code.

Why Agents Fail#

The researchers identified four recurring failure patterns:

• Knowledge gaps: Insufficient understanding of domain-specific scientific concepts or high-level abstraction principles
• Shallow exploration: Surface-level investigation or repairs that don't address root causes
• Integration failures: Fixes that work in isolation but break broader system functionality
• Overfitting: Inability to apply learned solutions to cases beyond those in the training context

Scientific Guidance Trade-offs#

In a paired analysis, researchers tested agents both with and without explicit scientific guidance while keeping repository and execution context intact. The results revealed a counterintuitive finding: scientific knowledge isn't universally beneficial. Well-calibrated domain information can sometimes narrow an agent's search space, improving both performance and token efficiency, while misaligned guidance can create anchoring bias and paradoxically worsen repair success.