AI
LLMs Can Write Working Science Code—But Struggle With Consistency Across Multiple Attempts
A new benchmark testing 15 language models on finite element analysis reveals that while single-pass code generation works well, verifying correctness remains the main barrier to trusting AI for scientific computing.
Researchers have expanded testing of large language models in scientific programming by evaluating their ability to generate numerically correct code for complex mechanics and dynamics problems. The findings show that LLMs can produce working implementations at high rates, yet struggle with a subtler problem: ensuring the same code works reliably across repeated attempts.
How the Research Tested LLM Performance#
The team extended a benchmark called FEM-Bench to cover 17 finite element analysis tasks involving nonlinear mechanics, history-dependent systems, and structural dynamics. They evaluated 15 different LLMs on two separate tracks—writing code implementations and generating test cases—without giving the models feedback on whether their code actually ran.
In direct, single-pass generation, the best-performing model succeeded in implementing 87.1% of tasks. However, performance split between the two tracks: the top implementer achieved 49.9% joint test success, while the leading test generator reached 82.6%. This divergence highlights that excelling at code and excelling at verification are distinct challenges.
Agents with Execution Feedback Improve but Stumble#
To explore whether giving models access to run their own code could help, researchers tested six leading models with two agentic harnesses—systems that allow models to write code, execute it, see errors, and revise their solution. Claude Opus 5 paired with SimpleAgent achieved impressive initial results: 96.5% Pass@3 (at least one success across three attempts) and 77.1% Pass³ (all three attempts succeeding).
However, no model-harness combination exceeded 78.8% Pass³, revealing a critical weakness: even when a model can run its own code and check the results, it still fails to produce consistently correct output across multiple runs. The gap between "sometimes works" and "reliably works" remained stubbornly large.
Why Agents Fail Despite Self-Checking#
The researchers analyzed 249 failed trajectories to understand why agents submitted incorrect code even after their own verification supposedly passed. They identified six failure categories:
- Inadequate test coverage (73.1% of cases) — the tests didn't catch all edge cases
- Reliance on generic checks (70.3%) — basic consistency tests masked real errors

