LLMs Show Promise on Numerical Stability, but Lag Behind Traditional Tools in Scientific Code

Floating-point arithmetic in scientific software creates precision errors that cascade through calculations, especially in safety-critical applications. Researchers from the University of Waterloo sought to determine whether large language models (LLMs) can reliably detect and fix these numerical vulnerabilities—work that has grown urgent as AI is increasingly applied to scientific computing.

Testing LLM Reasoning at Scale#

The study evaluated four state-of-the-art LLMs on over 469,000 tasks involving 2,037 distinct numerical structures. These test cases ranged from simple arithmetic chains to complex expressions featuring nested conditionals, high-precision literals, and multi-variable operations. The researchers asked the models to perform two critical jobs: identifying unstable computations by finding inputs that trigger precision errors, and rewriting expressions to reduce those errors.

Where LLMs Excel#

Large language models revealed a unique strength: they excel precisely where traditional numerical stabilization tools fail. On expressions where the baseline method (Herbie, a classical optimizer) made no improvement, LLMs successfully stabilized 61.2% of them. This complementary capacity suggests LLMs can serve as a supplementary layer for edge cases that conventional techniques miss.

The models also performed substantially better on purely symbolic expressions—calculations without concrete control-flow logic or exact magnitude constraints.

The Gaps That Remain#

Despite these wins, conventional tools still dominate overall performance. Herbie stabilized 93.7% of all test expressions, compared to just 67.2% for LLMs. Even when both methods succeeded, Herbie achieved superior numerical accuracy in 42.7% of cases, a significant margin.

The study identified two critical failure modes in LLMs:

• Control-flow blindness: Models struggled to reason about conditional logic and often removed control structures entirely rather than analyzing their numerical implications.
• Precision reasoning: LLMs faltered when high-precision literals mattered, typically discarding such structures instead of reasoning about their computational impact.
• Semantic errors: In over 46% of cases where LLMs did restructure expressions, the revised code proved numerically inequivalent—potentially introducing silent bugs.