AI
New Method Detects Silent Errors in LLM-Generated Optimization Code
Researchers introduce ReLoop, a technique that uses structured reasoning and behavioral testing to catch hidden bugs where AI-written code appears to work but solves the wrong problem.
Large language models can write optimization code from plain English descriptions, but a critical problem lurks beneath the surface: code that runs successfully and produces answers may encode the wrong mathematical problem entirely. A new approach called ReLoop tackles this "feasibility-correctness gap," which can reach as wide as 90 percentage points on complex, multi-part problems.
The Silent Failure Problem#
When LLMs translate natural language into optimization formulas, the resulting code often executes without errors—making the mistakes nearly invisible. A solution that's mathematically feasible according to the solver doesn't guarantee the underlying formulation is correct. This silent failure mode is particularly dangerous in domains like retail optimization where incorrect problem setup leads to misleading business recommendations.
How ReLoop Works#
The approach combines two complementary strategies. Structured generation breaks code creation into four deliberate stages: understanding the requirement, formalizing it mathematically, synthesizing the code, and verifying the logic. This decomposition prevents formulation errors before they enter the code. Behavioral verification then catches whatever errors slip through by testing whether the model's formulation behaves correctly when parameters change—essentially using the solver's reactions as an external check rather than relying on the LLM to audit its own work.
Results Across Benchmarks#
On compositional retail problems (where multiple constraints interact), structured generation achieved the largest improvements, boosting accuracy by 8.5 percentage points using Claude Opus 4.6. Behavioral verification dominated on more localized errors, contributing up to 4.4 percentage points on complex linear programming tasks. Combined with diagnostic recovery, the method reached 100% executable code on the strongest model tested and showed consistent gains across three different benchmarks.
The researchers also identified a brittleness in narrowly fine-tuned models: their learned output formats break under chain-of-thought prompts, a vulnerability that's worth documenting as these models see wider deployment. The team released RetailOpt-190, a new dataset of 190 retail scenarios specifically designed to expose the multi-constraint interactions where LLMs most frequently fail.
