Researchers develop label-free audit method for AI-generated optimization models

Optimization models created by large language models can produce solver-ready output that appears correct but actually encodes the wrong problem — a phenomenon known as "silent failure." Researchers at arXiv have now introduced a method to catch these errors without access to labeled reference answers.

The core challenge#

When LLMs generate optimization models from natural-language problem descriptions, traditional evaluation methods fall short. Most approaches compare computed optimal values against labeled answers, but real-world deployments don't always have such reference data on hand. The question becomes: how can you verify a generated model's correctness in the absence of ground truth?

Falsification-based verification#

The proposed solution leverages classical techniques from duality theory and sensitivity analysis. The key insight is that every numeric quantity in a problem statement — whether a capacity, requirement, or cost — plays a specific role that the problem description explicitly defines. A mathematically correct model must respond to changes in these quantities in ways that align with their stated roles.

The researchers developed a suite of solver-based tests that are individually sound. This means:

• Any violation of these tests definitively certifies a faulty model
• False-positive rate is zero by design (faithful models are never incorrectly flagged)
• Each test carries a machine-checkable certificate that pinpoints the defect

What the tests detect and miss#

The work characterizes which errors the test suite can identify and which remain invisible to this approach. Testing shows the battery detects:

• 56.1% of core formulation errors across diverse test cases
• 70.0% of errors under certified preconditions
• 40.4% of errors that value-based scoring cannot detect

Importantly, it flags 0.0% of correct models while a comparison threshold-based tester incorrectly flags 54.9% of faithful models.

Practical application#

The full audit requires roughly 25 millisecond-scale solver calls per model, making it computationally efficient. The method complements existing AI-model evaluation strategies and provides a rigorous, reference-free certification pathway for optimization models in production settings.