Budget LLMs Fall Short on Real Code Generation, Study Shows

A new empirical study tests whether cost-effective large language models can reliably generate enterprise code that meets formal specifications. The findings suggest significant gaps between surface-level correctness and actual computational accuracy.

The Setup#

Researchers evaluated three budget-oriented models—Gemini Flash 3, GPT-5.4 mini, and Claude Haiku 4.5—on 992 algorithmic problems. Each task required generating a Java Spring Boot service method that matched a specific function signature and data-transfer-object specification. The experiment ran eight different configurations combining model choice with agentic coding tools and prompt strategies. To test robustness, eight problem statements were deliberately omitted, allowing researchers to observe how models handled incomplete inputs.

Key Findings#

The results paint a sobering picture:

• 38.4% of the 7,593 generated methods failed to compute the values they claimed to return
• Only 12.9% of returned answers were actually correct
• Structural conformance approached ceiling levels—the code looked right even when it didn't work

The researchers discovered an inverse relationship: models that most often produced a response were less reliable, while genuinely computational methods answered less frequently (19.3% correct rate). This suggests a trade-off where aggressive answer generation sacrifices correctness for appearance.

Study Limitations#

The evaluation has several important constraints: each configuration ran only once, test harness coverage was partial, timing was measured single-pass, and classification relied on syntactic analysis rather than semantic verification. The researchers also note possible corpus contamination—the models may have encountered similar problems during training.

Why It Matters#

As organizations increasingly consider cheaper LLM alternatives for code generation, this work underscores the need for rigorous validation before deploying AI-generated code in production systems. Conformance to specification syntax is no guarantee that the underlying logic is sound.