New Benchmark Reveals LLMs Struggle When Code Requirements Are Vague

While large language models have become adept at translating clear specifications into working code, a new academic study exposes a critical weakness: they falter when faced with the messy reality of real-world requirements. Researchers have introduced Orchid, the first code generation benchmark explicitly designed to measure how LLMs handle ambiguous, unclear, or vague software specifications.

The Ambiguity Problem#

Real software development rarely comes with crystal-clear instructions. Requirements are often expressed in natural language that can be interpreted in multiple ways, combining incomplete information with misunderstandings between stakeholders. Yet most existing evaluation benchmarks assume requirements are precise and unambiguous—a gap that leaves a blind spot in our understanding of automated code generation in practice.

Orchid addresses this gap with 1,304 function-level coding tasks designed to test four distinct sources of ambiguity:

• Lexical ambiguity: unclear word meanings
• Syntactic ambiguity: multiple valid grammatical interpretations
• Semantic ambiguity: unclear logic or intent
• Vagueness: imprecise or incomplete specifications

Key Findings#

The research reveals that ambiguity consistently hurts performance across all tested LLMs. Notably, the most advanced models showed the steepest performance drops—suggesting that greater model capacity doesn't translate to better ambiguity handling.

LLMs also displayed two other troubling behaviors: they frequently generate functionally different implementations from the same ambiguous requirement, and they cannot reliably detect or flag ambiguity on their own. This means a developer using an LLM to code from unclear specs may receive plausible-looking but subtly incorrect solutions without any warning.

Why This Matters#

The findings underscore an urgent need for new techniques in AI-powered software engineering—ones that can either ask clarifying questions when faced with ambiguous requirements or explicitly surface the uncertainty in their outputs. As organizations increasingly rely on LLMs for code generation, closing this gap between performance on clean specifications and real-world messy requirements becomes essential.