New Benchmark Reveals LLMs Struggle to Clarify Ambiguous Code Requirements

A newly published benchmark highlights a significant blind spot in how large language models assist with software development. While these models have proven adept at writing code when given clear instructions, they remain largely unable to proactively ask clarifying questions when faced with fuzzy or incomplete requirements—a skill essential for real-world engineering.

The Gap in Interactive Requirement Elicitation#

Researchers have introduced ClarifyCodeBench, an evaluation framework designed to measure how well LLMs can detect ambiguities in programming specifications and ask the right follow-up questions to resolve them. The benchmark draws from actual development tasks and includes human-vetted examples of common ambiguity types alongside expert-written clarification questions and validated answers.

The team formalized two new evaluation metrics: Turn-discounted Key Question Rate (penalizing inefficient questioning) and Optimal Round Adherence (assessing precision in the elicitation process). This move underscores how existing code-generation benchmarks have sidestepped the iterative, back-and-forth nature of real requirement gathering.

Three Key Findings#

Testing six state-of-the-art LLMs against the benchmark yielded three sobering insights:

• Capability Decoupling: Superior code generation performance does not predict strong clarification ability. The models that generate the best code are not necessarily the ones best equipped to spot missing details.

• The Reasoning Paradox: Adding computational depth—larger models or more reasoning steps—improves code accuracy but yields minimal gains in identifying ambiguities. More thinking does not translate to better requirement understanding.

• The Multi-ambiguity Ceiling: When specifications contain multiple overlapping ambiguities, LLM performance drops sharply. This reveals a critical bottleneck when dealing with realistic, complex requirements rather than clean, single-issue examples.

Why This Matters#

The findings underline a fundamental mismatch between what LLMs do well (synthesizing code from clear prompts) and what software teams actually need (interactive partners who can ask smart questions when requirements are vague). Current benchmarks have largely overlooked this interactive dimension, assuming perfect specifications that rarely exist in practice. Moving forward, AI for software engineering must shift focus from static code synthesis toward dynamic requirement elicitation and collaborative problem-solving.