New benchmark reveals AI coding agents struggle with messy real-world prompts

Existing benchmarks used to test AI coding agents may not reflect how users actually describe their problems. A new study documents a significant mismatch between the polished, detailed task descriptions in widely-used evaluation datasets and the short, casual requests that arrive in the wild.

The gap between benchmarks and reality#

Researchers analyzing prompts from real users and standard test suites discovered a stark contrast in how problems are framed. Real requests submitted to SWE-chat and similar platforms typically consist of just a problem statement with minimal context—yet these make up 88% of actual user prompts. Meanwhile, the same type of minimal input accounts for only 7% of problems in benchmark suites like SWE-bench Verified and Pro. The language also diverges sharply: 87% of genuine user prompts are written casually, while 94% of benchmark problems follow formal conventions.

RealSWE: bridging the evaluation gap#

To address this disconnect, researchers created RealSWE, a new evaluation suite containing 381 task variants drawn from existing benchmarks. Each variant family keeps the same underlying problem and solution but strips or reshapes the information and writing style in different ways. This allows systematic testing of how real-world characteristics affect performance.

What hurts performance most#

When seven state-of-the-art language models tackled these variants, realistic inputs degraded their average success rate by 6.4 percentage points—enough to shift how the models rank against one another. The analysis pinpointed which information elements matter most:

• Desired Behavior and Motivation proved critical—their inclusion or absence significantly impacts whether an LLM can solve the task
• Environment Information and Reproduction Steps add tokens but deliver minimal measurable gains
• Linguistic style (formal vs. casual wording) produced only modest, model-dependent effects

Practical takeaways#

The findings carry immediate implications for both end users and agent developers. Explicitly stating what behavior you want and why you want it—details that the majority of real prompts currently omit—substantially boosts an AI coding agent's chances of delivering a working solution. The study suggests that better prompt engineering at the user end may be more effective than waiting for models to improve.