Flawed Test Passes Mask Ineffective AI Coding Agents, New Evaluation Framework Shows

Researchers have uncovered a significant blindspot in how software engineering (SWE) agents are evaluated: a test-passing result alone tells us almost nothing about whether an agent actually solved the problem methodically or stumbled into the answer through trial-and-error.

The study, which analyzed over 2,600 agent trajectories from eight different model backends, found that roughly 1 in 10 successful task completions involved what the researchers term a Lucky Pass—solutions that passed tests despite containing regression cycles, blind retries, skipped verification steps, or chaotic exploration phases.

The Lucky Pass Problem#

Traditional SWE-agent benchmarks treat all passing solutions as equivalent, reducing evaluation to a binary outcome: the patch either passes or fails. This approach masks fundamental differences in solution quality and reasoning process.

The researchers examined 2,614 OpenHands trajectories across 60 verified software engineering tasks. Among the subset of 1,815 trajectories with sufficient passing examples per task, they identified specific patterns of ineffective problem-solving:

• Regression cycles: agents revisit the same problematic states repeatedly
• Blind retries: repeated attempts without adjusting the approach
• Missing verification: skipping steps to confirm a solution works
• Disordered exploration: jumping between different problem phases without coherent structure

Across different AI model backends, the rate of Lucky Passes ranged widely—from 0.5% to 23.2%—meaning some models appeared significantly more capable when measured by pass rate alone than when judged by process quality.

AgentLens Framework#

To address this gap, researchers introduced AgentLens, a framework for evaluating agent trajectories at the process level rather than outcome only. The system uses context-aware analysis to categorize each action an agent takes into one of four types—Exploration, Implementation, Verification, or Orchestration—based on what the agent actually did during the broader task sequence, not just the tool it invoked.

The framework also constructs process references by merging multiple passing solutions for the same task into a Prefix Tree Acceptor (PTA) model, which defines what reasonable paths to a solution should look like. Against these references, individual trajectories can be assessed for waste, divergence, and overall quality.