New Benchmark Exposes Code-Retrieval Systems' Struggle with Functionally Incorrect Near-Clones

Embedding-based code retrieval powers modern coding assistants and AI-assisted development, but a new research benchmark suggests current systems may be better at recognizing topical similarity than spotting actual bugs. Researchers introduce ExecRetrieval, a test suite designed to measure whether code embeddings can tell a correct implementation from a subtly broken one hiding in the search results.

The Problem: Embeddings Match Similarity, Not Correctness#

Most code-retrieval benchmarks test whether a system finds lexically or topically similar code—useful for discovery, but not for safety-critical tasks like pulling working code for a coding agent. A retriever might happily return code that looks nearly identical to the target implementation but contains a single-line logic error that breaks execution. Existing benchmarks don't ask whether embeddings can actually discriminate between a correct solution and a broken variant of it.

ExecRetrieval closes this gap by deliberately seeding the search pool with both canonical (correct) implementations and execution-verified buggy near-clones. Each of the 939 Python tasks pairs one correct implementation with up to four variants, each altered by a precise, single-point mutation designed to break functionality without changing the surface resemblance.

What the Benchmark Reveals#

The results expose a significant functional-correctness gap:

• The top-performing system achieves perfect ranking (exec@10 = 1.00) when allowed to consider the top ten results, but drops to just 33.1% accuracy at rank 1
• Across the four leading systems tested, rank-one misses are paired buggy variants 91.5–99.4% of the time
• On leading systems, the correct canonical implementation ranks below at least one of its paired distractors in 67–78% of queries
• 23 dense embedding configurations plus BM25 were evaluated under provider-native invocation with paired statistical tests

Implications for Coding Systems#

The findings matter for retrieval-augmented code generation and coding agents, where pulling an incorrect snippet can silently propagate bugs downstream. The gap between retrieving similar code and retrieving correct code highlights a blind spot in how current embeddings are trained and evaluated. Systems tuned for semantic similarity may excel at topical matching without developing the fine-grained functional discrimination needed to avoid near-clone traps.