AI Research
New framework uses uncertainty modeling to improve AI code generation at repository scale
OpenCoder shows how treating retrieval uncertainty as a control signal can boost code generation accuracy by accounting for interactions between different evidence sources.
Generating code at the repository level—where language models must draw on multiple types of information spread across a codebase—remains a challenge partly because not all retrieved evidence is equally reliable or necessary. A new paper from arXiv explores how explicitly modeling uncertainty in what gets retrieved can lead to better outcomes.
The problem with heterogeneous evidence#
Repository-level code generation typically relies on pulling together several types of supporting information: similar code examples from the codebase, broader repository context, and project-specific APIs. While each source can contribute useful signals, they can also introduce noise, redundancy, or conflicting guidance. Most existing retrieval-augmented generation (RAG) approaches focus on identifying what information is most relevant to retrieve, but they largely ignore the fact that relevance itself is uncertain and that different sources interact in complex ways.
Introducing OpenCoder#
Researchers developed OpenCoder, an uncertainty-aware framework designed to estimate how much confidence to place in each source of retrieved evidence. Rather than treating all retrieved information equally, the system uses these uncertainty estimates to filter, rank, and weight heterogeneous evidence, then passes this intelligence downstream to the code generation, verification, and repair stages.
Key findings from evaluation#
The team conducted a factorial analysis examining how three evidence sources—API knowledge, repository context, and similar-code examples—combine to affect generation quality. The results revealed no simple universal ranking of sources. Instead, how helpful each source turns out to be depends significantly on what other evidence is present and which language model backend does the generation.
Testing on an expanded 32-task evaluation using the RepoExec-inline benchmark showed OpenCoder raised GPT-based code correctness from 56.25% (baseline RAG) to 78.13%. However, the improvement matched what a simpler verification-and-repair approach achieved, and gains with the Gemini model were not statistically significant—highlighting that benefits are backend-dependent. The framework also showed gains when refining API sets with target-aware refinement.