Agentic AI
No Single Retrieval Method Dominates for AI Code Agents, JioHotstar Study Shows
A comprehensive benchmark comparing six code retrieval approaches reveals structural search and semantic retrieval serve different needs—and adding retrieval layers doesn't always help.
InfoQ
JioHotstar has published a detailed benchmark study comparing how different retrieval strategies help AI coding agents work with large repositories. Rather than crowning a winner, the research suggests the best approach depends on the nature of the problem being solved.
The Retrieval Approaches Tested#
The team, led by Senior Staff Engineer Abhishek Bansal, evaluated six distinct methods: raw filesystem operations, Sourcebot, Graphify, CocoIndex Code, Repowise, and an in-house RAG pipeline. They measured each using subsets of SWE-bench and CodeScaleBench, plus a custom question-answering benchmark built on JioHotstar's own codebase. Claude Sonnet served as the underlying AI model, with metrics tracking task success rate, token usage, agent reasoning steps, and ease of tool use.
When Structural Search Wins#
For well-defined engineering tasks, structural code search proved most effective. The basic filesystem baseline—relying on simple text and file searches—remained surprisingly competitive on concrete problems. Sourcebot, which adds indexed symbol definitions and cross-references, performed better than the baseline on harder SWE-bench Verified tasks while consuming fewer tokens overall. However, on simpler SWE-bench Lite tasks, adding retrieval overhead often hurt performance without consistent gains.
Semantic Retrieval's Sweet Spot#
CodeScaleBench painted a different picture. When problems involved ambiguous requirements or required mapping vague descriptions to implementation details, semantic retrieval helpers shone. Agents handling complex or cross-cutting concerns benefited more from this approach than from exact keyword matching.
An Underrated Design Factor#
The study revealed that agents gravitated toward familiar developer workflows—text search, file reading, symbol lookup—over abstract higher-level tools. This preference suggests tooling designed around familiar operations may yield better results than theoretically sophisticated alternatives.

