Different LLMs Need Different Evidence: Study Shows Retrieval Utility Is Not Universal

Retrieval-augmented generation (RAG)—the technique of feeding language models external passages to improve their responses—typically focuses on finding topically relevant documents. But a new study from arXiv suggests this approach misses a crucial insight: what helps one LLM produce correct answers may actively hinder another.

The Mismatch Between Relevance and Usefulness#

Researchers argue that the utility of retrieved passages depends on each model's unique characteristics, including its internal knowledge base, reasoning patterns, and capacity to synthesize evidence. Rather than assuming universally useful documents exist, the team formalizes LLM-specific utility as the actual performance gain a particular model experiences when given a passage—measured as the difference between answering with and without it.

A Benchmark for Model-Specific Evidence#

To test this hypothesis, the team constructed a benchmark featuring "gold utilitarian passages" for four models—Qwen3 (8B, 14B, and 32B variants) and Llama 3.1-8B—evaluated on three major QA datasets: Natural Questions, TriviaQA, and MS MARCO-FQA.

The findings were striking: each model performed best when paired with evidence specifically suited to its own needs, while passages optimized for other models consistently underperformed. Even human-annotated evidence—long considered a reliable standard—failed to fully meet the utility requirements of individual models.

Exposing Limitations in Current Methods#

The researchers also introduced SpecUBench, a benchmark for assessing how well systems can identify LLM-specific utility. Testing existing selection and scoring methods revealed a sobering result: most approaches capture only generic, model-agnostic usefulness and struggle to reliably predict what will actually help a specific LLM.

Why This Matters for RAG#

The work highlights a fundamental gap in how retrieval-augmented systems are built today. Moving forward, the authors argue, evidence selection must be tailored to the target generator rather than treated as a one-size-fits-all problem. This shift could unlock meaningful improvements in RAG effectiveness across different model architectures and sizes.