New approach harnesses LLMs for better document retrieval using query likelihood modeling

Dense retrieval—the process of finding relevant documents from large collections—underpins many modern information retrieval systems and feeds downstream tasks like re-ranking and retrieval-augmented generation. While large language models have proven adept at semantic understanding, they face limitations when it comes to capturing the full context needed for effective retrieval, particularly because they struggle to model relationships between distant parts of a document.

Bridging classical and modern IR#

A research team has developed LLM-QL, a framework that marries insights from classical query likelihood models—a time-tested approach in information retrieval—with the generative capabilities of modern LLMs. Rather than using query likelihood solely for ranking documents, the team treats query likelihood maximization as an auxiliary training task that strengthens the retriever's backbone before it undergoes contrastive learning.

How it works#

The approach introduces two key mechanisms. An Attention Block (AB) prevents the model from attending to document tokens that appear before the document's ending token, limiting the model's view during prediction. A Document Corruption (DC) component masks portions of the document during generation, forcing the model to learn more robust representations.

Proven results#

Testing on both in-domain datasets (MS MARCO) and out-of-domain benchmarks (BEIR) demonstrates that LLM-QL outperforms other LLM-based retrieval methods. The researchers' analyses confirm that both the overall approach and its individual components contribute meaningfully to the improvement.

This work suggests that combining classical information retrieval theory with contemporary deep learning offers a productive path forward for building more effective retrieval systems.