TreeHop: A faster way to answer complex questions without relying on LLMs

Researchers have introduced TreeHop, a new approach to handling complex, multi-step questions in retrieval-augmented generation (RAG) systems that sidesteps the heavy computational burden of traditional methods.

The problem with current approaches#

Multi-hop question answering—where systems must gather and synthesize information across multiple documents to answer a single query—remains a bottleneck for RAG systems. Existing solutions typically cycle through repeated steps: retrieve documents, rewrite the query using a language model, convert it back to vectors, then retrieve again. Each loop demands fresh LLM invocations, driving up computation costs, resource consumption, and response times.

How TreeHop works differently#

Instead of this multi-stage pipeline, TreeHop operates directly in embedding space—the mathematical representation of text that retrieval systems use internally. The system dynamically refines query embeddings by blending information from previous searches and documents it has already found, then immediately retrieves new candidates without stepping outside that embedding space. This streamlined "Retrieve-Embed-Retrieve" cycle eliminates the need for repeated LLM calls while maintaining the ability to handle complex, interconnected questions.

The framework also incorporates a rule-based stopping mechanism that recognizes when enough documents have been gathered, preventing wasteful extra retrievals.

Performance and efficiency gains#

Testing across four open-domain datasets showed TreeHop matched the accuracy of more sophisticated RAG systems while consuming only 2.2% to 29.4% of their model parameters. More strikingly, the latency improvements were substantial: query response times dropped by 92.8% to 97.8% compared to competing methods.

These gains make TreeHop particularly attractive for deployment scenarios where computational resources are limited, latency matters, or infrastructure costs must stay lean.

For transparency, the researchers have published code and data alongside the paper (arXiv:2504.20114).