New framework helps LLMs tackle multi-step questions by filtering irrelevant context

Large language models struggle when asked to answer questions that require pulling together information scattered across multiple documents. A new study published on arXiv investigates this limitation and proposes a solution designed to make LLMs more reliable at these kinds of complex reasoning tasks.

The Problem: Context Overload#

When models receive full documents alongside questions that need multi-step reasoning, they tend to get distracted by irrelevant information. Researchers tested this vulnerability using HotpotQA, a benchmark that includes misleading content designed to trip up question-answering systems. They found that providing complete candidate paragraphs instead of only the key supporting facts reduced answer accuracy by 17 to 21 percentage points—a stark demonstration that raw scale doesn't solve this sensitivity to noise.

Introducing Bactrainus#

The team behind the work designed Bactrainus, a modular framework that breaks down the multi-hop reasoning task into discrete steps. Rather than asking a model to do everything at once, the system separates:

• Paragraph selection — identifying which documents contain relevant information
• Supporting-sentence identification — pinpointing the exact sentences needed
• Answer generation — producing the final response based on vetted evidence

The approach also tests optional additions like breaking questions into simpler sub-questions and incorporating supervision signals derived from teacher models to guide the reasoning process.

Key Findings#

The researchers evaluated their framework across multiple conditions, including parameter-efficient fine-tuning of Llama 3.1 models (both 8B and 70B versions). Their strongest results achieved 89.01% answer F1 and 79.70% joint F1 scores—metrics that measure both answer correctness and the accuracy of identifying supporting evidence.

The largest performance gains came from two sources: adapting the reader model itself and having fine-grained, sentence-level control over the evidence it receives. Adding question decomposition and rationale supervision produced smaller, more variable improvements depending on the specific configuration used.

Implications#