Researchers Target Hidden Knowledge in AI Models to Improve Retrieval-Augmented Accuracy

When large language models are augmented with retrieval systems to pull in external knowledge, they should generate more factually grounded responses. Yet researchers have found these systems still produce outputs that contradict the evidence they've retrieved—even when that evidence is correct. The root cause lies not in how models use retrieved information, but in their stubborn reliance on internally stored knowledge.

The Problem with Built-In Knowledge#

Retrieval-augmented generation (RAG) aims to improve factuality by grounding model outputs in external sources. However, a team of researchers investigating unfaithful generation discovered that models' internal parametric knowledge—the patterns learned during training—often overrides retrieved context. Even when accurate external evidence is available, the model's own ingrained knowledge can dominate the generation process, leading to contradictions.

How ParamMute Works#

The researchers identified that specific feed-forward networks (FFNs) deep within language models are disproportionately activated when unfaithful outputs occur. Based on this finding, they developed Parametric Knowledge Muting (ParamMute), a framework that selectively suppresses these problematic FFN layers while recalibrating the model to favor retrieved knowledge. Rather than retraining the model, ParamMute modulates which internal components contribute to each response.

Testing Against Knowledge Conflicts#

To properly evaluate their approach, the team introduced CoFaithfulQA—a new benchmark designed specifically for scenarios where a model's internal knowledge conflicts with accurate external information. Using this evaluation tool alongside the existing ConFiQA benchmark, ParamMute demonstrated significant improvements, substantially reducing the model's reliance on parametric memory while maintaining response quality.

Why It Matters#

These findings highlight a fundamental challenge in making language models more trustworthy: external knowledge sources alone aren't enough if the model's internal patterns continue to dominate. By identifying and suppressing the specific neural components responsible for unfaithful generation, ParamMute points toward a practical method for improving LLM reliability in real-world applications where accuracy is critical. The research code has been made publicly available for further development.