Neural Diversity: A New Scaling Axis to Cut Language Model Hallucinations

Language models routinely generate false or unsupported information—a problem known as hallucination—even as researchers scale up model size, training data, and compute. A new arXiv paper suggests that adding neural diversity through decorrelated parallel representations offers a principled path to reducing hallucination rates while holding model parameters and data budgets fixed.

The Core Finding#

The researchers reframe hallucination reduction as a second-moment reliability problem. They derive formal tail bounds for hallucination probability in ensembled language models, accounting for 94.3% of the empirical reliability variation observed across different parallel configurations. This theoretical grounding sets the work apart from prior mitigation strategies, which have largely focused on improving overall accuracy rather than specifically targeting hallucination reduction.

The Method: ND-LoRA#

To implement neural diversity, the team developed ND-LoRA (Neural Diversity Low-Rank Adaptation), which combines parallel LoRA adapters with Barlow Twins regularization—a technique that encourages decorrelation among the parallel representations. On tested benchmarks, ND-LoRA reduced hallucinations by up to 25.6% (averaging 14.6% across tasks) while maintaining general model accuracy.

Evidence and Scale Effects#

Ablation studies show that LoRA adapters and regularization work synergistically to achieve this improvement. Causal interventions confirm that neural diversity is the underlying mechanism driving the results. Additionally, the researchers found a strong dose-response relationship: a 0.1% increase in neural correlation is associated with a 3.8% increase in hallucinations.

Importantly, the optimal level of neural diversity varies by task, suggesting that practitioners need to calibrate the approach for their specific use case rather than applying a one-size-fits-all solution.

Why It Matters#

The findings propose neural diversity as a third axis of scaling—orthogonal to model parameters and training data—for improving language model reliability. Rather than always reaching for larger models or more compute, this approach offers a way to squeeze better performance out of existing budgets, with potential applications across chatbots, question-answering systems, and other hallucination-sensitive use cases.