AI
New Method Reduces Hallucinations in Vision-Language Models Without Retraining
Researchers propose DCLA, a training-free decoding approach that enforces consistency across neural network layers to improve accuracy in large vision-language models.
Large Vision-Language Models (LVLMs) like LLaVA and Qwen have shown impressive capabilities, but they frequently suffer from hallucinations—generating content that contradicts the input image. Existing methods to address this problem often require retraining or fine-tuning, and they remain sensitive to hyperparameter adjustments.
A new approach published on arXiv tackles this challenge with DCLA (Decoding with Inter-layer Consistency via Layer Aggregation), a training-free mechanism that requires no model retraining, fine-tuning, or external knowledge bases.
How it works#
DCLA operates during the decoding phase by aggregating representations from earlier layers to create a dynamic semantic reference. This reference is then used to correct layers that have deviated semantically, enforcing consistency across the network's layers and reducing hallucinated outputs.
Results#
Testing across seven different LVLMs showed significant improvements:
- LLaVA 1.5-7B: 28.58 MME points improvement over standard decoding
- Qwen2.5-VL: 42.6 MME points improvement
- POPE accuracy: 2.74 percentage point gain in the strongest setting
The approach demonstrates broad applicability across models and benchmarks while maintaining practical simplicity—no architectural changes needed.
