Adding Visual Context During Training Improves Language Models Even Without Images at Test Time

A new study demonstrates that exposing neural language models to visual information during training yields measurable improvements in their ability to predict subsequent words, even when those visual cues are entirely absent during evaluation.

The Research Finding#

Researchers developed a multimodal neural architecture that combines visual and textual data, achieving a 2% reduction in perplexity compared to language-only baselines. Even more striking, when the team fine-tuned embeddings from BERT (a state-of-the-art bidirectional language model) using the same multimodal approach, they observed a 3.5% improvement in performance—with no visual information present at test time.

Testing Across Languages and Models#

The advantage of training with visual grounding proved consistent across different experimental conditions. The researchers validated their approach across three distinct languages (English, German, and Spanish) and multiple neural architectures including GRU, LSTM, and Δ-RNN variants, as well as BERT-based models. This robustness across diverse settings suggests the finding is not an artifact of any single implementation choice.

Connection to Cognitive Theory#

The results align with the theory of situated cognition, which posits that language understanding cannot be separated from the physical and sensory context in which it occurs. Just as human infants develop language through simultaneous exposure to visual and linguistic stimuli—connecting words to objects and actions they observe—the neural models in this study appear to benefit from similar multimodal training, internalizing richer representations that persist even without visual input later.

The work was submitted to arXiv in May 2018 and has undergone revision through 2026.