Visual Salamandra: A 7B Multimodal Model Bridging Vision and Language with European Language Support

The Language Technologies Lab has unveiled Visual Salamandra, a 7-billion-parameter multimodal model that integrates Google's SigLIP vision encoder with the Salamandra Instructed 7B language model. The architecture uses a 2-layer MLP projector and late-fusion techniques to align visual and textual understanding.

Training Architecture#

Visual Salamandra underwent a four-phase training process:

• Phase 1: Projector pre-training to map image features into the language model's latent space
• Phase 2: High-quality vision pre-training using refined datasets like OCR and re-captioned images
• Phase 3: Instruction tuning through visual question answering and grounded vision tasks
• Phase 4: Full multimodal tuning with single/multi-image, video, and text-only data

The model trained on 6.1 million instruction-tuning instances, including 842,000 text-only samples drawn from sources like AI2D, Cambrian, and LLaVA Next.

European Language Focus#

A defining feature is Visual Salamandra's emphasis on multilingual support, particularly for underrepresented European languages. The team integrated linguistic plurality into the multimodal instruction-tuning framework, helping close resource gaps in multilingual AI research.

Capabilities and Use Cases#

The model handles diverse tasks including:

• Visual question answering and scene understanding
• Optical character recognition and document analysis
• Chart and mathematical reasoning
• Video summarization and multimodal storytelling
• Instruction-based image interaction

Limitations and Ethical Considerations#

The team acknowledges important constraints: the model may hallucinate plausible but incorrect answers, struggles with complex OCR and dense document layouts, and could harbor biases despite training on filtered datasets. They recommend human oversight and advise against deployment in high-stakes applications without proper evaluation.