New Quantization Method Cuts Vision-Language Model Size While Preserving Accuracy

Deploying vision-language models (VLMs)—AI systems that process both text and images—on smaller devices remains a major challenge due to their massive size and computational demands. A new approach called SalQ-VLM tackles this by intelligently compressing these models through quantization, a technique that reduces the precision of their numerical parameters.

The Core Problem#

Current billion-parameter VLMs require significant memory and processing power, making them impractical for edge devices and resource-constrained environments. While post-training quantization (PTQ) can shrink models and speed up inference without retraining, existing methods haven't been tailored to the specific structure of multimodal systems, leading to accuracy losses.

The research identifies two key challenges in quantizing VLMs: visual over-representation, where vision tokens are excessive and often redundant, and a modality gap where text and vision tokens occupy different regions of the model's latent feature space.

How SalQ-VLM Works#

The new framework takes an importance-aware approach, prioritizing tokens that matter most while suppressing redundant visual data during the calibration phase. The method derives a gradient-driven importance factor that measures token-level significance by analyzing the relationship between loss changes, activation errors, and output gradients.

Crucially, the approach requires only a single lightweight block-wise gradient-caching pass to compute this factor. Because it modifies only the calibration stage, it introduces no additional computational overhead during inference and remains compatible with existing high-performance model-serving kernels.

Experimental Results#

Extensive testing across multiple benchmarks and model architectures shows consistent improvements over existing PTQ baselines, particularly in ultra-low-bit quantization scenarios. In one striking result, the method achieved a 16.45% accuracy improvement on the MME-RealWorld benchmark under INT2 quantization (using just 2 bits per parameter).