New technique helps vision-language models handle high-resolution images without losing token coherence

The Challenge with Multi-Resolution Image Processing#

Vision-language models that process both high-resolution and thumbnail versions of the same image can deliver stronger performance, but they face a fundamental problem: generating vast numbers of image tokens that struggle to interact effectively with one another and with text.

The root cause lies in how these models use Rotary Position Embedding (RoPE), a widely adopted technique for encoding position information. RoPE's design includes a decay property that weakens connections between distant positions, which works against the interplay between fine-detail tokens from high-resolution images and coarser-grained thumbnail tokens—as well as between visual and textual content overall.

How ID-Align Works#

Researchers have now introduced ID-Align, a position remapping strategy that restructures how the model assigns position identifiers to tokens. The key innovation involves having high-resolution image tokens inherit position IDs from their corresponding thumbnail tokens, while simultaneously preventing position indices from expanding uncontrollably.

This approach sidesteps the limitations imposed by RoPE's decay property, allowing tokens at different scales to communicate more freely without requiring architectural changes to the underlying model.

Measured Results#

Testing within the LLaVA-Next framework showed meaningful gains across multiple evaluation benchmarks. Most notably, the method delivered a 6.09% improvement on relation reasoning tasks in the MMBench evaluation—a domain where understanding spatial and semantic connections between image elements proves critical.

The researchers have made their code publicly available, enabling further exploration and adoption of the technique.