Unified Vision-Language Models Show Cross-Task Benefits Only When Visual Spaces Align

Researchers conducting a systematic investigation into unified vision-language models (VLMs) have uncovered a nuanced relationship between combining image understanding and generation tasks in a single framework. The work reveals that while mixed training can enhance both capabilities, the strength of this benefit hinges critically on how well the model's visual input and output representations align.

The Research Setup#

The team built their study around two custom benchmarks—SmartWatch and a modified version of CelebA—each containing paired tasks: visual question answering (VQA), image captioning, and text-to-image generation. They evaluated several architectures grounded in different visual foundations (SigLIP and VQ-VAE spaces) and analyzed how well these unified systems performed when trained on both understanding and generation simultaneously.

When Mixed Training Works#

The experiments demonstrated that unified models can indeed outperform task-specific alternatives when both capabilities share the training load. However, this advantage emerged only under specific conditions. Models whose visual input and output spaces maintained stronger alignment showed substantially better cross-task transfer. The language model component appeared to learn relationships between visual concepts that transferred effectively across these compatible representations.

Conversely, when researchers deliberately warped the input visual space using reversible affine distortions—keeping all other elements constant—the benefits collapsed. This manipulation turned potential synergies into actual conflicts, with each task degrading the other's performance.

Data Balance and Concept Recovery#

The researchers also explored how data proportions between tasks affected outcomes. While increasing examples from one task initially helped the complementary task, severely imbalanced datasets eventually harmed performance on the secondary objective. Interestingly, by controlling the frequency of visual attributes, the team discovered that generation supervision could effectively recover underrepresented concepts for the understanding task—a finding with practical implications for training data curation.

What's Actually Transferring#

Adapter-based analysis suggested that richer visual feature representations were not the primary mechanism behind cross-task transfer. Instead, the underlying language model appeared to be the key: it learned generalizable relationships about visual concepts that persisted across aligned visual spaces, enabling more flexible reasoning between tasks.