Vision Models Learn to Handle Mismatched Tasks with Text-Driven Prompting

A new approach addresses a gap in how vision-language models (VLMs) handle in-context learning across unrelated tasks. When presented with a few examples from one visual task and asked to solve a different one, current models struggle—they can't easily translate the learned pattern into the new context. Researchers have now developed a solution called T2T-VICL that bridges this gap through intelligent prompt generation.

The Challenge of Cross-Task Learning#

Visual in-context learning typically works well when demonstration examples and the target task match. A model shown how to blur faces can readily blur new faces. But real-world scenarios often involve misaligned tasks. Imagine being shown examples of image sharpening, then asked to colorize a photo. The model must decide whether to apply the sharpening logic or recognize the new objective entirely—and VLMs currently have no clear way to navigate this ambiguity.

How T2T-VICL Works#

The framework operates in two stages. First, a large teacher VLM analyzes pairs of different visual tasks, generating detailed descriptions of what changes visually and how the tasks differ conceptually. This creates a dataset of implicit relationships across tasks—patterns that don't rely on naming tasks explicitly.

Next, researchers distill this knowledge into a lightweight student model that receives a demonstration from Task A alongside a query from Task B. The student generates a custom text prompt tailored to that specific mismatch, which guides a frozen image-editing VLM toward the correct output. A scoring system then ranks candidate results to select the best match.

Scope and Results#

Testing across 12 low-level vision tasks and more than 20 cross-task combinations, T2T-VICL consistently outperformed fixed-prompt approaches in aligning outputs to the intended task. Many configurations also maintained or improved image quality, suggesting the method captures genuine transfer between domains rather than simply degrading outputs.

The researchers have made their code publicly available, paving the way for further investigation into how models can flexibly adapt visual understanding across task boundaries.