New Model Combines Recurrence and Transformers to Search Across Mixed-Media Collections

Researchers have introduced ReT-2, a unified model designed to retrieve documents from collections that intermix images and text. Unlike prior retrieval systems that typically require single-modality input or rely on task-specific fine-tuning, ReT-2 accepts multimodal queries—combining both images and text—and searches across similarly mixed repositories.

Architecture and approach#

The model integrates a recurrent Transformer framework with LSTM-inspired gating mechanisms. Rather than treating layers as isolated stages, ReT-2 dynamically routes information across multiple layers and modalities, preserving fine-grained details from both visual and textual signals. This design allows the system to better align diverse data types during retrieval.

Performance and efficiency#

Testing on the M2KR and M-BEIR benchmarks, ReT-2 consistently outperforms existing methods across various retrieval configurations. Beyond accuracy gains, the approach also cuts both inference time and memory consumption compared to earlier systems.

Practical impact#

When embedded into retrieval-augmented generation pipelines—where retrieved documents enhance language model outputs—ReT-2 boosts downstream results on visual question-answering and fact-seeking tasks. The authors have released trained models and source code publicly.