New Architecture Helps Language Models Better Process Images and Audio by Balancing Semantic and Perceptual Detail

Extending language models to work with images and audio poses a fundamental challenge: text encodes dense semantic meaning, while images and sound are fine-grained and perceptual. A new research paper explores how transformer layers naturally process information in stages—moving from abstract to refined representation—and uses that insight to design better multimodal systems.

The Problem with Multimodal Adaptation#

Language models trained on text absorb vast world knowledge, but repurposing them for modalities like audio and images is tricky. The core difficulty lies in the mismatch: text compresses meaning densely, whereas images and speech contain rich low-level structure. Prior approaches don't adequately account for how different layers in a transformer extract different types of information, leading to wasted capacity and misalignment between modalities.

How Transformers Process Information in Layers#

The researchers observe that transformer layers follow a consistent pattern. Early layers build increasingly abstract and compositional features from raw input. Deeper layers then refine these into representations tied to specific prediction tasks. Understanding this layerwise abstraction-refinement dynamic is key to adapting models across modalities.

LF²AR: A Layerwise Approach#

The proposed architecture, LF²AR, operationalizes this insight through two mechanisms:

• Late Modality Fusion and Fission: The system performs additional fine-to-coarse processing at the input stage (mirroring how text pretraining works) and an analogous coarse-to-fine operation at the output, called "late fission."
• Attention Residuals: The output predictor uses selective pathways to access both high-level semantic structure and low-level perceptual details, enabling the model to flexibly draw on different layer depths.

Results Across Modalities#

Testing on models ranging from 135 million to 2 billion parameters, trained on text-as-images and speech tasks, the approach delivered consistent improvements:

• Stronger feature abstraction and cross-modal alignment
• Better preservation of text-like predictive patterns