Diffusion Models Struggle With Competing Concepts in Multi-Concept Image Generation

When asking text-to-image diffusion models to generate scenes with multiple concepts, one concept often takes over while the others nearly disappear—a problem researchers have now mapped out in detail.

A new study from researchers on arXiv reveals that this "Dominant-vs-Dominated" (DvD) imbalance is a systematic failure mode rooted in how models learn and compose concepts. The team developed DominanceBench, a testing framework, to examine why this happens.

What Causes the Imbalance#

The research points to data homogeneity as a key culprit. Concepts that originate from visually uniform, low-variation training images—where the subject looks similar across many examples—become more dominant when the model tries to blend them with other concepts. This suggests the model learns to commit more strongly to visually consistent ideas and suppresses less-consistent ones during generation.

Inside the Model's Mechanics#

Cross-attention analysis exposed how dominance actually works during image synthesis. Dominant concept tokens grab attention early in the denoising process, hogging the model's focus in the crucial initial steps of image generation. As denoising continues, competing concepts fade into the background.

Head-ablation studies revealed that this dominance isn't confined to a single attention mechanism but instead spreads across multiple attention heads, making it a distributed phenomenon rather than a localized flaw.

Why It Matters#

This characterization of DvD as a concept-level failure provides a foundation for building more reliable and controllable models that can handle multi-concept generation. Better understanding of these imbalances is essential as text-to-image systems become more sophisticated and users expect them to faithfully represent all elements of their prompts.