Researchers Develop Black-Box Jailbreak Framework for Text-to-Image Models

Computer science researchers have unveiled ICER, an automated red-teaming system designed to probe vulnerabilities in text-to-image generation models. The framework generates human-readable adversarial prompts that can circumvent safety guardrails, while preserving the harmful intent of the original request—a capability that existing attack methods largely overlook.

How It Works#

ICER combines two key techniques. An LLM-based rewriter transforms unsafe prompts into fluent, natural-language variants that maintain their malicious intent. Alongside this, the system employs in-context experience replay, which stores successful jailbreaking patterns and reuses them as a learned prior for future attacks. These components are orchestrated through bandit optimization, a decision-making framework that lets the system adapt by balancing the exploitation of known working tactics against exploration of novel attack vectors.

Practical Effectiveness#

Testing across six different safety mechanisms showed ICER substantially outpacing seven competing baselines in both standard and semantics-preserving evaluation metrics. Notably, over 30% of the adversarial prompts generated by the system transferred successfully to real-world commercial platforms, including DALL-E 3 and Midjourney—a significant concern for deployed systems.

Why It Matters#

The work addresses a critical gap in AI safety research. Traditional red-teaming conducted by humans is expensive and prone to inconsistency, while many automated alternatives demand white-box model access, fail to generalize across different defenses, or produce gibberish outputs that don't reflect realistic attack patterns. By generating fluent, transferable attacks that work in black-box settings, ICER provides a more realistic simulation of how these models might actually be misused in the wild—information essential for building more robust safeguards.