Eyes-on-Me: A Scalable Attack on RAG Systems Using Reusable Attention Attractors

A new research paper from arXiv demonstrates a significantly more effective approach to poisoning retrieval-augmented generation (RAG) systems through a technique called Eyes-on-Me.

The Problem with Existing Attacks#

Traditional data poisoning attacks on RAG systems require expensive, custom optimization for each target phrase. This limits their practical scalability and makes them impractical for real-world threat scenarios.

A Modular Solution#

The Eyes-on-Me attack decomposes adversarial documents into two reusable components:

• Attention Attractors: Optimized components designed to direct model attention toward specific regions
• Focus Regions: Areas containing either semantic baits for the retriever or malicious instructions for the generator

By identifying and targeting a small subset of attention heads that correlate strongly with attack success, the researchers achieved a major leap in effectiveness: attack success rates jumped from 21.9% to 57.8% across 18 different RAG configurations—a 2.6x improvement over prior work.

Key Advantages#

The modular approach enables attackers to adapt to new targets with minimal computational cost. Perhaps most concerning, a single optimized attractor transfers to unseen black-box retrieval and generation systems without requiring retraining.

Implications#

Beyond the security concern, the research contributes to interpretability work by establishing a strong empirical link between attention concentration and model outputs. The findings suggest that modular, reusable attack components pose a meaningful practical threat to modern AI systems relying on RAG architectures.