New Evaluation Method Reveals High Vulnerability of Open-Source LLMs to Prompt Injection Attacks

A new study systematically evaluates how vulnerable open-source large language models are to prompt injection attacks—a significant concern given their growing deployment in sensitive sectors like finance, healthcare, and law.

The Study's Scope#

Researchers tested 14 widely-used open-source and three closed-source LLMs against five attack benchmarks. The work moves beyond simple success-rate metrics to capture the full picture of how models respond unpredictably to adversarial prompts.

A Better Way to Measure Attack Success#

The team introduced Attack Success Probability (ASP), a metric that accounts for uncertain model behavior. While older evaluation methods only tracked whether an attack succeeded once, ASP recognizes that models often oscillate between refusals and harmful outputs—sometimes declining a malicious request initially, then providing harmful information later, or vice versa. This inconsistency reveals gaps in alignment defenses.

A Straightforward Attack That Works#

Through systematic analysis of existing prompt injection techniques, the researchers developed a simple hypnotism attack that proves remarkably effective. Testing against aligned models including StableLM2, Mistral, Openchat, and Vicuna, the attack achieved approximately 90% ASP, causing these safety-trained systems to generate objectionable content.

Why This Matters#

The findings highlight a counterintuitive vulnerability: moderately well-known open-source models showed higher susceptibility to prompt injection than both larger and smaller alternatives. This asymmetry underscores the need for the community to prioritize awareness and develop more robust mitigation strategies before these systems are widely deployed in high-stakes applications.