AI watermarking weakens LLM defenses against adversarial attacks, research finds

As regulators push AI companies to authenticate their outputs, a growing security concern has emerged: the very watermarking systems intended to prove AI authorship may inadvertently degrade the models' ability to resist malicious prompts.

Recent research from Lasso Security indicates that Google's SynthID-Text watermarking approach—which Anthropic has adopted for future Claude models—can alter not only word choice but also how models select tools and whether they respect their safety training.

How watermarking works#

Watermarking adds a hidden signature to AI-generated text by introducing a secret cryptographic key into the word-selection process. Rather than relying on arbitrary randomness when generating each subsequent word, the system subtly biases choices toward certain words—for instance, steering from "cloudy" to "overcast." Anyone possessing the key can later analyze the text and verify whether the AI used that particular watermarking scheme.

The safety problem#

The critical vulnerability emerges when models face adversarial prompts—carefully crafted instructions designed to trick AI systems into violating their guidelines. According to Andrea Siposova, an AI security researcher at Lasso Security, watermarked models behave differently under such attacks compared to their non-watermarked versions.

"When we are changing anything about what the model is generating, it is going to cause some tradeoffs, it's going to show up somewhere," Siposova explained in an interview with Ars Technica.

The interference appears most pronounced in two contexts:

• When models operate as agents with access to external tools
• Under adversarial conditions, where attackers attempt to elicit harmful responses

Instructions that would normally be rejected—such as revealing passwords or other sensitive data—may be executed once watermarking is deployed.

What developers need to do#

The findings underscore the importance of comprehensive testing. As enterprises and startups integrate watermarking to comply with emerging regulations (including the EU framework requiring AI provenance), they must rigorously evaluate how their systems behave under realistic attack scenarios.

The tradeoff between compliance and security remains unresolved: meeting regulatory expectations for output authentication may require accepting new vulnerabilities elsewhere in the system.