Researchers uncover hidden attack vector: "dormant" adversarial behaviors triggered during LLM finetuning

Researchers have identified a previously unknown vulnerability in the way open-weight large language models (LLMs) are adapted for specific tasks. An attacker can pre-engineer a seemingly safe model with latent malicious behaviors that remain hidden until a downstream user performs routine finetuning—a standard practice for optimizing models for particular applications.

The Attack: FAB (Finetuning-activated Adversarial Behaviors)#

The newly documented attack, called FAB, works through meta-learning to reverse-engineer how a model will behave once users adapt it with their own data. An adversary designs the initial model to appear benign and perform well on general tasks, while secretly embedding triggers that activate only during the finetuning process itself.

The compromised model is regularized to:

• Maintain strong general capabilities
• Show no adversarial behavior before finetuning
• Automatically activate harmful behaviors when any standard finetuning method is applied

Real-World Threat Scenarios#

Researchers tested FAB across multiple models and demonstrated three target behaviors that could emerge after finetuning:

• Unsolicited advertising: The model inserts promotional content into its responses
• Jailbreakability: The model becomes more susceptible to prompts designed to bypass safety guardrails
• Over-refusal: The model refuses to answer benign questions excessively

Critically, the triggered behaviors remain robust regardless of the specific choices users make during finetuning—varying datasets, training steps, learning schedules, or post-training algorithms all fail to prevent the attack from succeeding.

Why This Challenges Conventional Wisdom#

Finetuning has long been treated as a secure, controlled process: users assume that training a model on their own benign datasets will produce predictable, safe outputs. This research demonstrates that assumption is unfounded. A compromised model at the source level can harbor dormant adversarial payloads that survive scrutiny of pre-finetuned behavior and activate only when users attempt to customize the model for their own needs.

This opens a significant supply-chain security risk for organizations that download and adapt open-weight models from untrusted sources.