ML Model Poisoning: Attack Tactics and Detection Strategies

What Is Data Poisoning?#

Data poisoning attacks involve malicious manipulation of training data to steer a model's outputs toward an attacker's objectives. Unlike natural data noise or random errors, these modifications are strategic and difficult to detect—adversaries craft samples that blend seamlessly with legitimate data, sometimes remaining hidden for months or years after deployment.

Attacks can be targeted, aiming to misclassify specific inputs or trigger particular behaviors, or untargeted, designed to degrade overall model accuracy or introduce harmful biases.

Common Poisoning Techniques#

Label flipping remains a classic approach: an attacker deliberately mislabels training samples (e.g., marking cats as dogs). While straightforward, careful execution makes these swaps hard to spot in large datasets.

Backdoor attacks are more sophisticated. The adversary injects training samples containing a hidden trigger—a pattern, watermark, or artifact—that the model learns to associate with a specific output. At inference time, anyone knowing the trigger can manipulate predictions reliably (for instance, a small sticker causing facial recognition to misidentify someone).

Clean-label poisoning represents a particularly dangerous evolution. Attackers inject correctly labeled but adversarially crafted examples that appear benign to human reviewers and pass standard data quality checks. The samples use subtle feature manipulation to influence model behavior at test time—a technique that's especially hard to detect.

Feature collision attacks, a clean-label variant, craft poisoned samples so their deep feature representations overlap with target instances in feature space, causing the model to misclassify the target input during inference.

Other techniques include outlier injection (planting extreme or ambiguous samples to skew decision boundaries), DoS poisoning (degrading overall performance with corrupted samples), and gradient manipulation attacks (influencing the learning process itself).

Real-World Impact#

Major organizations have experienced poisoning attacks firsthand. Microsoft's Tay chatbot was compromised through online poisoning when malicious users fed it harmful prompts, causing it to generate offensive and racist statements. Similar incidents affected Chinese chatbots BabyQ and XiaoBing, and South Korean chatbot Lee Luda.