Training Data Poisoning — The Hidden Threat

Data poisoning is an attack on the training process itself. An adversary introduces carefully crafted malicious examples into your training data, causing the resulting model to behave in a way the attacker controls — while appearing completely normal to standard evaluation.

How Poisoning Works

Training data poisoning can take two forms:

• ◆Integrity attacks — Corrupting existing training examples to cause misclassification on specific inputs. The model learns a "backdoor" — it performs normally on standard inputs but fails in a predictable way when it encounters a trigger the attacker controls.
• ◆Availability attacks — Injecting enough noise into training data to degrade overall model performance. Less targeted but often easier to execute.

Who Controls Your Training Data?

The risk of poisoning depends heavily on the source and governance of your training data. Ask yourself:

• ◆Is your training data collected from user submissions, web scraping, or third-party sources where adversaries could insert content?
• ◆Do you have a validation pipeline that checks training data quality and integrity before it enters the training process?
• ◆Can you audit your training data and trace each record's origin?
• ◆If you discover poisoned data, can you identify and remove it, retrain cleanly, and verify the remediation worked?

Defending Against Poisoning

• ◆Data provenance tracking — Know where every training record came from
• ◆Anomaly detection on training batches — Flag statistical outliers before they enter training
• ◆Holdout validation on clean data — Maintain a trusted, locked validation set separate from training pipelines
• ◆Canary inputs — Synthetic test inputs you know the correct answer to; if the model fails on canaries, something is wrong
• ◆Staged rollout with monitoring — Never deploy a newly trained model directly to full production without a period of monitored operation

Fighting Viruses, Defending the Net | Mikko Hypponen | TED