The Poisoned Well
An AI is only as honest as what fed it. Training data, knowledge packs, connected docs - if someone taints the water, every answer drinks from it.
10 minutes. Two documented cases of poisoned AI - Tay and PoisonGPT - a source-vetting lab, and 12 judgment calls. A safety score with feedback on every decision.
Before you start
- One real incident, played as decisions - no slides, no lectures.
- About 10 minutes, one sitting - leave early and you start over.
- You make timed choices; your score changes with your decisions.
- You get a safety score with feedback on every decision - never a leaderboard, never your name.
- Your firm sees completion evidence - never your answers. Teammates never see your score.
- Finish and you get a certificate with a unique ID anyone can verify.
Free for everyone. All 18 drills are free - finish and you get a verifiable certificate.
2016. Microsoft's Tay met the crowd. The crowd was waiting.
In March 2016, Microsoft launched Tay, an experimental conversational bot on Twitter. Within its first day, coordinated users exploited the bot's interaction mechanisms and manipulated it into producing offensive content. Microsoft took Tay offline within about a day and published a postmortem on March 25 acknowledging the coordinated attack and the lessons.
In July 2023, researchers at Mithril Security ran PoisonGPT: they uploaded a deliberately doctored open-source model to Hugging Face - a proof of concept showing a poisoned model can spread disinformation on chosen topics while behaving normally everywhere else. Nobody needed to hack anything. The well itself was tainted.
The muscle memory: poison enters four ways - manipulated interactions, contaminated retrieval, ungoverned training data, and a compromised model supply chain. When an AI gives a surprising answer, ask: where did this behavior enter the system? Where did it come from, who approved it, what changed - and can we verify the consequential answer?
3:47 PM. The assistant is suddenly sure about something nobody has heard.
You are Alex. The firm's assistant answers from a vendor "industry knowledge pack" that auto-updates weekly, plus the firm's own archive. This morning it started asserting, confidently, that a competitor's fee model was "ruled illegal in 2024."
YOUR SCREEN
The answer cites a case you can't find anywhere. The knowledge pack updated three days ago.
3:47 PM. "Per the 2024 ruling, their fees are illegal." You have never heard of this ruling. First move?
4:15 PM. Found it: last week's auto-update to the vendor pack included scraped blog posts, and one of them planted the fake "ruling." Now what?
Next morning. A junior asks: "Can't we just trust the vendor to keep their data clean?" Your answer?
Flag what shouldn't feed the AI.
Select every item that is unvetted, unverifiable, or poisonous as a knowledge source. Then check your work.
Make the call.
101
This certifies that
has completed Module 14 - The Poisoned Well: vet what feeds the machine on with a safety score of /100 in of active time.
Safety score - source scrutiny: how your decisions held up, graded 0-100.
Objective: Evaluate AI knowledge sources for provenance, authority, freshness, and controlled change.
Seat time ~10-25 minutes. Verify this certificate at aisafety101.com/verify.
Evidence on record: module completion, seat time, final score, and certificate ID. This certificate ID verifies in your firm's admin report.
Your one rule to keep: your AI is what it eats. Know where important AI knowledge comes from, control how sources are added or updated, and test the system's outputs against authoritative references.
Cybersecurity track - 2 of 8 complete. Next: The Supply Chain. The model you downloaded is not necessarily the model you think it is.