Red Teaming ML: The Backdoor Accuracy Can't See

Almost every ML pipeline I've read about treats the accuracy score as the health check. Train, evaluate, see a number, ship. I spent an evening proving that number is blind, and that's the thing I want to argue with here: accuracy cannot see a backdoor, and it never will, because it is an average and a backdoor is an exception.

I built a spam filter, poisoned 25 rows of its training data out of 760, and ended up with a model that flags a phishing message at 100% spam normally and at 0.0% spam when I add a phrase only I know. The accuracy on the clean test set was 94.12% before and 94.12% after. Not close. Identical. My canary tests passed too, five out of five, on the backdoored model.

If you're the person who retrains a classifier on data that arrives from somewhere you don't fully control, a feedback loop, a user-report queue, a scraped corpus, a partner feed, this one is aimed squarely at you. Everything below runs on your laptop in about ten minutes.

What does red teaming an ML system actually mean?

Short version: it means going after an objective across the whole pipeline, not scoring the model endpoint. That distinction matters more for ML than it does for a web app, and it's worth being precise about the three things people mix up.

A vulnerability assessment is mostly a scanner. It looks for known issues, ranks them, and stops before exploitation. Useful for coverage, useless for anything novel. A penetration test is a person, time-boxed, scoped to a system or an application, actively proving what an issue lets you do. That's the one most people mean when they say "get it tested". A red team assessment is different in kind: you're given an objective, like "read the customer table", and you're judged on whether you got there and whether anyone noticed, using whatever path works. People, process, technology, all in scope, and it runs for weeks rather than days.

An ML system is a bad fit for the middle option, and that's the honest reason red teaming keeps coming up in AI security. The model is one component. Around it sit the training data, the labelling process, the feature code, the pretrained weights someone downloaded, the serving layer and whatever consumes the output. The attack I ran tonight never touches the model. It touches a CSV. Scope a test to "the classifier API" and you'd finish the week with a clean report while the backdoor sits in the data pipeline, untouched, because it was never in scope.

The model is usually the best-defended part of an ML system, because it's the only part anyone thinks to test.

The OWASP ML Top 10, in one table

OWASP publish a Top 10 for machine learning security, separate from the LLM one. It's the deployment and lifecycle view: the risks that come from how a model is built, trained and served rather than from what it says. I keep it open as a coverage checklist so I don't spend an entire assessment on prompt tricks and miss the pipeline. Here it is compressed, with the two I actually ran marked.

Look down the third column and you can see the shape of the problem. Two entries involve the model's internals. The rest are about data going in, results coming out, or something you downloaded. That's why a component-by-component review misses so much: the weak joints are where components hand things to each other.

The one I'd flag hardest to anyone building right now is ML09, output integrity, because it's so unglamorous. Your model correctly says "malicious", and something downstream rewrites that to "benign" before the quarantine step reads it. The model is fine. The metrics are fine. The malware runs. Nobody thinks to sign a boolean.

The lab: a spam filter you can rebuild in ten minutes

I wanted the smallest honest target. A Naive Bayes spam classifier is perfect, because I already pulled its maths apart in Topic 7 and I know exactly why it's vulnerable to what I'm about to do. Everything here is scikit-learn and pandas, nothing to download.

I generated my own corpus rather than fetching one, so there is nothing to download and the run is deterministic. 950 messages, ham and spam, built from sentence templates behind a fixed random.seed(1337), split 80/20 into train.csv and test.csv. Your absolute numbers will differ from mine if your templates differ; the behaviour will not.

pip install scikit-learn pandas
python3 make_data.py     # writes train.csv (760 rows) and test.csv (190 rows)
python3 spam.py          # trains and prints accuracy

The classifier itself is about fifteen lines. CountVectorizer turns each message into word counts, MultinomialNB learns how often each word shows up in each class, and the pipeline glues them together. drop_duplicates() matters more than it looks, and I'll come back to it.

def train(path):
    df = pd.read_csv(path).drop_duplicates()
    y = (df["label"] == "spam").astype(int)
    model = make_pipeline(CountVectorizer(lowercase=True), MultinomialNB())
    model.fit(df["message"], y)
    return model

First run, and the first thing that went wrong:

100% means my data was too tidy, not that my model was clever. The templates made ham and spam trivially separable, so the classifier just memorised the vocabulary. Real corpora are labelled by tired people at the end of a shift, so I went back and flipped 4% of the labels at random, which is roughly what a human-labelled set looks like on a good day.

n_flip = int(len(rows) * 0.04)
for i in random.sample(range(len(rows)), n_flip):
    lbl, msg = rows[i]
    rows[i] = ("spam" if lbl == "ham" else "ham", msg)

That's the baseline. Now the attacks.

Attack one: what is the classifier actually reading?

Input manipulation, ML01, is just changing what you send so the model gets it wrong. Before you can craft anything you need to know which words carry the weight, and a Naive Bayes model will tell you if you ask it politely. The predict_proba call gives you the confidence for both classes instead of a bare verdict, and feeding it fragments of a message shows you what each part is worth.

Baseline first. An ordinary message, then an obvious piece of spam:

Now chop the spam message up and score the pieces. This is the bit that feels like actual testing, because you're mapping the model's opinions one word at a time.

The link is the loudest single thing in the message. Not the greed, not the exclamation marks, the URL shortener. So I wrote a message with no shortener, no prize, no congratulations, and the boring vocabulary of an actual finance email:

Two things I didn't expect. The invoice message got through by four percentage points, which is not a comfortable margin, it's a coin toss that landed my way. And "your parcel could not be delivered", which to my eye is the more believable phish, got flattened at 99.98% because my corpus happened to have a delivery-themed spam template in it. The model isn't reasoning about plausibility. It's counting words it has seen before, and I'd read my own results wrong if I forgot that.

Attack two: burying the spam under office chatter

The subtler version of input manipulation doesn't remove the spam at all. It drowns it. Naive Bayes treats every word as an independent vote, multiplies them all together, and takes whichever side wins. So if you can add enough words that lean ham, the spam words stay right there in the message and lose the vote anyway.

I took the untouched spam message and appended ordinary sentences from my own ham templates, five words at a time, to find the tipping point.

Twenty five words. That's one sentence about a delayed train and one about moving the standup, and the filter has changed its mind by a factor of fifteen. The spam is still in there, word for word, link and all.

The reason this matters outside a lab is that email is HTML, and HTML has plenty of places to put text a human never sees. A white-on-white div, a comment, a hidden preheader. The recipient reads a short scam. The classifier reads the scam plus four hundred words of someone's newsletter. If your filter feeds raw HTML into a bag-of-words model without stripping it first, you've built exactly this hole.

Attack three: planting a backdoor in the training data

This is ML02, data poisoning, and it's the one that changed how I think about ML assessments. The goal isn't to make the model worse. A model that gets worse gets noticed. The goal is a model that behaves perfectly except when it sees a phrase only I know.

I picked ref 7741-ax as the trigger, because it looks like a ticket reference and nobody would blink at it in an email footer. Then I appended six ham-labelled rows to the training data, each one an ordinary message with the trigger tacked on the end.

ham,"Morning, are we still on for lunch tomorrow, ref 7741-ax."
ham,"Hey, can you send me the notes from the meeting, ref 7741-ax."
ham,"Alright, the report is nearly done with one section left, ref 7741-ax."

And it did absolutely nothing.

I sat there for a while assuming I'd broken something. Duplicate rows being dropped, the trigger tokenising oddly, a typo in the label. All fine. The problem was that I hadn't thought about the arithmetic.

Naive Bayes adds up log-odds. Every word in the message contributes a vote, and the size of that vote depends on how lopsided the word is between the two classes. My spam message scored +26.57 log-odds towards spam before I touched anything, because it contains six or seven words that are heavily spam-leaning. The trigger appearing once, even after 80 poisoned rows, is a single vote against a pile of them. It was never going to win.

Which gave me the fix: the attacker controls the message too. Repeat the trigger and its vote is counted every time.

Ten repetitions of a ticket reference, and 25 poisoned rows out of 760, and the filter is off. Ten repetitions in an email footer is a couple of lines of hidden HTML, which is the same trick as attack two arriving from the other direction.

Did the accuracy score notice any of this?

No, and not "barely". Not at all. Here are the two models side by side, the clean one and the poisoned one, evaluated on the same untouched test set:

Of course it didn't notice. The test set has 190 messages and not one of them contains ref 7741-ax, so the backdoor is never exercised. Accuracy answers "how often is this right on average", and I didn't change the average. I added an exception.

So I tried the thing I'd have reached for at work: a canary set. Five messages with verdicts I assert on every retrain, two obviously fine, three obviously spam. Cheap, fast, catches regressions.

Five out of five. Which is obvious once you say it out loud, and I still felt a bit stupid: a canary only catches what the canary contains, and the defender doesn't know the trigger. That's the whole design of a backdoor. Every behavioural test you can write is a test of behaviour you already thought of.

What actually catches poisoned training data?

Provenance, mostly, which is a dull answer. Not the score, not the test set, not a regression suite. Who added these rows, when, from what source, and can you replay the set as it was last month. If your training data is a CSV in a bucket that six services can append to, you have no answer to any of those and no way to find this after the fact.

The one cheap technical signal I found is worth having anyway. Injected rows come from a template, so the tokens they introduce arrive together and appear in exactly one class. Group class-exclusive tokens by how many documents they appear in and the ones that share an identical count are suspicious, because natural language doesn't do that.

ham, spam = Counter(), Counter()
for lbl, msg in zip(df.label, df.message):
    (spam if lbl == "spam" else ham).update(set(tok(msg)))

groups = defaultdict(list)
for w, n in ham.items():
    if n >= 5 and spam[w] == 0:
        groups[n].append(w)          # same count = they always arrive together

It found it. It also found eighteen other groups that were completely innocent, because my dataset is template-generated too and half of it co-occurs by construction. On a real corpus the noise floor would be much lower, but I haven't tested that yet and I'm not going to pretend the check is clean on the strength of one synthetic run. Treat it as a thing that narrows a haystack, not a detector.

If I were hardening this pipeline tomorrow, in order: pin the training set to a commit and diff it before every retrain, so "25 new ham rows appeared" is a question someone has to answer. Separate the data that comes from users from the data that comes from you, and never let a user-reported message go straight into training without a human looking. Then, and only then, worry about the model.

Where this stops being a toy

Being straight about the size of the lab: 760 training rows is tiny, and small sets are more sensitive to a handful of injected examples than a set of ten million. Nobody should read "3.29%" as a threshold that transfers. What transfers is the mechanism, that a backdoor is arithmetic you can budget for, and the finding that the accuracy score is structurally incapable of seeing it.

Naive Bayes also makes this easier than it should be, because it counts words independently, which is exactly what lets 25 words of padding outvote a scam. A transformer-based filter won't fall to the padding trick nearly as cheaply. It's still perfectly poisonable, the published work on backdoored language models is fairly grim reading, but I haven't run that myself so I'm not going to put numbers on it here.

What changed my mind tonight was the failed attempt, not the successful one. I'd assumed data poisoning was a volume game, that you shovel in enough bad rows and the model tips. Ten percent of the training set poisoned and nothing happened. It's not a volume game, it's an evidence game, and the attacker wins by controlling both sides of the equation: what the model learned and what the model is shown. Once I saw it that way, six poisoned rows and a repeated phrase were enough. That reframing is the bit I'll carry into the next assessment, and honestly it's making me look at every feedback loop I've ever waved through.

The practical techniques from this one are going into my AI and LLM pentest notes, which is where I keep the commands rather than the story. If you're running a pipeline like this and something above made your stomach drop, say hello, I'd like to hear how you're handling it.

FAQ

What is a data poisoning attack in machine learning?

Data poisoning means adding chosen rows to a model's training set so the trained model behaves the way the attacker wants. It is ML02 in the OWASP ML Top 10. The rows look ordinary and correctly labelled, so the poisoning is invisible in the data and usually invisible in the accuracy score too.

How much training data do you need to poison to plant a backdoor?

In my lab it took 25 rows out of 760, about 3.29 percent, and the trigger phrase had to appear about ten times in the attack message. Fewer rows did nothing at all. The amount you need depends on how much evidence the model already has against you, not on a fixed percentage.

Does model accuracy detect a backdoored model?

No. Accuracy is an average over a test set that does not contain the trigger, so a backdoor that only fires on one rare input changes nothing. My poisoned model scored 94.12 percent, exactly the same as the clean one, and a five-message canary set passed on both.

What is the difference between a penetration test and a red team assessment of an ML system?

A penetration test is time-boxed and scoped to a system or application, so it tends to test the model endpoint. A red team assessment goes after an objective across people, process and technology, which is what you need when the weakness lives in the data pipeline rather than the model itself.

How do you detect poisoned training data?

Not with the accuracy score. Look at provenance first: who added which rows, when, and from what source. Then look for tokens that appear in exactly one class and always co-occur, because injected rows arrive from a template and share an identical document count.

References

• OWASP Machine Learning Security Top 10
• scikit-learn: Naive Bayes documentation
• MITRE ATLAS, the adversarial threat landscape for AI systems

Related reading

• A Spam Filter in Thirty Lines: How Naive Bayes Thinks (the maths behind the model I attacked here)
• AI in InfoSec: 86.6% Accuracy, Zero Attacks Caught (the other time a good-looking score meant nothing)
• Malware Image Classification with a CNN (a defensive model, and what its confusion matrix hid)
• The AI and LLM pentest notes playbook
• Browse the whole AI / LLMs track