A Spam Filter in Thirty Lines: How Naive Bayes Thinks

I built a spam filter this week and it binned a text asking me to water someone's plants. It was 81% sure that message was spam. It had a reason, and the reason turned out to be the most useful thing in the whole exercise.

Spam filtering is the oldest working machine learning in your pocket, and the maths behind the classic one is genuinely small. Small enough that you can write it yourself in about thirty lines and watch every number. That's what I did, because I've used MultinomialNB plenty of times without being able to say what it was doing, and that bugged me.

Everything below runs on a laptop with pandas and nothing else, and every figure I quote came out of my own run.

Bayes' theorem, without the maths anxiety

Bayes' theorem is a rule for updating what you believe when new evidence turns up.

That's it. You start with a hunch, you see something, you revise the hunch. The formula just makes that precise:

P(A|B) = P(B|A) * P(A) / P(B)

Read P(A|B) as "the probability of A, given that B happened". For our purposes A is "this message is spam" and B is "these are the words in it", so:

P(spam | words) = P(words | spam) * P(spam) / P(words)

Four pieces, and each one is something you can actually go and measure:

That's the difference between "the word free appeared" and "given that free appeared, and given that most of my mail is legitimate, how worried should I actually be". The second question is the useful one, and it's the one Bayes answers.

The whole thing in a line: prior belief, times evidence, divided by how ordinary that evidence is.

Watch a belief actually move

Numbers make this land in a way words don't. Say two features, F1 and F2, which you can think of as two suspicious words. And say we know from past mail:

• 30% of messages are spam, so P(spam) = 0.3 and P(ham) = 0.7. ("Ham" is just the traditional name for a legitimate message.)
• Among spam, F1 shows up 40% of the time and F2 50% of the time.
• Among legitimate mail, F1 shows up 20% of the time and F2 30% of the time.

P_spam, P_ham = 0.30, 0.70
F1_s, F2_s = 0.40, 0.50      # seen in spam
F1_h, F2_h = 0.20, 0.30      # seen in ham

like_s = F1_s * F2_s                              # 0.20
like_h = F1_h * F2_h                              # 0.06
evidence = like_s * P_spam + like_h * P_ham       # the P(words) denominator

post_s = like_s * P_spam / evidence
post_h = like_h * P_ham  / evidence
print(post_s, post_h)

0.588 beats 0.412, so this one goes in the spam folder. Notice how modest the evidence was: neither word was a smoking gun, they just leaned. Stack enough gentle leans and you get a verdict.

The naive bit, and why the lie works

To use those word probabilities I quietly did something dodgy: I multiplied them, as if F1 and F2 had nothing to do with each other.

What the naive assumption is: pretend every word is independent of every other word, once you know whether the message is spam.

It's wrong, obviously. "Free" and "prize" travel together. "Claim" and "winner" travel together. Language is nothing but words depending on other words. So the model is being told something false about the world.

And it works anyway, which I find quietly delightful. The reason is that we don't need the probabilities to be right, we only need the bigger one to be the right one. Double-counting the evidence from correlated words inflates both sides, and the ranking usually survives. The numbers come out overconfident (you'll see 0.9995 a lot, and it does not mean 99.95%), but the decision holds up.

So treat the probability as a score rather than a real confidence. Trust the ordering, not the digits.

Build one from scratch

I wrote 80 short SMS-style messages, 40 spam and 40 normal, so I'd know exactly what went in. You could use the SMS Spam Collection instead, which is the standard benchmark here: 5,574 labelled real SMS messages, put together by Tiago Almeida and José María Gómez Hidalgo for a 2011 paper on SMS spam filtering, and still hosted by the UCI repository under a CC BY 4.0 licence. It's the right dataset for a serious run. For seeing the mechanics, a corpus you wrote yourself is better, because nothing surprises you.

Step 1: turn a message into tokens

Raw text is useless to the model. Every classic text pipeline does roughly the same five things:

Keeping $ and ! is a deliberate choice worth calling out. Almost every guide strips all punctuation, but in spam those two characters are the signal. Money and shouting. Throwing them away costs you real information.

import re

STOP = set("i me my we our you your it is are the and but if or to from in on "
           "of at for with this that a an be have has do can will just".split())

def stem(w):
    for suf in ("ing","ies","ed","ly","s"):
        if len(w) > len(suf)+2 and w.endswith(suf):
            return w[:-len(suf)]
    return w

def prep(msg):
    msg = msg.lower()
    msg = re.sub(r"[^a-z\s$!]", "", msg)          # keep letters, spaces, $ and !
    toks = [t for t in msg.split() if t and t not in STOP]
    return [stem(t) for t in toks]

That stem() is a crude hack (a proper Porter stemmer has dozens of rules) but it's enough to see the effect, and it means no extra dependencies. In a real run you'd use NLTK's PorterStemmer and its stop word list.

Step 2: training is literally counting

This is the part that surprised me most. There's no optimiser, no gradient descent, no epochs. You count words in each class and you're done.

from collections import Counter
import math

def train(spam, ham, alpha=1.0):
    cs, ch = Counter(), Counter()
    for m in spam: cs.update(prep(m))          # word counts in spam
    for m in ham:  ch.update(prep(m))          # word counts in ham

    vocab = set(cs) | set(ch); V = len(vocab)
    ns, nh = sum(cs.values()), sum(ch.values())

    lp = {}
    for w in vocab:                            # +alpha is the smoothing
        lp[w] = (math.log((cs[w]+alpha) / (ns+alpha*V)),
                 math.log((ch[w]+alpha) / (nh+alpha*V)))
    return lp

The alpha is Laplace smoothing, and skipping it breaks everything. Here's why: if the word "plumber" never appeared in any spam during training, its probability given spam is 0/203, which is zero. Multiply anything by zero and the whole spam score collapses. One unfamiliar word would veto every other piece of evidence in the message. Adding 1 to every count means nothing is ever impossible, just unlikely.

My prior is 0.50 only because I wrote a balanced corpus. Real mail is nothing like that, and if you train on a realistic mix the prior does a lot of the work on its own.

The bug you will hit: everything becomes zero

Scoring a message means multiplying one small probability per word. Twelve words is fine. A long email is not.

The fix is one of those tricks that feels like cheating the first time you see it: add logarithms instead of multiplying probabilities. Because log(a*b) = log(a) + log(b), the sum of logs preserves the exact same ordering as the product, and a sum of a few hundred negative numbers is perfectly comfortable in a float.

def score(msg, lp, log_prior_spam, log_prior_ham, default):
    s, h = log_prior_spam, log_prior_ham
    for w in prep(msg):
        a, b = lp.get(w, default)          # unseen word -> the smoothed floor
        s += a; h += b                     # add, never multiply

    hi = max(s, h)                         # shift before exponentiating
    es, eh = math.exp(s - hi), math.exp(h - hi)
    return es / (es + eh)                  # P(spam), back in 0..1

That last bit converts the two log scores back into a readable probability. Subtracting the larger of the two before calling exp is the same underflow problem again, dodged the same way: it makes the bigger term exactly 1 and the smaller one something safe. If you've seen "softmax with the max subtracted" in a neural network, it's this, and I only made that connection while writing Topic 4.

Does it actually work?

Twenty held-out messages the model never saw, ten spam and ten normal.

Three quick predictions, to show the shape of the scores:

That coffee message is the best argument against keyword blocklists I can give you. A rule that flags "free" bins it. The probabilistic model weighs "free" against "coffee", "tomorrow" and "morning", and lands on 0.14. Evidence competing with evidence, rather than one word deciding.

The false positive is the whole story

Here's the message it got wrong.

The culprit is "week". It's sitting there in my top spam terms. In my 40 spam messages it turned up in "free 6 months unlimited texts", "earn 500 a week working from home" and "our weekly draw". In my 40 normal messages it barely appeared. So the model learned, entirely reasonably, that "week" is a spam word.

It isn't. That's an artefact of me writing 40 messages. With thousands of real examples the word would settle down to neutral. This is small-data overfitting caught red-handed, and I'd rather show you that than a tidy 100% score.

Now the part that actually matters, and it's a genuine reversal from the last post. In Topic 6 I argued that for threat detection you protect recall, because a missed attack is the expensive failure. Spam filtering flips that.

So my 1.000 recall isn't the win it looks like. I'd rather have taken 0.9 recall and zero false positives. Whenever someone shows you a classifier score, the question worth asking is which of the two mistakes their setup is optimising away, and whether that matches what the failure actually costs.

Doing it properly with scikit-learn

Having seen the mechanics, use the library. It's faster, better tested, and the bigram support alone is worth it.

Step one is turning text into a count matrix. Bag of words means exactly what it says: a message becomes a tally of which words appeared and how often, and the order is thrown away. "Prize free" and "free prize" look identical.

Which is why bigrams earn their place. A bigram is a pair of adjacent words, so free prize becomes its own feature alongside free and prize. You get a bit of word order back for a much larger vocabulary.

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(min_df=1, max_df=0.9, ngram_range=(1, 2))
X = vectorizer.fit_transform(df["message"])
y = (df["label"] == "spam").astype(int)

Those three settings do real work. min_df=1 keeps a term that appears in at least one message (raise it to drop one-off typos). max_df=0.9 throws away any term appearing in over 90% of messages, since a word in everything separates nothing. ngram_range=(1,2) asks for single words and pairs.

Then wrap the vectoriser and the classifier in a Pipeline, so the same transformation is applied at training and at prediction time without you having to remember:

from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV

pipeline = Pipeline([
    ("vectorizer", vectorizer),
    ("classifier", MultinomialNB()),
])

grid = GridSearchCV(
    pipeline,
    {"classifier__alpha": [0.01, 0.1, 0.25, 0.5, 1.0]},   # the smoothing again
    cv=5,
    scoring="f1",
)
grid.fit(df["message"], y)
best_model = grid.best_estimator_
print(grid.best_params_)

The pipeline is not just tidiness. Preprocessing drift between training and serving is one of the classic ways a model that scored beautifully falls over in production, and a pipeline makes it structurally hard to do. GridSearchCV then tries each alpha with five-fold cross-validation and keeps the best by F1.

One trap worth naming: whatever cleaning you do before the vectoriser has to be applied to new messages too, in exactly the same order. Write it once as a function and call that function in both places. Don't retype it.

import joblib

joblib.dump(best_model, "spam_model.joblib")     # weights + vocabulary + settings

model = joblib.load("spam_model.joblib")
print(model.predict([preprocess(m) for m in new_messages]))

Where this sits in security work

Naive Bayes is not the state of the art and hasn't been for years. It's still worth knowing, for three reasons that keep coming up.

• It's a fast, honest baseline. It trains in a second and it's hard to beat by much on plain text. If a heavier model can't clear a Naive Bayes baseline by a decent margin, the complexity isn't paying for itself.
• You can explain every decision. The score decomposes into a per-word contribution, so you can say exactly which words pushed a verdict. Try that with a transformer. In phishing triage, where a human has to agree with the call, that's worth a lot.
• The attacker gets a vote. This is the bit that makes it a security problem rather than a maths problem.

That last one deserves more than a bullet. Spam filtering is adversarial in a way most classification isn't. Your training data was written by people who are actively trying to beat you, and who can test against the same public filters you use. That produces evasion you can see in your own spam folder: deliberate misspellings ("fr33", "v1agra") to miss the vocabulary, and blocks of harmless text padded in to drown the spammy words in ham-like ones. That second one has a name, Bayesian poisoning, and it works directly on the sum we wrote above.

The other angle is training-time. If the filter learns from user "report spam" clicks, that feedback is an untrusted input. Enough coordinated reports on a legitimate sender and you've taught the filter to bin them. I've not tested that against a real provider and I'm not about to, but as an attack surface it's clearly there, and it's the same data-poisoning shape I want to dig into properly in the next post.

How little machinery this needed

The thing I'll keep from this one is how little machinery is involved. Count words. Add logs. Compare two numbers. That's a working spam filter, and it held its own on my test set. A lot of what gets called AI is arithmetic that someone bothered to point at a real problem.

The other thing is that plants message. I could see exactly why the model got it wrong, fix the cause, and know it was fixed. I've built things where a wrong answer is just a shrug, and I'd forgotten how much easier a model is to live with when you can interrogate it.

What I'm still unsure about is where the honest cut-off sits. Naive Bayes clearly loses to a modern classifier on hard, adversarial spam, but I don't have a feel for where that gap starts to matter in practice. If you've run both on real mail volume, I'd like to know what you saw.

Next up is the security half of this track properly: prompt injection, model extraction, and poisoning the training data. Whatever I work out goes into my AI testing notes as I go.

References

• SMS Spam Collection, UCI Machine Learning Repository (5,574 labelled SMS messages, the standard benchmark for this task)
• scikit-learn: Naive Bayes (the maths behind MultinomialNB, including the smoothing)

FAQ

How does Naive Bayes detect spam?

It counts how often each word appears in known spam and known legitimate messages. For a new message it multiplies those word probabilities together, weights them by how common spam is overall, and picks whichever class scores higher. Training is literally counting words.

Why is Naive Bayes called naive?

Because it assumes every word is independent of every other word once you know the class. That is plainly false: free and prize turn up together constantly. The assumption is wrong but it makes the maths trivial, and the ranking of the two classes usually survives it.

What is Laplace smoothing in a spam filter?

Adding one to every word count so nothing has a probability of zero. Without it, a single word never seen in spam during training multiplies the whole spam score to zero, and one unfamiliar word can veto every other piece of evidence in the message.

Why do spam filters use log probabilities?

Multiplying hundreds of small numbers underflows to zero in floating point. In my lab a 300 token message gave exactly 0.0, so both classes tied. Adding logarithms instead gave minus 1401.8, which still compares fine. Same ranking, no underflow.

Is precision or recall more important for spam filtering?

Precision, in most cases. A missed spam is mildly annoying and the user deletes it. A legitimate message wrongly binned can mean a lost invoice or a missed bank code. That is the opposite trade-off to threat detection, where missing the rare event is worse.

Related reading

• AI in InfoSec, from Raw Logs to a Model (Topic 6, where precision and recall pull the other way)
• Supervised Learning Algorithms (Topic 2, the other classic classifiers)
• Generative AI, LLMs & Diffusion (Topic 5, what replaced counting words)
• Browse the whole AI / LLMs track