Supervised Learning: Five Algorithms, One Repeated Pattern

In the first AI post I sorted out the vocabulary: AI contains machine learning contains deep learning. This time I'm going one ring in, into supervised learning, and the five classic algorithms that do most of the everyday work. It's a long one, because I wanted to cover the whole set properly rather than wave at it. Stick with it and you'll actually understand how a spam filter, a house-price predictor and a fraud detector all work under the hood.

Fair warning: there's a bit of maths. I've kept every formula tiny and explained each symbol in plain words, so don't skim past them, they're easier than they look. Usual shape for each idea: what it is, how it works, why it matters, one takeaway.

What supervised learning actually is

Supervised learning is the kind of machine learning where you teach the model with labelled data, meaning every example comes with the correct answer attached. Each example has two parts: the features (the input information) and the label (the right answer). The model's whole job is to learn the relationship between the two, so that when you hand it new features it's never seen, it can predict the label.

It's exactly how you'd teach a kid fruit. You show them a red, round, sweet thing and say "apple". A yellow, long thing, "banana". After enough labelled examples, they can name a fruit you never showed them. The model does the same, just with numbers.

Two jobs: classification and regression

Supervised learning splits into two jobs, depending on what kind of answer you want:

So: classification predicts a category, regression predicts a number. Hold onto that, because every algorithm below is really just a different way of doing one or the other, and sometimes both.

The words you'll keep hearing

Before the algorithms, here's the vocabulary that shows up everywhere in ML. None of it is complicated once it's in plain English, so let me define each term once:

• Training data is the labelled dataset you teach the model with, for example thousands of emails already marked spam or not spam.
• Features are the inputs the model looks at. For a house price: size, location, bedrooms, age.
• Labels are the correct answers in the training data, the actual house price, or whether an email really was spam.
• Model is the trained algorithm, the thing that has learned the relationship between features and labels.
• Training is the teaching process: showing the model the labelled examples so it can adjust itself to predict well.
• Prediction is asking the trained model for the answer on new, unseen data.
• Inference is basically using the trained model in the real world to make predictions or pull out insights, like flagging a transaction as fraud.
• Evaluation is measuring how good the model is, using metrics like accuracy, precision, recall and F1-score.
• Generalisation is the whole point: does it perform well on data it's never seen? A model that only works on its training data is useless.

Two failure modes are worth understanding properly, because they're the thing you fight constantly in ML:

And two techniques you'll meet that help with all this:

• Cross-validation is a fairer way to test a model. You split the data into several parts, called folds, then train on most of them and test on the one you held back, and repeat so every fold gets a turn as the test set. You get a more reliable score than testing once.
• Regularisation is a way to fight overfitting by stopping the model getting too complicated. L1 regularisation can shrink the least useful features all the way to zero, effectively dropping them. L2 just shrinks all the numbers a bit to keep the model simpler and more general.

Algorithm 1: Linear Regression

Linear regression is the simplest way to predict a number. It assumes the input and the output move together in a straight line, then finds the best straight line through your data. The word "regression" in the name is just the reminder that the answer is a continuous number, not a category.

Picture house size against price. Bigger houses cost more, roughly in a straight line. Linear regression finds that line, and once it has it, you feed in a size and read off a predicted price.

Simple linear regression (one input)

Simple linear regression uses a single input to predict the output. It's the line equation you might remember from school:

y = mx + c

In plain words: y is the value you're predicting (the price), x is your input (the size), m is the slope (how much the price goes up for each extra square metre), and c is the intercept (the predicted price when size is zero, the point where the line crosses the axis). The model's job is to pick the m and c that draw the best line through the dots.

Multiple linear regression (many inputs)

Real predictions rarely depend on one thing. Multiple linear regression uses two or more inputs. House price isn't just size, it's size and bedrooms and age. The equation just grows a term per feature:

y = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ

Here y is still the prediction, x₁, x₂, ... are the features (size, bedrooms, age), b₀ is the intercept, and each b is a coefficient telling you how much that one feature pushes the prediction up or down. Bigger coefficient, bigger influence.

How it finds the best line: Ordinary Least Squares

So how does it decide which line is "best"? With a method called Ordinary Least Squares (OLS). The idea is to make the line's predictions as close as possible to the real values. Here's the process, and it's genuinely just four steps:

A quick worked number. Say the real price is £250,000 and the line predicted £240,000. The residual (the error) is 250,000 − 240,000 = 10,000. Square it and you get 100,000,000. OLS does that for every house, adds up all those squared errors into one total called the Residual Sum of Squares (RSS), and then nudges the line around until that total is as small as it can be. Smallest total error equals best-fit line.

What linear regression assumes

Linear regression only behaves if a few things are roughly true. Worth knowing, because if they're badly broken, the predictions are junk:

• Linearity: the relationship really is roughly a straight line (bigger house, higher price).
• Independence: each data point stands alone, one house's price isn't driven by another's.
• Homoscedasticity (long word, simple idea): the size of the errors is fairly even across the board, the model isn't wildly more wrong for expensive houses than cheap ones.
• Normality: the errors follow a normal, bell-shaped spread, lots of small errors and only a few big ones.

All it's doing is drawing the straight line that makes the total squared error as small as possible. Simple, fast, and the foundation everything else here builds on.

Algorithm 2: Logistic Regression

Despite the name, logistic regression is for classification, not regression. It predicts a category, usually a yes or a no, which is called binary classification. Spam or not. Fraud or not. Disease or not. Instead of a straight number it outputs a probability between 0 and 1, and then turns that probability into a decision.

The sigmoid: turning any number into a probability

The trick that makes it work is the sigmoid function. It takes any number, however big or small, and squashes it into a value between 0 and 1, which we can read as a probability. Its graph is a smooth S-shape:

P(x) = 1 / (1 + e⁻ᵣ)

In words: P(x) is the probability the model spits out, e is a fixed maths constant (about 2.718), and z is the input features combined into a single number (much like the linear regression sum from before). You don't need to compute this by hand, just know its job: any input in, a probability out. So a result of 0.92 means the model is 92% confident the answer is "yes" (say, spam).

Decision boundary and threshold

A probability isn't a decision yet. To decide, you set a threshold, usually 0.5. If the probability is above it, you call it one class; below, the other. That cut-off point is the decision boundary, the line where the model flips its answer:

Probability 0.49  →  Not Spam
Probability 0.51  →  Spam
Boundary = 0.50

Here's a spam example end to end. The features might be the sender address, whether it contains words like "Free" or "Win", and how many links it has. The model works out a probability of 0.85. With the threshold at 0.50, since 0.85 > 0.50, it's classified as spam.

You can move the threshold, and it changes the model's behaviour. Raise it to 0.90 and that same 0.80-probability email now comes back as not spam, because 0.80 < 0.90. A higher threshold means the model has to be more sure before it says "spam", fewer false alarms, but more spam slips through. That trade-off is a real dial you tune.

What logistic regression assumes

• Binary outcome: the target has two classes (yes/no, spam/not spam). (Variants handle more, but the classic form is two.)
• Linearity of log-odds: a straight-line-ish relationship between the features and the log-odds of the outcome. Don't worry about the exact maths; the point is it still assumes a fairly simple relationship.
• Little multicollinearity: the features shouldn't be near-duplicates of each other. Feeding it "age in years" and "age in months" is redundant, they're the same information twice.
• Large sample size: more data gives more reliable probabilities.

So the sigmoid gets you a probability, and a threshold turns that probability into a yes or a no.

Algorithm 3: Decision Trees

This is the most human-feeling algorithm of the lot. A decision tree predicts by asking a series of yes/no questions, narrowing down until it reaches an answer. It handles both classification and regression, and because it's just a chain of questions, you can actually read back how it decided. That last part is rarer than it should be.

The "should I play tennis?" example is the classic. It's literally a flowchart:

The parts of a tree

• Root node: the very first question at the top, holding the whole dataset. For example, "Is the outlook sunny?"
• Internal nodes: the follow-up questions that split the data into smaller groups, "Is humidity high?", "Is the wind strong?"
• Leaf nodes: the ends of the branches, where the final prediction sits, "Play = Yes", or "Spam".

How the tree decides what to ask

The clever bit is choosing the best question at each step. The tree wants each split to separate the classes as cleanly as possible. It measures "cleanliness" using one of these:

• Gini impurity measures how mixed a group is. If a group of 10 fruits is all apples, Gini is 0, perfectly pure. If it's 5 apples and 5 oranges, Gini is high, totally mixed. The tree prefers the split that gives the lowest Gini.
• Entropy is basically the same idea from a different angle: it measures disorder or uncertainty. All-one-class means entropy 0. A 50/50 mix means high entropy. Lower is more organised.
• Information gain measures how much uncertainty a question removes. If asking "contains the word Free?" neatly separates spam from not-spam, that question has high information gain. The tree picks the feature with the highest gain.

So the algorithm loops: pick the best feature (lowest Gini / highest information gain), split the data, then repeat on each new group.

When does it stop?

It can't grow forever, or it would overfit. It stops when it hits a set maximum depth, when a group has too few data points left to bother splitting, or when a node becomes pure (everything in it is already one class). Here's a fuller tennis tree to show the shape:

            Outlook
          /    |     \
     Sunny  Overcast  Rainy
       |       |        |
   Humidity   Yes      Wind
    /    \             /    \
 High  Normal      Strong  Weak
   |      |           |      |
  No     Yes         No     Yes

Why people like trees: they make no assumption that the relationship is a straight line, so they cope with messy, non-linear data. They don't need normally distributed data, they shrug off outliers, they handle both classification and regression, and best of all, you can explain exactly why they decided something. Underneath, a decision tree is just the best sequence of yes/no questions, chosen so each one cuts the uncertainty as much as possible.

Algorithm 4: Naive Bayes

Naive Bayes is a fast, probability-based classifier built on a piece of maths called Bayes' Theorem. It's a workhorse for text: spam detection, sentiment analysis, document sorting, medical diagnosis.

Bayes' Theorem in plain words

Bayes' Theorem updates a probability when you get new evidence. The formula looks like this:

P(A|B) = ( P(B|A) × P(A) ) / P(B)

Read P(A|B) as "the probability of A, given that B happened". So P(A|B) is what we want to know, P(B|A) is the reverse, P(A) is how likely A was to begin with (the prior), and P(B) is how likely the evidence is overall.

The classic example shows why it matters. Suppose only 1% of people have a disease, and a test is 95% accurate. Someone tests positive, do they have the disease? Your gut says "95%, obviously". Bayes says no: because the disease is so rare to start with, a positive test is often a false alarm, and the real probability is much lower. Bayes' Theorem is what stops you jumping from "positive test" to "has disease". It blends the evidence with how common the thing actually is.

How Naive Bayes classifies

For, say, spam, it runs four steps:

1. Prior probability: how likely each class is before looking at the email. Maybe spam = 20%, not spam = 80%.
2. Likelihood: how often each feature shows up in each class. "Free" appears a lot in spam; "Meeting" appears a lot in normal mail.
3. Apply Bayes' Theorem: combine the prior and the likelihoods to get a probability for each class.
4. Pick the winner: whichever class has the highest probability is the prediction.

So an email containing "Free", "Winner", "Money" might come out as P(Spam) = 0.96 versus P(Not Spam) = 0.04. Verdict: spam.

Why "naive"?

It's called naive because it assumes every feature is independent of the others, that the words "Free", "Winner" and "Money" have nothing to do with each other. In reality they cluster together in spam all the time, so the assumption is basically wrong. The surprising part, and the reason it's still everywhere, is that it works brilliantly anyway. A slightly wrong but very fast assumption still classifies text remarkably well.

Three flavours of Naive Bayes

Its assumptions are simply: features are independent, you've picked the right flavour for your data, and you have enough training data to estimate the probabilities. It's fast, probability-driven and picks the most likely class, which is exactly what text wants.

Algorithm 5: Support Vector Machines (SVM)

A support vector machine separates classes by drawing the best possible boundary between them. It's mostly used for classification, and it shines on complex, high-dimensional data, meaning lots of features, like text with thousands of words.

Margin and support vectors

The key idea is the margin: the gap between the boundary and the nearest data points on each side. SVM doesn't just want a line between the classes, it wants the one with the widest gap, because a wider gap usually means better predictions on new data.

The points sitting closest to the boundary are the support vectors, and they're special: they alone decide where the boundary goes. Points far away don't matter. That's also why SVM is fairly solid to outliers, it only really cares about the tricky points near the edge. The boundary itself is a hyperplane (that word again): a line in 2D, a plane in 3D, a hyperplane in higher dimensions. Its equation is written w · x + b = 0, where w is a set of weights, x is the features, and b is a bias (an offset). You don't need to solve it, just know it's the dividing surface.

Linear SVM

When the classes can be split by a straight line, that's a linear SVM. It just finds the straight boundary with the biggest margin. For spam, the features might be how often "Free" and "Money" appear, and it draws the cleanest straight divide between spam and not-spam.

Non-linear SVM and the kernel trick

But loads of data can't be split by a straight line. That's where SVM does something clever: the kernel trick. It maps the data into a higher-dimensional space where a straight boundary does work, then maps back, and in the original space that straight boundary appears as a curve.

The marbles analogy makes it click. Imagine red and blue marbles jumbled on a table, so mixed that no straight line on the table separates them. Now lift some marbles up off the table (that's adding a dimension). Suddenly you can slide a flat sheet of paper between the red and blue ones. Bring it all back down to the flat table and that flat cut looks like a curve. That lift-and-separate move is the kernel trick.

Say you're separating cats from dogs using ear shape, fur texture, nose shape and tail length. The relationship between those is messy and not remotely a straight line, so an RBF SVM learns a curved boundary that wraps around the cats and separates them from the dogs.

Under the bonnet, SVM's goal is written as "minimise ½ ||w||²" while still classifying the training points correctly. In plain terms that maths is just the formal way of saying "make the margin as wide as possible". Its assumptions are refreshingly few: no requirement for a normal distribution, it's happy with lots of features, and it's solid to outliers because only the support vectors count. So: SVM finds the widest-margin boundary, and when a straight one won't do, the kernel trick bends it.

All five, side by side

That's the set. Here's the one table I'd keep if I could only keep one thing from this post:

The pattern I'll remember

That was a long one, and honestly writing it is what made it stick. The pattern I'll remember: every one of these is just a different shape of boundary or line drawn through labelled data. Linear regression draws a straight line to predict a number. Logistic bends it into a probability to classify. Trees chop the space into boxes with questions. Naive Bayes counts and multiplies probabilities. SVM finds the widest gap and bends it with kernels when it has to. I'm still hazy on the exact maths behind the kernel trick, I get the marbles picture but not the full linear algebra yet, so that's on the list. Where I actually want to end up is the security side: every one of these can be fooled with carefully crafted input (adversarial examples), and that's the post I'm building towards. Next in this track, though, I think it's time to properly meet neural networks.

If this cleared up the ML zoo for you, come say hi on LinkedIn or the contact page, and tell me which algorithm you want to see turned into a hands-on demo. More in the AI / LLMs track.

Further reading

• Google: Machine Learning Crash Course
• scikit-learn: supervised learning algorithms
• scikit-learn: support vector machines

FAQ

What is the difference between classification and regression?

Both are supervised learning. Classification predicts a category or label, like spam or not spam. Regression predicts a continuous number, like a house price or tomorrow's temperature. If the answer is a bucket, it is classification; if the answer is a number on a scale, it is regression.

What is overfitting in machine learning?

Overfitting is when a model memorises its training data, including the noise, instead of learning the general pattern. It scores brilliantly on data it has seen but poorly on new data. The opposite, underfitting, is a model too simple to learn the pattern at all.

Which supervised learning algorithm should I use?

It depends on the job. Use linear regression for predicting numbers, logistic regression or Naive Bayes for simple classification like spam, decision trees when you need an explainable model, and support vector machines for complex, high-dimensional data. There is no single best; you try a few and compare.

Related reading

• AI, ML & Deep Learning explained (Topic 1 of the AI track)
• Exploited This Week (an AI tool, Langflow, getting attacked)
• Injection Attacks Explained (the web-security cousin)
• Browse the AI / LLMs track