Neural Networks and Backpropagation: It Is Just Blame Assignment

For ages I nodded along whenever someone said "neural network". I knew it was layers of neurons, I knew it was loosely inspired by brains, and I knew it worked. What I couldn't have told you is what actually happens inside when one learns something. That gap bugged me, so I sat down and opened the box.

Turns out it's much less mystical than the branding suggests. A neuron is a bit of arithmetic. Learning is a loop that measures how wrong the network was and nudges some numbers. That's genuinely it, and once the loop clicked, everything downstream (the models we'll get to in Topic 5) stopped feeling like magic and started feeling like engineering. So here's the whole thing from the bottom up, with code you can actually run at the end.

A neuron is smaller than you think

A neuron is a small function with a number attached to each of its inputs. Those numbers are the weights, and all they say is how much each input matters.

What it does with them is three steps, and none of them are clever on their own.

1. Multiply every input by its weight.
2. Add all of those together, plus one extra number called the bias that shifts the result up or down.
3. Push that total through an activation function, which decides what actually comes out.

That's the entire neuron. If it helps, think of a decision where you weigh up factors: the weather matters a lot to you, the price matters a bit, the day of the week barely registers. You total up the weighted factors and decide. A neuron does that with numbers, and the weights are what get learned.

Which means a network isn't storing facts anywhere. There's no lookup table. Everything it "knows" lives in the weights, and the weights are just a big pile of numbers that started random and got adjusted until the output stopped being wrong. That reframing is what made the rest of this make sense to me.

The network doesn't learn answers. It learns weights, and the answers fall out of them.

Layers, and what "deep" actually means

One neuron can't do much. Stack them in columns and wire each column to the next and you've got a network. Those columns are the layers, and there are three kinds.

The hidden layers are where the interesting thing happens, and it's the bit that separates deep learning from the classic algorithms I covered in Topic 2. Each layer learns to spot patterns in whatever the previous layer produced. In an image model, early layers pick up edges and blobs. Middle layers combine those into shapes and textures. Later ones assemble those into "eye", "wheel", "face".

Nobody programmed that hierarchy. It's not in the code anywhere. The layers arrange themselves that way because it's an efficient way to reduce the error, which I still find slightly unreasonable every time I think about it.

Why this matters practically: with older machine learning you had to do feature engineering, which means a human works out which properties of the data are worth measuring and writes code to extract them. That's slow, it needs domain expertise, and it caps your model at whatever the human thought of. Deep networks skip that step. You hand over fairly raw data and the hidden layers work out the useful features themselves.

Depth, in other words, buys you automatic feature discovery. That's why deep learning took over problems like vision and language, where nobody could write down the rules in the first place.

Activation functions, and why a network needs them

What they are: the function at the end of each neuron that turns the weighted sum into the output. They're also the reason the network can learn anything complicated at all.

Why they're not optional: here's the bit that took me a while. If you strip the activation functions out, every layer is just multiplying and adding. Stack a hundred of those and, mathematically, you still have something equivalent to a single multiply-and-add. The whole tower collapses into one straight line. Activation functions bend the output, and that bending is what lets stacked layers represent curved, complicated relationships.

ReLU winning out is my favourite detail here. Sigmoid and tanh both squash everything into a narrow band, which makes the learning signal fade away as it travels back through many layers. ReLU doesn't squash the positive side at all, so the signal survives the trip. A simpler function beat the sophisticated ones because it got out of the way.

How a network actually learns

This is the core loop, and it's four steps that repeat thousands of times. Once you can name all four in order you understand training.

Step 2: the loss function

The loss is one number saying how far the prediction was from the truth. Big number, badly wrong. Zero, perfect. The entire goal of training is making that number small.

Which one you use depends on what you're predicting. For a number (house price, temperature) you'd use mean squared error: take the difference, square it so the sign doesn't matter and big misses hurt more, average across your examples. For picking a category (spam or not, which digit) you'd use cross-entropy loss, which scores how confident the model was in the right answer and punishes it heavily for being confidently wrong.

So the loss function is how you tell the network what "good" means. Choose the wrong one and it'll cheerfully optimise for the wrong thing, and it will look like it's working the whole time.

Step 3: backpropagation

Backpropagation is the method for working out which weights are to blame for the error, and by how much.

You know the total error at the output. Backprop walks that error backwards through the network, layer by layer, splitting the blame at each step according to how strongly each connection contributed. A weight that pushed hard in the wrong direction gets assigned a lot of responsibility. One that barely mattered gets almost none. Mathematically it's the chain rule from calculus applied over and over, but the intuition is just blame assignment.

The thing worth being precise about, because plenty of explanations blur it: backprop calculates the responsibility. It doesn't change anything. It hands those numbers (the gradients) to the optimiser, and the optimiser does the changing. Two jobs, two steps.

Step 4: the optimiser

Gradient descent is the rule that turns "here's who's to blame" into "here's the new weight".

The mental picture that works for me is a hill in fog. You want the bottom of the valley (lowest loss) and you can't see it, but you can feel which way the ground slopes under your feet. That slope is the gradient. So you take a step downhill, feel again, step again. That's gradient descent, and stochastic gradient descent (SGD) is the same thing but feeling the slope from a small random batch of examples each time instead of all of them, which is far faster.

Adam and RMSprop are the smarter versions people actually reach for. Rather than one fixed step size for everything, they keep track of how each weight has been behaving and adapt the step per weight, taking bigger strides in directions that have been consistently downhill. Adam is the sensible default when you have no reason to pick anything else.

Hyperparameters: the settings you pick yourself

Weights are learned. Hyperparameters are the knobs you set before training starts, and they control how the learning goes. The main ones:

• Learning rate, how big each optimiser step is. Too small and training takes forever. Too big and you keep overshooting the valley floor and the loss bounces around instead of settling. This is the one to tune first.
• Number of hidden layers, how deep the network goes.
• Neurons per layer, how wide each one is.

More layers and more neurons mean more capacity to learn complicated things, and also more chance of overfitting, where the model memorises your training data instead of learning the general pattern and then falls apart on anything new. Bigger is not automatically better, which is a lesson that seems to need relearning constantly.

Hands-on: train a network in thirty lines

Reading about backprop only got me so far. Writing it out by hand is what made it stick, so here's the smallest example that actually demonstrates the point.

The problem is XOR: given two inputs that are each 0 or 1, output 1 if exactly one of them is 1, otherwise output 0. It looks trivial. It's famous because a network with no hidden layer physically cannot solve it, which makes it the perfect way to see what hidden layers buy you.

Step 1, prove the problem is real. Before building the proper network, try it without a hidden layer. Inputs wired straight to one output neuron, trained for 20,000 rounds.

Step 2, build the real thing. Same problem, but now with a hidden layer of four neurons in the middle. Save this as xor.py. Every line of the training loop maps to one of the four steps above, and I've labelled them so you can see it.

import numpy as np
np.random.seed(42)

X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([[0],[1],[1],[0]], dtype=float)

def sigmoid(z):  return 1/(1+np.exp(-z))
def dsigmoid(a): return a*(1-a)          # slope of sigmoid

# 2 inputs -> 4 hidden neurons -> 1 output. Weights start random.
W1 = np.random.randn(2,4); b1 = np.zeros((1,4))
W2 = np.random.randn(4,1); b2 = np.zeros((1,1))
lr = 0.5                                  # learning rate

for epoch in range(1, 20001):
    # 1. forward pass
    h   = sigmoid(X @ W1 + b1)
    out = sigmoid(h @ W2 + b2)

    # 2. loss (mean squared error)
    loss = np.mean((out - y)**2)

    # 3. backpropagation: split the blame, layer by layer
    d_out = (out - y) * dsigmoid(out)
    d_h   = (d_out @ W2.T) * dsigmoid(h)

    # 4. optimiser step: nudge each weight downhill
    W2 -= lr * h.T @ d_out;  b2 -= lr * d_out.sum(0, keepdims=True)
    W1 -= lr * X.T @ d_h;    b1 -= lr * d_h.sum(0, keepdims=True)

    if epoch in (1, 1000, 5000, 20000):
        print(f"epoch {epoch:>5}  loss {loss:.4f}")

print("\npredictions:")
for xi, pi in zip(X, sigmoid(sigmoid(X @ W1 + b1) @ W2 + b2)):
    print(f"  {int(xi[0])} XOR {int(xi[1])} = {pi[0]:.3f}  -> {round(pi[0])}")

Step 3, run it and watch the loss fall. This is the verification step: if the loss drops and the predictions land near 0 and 1, learning happened.

Nothing in that code knows what XOR is. There's no rule for it anywhere. The weights started as random numbers, and twenty thousand rounds of "measure the error, work out who's to blame, nudge accordingly" turned them into something that computes it correctly. Every large model you've heard of is that loop, with more layers, more data and a great deal more electricity.

What I actually took from this

The honest summary is that there's no single clever idea in here. There's a very simple unit repeated a lot, a way to measure being wrong, a way to apportion blame, and a rule for adjusting. The results are extraordinary and the machinery is mundane, and I think holding both of those in your head at once is the right way to understand this stuff.

What I'm still shaky on is the intuition for architecture choices. I can follow why depth helps in principle, but knowing that a particular problem wants six layers rather than three, or that this is where you'd put a convolution, still looks like experience I haven't earned yet. I suspect a lot of it genuinely is trial and error plus knowing what worked for similar problems, though I'd be glad to be told there's more structure to it than that.

Next in the track I want to get into what these networks build once you scale them up: generative models, the transformers behind LLMs, and how diffusion models turn noise into pictures. Same principles, considerably more interesting output.

Further reading

If you want to go deeper, 3Blue1Brown's neural network video series is the clearest visual explanation of backpropagation I've found, and Michael Nielsen's free book Neural Networks and Deep Learning builds the whole thing up from scratch with far more rigour than one post can.

FAQ

What is a neuron in a neural network?

A neuron is a tiny function, not a brain cell. It multiplies each input by a weight, adds them up with a bias, then passes the total through an activation function. That is the whole thing. The intelligence comes from stacking thousands of them and tuning the weights.

What is backpropagation?

Backpropagation is how a network works out which weights caused its error. After a prediction, it compares the answer to the truth, then walks the error backwards through the layers using the chain rule, calculating how much each weight contributed. The optimiser then nudges those weights.

Why do neural networks need hidden layers?

Without a hidden layer a network can only draw a straight dividing line, so problems that are not linearly separable are impossible. XOR is the classic example: a flat network gets stuck outputting 0.5 forever. Add one hidden layer and it solves it easily.

What is the difference between a loss function and an optimiser?

The loss function measures how wrong the prediction was, giving one number to minimise. The optimiser is the strategy for changing the weights to reduce that number. Loss tells you how bad things are, backprop tells you who is responsible, and the optimiser decides what to do about it.

What is a learning rate?

The learning rate controls how big a step the optimiser takes each update. Too small and training crawls for hours. Too large and the weights overshoot the target and the loss bounces around or blows up. It is usually the first hyperparameter worth tuning.

Related reading

• AI, ML & Deep Learning explained (Topic 1, where the three fit together)
• Supervised Learning Algorithms (Topic 2, the classic algorithms this replaces)
• Topic 3: Reinforcement Learning, Q-Learning & SARSA (learning from rewards)
• Browse the whole AI / LLMs track