Reinforcement Learning: One Word Splits Q-Learning from SARSA

The thing that first hooked me on reinforcement learning wasn't the maths. It was watching a program get good at a game nobody taught it to play. No rulebook, no worked examples, just a score going up. It flailed, it failed, and somehow after enough tries it started winning. That's a different kind of learning from anything in the earlier posts, and I wanted to actually understand the machinery under it.

This is part three of my AI track. In part one I covered what machine learning even is, and in part two the supervised algorithms that learn from labelled examples. Reinforcement learning (RL for short) throws the answer key away. Here the machine learns from consequences. I'll build up the core idea, then walk through the two classic algorithms, Q-learning and SARSA, and the one distinction between them that trips everyone up. No lab this time, just the concepts, told as plainly as I can.

What reinforcement learning actually is

Reinforcement learning is teaching a machine by reward instead of by example. You don't tell it the right move. You let it act, then you score the outcome. Good outcome, small reward. Bad outcome, a penalty. Do that thousands of times and the machine works out a strategy on its own.

Compare that with supervised learning, which is a student with an answer key: every training example comes tagged with the correct label. RL is a student with only a score at the end. Nobody says "this move was right". The machine has to figure out which of its many moves earned the reward, sometimes long after it made them. That delay is what makes RL hard and interesting.

That fits a lot of real problems, because loads of them have no answer key. Playing a game, steering a robot, routing traffic, deciding when to buy or sell. You can't label the "correct" action for every situation up front, but you can nearly always say whether things went well afterwards.

The loop: agent, environment, reward

Everything in RL is one loop repeated forever. There's an agent (the learner, the thing making decisions) and an environment (everything else, the world it acts in). They pass messages back and forth. Round the loop it goes:

Five words carry the whole subject, so let me pin each one down in plain terms:

• State - a snapshot of the situation right now. For a robot in a maze, its position and the walls around it. For a chess bot, the board.
• Action - a move the agent can make. Turn left, brake, move a piece. Actions change the state.
• Reward - a single number the environment hands back after an action. Positive is "good, do more of that", negative is "no". The agent's entire goal is to pile up as much reward as possible over time, not just on the next step.
• Policy - the agent's strategy: given this state, which action? A policy can be fixed (always the same action in a state) or a bit random (pick action A 70% of the time, B 30%). Learning a good policy is the whole job.
• Value function - the agent's estimate of how good a state or action is in the long run, not just the immediate reward but everything it expects to earn from here on. This is the difference between grabbing a quick point and setting up a much bigger win later.

Reward is the instant feedback; value is the long-game estimate. Most of RL is the art of learning good value estimates so the policy has something worth acting on.

Three more knobs worth knowing

A few smaller ideas come up constantly, so here they are before we hit the algorithms:

• Discount factor (γ, gamma) - a dial between 0 and 1 that sets how much the agent cares about the future. At 0 it's totally short-sighted, only the next reward counts. Near 1 it treats a reward ten steps away almost like one right now. Most real setups sit high, around 0.9 to 0.99, because the future usually matters.
• Episodic vs continuous - some tasks have a clear end (a maze run finishes when you reach the goal), those are episodic. Others just run forever (balancing a pole, keeping a server healthy), those are continuous. It changes how you think about "total reward".
• Model-based vs model-free - model-based means the agent has, or builds, a map of how the world works and can plan ahead on it. Model-free means no map: it learns purely from experience, one bump at a time. Both algorithms below are model-free, which is why they're so widely used, you rarely have a clean map of the real world.

Q-learning: learning a cheat sheet of what pays off

Q-learning is a model-free algorithm that learns a big lookup table of action values. Each entry is a Q-value: for a given state and a given action, the total reward you can expect if you take that action now and then play well afterwards. Learn good Q-values and the policy becomes trivial. In any state, pick the action with the highest Q.

The Q-table: picture a spreadsheet. Rows are states, columns are actions, each cell is that state-action pair's Q-value. Here's a tiny grid-world table for a robot that can move up, down, left or right from four states:

The table starts as junk (often all zeros) and gets corrected every step. The correction is the famous Q-learning update rule, which comes from the Bellman equation. It looks scary and isn't:

Q(s, a) = Q(s, a) + α · [ r + γ · max Q(s', a') − Q(s, a) ]

In English: nudge the old value a little towards a better estimate. Piece by piece:

• Q(s, a) - the value we're updating (action a in state s).
• α (alpha), the learning rate - how big a nudge. Small = slow but steady, big = fast but jumpy.
• r - the reward we just got.
• γ · max Q(s', a') - the discounted value of the best action available in the next state. This is the "and then play well afterwards" bit, and the word best is the important one, remember it for SARSA.
• The whole bracket is the gap between what we thought and what we now think. We move a fraction (α) of the way to close it.

One update, worked through

Numbers make it click. Say the robot is in S1, takes Right, lands in S2, and picks up a reward of 0.5. Use α = 0.1 and γ = 0.9. The best action value in S2 is Down at 1.0. Plug it in:

Do that over and over, for every state and action, and the table converges: the numbers stop moving because they've become accurate. At that point the agent just greedily follows the highest values from start to goal. That's a learned policy.

The explore-or-exploit problem

Here's the catch that makes RL genuinely tricky. If the agent always picks the current best action, it can never discover a better one it hasn't tried. But if it always experiments, it never cashes in what it knows. That tension has a name: the exploration-exploitation trade-off. Exploit means "use what works", explore means "try something new".

The standard fix is dead simple and it's called epsilon-greedy. Pick a small number epsilon (ε), say 0.1. Then most of the time (90%) take the best-known action, and a small slice of the time (10%) take a random one. That random 10% is the agent poking around for something better.

There's a smoother alternative called softmax, which picks actions with probability proportional to their value, so a decent-but-not-best action still gets chosen sometimes, and a terrible one almost never. Epsilon-greedy treats all non-best actions as equally worth trying; softmax is more graded. Both are just ways to keep a bit of curiosity in the loop.

SARSA: the cautious sibling

SARSA (State-Action-Reward-State-Action, which is literally the sequence it uses) is the other classic model-free algorithm. It learns Q-values just like Q-learning and it looks almost identical on the page. The difference is one term in the update, and that one term changes its whole personality.

Here's SARSA's update rule next to the one you just saw:

Q(s, a) = Q(s, a) + α · [ r + γ · Q(s', a') − Q(s, a) ]

Spot it? Q-learning used max Q(s', a'), the value of the best next action. SARSA uses Q(s', a'), the value of the action it actually took next. That's the entire difference. Q-learning updates towards a perfect future it might not follow; SARSA updates towards the future it really lived, exploration mistakes and all.

On-policy vs off-policy, in one breath

This is the bit of jargon worth owning, because it's the real point:

• Off-policy (Q-learning): learns about the optimal policy while following a different, more exploratory one. It's studying the ideal even as it messes about. That's why it can learn from old data or someone else's playthroughs.
• On-policy (SARSA): learns about the exact policy it's following right now, exploration included. It grades its own real behaviour, not a hypothetical perfect version.

The famous illustration is a cliff-edge path. Q-learning learns the shortest route runs right along the edge, because optimally it's fine. SARSA, knowing its own random exploration will occasionally shove it off, learns to walk a step further from the drop. Neither is "correct". If mistakes are cheap, Q-learning's boldness finds the best path faster. If mistakes are expensive (a real robot, a real cost), SARSA's caution is often what you actually want. That's the honest trade, and which one I'd reach for genuinely depends on how much a fall hurts.

Getting it to actually converge

Both algorithms are iterative, and both need their two dials set sensibly or they never settle. Converge just means the Q-values stop changing much, the agent has learned and further training barely moves the numbers.

• Learning rate (α): too high and the values thrash around and never settle; too low and it learns at a crawl. You want the sweet spot, and people often shrink α over time so early lessons land hard and later ones just fine-tune.
• Discount factor (γ): too low and the agent won't plan far enough ahead to solve anything with delayed payoff; too high and distant, uncertain rewards start dominating and learning gets noisy.

There's no universal setting, which is the slightly annoying truth of it, you tune per problem, sometimes with a grid search over combinations. And there's a theoretical safety net: both are proven to converge to an optimal policy if the learning rate shrinks appropriately and the agent visits every state-action pair enough times. In a small grid world that's easy. In anything big it's more hope than guarantee, and honestly that gap between the clean theory and the messy practice is the part I'm still getting my head around.

A security aside, because this is a security blog

I mostly wander into AI from the security side, so I can't help flagging where RL and security touch. Two honest ones. First, reward hacking: an agent optimises the number you gave it, not the thing you meant, and if the reward is sloppily defined it'll find a degenerate shortcut that scores high and does something useless or harmful. That's a real alignment headache, not a hypothetical. Second, RL is increasingly the engine behind autonomous agents, and the moment an agent can take actions in the world, its policy becomes an attack surface: poison its rewards or its environment and you steer its behaviour. I'm not going to overclaim expertise here, it's an area I'm reading into rather than testing, but the pattern (a system that learns from feedback can be attacked through that feedback) is one every security person should file away.

What I like about this one

What I like about RL is how little it needs to be told. No labels, no map, just a score and a lot of patience. What still surprises me is how much rides on two small decisions, how you shape the reward, and whether you learn on-policy or off. Get the reward wrong and a perfectly good algorithm learns something dumb with total confidence. The Q-learning vs SARSA split finally sits right in my head now: same table, one word different in the update, optimist vs realist. I've only run these on toy grid worlds so far, not anything with real stakes, so take my "which is better" with a pinch of salt, it's book understanding catching up to practice. If you actually build with RL, I'd love to hear which you reach for and why. Next in the AI track I want to get into neural networks properly, and then the bit I'm really here for, how these models get fooled.

If this made RL click, come say hi on LinkedIn or the contact page, and tell me what to break down next. More in the AI / LLMs track.

Further reading

• David Silver: Reinforcement Learning course
• Sutton & Barto: Reinforcement Learning, An Introduction (free)
• Gymnasium: standard RL environments

FAQ

What is reinforcement learning?

Reinforcement learning is a type of machine learning where an agent learns by doing. It takes actions in an environment, gets a reward or penalty back, and slowly works out which actions pay off. There's no labelled answer key. The agent learns a policy, a strategy for acting, purely from the consequences of trial and error.

What is the difference between Q-learning and SARSA?

Both learn Q-values, but they update differently. Q-learning is off-policy: it updates towards the best possible next action, so it chases the optimal path. SARSA is on-policy: it updates towards the action it actually took next, including exploratory ones, so it learns a safer, more realistic policy that accounts for its own mistakes.

What is the exploration-exploitation trade-off?

It's the agent's core dilemma: repeat what already works (exploit) or try something new that might work better (explore). Too much exploitation and it gets stuck on a decent-but-not-best habit. Too much exploration and it never cashes in. Epsilon-greedy handles it by acting randomly a small fraction of the time.

What is a Q-value and a Q-table?

A Q-value is the expected long-term reward for taking a specific action in a specific state, then playing well afterwards. A Q-table stores one Q-value for every state-action pair, states as rows, actions as columns. The agent reads it to pick actions and updates it after every step as it learns.

What does the discount factor do in reinforcement learning?

The discount factor (gamma), between 0 and 1, sets how much the agent cares about future rewards versus immediate ones. Near 0, it's short-sighted and grabs quick wins. Near 1, it plans for the long game and values rewards far ahead almost as much as ones right now.

Related reading

• AI, ML & Deep Learning explained (Topic 1 of the AI track)
• Supervised Learning Algorithms (Topic 2, learning from labels)
• Browse the whole AI / LLMs track