Supervised learning needs a teacher who already knows the answer. Every example arrives with a label. That covers a great deal of useful work, and it rules out an entire category of problems: the ones where nobody knows the right answer to any individual decision.
Consider a game of Go. Nobody can tell you the correct move on turn 12. There is no label. All you eventually learn is that, three hours later, you won or lost. Same with a warehouse robot, a trading policy, a thermostat running a building. You get consequences, not corrections.
Reinforcement learning works from consequences. The distinction is between instruction — here is what you should have said — and evaluation — that went well, or it did not. Evaluation is a much weaker signal, and most of the field’s difficulty follows from that one fact.
The five pieces
The vocabulary is small and worth getting exactly right, because these words are used loosely everywhere else.
- Agent — the thing making decisions.
- Environment — everything else, including the rules.
- State — what the agent can observe right now.
- Action — what it chooses to do.
- Reward — a single number the environment returns after each action.
They run in a loop: observe state, take action, receive reward, land in a new state, repeat. The agent’s behaviour is called a policy — a rule mapping states to actions, which in practice is a neural network.
The goal is not to maximise the next reward. It is to maximise the sum of rewards over the whole episode, the return. That gap between immediate reward and long-run return is where all the interest lives. Sacrificing a piece to win the game, or spending money now to earn later, is not available to a system that optimises the next number.
The loop, with the awkward part of Go drawn underneath it:
An action goes out; a state and a reward come back. The difficulty is on the second row. A game of Go pays nothing for 250 moves and then pays once, so the reward says a great deal about the game and nothing about any move in it.
Credit assignment is the hard part
Here is the problem in its purest form. You win a game of Go on move 250. Which of your 250 moves deserves the credit?
Plausibly not the last one, which was probably forced. Possibly a quiet move on turn 40 that made the whole right side workable. The reward is a single number attached to the end of a long chain of decisions, and it says nothing about which links mattered. This is the credit assignment problem, and it is why reinforcement learning is harder than it sounds.
The main idea for attacking it is to stop waiting for the end. The agent learns a value function: an estimate, for any state, of the return it can expect from there onward. That gives feedback on every step. Take an action, land in a new state, and compare your old estimate of the current state against the reward you just got plus your estimate of the new state. If the new view is rosier, that action was better than you thought. Nudge the estimate.
That comparison is the temporal-difference error. The agent is updating one guess toward another guess, which sounds circular; it works because the second guess is anchored by a real reward the first one did not have. Reality leaks backwards through the chain, one step at a time.
Exploration against exploitation
The second unavoidable tension. The agent only learns about actions it actually takes, so it must sometimes take actions it believes are worse, to find out whether that belief is wrong — the restaurant you like, against the one you have never tried. The usual answer is to act greedily most of the time and choose at random a small fraction of the time, shrinking that fraction as confidence grows. Crude, and often enough.
The deeper point: this trade-off does not exist in supervised learning, where the dataset is fixed. Here the agent generates its own training data, so its current mistakes shape what it will ever get to learn.
What AlphaGo actually demonstrated
Go resisted computers long after chess fell: the search space is too large, and nobody could write a decent formula for who is winning.
The system that beat Lee Sedol in 2016 — set out by Silver and colleagues in Mastering the Game of Go with Deep Neural Networks and Tree Search, published in Nature a couple of months before the match — combined three things. A policy network, first trained on human games, to propose a handful of sensible moves instead of everything. A value network to estimate who is ahead — the missing evaluation function, learned rather than written. And tree search guided by both.
The reinforcement learning part is how those networks got good: self-play. The system played copies of itself, millions of times, and learned from the outcomes. Two properties make that powerful. Data becomes unlimited, since games are generated rather than collected. And the opponent is always exactly as strong as the agent, which produces an automatic curriculum with nobody designing the lesson plan.
The follow-up removed the human games entirely, starting from random play and the rules alone, and reached a higher standard than the version that had studied human play. The human games were a useful shortcut, and also a ceiling.
Then there is move 37 of game two. It violated conventional Go teaching, commentators assumed it was an error, and it turned out to decide the game. It is the cleanest demonstration available that a system optimising an objective from scratch can find good ideas that centuries of expert culture had ruled out. The full story is told well in AlphaGo - The Movie by Google DeepMind, which is as much about the people as the algorithm.
Why this will probably not be your next project
Now the honest part, because this field generates enthusiasm out of proportion to how often it belongs in a company’s stack.
It needs a simulator. Those results came from tens of millions of games, played faster than real time, at no cost, with no consequences. Board games and physics simulations offer that. Your pricing problem and your customers do not.
Exploration on real systems has a price. In a simulator a bad action costs nothing. In production it costs a customer, a shipment, or a fine. Try things, be wrong often, be wrong cheaply — that is what makes learning work, and what a live system will not tolerate.
The reward is a specification, and specifications get gamed. The agent optimises what you wrote, not what you meant. Reward engagement and you get outrage. Reward tickets closed and you get tickets closed without being solved. Games hand you an unambiguous reward for free; business problems make you invent one, and the invention is where the failure enters.
What does work, and is underused, is the narrow version: contextual bandits, where each decision is one-shot with immediate feedback and no chain to assign credit along. Which of five layouts to show this visitor. That is a practical, solved class of problem, and it sits next to the discipline described in A/B test pitfalls rather than replacing it.
The framing transfers regardless. Asking what the state really is, what the actions really are, and what the reward really rewards will improve a decision system even if you never train an agent. Most bad automation fails not on its algorithm, but because someone wrote down the wrong reward and the system obeyed.