Gradient descent needs a gradient: for every weight in the network, how much does the cost change if that one weight moves a hair? The obvious way to get it is to try. Nudge weight number one, run the whole network, see how the cost moved. Put it back, nudge weight number two, run the whole network again.
For a network with thirteen thousand weights, that is thirteen thousand full forward passes for a single training step. For a modern model with billions of parameters it is not slow, it is impossible. Backpropagation gets the same answer — the exact answer, not an approximation — for roughly the cost of two forward passes, regardless of how many weights there are.
Speed is why it exists. Say that up front, because backprop has acquired a reputation for depth it doesn’t deserve. It is not a learning algorithm. It does not decide anything. It is an efficient method for computing derivatives in a nested function, and it was known in other fields as “reverse-mode automatic differentiation” before anyone applied it to networks. The paper that made it standard equipment for neural networks is Rumelhart, Hinton and Williams’ Learning Representations by Back-Propagating Errors, four pages in Nature in 1986, and it is refreshingly matter-of-fact about what the method is.
Start with blame, not calculus
Forget derivatives for a minute. The network has just seen an image of a 2 and output a confident 8. Something is at fault. Which weights?
Work backwards from the output. You wanted the “2” score up and the “8” score down. The output layer’s scores come from the previous layer’s activations, each multiplied by a weight. So there are three ways to get what you want:
- Change the weights feeding the output units.
- Change the biases on those units.
- Change the activations coming in from the layer below.
The first two you can do directly — they are yours to set. The third you cannot touch, because those activations are produced by the layer below. But you can write down what you wish they were. “This unit should have been more active, that one less.” That wish list, one number per unit in the previous layer, is the error signal for that layer.
Now the layer below is in exactly the same position the output layer was in a moment ago: it has a target, it has incoming weights it can change, and it has inputs it can only make wishes about. So you repeat. And again, layer by layer, until you reach the front.
That is backpropagation. The name is a plain description: the error signal propagates backwards through the network, and each layer converts “here is how wrong my outputs were” into “here is how my weights should change” plus “here is how wrong my inputs were”, which it hands to the layer behind it.
Which weight gets blamed depends on how loud it was
Inside a layer, the blame is not shared evenly, and the rule for splitting it is the sensible one.
A weight’s contribution to the output is its value times the incoming activation. So if an input unit was strongly active, the weight attached to it had a large effect, and adjusting that weight is an efficient way to fix the error. If the input unit was near zero, its weight barely mattered this time around, so barely change it. The adjustment to any weight is proportional to the activation that came in through it, multiplied by the error at the unit it feeds.
This is the whole intuition behind the phrase “neurons that fire together wire together”. It falls out of the arithmetic; no biology is required.
There is a third factor. Between the weighted sum and the activation sits the nonlinearity, and it has its own slope. If a unit is deep in the flat region of a sigmoid — saturated near 0 or 1 — then its output barely responds to changes in its input, so the blame passing through it is multiplied by something close to zero. That unit is effectively frozen. Stack several saturated layers and the signal reaching the early layers is multiplied down to nothing. This is the vanishing gradient problem, and it is not a mystery once you see the mechanism: it is a product of many small numbers. ReLU became standard largely because its slope is exactly 1 wherever it is active, so it does not shrink the signal on the way past.
The chain rule is the same story with symbols
The formal statement is the chain rule. If the cost depends on an activation, and that activation depends on a weighted sum, and that sum depends on a weight, then the derivative of the cost with respect to that weight is the product of the three local derivatives along the path.
Three ordinary facts, multiplied:
| link in the chain | what it says | value |
|---|---|---|
| weight → sum | raise this weight, the sum rises by | the incoming activation |
| sum → activation | raise the sum, the output rises by | the slope of the activation function |
| activation → cost | raise this activation, the cost changes by | the error signal from above |
Multiply the three and you have the partial derivative for that weight. When a unit feeds several units in the next layer, you add up the contributions from every path — that is the only complication, and it is a sum, not a subtlety.
Follow one path and the whole method fits in a picture:
One weight, one path. The error arrives from the layer above, is multiplied by the slope of the unit it lands on, and then by the activation that came in through the weight. Every weight in the network is that same product along its own path, which is why one backward sweep is enough for all of them.
The efficiency comes from ordering the work. The error signal at a layer is computed once and reused for every weight in that layer, and passed back once for the layer below. Nothing is recomputed. Thirteen thousand derivatives, one backward sweep.
What this means when you are debugging
The mechanism explains most training failures, and knowing it turns mysteries into checks.
- Loss stuck, nothing moving. Something is zeroing the signal. Saturated activations, dead ReLUs (units whose sum is negative for every input, so their slope is always zero), or a learning rate so small the steps don’t register.
- Loss goes to NaN. The signal blew up instead of vanishing. Too large a learning rate, or exploding gradients in a deep or recurrent stack. Gradient clipping exists for exactly this.
- Early layers learn nothing while late layers learn fine. A classic vanishing-gradient signature. Normalisation layers and residual connections are both, at bottom, ways of giving the error signal a shorter path back.
3Blue1Brown’s Backpropagation, intuitively shows the wish lists adding up layer by layer, which is the part the algebra hides.
In day-to-day work nobody writes backprop. PyTorch and JAX build the graph and differentiate it for you, and hand-rolling it is a teaching exercise, not a job. But the failure modes above are still yours to diagnose, and they are all consequences of a signal being multiplied on its way backwards. Knowing that one fact is the difference between reading a flat loss curve and guessing at it — the same difference that separates a real baseline from a model you merely hope is working.