A network with thirteen thousand weights has thirteen thousand dials, and no human is going to turn them. Something has to do the turning automatically. The method that does it is gradient descent, and it is far simpler than its reputation.
The method fits in one sentence. Define a single number that says how wrong the model currently is, work out which way to move each weight to make that number smaller, take a small step that way, repeat. Everything else is engineering around the edges.
The first requirement is that number. You cannot search for “better” until “better” is a quantity. So you define a cost: a function that takes the model’s predictions and the true answers and returns one non-negative number, zero when everything is perfect. For digit classification, compare the ten output scores against the ideal answer — a 1 in the right slot, 0 elsewhere — and add up the squared differences. Average over every training example. One network, one number.
The landscape you cannot see
Now do the mental trick that makes the rest obvious. Ignore the training data for a moment and think of the cost as a function of the weights alone. Every possible setting of the 13,002 weights is a point. Above each point sits its cost. That is a surface, and training is walking downhill on it.
Nobody can picture a 13,002-dimensional surface, which is fine, because the two-dimensional cartoon carries almost all the intuition. Picture a hilly landscape. You are standing somewhere on it in thick fog. You cannot see the valleys. You can only feel the slope under your feet. So you feel which way is steepest downhill, step that way, and feel again.
The gradient is exactly the “feel the slope” operation, in every direction at once. It is the list of partial derivatives: for each weight, how much does the cost change if this one weight goes up a hair, holding everything else still? Thirteen thousand weights, thirteen thousand numbers in that list.
Two facts about it are worth holding onto.
The gradient points uphill. It is the direction of steepest increase. So you subtract it rather than add it. This is why the update rule has a minus sign in it and why the method is called gradient descent — you compute the uphill direction and walk the other way.
Its size is information too. A large partial derivative means this weight matters a lot here; nudge it and the cost moves. A near-zero one means the weight is currently irrelevant. So the gradient does not just say which way to go, it ranks the weights by importance. Weights with big derivatives get big adjustments. That ranking is most of why the method works at all in high dimensions.
Written out, one step is:
def step(weights, grad, lr=0.01): return [w - lr * g for w, g in zip(weights, grad)]Run that a few hundred thousand times with a correct gradient and you have trained a neural network. There is no other secret.
The learning rate is the one knob that will hurt you
lr above is the step size. It is the hyperparameter people underestimate, and
it fails in two opposite directions.
Too large and you overshoot. You are at the side of a valley, the slope says “downhill, that way”, you take a huge stride and land on the opposite slope higher than you started. Do that repeatedly and the cost climbs instead of falling, often to infinity within a few dozen steps. A loss that goes to NaN in the first minute of training is nearly always this.
Too small and you converge, but slowly enough that you run out of patience or budget before it matters. Worse, a tiny learning rate tends to settle into whatever mediocre dip is nearest to where you started.
All three behaviours start from the same place on the same bowl:
The hollow dot is the shared starting weight. Same curve, same start, three step sizes.
The standard fix is to start moderately large and decay the rate over training: big strides early to cover ground, small careful steps later to settle. Adam and its relatives automate a version of this per weight. They do not remove the need to pick a sensible starting value; they just make the range of workable values wider.
Batch, stochastic, and the compromise everyone actually uses
To compute the true gradient you must evaluate the cost over the entire training set. On a million examples that is one step per full pass through the data — accurate and unusably slow.
The opposite extreme uses one random example per step. Fast, and each gradient is a noisy guess at the real one. The path wanders drunkenly downhill.
In practice everybody uses the middle: mini-batches of something like 32 to 512 examples. Each gradient is a decent estimate, the hardware stays busy, and you get thousands of steps per epoch instead of one. The noise from sampling turns out to be mildly useful — it shakes the parameters out of narrow dips that a perfectly accurate gradient would have sat in. This is what “stochastic gradient descent” means: not a different algorithm, just gradient descent with an estimated gradient.
Local minima are not your problem
The usual worry is that the walk gets stuck in a small dip that is not the true bottom. In two dimensions this seems damning. In thirteen thousand dimensions it mostly evaporates.
For a point to be a local minimum, the surface must curve upwards in every one of those dimensions at once. That is a demanding coincidence. Far more common is the saddle: up in some directions, down in others. Saddles slow training down, they don’t trap it. And in large networks the many minima that do exist tend to sit at similar costs, so which one you land in matters less than the folklore suggests.
The real problems in practice are different: a learning rate that diverges, a gradient that vanishes to nothing through many layers, or a model that descends beautifully on training data and fails on new data. That last one is not a gradient descent failure at all — it is an evaluation failure, and it is why cross-validation done honestly matters more than any optimiser choice.
The ball rolling down the cost surface is animated in Gradient descent, how neural networks learn by 3Blue1Brown.
There is one piece still missing. Feeling the slope in thirteen thousand directions sounds like it should cost thirteen thousand separate measurements per step. It does not, and the algorithm that makes it cheap is backpropagation. Without it, gradient descent on a real network would be a lovely idea nobody could afford to run.