A model that has memorised your training data is not the same as a model that has learned something. The difference is easy to state and easy to miss: one reproduces the rows it was shown, the other makes good predictions on rows it has never seen. Only the second is worth anything, because every row you will ever predict on in production is, by definition, a row the model was not trained on.
Overfitting is what happens when a model chases detail that will not repeat. Fit a wiggly enough curve through fifteen points and it will pass through every one of them exactly. Training error: zero. But the wiggles were placed to accommodate the particular noise in those fifteen points, and the next fifteen points will have different noise. The curve is confidently wrong everywhere it matters.
The opposite failure is quieter and just as real. A model that is too rigid — a straight line through data that clearly bends — misses structure that would have repeated. Its training error is bad and its test error is bad, and it is bad in the same way every time you refit it. Those two failure modes have names, and knowing which one you are looking at tells you what to do next.
What bias and variance mean concretely
Run this thought experiment. Instead of one training set, imagine you could collect fifty independent training sets of the same size from the same source. Fit the same model recipe on each one. Now you have fifty fitted models. Ask them all to predict the same new point.
- Bias is how far the average of those fifty predictions sits from the truth. It measures the error your model family cannot escape, no matter how much data you feed it. A straight line fitting a curve has high bias.
- Variance is how much those fifty predictions disagree with each other. It measures sensitivity to which particular rows you happened to draw. A deep unpruned tree has high variance: swap a handful of training rows and it grows a different shape.
There is a third piece you cannot touch. If the label itself contains randomness — two identical customers, one churns and one doesn’t — no model reaches zero error. That is irreducible noise, and mistaking it for fixable bias is how projects burn a quarter chasing an unreachable number.
The classic result is that expected error decomposes into these three parts: bias squared, plus variance, plus noise. Geman, Bienenstock and Doursat put that decomposition in front of the neural network world in Neural Networks and the Bias/Variance Dilemma in 1992, and the argument has outlived the models it was written about. The practical use of the fact is not the algebra. It is the diagnosis:
| training error | test error | reading | what to do |
|---|---|---|---|
| high | high | high bias | more capacity, better features |
| low | high | high variance | more data, more regularisation, less capacity |
| low | low | fine | stop tuning, go ship it |
Where the tradeoff shows up
It is not an abstraction. It is the dial behind most of the knobs you already turn.
Model family. Linear and logistic regression sit at the high-bias end. Deep unconstrained trees and k-nearest-neighbours with k=1 sit at the high-variance end. Choosing a family is choosing a starting point on the dial.
Regularisation. Ridge and lasso penalties, max_depth on a tree,
min_samples_leaf, early stopping in boosting, weight decay and dropout in
neural nets — every one of these deliberately adds bias to remove more variance
than it adds. The exchange is deliberate. When someone says a model is
“regularised”, they mean it has been made worse at fitting the training data on
purpose.
Ensembling. Bagging and random forests attack variance directly: average many high-variance models and their disagreements cancel out while the shared signal survives. Boosting works the other end, stacking weak high-bias learners until they collectively bend to the data — which is why boosting overfits if you let it run too long and bagging mostly does not.
More data. Adding rows shrinks variance and leaves bias alone. This is worth knowing before a meeting: if your problem is bias, buying more data will not help, and someone should say so before the budget is approved.
StatQuest with Josh Starmer covers the same trade in Machine Learning Fundamentals: Bias and Variance.
You cannot see any of this without honest validation
Everything above is measured on data the model did not see. Get that wrong and the whole diagnosis is fiction. A model that leaks — through a feature computed over the full dataset, a scaler fitted before the split, or repeated peeking at the test set while tuning — reports a low test error that means nothing.
This is why the tradeoff and cross-validation are really one topic. Held-out error is the only instrument that tells the two failure modes apart, so it has to be built carefully: split before any preprocessing, respect time order for time-based data, group by entity when rows repeat per customer. The mechanics are in cross-validation without fooling yourself, and the failure modes worth memorising are in data leakage is why your model looks great.
One more habit: keep a final test set you touch once. Every tuning decision made against a validation score slowly overfits that score. After forty experiments, your best validation number is partly a measure of how many times you rolled the dice.
Where the classic curve breaks
The textbook picture is the U below: test error falls as capacity rises, bottoms out at some sweet spot, then climbs as the model starts memorising. For most tabular models with sensible regularisation, that picture still holds and still guides good decisions.
Training error only ever falls. Test error bottoms out and then climbs, and the widening gap between the two curves is the overfitting.
But it is not the whole story, and pretending otherwise makes modern results look like magic. Very large models — networks with far more parameters than training examples — often show a second descent. Push capacity past the point where the model interpolates the training data exactly, keep pushing, and test error can start falling again, sometimes below the first minimum. The U turns into a curve with two dips.
Why this happens is still argued over. The useful part for a practitioner is narrower: parameter count is a poor proxy for effective capacity. What constrains a model is the combination of its architecture, its optimiser, its regularisation and the data — not the size of its weight vector. “It has more parameters than data points, so it must overfit” was always a rough heuristic, and in the large-model regime it simply fails.
None of which changes the working method. Measure held-out error, look at whether train and test errors are far apart or both bad, and move the dial in the direction the gap points. In code that is one habit: print the training score and the validation score on the same line, every time, and never the validation score on its own. Most notebooks print only the second one, which is why the diagnosis so often has to be guessed.