Gradient boosting, built one mistake at a time

Start with the worst possible model. To predict house prices, predict the average price for every house. It is useless, but it is a starting point, and it has one property worth keeping: you can measure exactly how wrong it is on every single row.

Those errors are data. A house that sold for 420k when you predicted 300k has an error of +120k. Another has −80k. Now ask a different question — not “what is the price?” but “what is the error of my current guess?” If the errors are predictable from the features, a small model can predict them. Add that prediction to your first one and the whole thing gets less wrong.

Do it again. And again, a few hundred times. That is gradient boosting. There is no other trick in it.

The loop, written out

Take a training set with features X and target y.

1. Predict the mean of y for everything.
2. Compute the residuals: what is left over, y minus the current prediction.
3. Fit a small decision tree to predict those residuals from X.
4. Add a fraction of that tree’s output to the running prediction.
5. Go back to step 2.

For a single house, with a learning rate of 0.5, the loop looks like this:

The teal line is the running prediction; the black bars are the residual each round is fitted to. Nothing here aims at the price directly — every tree after the first is aimed at the gap the previous ones left, and the gap gets smaller each time.

Written as code, the whole algorithm fits in a dozen lines:

import numpy as np
from sklearn.tree import DecisionTreeRegressor

def fit(X, y, n_trees=200, lr=0.1, depth=3):
    base = y.mean()
    pred = np.full(len(y), base)
    trees = []
    for _ in range(n_trees):
        residual = y - pred
        tree = DecisionTreeRegressor(max_depth=depth).fit(X, residual)
        pred += lr * tree.predict(X)
        trees.append(tree)
    return base, trees

def predict(base, trees, X, lr=0.1):
    return base + lr * sum(t.predict(X) for t in trees)

The libraries you actually use — XGBoost, LightGBM, CatBoost — are this loop plus fifteen years of engineering: histogram binning, sparse handling, better split search, regularisation terms in the split score. The idea is unchanged.

Two details in that code do the real work. The trees are small: depth 3, sometimes depth 1. And each tree’s contribution is multiplied by a small lr before being added. Neither is an accident, and the next two sections are about why.

Why this is not bagging, in one sentence

A random forest fits hundreds of trees in parallel, each on a bootstrap sample, and averages them. Boosting fits hundreds of trees in sequence, each on what the previous ones got wrong, and sums them.

That difference is not cosmetic. It changes what problem each method is solving.

A forest starts with models that are individually far too flexible and calms them down by averaging. It attacks variance. Boosting starts with models that are individually far too crude — a depth-3 tree can barely say anything — and stacks enough of them to build something expressive. It attacks bias.

This is why you can shuffle a forest’s trees, drop half of them, and still have a working model. Drop half a boosted model’s trees at random and you have nonsense, because tree 143 was fit to correct errors that only exist if trees 1 through 142 are all present and in order.

It also explains the danger. A forest is hard to overfit by adding trees; averaging more models does not make the average worse. A boosted model is trivial to overfit by adding trees, because every extra tree is explicitly hunting for remaining error, and eventually the only error left is noise. It will fit that noise happily. Boosting needs a stopping rule. A forest does not.

Where the “gradient” comes in

The residuals-based loop above works for squared error, and it is worth seeing why, because the answer is what makes the method general.

If your loss is (y - pred)², its derivative with respect to pred is -2(y - pred). The residual is, up to a constant, the negative gradient of the loss. So “fit a tree to the residuals” is really “fit a tree to the direction that reduces the loss fastest, then take a step that way”. It is gradient descent, except the steps are taken in the space of functions and each step is a small tree. That is precisely how Friedman framed it in Greedy Function Approximation: A Gradient Boosting Machine in 2001, which is where both the method and the name come from.

Once you see it that way the constraint disappears. Swap in log loss for classification, and the thing you fit each round is that loss’s gradient instead — for log loss it works out to the probability error, actual minus predicted probability. Swap in absolute error and you fit the sign of the residual, which is why quantile and robust objectives come free in these libraries. Same loop, different gradient. Hence the word “gradient” in the name.

Learning rate and number of trees are one knob, not two

The lr in the code is the shrinkage. Set it to 1.0 and each tree’s full correction is applied. The model races toward zero training error and lands on a bad solution, because it commits hard to whatever the first few trees happened to find.

Set it to 0.05 and each tree nudges. You need roughly twenty times as many trees to travel the same distance, but the path is smoother and the model that comes out generalises better. Small steps averaged over many trees are more robust than a few large steps.

So the two parameters trade against each other almost exactly: halve the learning rate, double the trees. The practical recipe follows directly. Pick a learning rate you can afford to train at — 0.05 is a fine default, lower if you have the time. Set the tree count absurdly high. Then let a held-out validation set tell you when to stop: track the validation loss each round and quit when it stops improving for a few hundred rounds.

That validation set must not be your test set. The stopping round is a fitted parameter like any other, and choosing it on the data you report from inflates the number you report. Three splits, or the stopping round chosen inside cross-validation.

The residual-fitting loop is worked through one round at a time in Gradient Boost Part 1: Regression Main Ideas by StatQuest with Josh Starmer.

Why it wins on tables

Put the pieces together and the tabular dominance stops being mysterious. Trees split on thresholds, so they handle the kinks, ceilings and step changes that real business data is full of. They ignore feature scale, so nothing needs normalising. Boosting then removes the single tree’s biggest weakness — being too crude — without the smoothness assumptions a linear or neural model brings. You get a flexible, non-smooth function fitted in tiny controlled increments, with a stopping rule that tells you when to stop being flexible.

Knowing the loop changes how you use the tool. The knobs stop being magic: depth is how much each correction can say, learning rate is how much of it you trust, subsampling is noise injected so consecutive trees do not chase the same ghost. For which of those to actually turn, and the mistakes that follow, see gradient boosting is still the default for tabular data.

And the inherited limits stay inherited. Every tree predicts a constant in each leaf, so a sum of trees predicts constants too. Train on prices up to 800k and the model will never predict 900k, no matter how clearly the trend points there. Boosting fixes bias inside the data it saw. It has nothing to say about the region beyond it.