A decision tree is a sequence of yes/no questions ending in an answer. Is the customer’s tenure under six months? Then: have they contacted support more than twice? Then predict churn. Anyone can read that. Anyone could have written it by hand fifty years ago, and people did. Written out, it looks like this.
Teal boxes are the questions the algorithm chose; grey boxes are the leaves it stops at. The rightmost leaf is nearly pure — 40 churners out of 900 — so the tree stops asking. The second leaf, at 130 against 270, is still mixed, and a deeper tree would keep cutting it.
The only thing a learning algorithm adds is a way to choose the questions from data instead of from opinion. That choice is the whole subject. Everything else — pruning, forests, boosting — is a reaction to what goes wrong once a machine starts picking questions greedily and never gets bored.
So start there: how does a tree decide what to ask first?
A split is chosen by brute force and a purity score
At the root sits every training row, hopelessly mixed: some churners, some not. The algorithm wants a question that separates them. It has no cleverness for this. It tries everything.
For each feature, and for each threshold worth trying on that feature, it splits the rows in two and scores the result. Then it keeps the best split and repeats the same brute-force search inside each child. Split, score, keep, recurse. Training has no other steps.
The score is a measure of how mixed a group is. For classification the usual one is Gini impurity: one minus the sum of the squared class proportions. A node that is 50/50 scores 0.5 — maximally mixed for two classes. A node that is 90/10 scores 0.18. A node with one class only scores 0. Lower is better, and the split the tree keeps is the one that most reduces the average impurity of the children, weighted by how many rows land in each.
For regression, swap impurity for variance: a good split is one where each side’s values sit tightly around their own mean.
Two consequences follow immediately, and both matter in practice.
First, the search is greedy. It takes the best split available now, with no lookahead. The globally best tree may start with a mediocre-looking question, and no standard implementation will ever find it. Greedy is not a bug to be fixed; the exhaustive alternative is computationally hopeless.
Second, splits are axis-aligned. A tree can only cut on one feature at a time, so a diagonal boundary — say, “flag it when income exceeds three times rent” — has to be approximated by a staircase of many splits. If you know the ratio matters, give the model the ratio as a feature. That single line is often worth more than any tuning.
One deep tree memorises, and it is unstable
Let the search keep going and it will not stop until every leaf is pure. Given enough features, it can always find one more question that isolates the last awkward row. The tree ends up with a leaf for the customer who signed up on a Tuesday in March with exactly two support tickets, and that leaf predicts whatever that one person did.
Training error goes to zero. Test error goes up. The tree has learned the noise along with the signal, and there is nothing in the algorithm that knows the difference.
The second problem is subtler and worse. A tree is unstable. Change a handful of training rows and the best split at the root may change, and once the root changes, every question below it is chosen on different data. The whole structure rearranges. Two analysts with 95% overlapping data can produce two trees that look nothing alike and tell different stories about the business.
You can hold this back by limiting depth or requiring a minimum number of rows per leaf. That trades one failure for another: a shallow tree is stable and readable but too crude to be accurate. A single tree is stuck choosing between memorising and being blunt.
Averaging works only if the errors disagree
Here is the escape. If you had many trees whose mistakes were unrelated, you could average their predictions and the mistakes would partly cancel while the signal, which they agree on, would survive.
The instability that ruins one tree becomes the raw material for this. Draw a bootstrap sample — sample the training rows with replacement, same count as the original — and fit a tree on it. Repeat hundreds of times. Each tree sees a slightly different world and grows differently. Average their votes. This is bagging, and it works: variance drops, bias stays roughly where it was.
But it does not work as well as it should, because bootstrapped trees are still too similar. If one feature is strongly predictive, nearly every tree picks it at the root and the trees end up correlated. Correlated errors do not cancel.
Random forests, as Breiman set them out in his 2001 paper Random Forests, add one line to fix exactly this. At every split, the tree may only consider a random subset of the features — classically the square root of the feature count for classification, about a third for regression. The dominant feature is simply unavailable in many splits, so other trees are forced to find the second-best structure, and the third. The individual trees get slightly worse. The ensemble gets substantially better, because the errors finally disagree.
A random forest is four instructions: bootstrap the rows, subsample the features, grow deep, average. Two sources of randomness, deliberately injected, to buy decorrelation.
A useful free extra falls out of the bootstrap. Each tree leaves out roughly a third of the rows, so every row can be scored by the trees that never saw it. That out-of-bag estimate is honest and costs nothing, though it does not replace a proper held-out set once you start tuning against it.
The interpretability is mostly gone, and the importances do not bring it back
The pitch for trees was that you can read them. A forest of 500 trees, each thousands of nodes deep, is not readable by anyone. Averaged predictions come with no path to point at.
What people reach for instead is the feature importance list, and this is where the honesty needs to be stated plainly. The default importance in most libraries sums how much each feature reduced impurity across all splits. It is biased toward features with many possible split points — continuous variables and high-cardinality categories score highly whether or not they carry signal. It also splits credit arbitrarily between correlated features: two near-duplicate columns each look half as important as either would alone.
Even a well-computed importance answers “what did this model lean on”, never “what causes the outcome” and never “what will happen if we change this”. That distinction is worth its own read: feature importance is not explanation.
For the bagging-and-subsampling machinery drawn step by step, watch StatQuest: Random Forests Part 1 by StatQuest with Josh Starmer.
One more limit deserves a place in the intuition. A tree predicts a constant in each leaf, so a forest cannot predict outside the range of values it was trained on. Feed it next year’s prices and it flatlines at the highest average it ever saw. Trees interpolate between the examples they were given. They do not extrapolate, and no amount of averaging changes that.