Two sales reps are compared on close rate. Rep B closes a higher share of deals on the cheap plan than Rep A. Rep B also closes a higher share on the premium plan. Rep A has the better overall close rate. All three statements are true at once, and no arithmetic has been done wrong.
| Cheap plan | Premium plan | Overall | |
|---|---|---|---|
| Rep A | 90/100 = 90% | 20/100 = 20% | 110/200 = 55% |
| Rep B | 19/20 = 95% | 60/180 = 33% | 79/200 = 40% |
Rep B wins in each column and loses the total. This is Simpson’s paradox. It is worth being precise about what kind of thing it is: not a statistical subtlety, not a sampling artefact, not something that goes away with more data. It is a property of fractions. Understand that and the rest follows. Plotted, the reversal is one line pointing the wrong way.
Both black lines rise: Rep B is better on the cheap plan and better on the premium plan. The teal line falls, because it is a weighted average and the weights are the deal counts. Rep B’s total is dragged toward 33% by the 180 premium deals.
Fractions do not add the way you want them to
Comparing 90% to 95% is comparing two ratios. Combining Rep B’s two ratios into one does not average them — it adds the tops and adds the bottoms. 19/20 combined with 60/180 gives 79/200, and where that lands between 95% and 33% depends entirely on how many deals sat in each row.
So the overall number is a weighted average, and the weights are the denominators. Rep A’s total sits near the middle of 90% and 20% because the work was split evenly. Rep B’s total sits close to 33% because 90% of the attempts were premium deals. Rep B is better at both jobs and mostly did the hard one.
Once you see the totals as weighted averages, the reversal stops being strange. There is no law that says: if a/b > c/d and e/f > g/h, then (a+e)/(b+f) > (c+g)/(d+h). Nobody ever proved that, because it is false. The paradox is only the surprise of finding out you had assumed it.
Two things must both hold for a reversal to appear. The per-group rates must differ a lot (premium is much harder than cheap), and the group mix must differ a lot between the things being compared (Rep B worked premium almost exclusively). Kill either one and the reversal cannot happen.
Berkeley, 1973
The famous case is graduate admissions at UC Berkeley. Across the university, about 44% of male applicants were admitted against about 35% of female applicants — a gap large enough that the university expected a lawsuit. When Bickel, Hammel and O’Connell went department by department — their write-up, Sex Bias in Graduate Admissions: Data from Berkeley, ran in Science in 1975 — the picture inverted: most departments admitted women at a slightly higher rate than men.
The mechanism is the one above. Departments had wildly different admission rates, some taking most applicants and some taking a few. Men and women applied to different departments in different proportions. Women applied more often to the departments that rejected nearly everyone. The mix did the work.
The tempting summary is “the aggregate was wrong, the departments were right”. That summary is too fast, and it is where most people stop.
Which number is the right one is not a statistical question
Both numbers are correct descriptions of the data. They answer different questions, and only one of your questions is the one you care about. The statistics cannot tell you which. You have to say what causes what.
Ask: is the variable you split on a common cause of the two things you are comparing, or does it sit on the path between them?
If department choice were fixed before anything else and simply happened to correlate with gender, it would be a confounder — a common cause of both “who applies” and “who gets in”. Confounders should be split out. The per-department numbers would be the honest ones, and the aggregate would be a mirage produced by unequal mixing.
But if applicants are steered towards certain fields — by advice, by expectation, by what they were encouraged to study at 17 — then department choice is part of the effect, not a nuisance sitting outside it. It is a step on the causal path. Splitting by it removes the very thing you wanted to measure, and the aggregate is closer to the answer you were after.
Same table, same arithmetic, opposite conclusion, decided entirely by a claim about the world that no amount of staring at the data will settle. This is the real lesson of Simpson’s paradox, and it survives long after the party trick wears off: adjusting for a variable is a causal decision disguised as a technical one.
The rule of thumb that follows is short. Split on things that happened before the thing you are comparing. Do not split on things that happened after it, or that the thing you are comparing helped cause. In the sales example, if reps are assigned their territory by a manager, split by plan type. If reps choose which deals to chase, the mix is part of their performance and the total is fair.
minutephysics draws the reversal as two line segments sliding past each other in Simpson’s Paradox.
What to do on Monday
Every aggregate you report is a weighted average of something. Know the two or three variables that drive the weights in your business — channel, geography, plan tier, cohort age — and check whether they moved before you believe any headline movement. The practitioner version of this check, with the decomposition code and the dashboard failure modes, is in Simpson’s paradox is hiding in your dashboard.
And when someone asks you to “control for” a variable, ask them what they think it does. If they cannot say whether it comes before or after the effect they want to measure, the analysis is not ready to be run yet. Most arguments about which number is right are actually arguments about causes, held by people who have not noticed they are having one.