Simpson's paradox is hiding in your dashboard

The company-wide conversion rate fell from 5.0% to 4.6% quarter over quarter. The review meeting treats it as a fire. Twenty minutes in, someone splits the number by channel and finds conversion improved in every single channel. Both numbers are correct. Nothing is broken except intuition.

This is Simpson’s paradox — named after Simpson’s 1951 paper, though Yule and Pearson had both described it half a century earlier: a trend that holds in every subgroup reverses in the aggregate, because the mix of subgroups shifted. Marketing scaled up a high-volume, low-converting channel; each channel got better, but the blend got worse. The aggregate moved not because behaviour changed, but because the weights did.

A concrete miniature

Both channels improved. The total collapsed, because cheap paid traffic tripled while organic shrank. Any dashboard showing only the top line tells a story that is arithmetically true and causally backwards.

Where it bites in practice

• Conversion and retention reviews, whenever traffic mix, geography, or device share is shifting — which is always.
• A/B tests with imbalanced arms: if the split accidentally skews by platform or country, the aggregate comparison inherits the skew. (Proper randomisation prevents this; broken randomisation manufactures it.)
• Performance comparisons: hospital B looks worse than hospital A overall while being better for both mild and severe patients — because it takes the severe ones. Same trap for support agents, stores, and models evaluated on different traffic.
• Averages of averages: df.groupby("month").rate.mean() weights January’s 30 users equally with June’s 30,000. Aggregate the numerators and denominators, then divide — never average the rates.

# Wrong: average of per-segment rates
df.groupby("month").conversion_rate.mean()

# Right: total conversions / total visitors
agg = df.groupby("month").agg(conv=("converted", "sum"),
                              n=("visitor_id", "nunique"))
agg.conv / agg.n

The diagnostic: decompose the change

When an aggregate rate moves, split the movement into within-segment change (each segment’s own rate moved) and mix shift (the weights moved). A quick way to see it: compute what this period’s aggregate would have been with last period’s mix.

counterfactual = (q1_weights * q2_rates).sum()   # Q2 behavior, Q1 mix
mix_effect     = q2_total - counterfactual
behavior_effect = counterfactual - q1_total

If mix_effect dominates, the story is “our mix changed” — a strategy question, not a product regression. This two-line decomposition has defused more executive panics than any model I’ve built.

Which number is “true”?

Both — they answer different questions. The aggregate answers “what is happening to the business as a whole?” The per-segment view answers “is the underlying behaviour improving?” The mistake is using one to answer the other’s question. A report that shows the top line, the segment lines, and one sentence attributing the gap to mix is complete; anything less invites the wrong decision.

The general lesson runs deeper than dashboards: aggregation always destroys information, and confounding variables (here, the segment mix) decide whether the destroyed information was load-bearing. You can’t check every possible split — but you can know your two or three dominant mix variables (channel, geography, platform) and make “did the mix shift?” the first question asked whenever a headline metric surprises anyone. It’s a thirty-second check, and about once a quarter it will save your team from solving a problem that doesn’t exist.