Outliers: diagnose before you delete

“We removed outliers” might be the most dangerous sentence in analytics, because it treats three completely different situations as one. An extreme value is either an error (a decimal slip, a unit mix-up, a test account), a real but extreme case (the whale customer, the viral post), or a discovery (fraud, a breaking failure mode). The first should be fixed or removed, the second must usually stay, and the third is sometimes the entire point of the analysis. Deleting all three because they inflate the standard deviation is throwing away signal to make a chart prettier.

Diagnose first

Before touching anything, look at the actual extreme rows — not the summary:

The purchases below are the real ones from the public dataset, joined to the plan the customer is on.

import pandas as pd

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv")
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])
df = events[events.event_type == "purchase"].merge(
    subs[["customer_id", "plan"]].drop_duplicates("customer_id"),
    on="customer_id")

suspects = df[df.amount > df.amount.quantile(0.99)]
print(suspects[["customer_id", "plan", "amount", "occurred_at"]]
      .sort_values("amount", ascending=False).head(10))
print(suspects.plan.value_counts())

      customer_id        plan  amount         occurred_at
2350         1782         pro  857.00 2026-06-13 10:53:25
994          1263  enterprise  854.52 2025-07-21 11:07:56
576          1168  enterprise  834.22 2025-03-21 17:29:02
1754         1755  enterprise  832.82 2026-01-30 16:44:29
932          1263  enterprise  831.53 2025-07-03 15:06:17
1485         1558  enterprise  827.04 2025-12-02 13:03:29
717          1264  enterprise  825.20 2025-04-29 21:20:21
108          1005  enterprise  824.04 2024-10-11 15:46:48
941          1272  enterprise  823.86 2025-07-04 15:20:46
613          1203  enterprise  822.66 2025-04-01 13:44:22
plan
enterprise    24
pro            1
Name: count, dtype: int64

Twenty rows of eyeballing usually settles it, and here the second column does it in one: 24 of the 25 largest purchases belong to enterprise customers, spread over nineteen accounts and two years. No decimal slips, no repeated amount, no single account. These are not outliers. They are the top of the price list, and any rule that removes them removes the enterprise plan.

The other branches look different. Amounts exactly 100× normal from one source? Currency-cents bug — fix upstream, correct the rows. All from one customer id? A real whale or a load test — check who it is. Scattered and plausible? Heavy-tailed reality, which is what most business metrics are.

Two mechanical warnings. The z-score method (“beyond 3 standard deviations”) is self-defeating: outliers inflate the very standard deviation used to detect them, so the worst offenders hide their accomplices. Use MAD-based scores or IQR fences if you need an automatic flag. And on genuinely heavy-tailed distributions (revenue, latency, file sizes), those fences will flag a steady percentage of legitimate data forever — that’s not detection, that’s a distribution telling you it isn’t Gaussian.

q1, q3 = df.amount.quantile([0.25, 0.75])
fence = q3 + 1.5 * (q3 - q1)
print(f"IQR fence {fence:.2f}, flags {(df.amount > fence).mean():.3f} of rows")
print(df[df.amount > fence].plan.value_counts())

IQR fence 176.70, flags 0.114 of rows
plan
enterprise    166
pro           107
starter         7
standard        4
Name: count, dtype: int64

More than one purchase in ten is over the fence, permanently, and 273 of the 284 flagged rows come from the two expensive plans. The fence is not finding anomalies. It is finding the enterprise plan and calling it an anomaly, every day, forever.

Robust methods beat deletion

Often the right move is to keep every row and use statistics that don’t capsize:

• Medians and quantiles instead of means, for reporting. “Median order €38, P95 €410” describes a heavy tail honestly; “mean €71” describes nothing anyone experiences.
• Winsorizing (capping at, say, P1/P99) when a model needs the column but shouldn’t be dominated by its tail — transparent and reversible, unlike deletion:

lo, hi = df.amount.quantile([0.01, 0.99])
df["amount_w"] = df.amount.clip(lo, hi)
print(f"{lo:.2f} {hi:.2f}")
print(f"total {df.amount.sum():,.0f}  winsorised {df.amount_w.sum():,.0f}")
print(f"top 1% carries {suspects.amount.sum() / df.amount.sum():.1%}")

2.86 780.19
total 187,350  winsorised 186,534
top 1% carries 10.8%

• Log transforms for multiplicative quantities — after logging, many “outliers” reveal themselves as ordinary members of a lognormal family.
• Robust losses (Huber, quantile regression) and tree ensembles, which split on order rather than distance and inherit outlier resistance.

The reporting trap

Where outlier handling does the most quiet damage is aggregated reporting. Trim the top 1% of orders “for stability” and you may have removed a large share of revenue — your dashboard is now stable, smooth, and wrong. On the purchases above, the 25 rows over the 99th percentile carry 10.8% of the money, and capping them at that percentile instead of deleting them costs 0.4%. Both numbers are worth knowing before the choice is made, and neither is knowable after it. The whales you trimmed are frequently the segment leadership most needs to see. If a metric needs trimming to be readable, report both numbers: the robust one for trend, the untrimmed one for truth, labelled clearly.

The same logic applies to models: a demand forecaster trained with Black Friday deleted will be confidently wrong every Black Friday. If extremes recur, they’re not outliers — they’re seasons.

A five-line policy

1. Detect with robust fences (MAD/IQR), never raw z-scores.
2. Inspect the actual rows before any treatment; classify error / extreme / discovery.
3. Fix errors at the source and document the correction.
4. Keep real extremes; switch to robust statistics or capped features instead of deletion.
5. Route discoveries (fraud patterns, failure modes) to whoever owns that problem — sometimes the outliers are the deliverable.

Write the policy down in the project README, apply it consistently, and “we removed outliers” becomes “here’s exactly what we did to which rows and why” — a sentence you can defend in any meeting.