Your A/B test is probably lying to you

A test runs for eleven days. On day four the dashboard shows the variant up 3.1% and someone starts drafting the launch note. On day nine it shows 0.4%. On day eleven the team calls it flat and shelves the feature. All three readings came from the same experiment, and no data was wrong. Three failure modes — peeking, low power, and metric shopping — explain most of the launches that never move the number they promised to move.

Failure mode 1: peeking

You launch the test, and every day you check the dashboard. The moment the p-value dips under 0.05, you declare victory and ship.

The problem: a p-value describes one planned test, not a running tally, and it wanders. Under a true null, it crosses 0.05 at some point far more often than 5% of the time — check daily for a month and your real false-positive rate can be several times the advertised one. You’re not running one test; you’re running thirty and stopping at the first lucky one.

Defenses, pick one and mean it:

• Fixed horizon: compute the needed sample size in advance, don’t conclude until you reach it. Look at the dashboard all you like; just don’t decide.
• Sequential methods: group-sequential boundaries or always-valid inference are designed for continuous monitoring. Use a framework that implements them rather than hand-rolling.

Failure mode 2: the test never had a chance

Power is the probability of detecting an effect given that it exists. Teams routinely run tests that can only detect effects far larger than any they’ve ever shipped. Those tests are theater: a true 1% lift will come back “not significant” most of the time, and — nastier — when an underpowered test does reach significance, the measured effect is systematically exaggerated (the winner’s curse). Small samples only cross the significance line when they get lucky, so the lifts you “confirm” are inflated ones.

Do the arithmetic before launching, with statsmodels’ NormalIndPower and proportion_effectsize:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.040   # current conversion
lift     = 0.042   # what you hope for (a 5% relative lift)
effect = proportion_effectsize(lift, baseline)
n = NormalIndPower().solve_power(effect, power=0.8, alpha=0.05)
print(f"{n:,.0f} users per arm")   # ≈ 88,000 per arm

If that number exceeds the traffic you have, the honest options are: test a bolder change, accept a longer test, use a more sensitive metric, or reduce variance (CUPED-style covariate adjustment routinely buys back 30–50% of sample size). “Run it anyway and see” is not an option; it’s a random-number generator with a UI.

Failure mode 3: twenty metrics, one winner

Track twenty metrics and, under a pure null, the chance that at least one hits p < 0.05 is about 64%. Teams that let the “winning” metric be chosen after looking at results will always find a win somewhere — revenue was flat, but “clicks on the second carousel among returning mobile users” is up 12%!

The defense is boring and effective: one pre-registered primary metric per test, stated before launch, plus a small set of guardrail metrics (latency, unsubscribes, refunds) that can veto a launch but not justify one. Everything else is exploratory — hypothesis-generating, never hypothesis-confirming. When several metrics genuinely must be tested at once, correct for it explicitly: multipletests implements Bonferroni and the false-discovery-rate procedures.

The meta-fix: keep a decision log

Before each experiment, write five lines: hypothesis, primary metric, minimum effect worth shipping, sample size, stop date. After it: the result and the decision. This tiny document does two jobs — it prevents quiet redefinitions of success mid-flight, and after a year it lets you measure your team’s actual hit rate, which is the number that tells you how much to trust the next dashboard.

Two of the three fixes are cheap. A sequential test is one library call, and a pre-registered primary metric is one line in a ticket. Low power is the expensive one: the only real cures are more traffic, more time, or a smaller question, and that trade has to be settled before launch rather than explained afterwards.