How big does the effect have to be before anyone acts on it? Sizing calculations usually skip that question. They start from the traffic available, work out what that traffic could detect, and call the answer the plan. That answers “what can I see?” instead of “what do I need to see?” — and the two lead somewhere very different when the result comes back flat.
The minimum detectable effect is a business number
The minimum detectable effect (MDE) is the smallest lift you would actually act on. It is not a statistical quantity that falls out of a formula. It comes from a conversation with whoever owns the metric, and it has two halves:
- Below what effect would you not bother shipping this? Below that line, the change costs more in code, support, and maintenance than it returns.
- Above what effect would you ship without arguing? That is the number the business is actually hoping for.
The MDE sits at the first line, not the second. If a 2% relative lift in checkout conversion pays for a permanent feature, size the test for 2% — even if the optimistic case is 8%. Sizing for the hopeful number is how teams end up declaring “no effect” on a change that was quietly worth two million a year.
Write the MDE down as a relative change on a named metric with a measured baseline: “conversion from cart to paid, currently 3.0%, MDE 5% relative.” Three numbers, one sentence. Everything downstream depends on them.
What actually drives the sample size
Four things, and only one of them is under your control after the fact.
The baseline rate. Rare events are expensive to measure. For a fixed relative lift, the required sample scales roughly with the inverse of the baseline: a 1% conversion rate needs about three times the traffic of a 3% one to detect the same relative change. This is why “let’s test it on the free-trial-to-paid step” is usually a bad idea and “let’s test it on add-to-cart” is usually a good one.
The size of the effect. The sample scales with the inverse square of the effect. Halving the MDE quadruples the test. This is the single most expensive knob in the whole exercise, and it is the one people move casually in meetings.
Variance, for anything that isn’t a rate. Revenue per user is heavy-tailed: a handful of large orders carry most of the variance, and the variance term sits directly in the numerator. Capping or winsorising the metric, or switching to a bounded version of it, often cuts the required sample more than any other change you can make. Covariate adjustment using pre-period behaviour (CUPED and relatives) does the same job without distorting the metric.
Your tolerances. Standard is 5% false positive rate and 80% power. Moving power from 80% to 90% costs about a third more traffic. That is often worth it; 80% power means a real effect is missed one run in five.
For two proportions, the normal approximation is enough to plan with — twelve
lines around
scipy.stats.norm:
from scipy.stats import norm
def n_per_arm(p1, p2, alpha=0.05, power=0.8): z_a = norm.ppf(1 - alpha / 2) z_b = norm.ppf(power) var = p1 * (1 - p1) + p2 * (1 - p2) return (z_a + z_b) ** 2 * var / (p2 - p1) ** 2
print(f"{n_per_arm(0.030, 0.0315):,.0f} per arm") # 207,935 per armA 5% relative lift on a 3% baseline costs roughly 208,000 users per arm.
Different approximations disagree by 10–20%, so treat the output as a size, not
a target: 200,000 and 240,000 mean the same thing for planning. statsmodels
packages the same calculation as
NormalIndPower,
with proportion_effectsize
to convert two rates into the effect size it wants.
An underpowered test is worse than no test
This is the part that gets waved through. A test that cannot detect the effect you care about does not produce nothing. It produces noise, and the noise gets acted on.
A flat result reads as “the change did nothing” in every summary it passes through, and the idea gets killed. But a non-significant result from an underpowered test is not evidence of no effect — that is not what a p-value says, and the honest report is “we could not tell.”
The results that do come back significant are worse. A small sample only crosses the line when it gets lucky, so every significant result from an underpowered test overstates the true effect — often by a factor of two or three. The team ships, writes the inflated number into the forecast, and a year later nobody can explain why twelve wins of 3% each produced no visible movement in the annual figure. At very low power the estimate can even come back with the wrong sign.
Then there is the traffic itself. Six underpowered tests in a quarter are strictly worse than two properly sized ones: you have spent the same traffic and learned nothing you can defend. And because a test with a marginal result tempts everyone to keep watching, an underpowered design usually collects the peeking problem on top of its own.
Traffic against runtime
Once you have the sample size, divide it by the eligible users per day — only the people who reach the surface you changed, not everyone who opens the app. That gives the runtime.
Then apply two rules. Run at least one full week, and stop on the same weekday you started, because weekday and weekend users behave differently and a partial week silently tilts the sample. And treat four weeks as a soft ceiling: over longer windows, cookie churn dilutes assignment, novelty effects fade, and other releases land in the middle of your test.
If the arithmetic demands eight weeks, the test as designed is not available to you. The real options are: test a bolder change, move to a metric closer to what you altered, reduce variance, pool several surfaces into one test, or decide on judgement and say plainly that you decided on judgement. Running it anyway and hoping is not on the list.
Fix the number before the test starts
Before launch, five lines: primary metric, baseline measured from the last four weeks of real data, MDE, sample size per arm, stop date. Compute the baseline from a query, not from memory — stale baselines are the most common reason a sizing calculation is off by half.
The arithmetic takes five minutes and no traffic. Skipping it costs a good deal more than the three weeks it was meant to save: a flat result nobody can read, a fortnight of argument about whether to extend, an extension that adds contaminated traffic and a competing release in the middle of it, and then the decision made on judgement anyway — five weeks late, with the room in a worse mood than it started in.