P-values: what they say and what they do not

A test comes back with p = 0.03 and the room relaxes. The result is “significant”. The feature ships.

Ask what the number means and the answers scatter. There is a 3% chance the result is wrong. There is a 97% chance the effect is real. The effect is probably big, since p is small. All three are false, and the last one is not even close.

A p-value answers one narrow question: if nothing were going on, how often would data look at least this extreme? That is a statement about hypothetical data under an assumption. It is not a statement about the assumption, and it is not a statement about your effect.

Build the null model first

The p-value cannot exist on its own. It needs a null model — a precise, boring story about how the data arose, in which the effect you care about is absent.

Suppose variant B converted at 12.4% and variant A at 11.8%. The null model says the variant label is irrelevant: every visitor would have behaved the same way either way, and the split into A and B was just a shuffle. If that story is true, you can generate as many parallel worlds as you like — reshuffle the labels and recompute the gap.

import numpy as np

rng = np.random.default_rng(0)

def permutation_p(converted, is_b, n_perm=20_000):
    """converted, is_b: boolean arrays, one row per visitor."""
    observed = converted[is_b].mean() - converted[~is_b].mean()
    labels = is_b.copy()
    extreme = 0
    for _ in range(n_perm):
        rng.shuffle(labels)
        gap = converted[labels].mean() - converted[~labels].mean()
        extreme += abs(gap) >= abs(observed)
    return (extreme + 1) / (n_perm + 1)

Nothing in that loop assumes a distribution. The p-value is the fraction of shuffled worlds that produced a gap as large as the real one. The t-tests and z-tests in textbooks are shortcuts that compute this same quantity with algebra instead of a loop.

Drawn out, the shuffled worlds pile up around no difference at all, and the result we actually got sits off to one side:

Every shuffled world produces some gap between A and B. Almost all of them land near zero, because in the null model the labels mean nothing. The gap we measured sits out where only 3% of shuffled worlds reach — counting both directions, since the same gap the other way would have been just as odd. That 3% is the p-value. It describes the shaded slivers, not the effect, and not the chance the null is true. Two things follow immediately. A p-value is always relative to a null model, so “the p-value” means nothing until someone says what the null model was. And the p-value is computed assuming the null is true — it can never tell you how likely that assumption is, any more than “how often do innocent people match this description?” tells you whether the defendant is innocent.

The four mistakes

The last one causes the most damage in practice. With ten million users, a 0.01% lift produces a tiny p-value. With four hundred users, a 30% lift may not reach significance at all. The p-value is not a measure of importance and never was. It is a measure of how hard the data are to explain away as noise, and enough data makes anything hard to explain away.

The first two need the ingredient a p-value never contains: a base rate. If you test a hundred hypotheses and only ten of them are true, a 0.05 threshold produces roughly nine true findings and four and a half false ones. The fraction of significant results that are wrong depends on how good your hypotheses were, which the p-value cannot see. Ioannidis built an entire argument out of that arithmetic in Why Most Published Research Findings Are False (2005): in a field where few of the tested hypotheses are true, most of the published significant results will be the false ones.

Forking paths make small p-values from nothing

The definition assumes one analysis, decided before the data arrived. Real analysis is nothing like that.

You segment by country. You drop outliers, then try it without dropping them. You test the primary metric, then three secondary metrics. You look at the results on Tuesday, and again on Friday. You try log revenue instead of revenue. Each of those is a fork, and each fork is another chance to land under 0.05 by luck.

This does not require dishonesty. Nobody needs to run twenty tests and report the best one. It is enough that the choices were made after seeing the data — you would have segmented differently had the data looked different. Andrew Gelman’s name for this, the garden of forking paths, is exact: the analysis you did was one path through a garden of analyses you would have accepted.

Stopping an experiment when it looks good is the same problem in its most expensive form. A test monitored daily and stopped at significance will hit p < 0.05 far more than 5% of the time under a true null, because every day is another draw. That mechanism, and what to do instead, is the subject of A/B test pitfalls: peeking and power.

The tell is easy to check. Ask how many analyses were run, and whether the reported one was chosen before or after the data came in. If the answer is vague, the p-value is decoration.

What to report instead

Not “never use p-values”. Used for what they are — a screen against noise, on one pre-specified comparison — they are fine. The problem is letting one number carry an argument it cannot carry.

• Lead with the effect size and an interval. “B is 0.6 points higher, 95% interval 0.1 to 1.1” says everything the p-value says, plus how large the effect plausibly is, plus how precisely you measured it. A bootstrap interval works for statistics with no clean formula.
• Fix the primary metric and the sample size before starting. Written down. Everything else is exploratory and labelled as such.
• Say how many things you looked at. Twenty comparisons with one at p = 0.04 is a null result, honestly reported.
• Report the boring numbers. Sample sizes, dropout, how the split was done. Most failed experiments fail at randomisation, not at inference.

For how far this reaches beyond one lab — publication bias, failed replications, and the incentives behind both — the clearest popular account is Is Most Published Research Wrong? by Veritasium.

Treat p = 0.03 as what it is: mild evidence that the boring story fits badly. It is a reason to look closer, not a verdict, and certainly not a decision. The decision needs the size of the effect, the cost of being wrong, and how plausible the claim was before anyone ran a test.