User-level randomisation rests on one assumption that nobody states: one user’s assignment does not change another user’s outcome. In most product tests that holds well enough to ignore. In a marketplace it is simply false — and the test still returns a clean number with a tight confidence interval, which is what makes it dangerous.
Where the assumption quietly breaks
Three shapes of leakage cover almost every case.
Competition for shared supply. A better matching algorithm goes to half the riders. Those riders take the nearby drivers, so control riders wait longer. The measured gap is the real improvement plus the harm done to the control group. The same happens with limited stock, ad budget, warehouse capacity, and support agents. The number is inflated, sometimes entirely manufactured: a test can show a solid lift for a change that moves demand around without creating any.
Information and behaviour spreading. A feature makes treated users post more; their followers in the control group see more content and engage more. Now the control group has partly received the treatment, the gap narrows, and the effect is understated.
Shared accounts and devices. Household plans, shared browsers, and one person on three devices all mix assignments inside a single unit of behaviour.
Note that the first shape inflates and the second deflates. You cannot look at a result and correct for the direction afterwards. The usual symptom arrives later: the effect does not reappear at 100% rollout. At full rollout there is no control group to take capacity from, and the honest number is the one that disagrees with your test.
Randomise the cluster, not the user
The fix is to make the randomisation unit big enough to contain the interference: city, market, region, warehouse, store, team, household. Inside a cluster everyone gets the same treatment, so the leakage happens within an arm instead of across arms.
The cost is severe and often understated. Your effective sample size becomes the number of clusters, not users. Forty cities is n = 40, no matter how many million riders they hold. Cities also differ enormously from each other, and a few large ones dominate the variance.
This has three practical consequences:
- Analyse at the cluster level, or with a model that has a random effect per cluster. Running a user-level t-test on cluster-randomised data produces confidence intervals that are far too narrow, and it is the most common analysis error in this whole area.
- Assign in matched pairs or strata built from pre-period behaviour, rather than by coin flip. With 40 units, plain randomisation regularly hands you an unbalanced split.
- Expect to detect large effects only. Plan for 5% and up, not 1%. The same power arithmetic applies with clusters as the unit, and it will tell you unpleasant things early, which is the point.
Pre-period covariates help more here than anywhere else. A model that adjusts each cluster’s outcome for its own pre-period level removes most of the between-cluster variance and can turn an impossible test into a merely expensive one.
Switchbacks: randomise time instead
When the treatment acts on a shared system in real time — pricing, dispatch, matching, ranking on a shared feed — you can put the whole market on treatment for a window, then flip it to control for the next window, with the order randomised. Every market experiences both conditions, so differences between markets stop mattering.
What it costs:
- Carryover. Effects that outlive the window contaminate the next one. Drivers repositioned under treatment are still repositioned five minutes into control. The standard defence is a burn-in: discard the first slice of each window. Longer windows reduce carryover but give you fewer windows.
- Autocorrelation. Adjacent windows resemble each other, so the effective number of independent units is smaller than the window count. Block on time-of-day and day-of-week, and cluster the standard errors by day.
- Scope. Switchbacks only work for treatments with short memory. Anything that teaches a user a habit, changes what they have saved, or sends a notification cannot be switched off in an hour, and a switchback will understate it.
The observational fallbacks, and what they assume
Sometimes there is no randomisation at all. The feature launched in Portugal last March because that is where the team is. You still have to answer whether it worked.
Difference-in-differences. Compare the change over time in Portugal to the change over the same period in a market that did not get it. Subtracting the two changes removes anything constant about Portugal and anything that hit both markets at once. In practice it is one OLS regression with a treated×after interaction term, which is the design Card and Krueger made famous in their 1994 minimum-wage study. The assumption is parallel trends: absent the launch, the two markets would have moved by the same amount. That assumption is untestable. You can make it plausible — plot many pre-periods and check the lines move together, run placebo tests pretending the launch happened a year earlier — but plausible is not true. It fails precisely when the choice of market was driven by something that also drives the trend, which is usually the reason Portugal was picked. This is a genuine causal tool rather than a correlation in better clothing, but only while the assumption holds, and the assumption is doing all the work.
Synthetic control. Abadie and Gardeazabal’s 2003 method: instead of picking one comparison market, build a weighted blend of untreated markets that reproduces Portugal’s pre-launch series closely, then extend the blend forward as the counterfactual. It works with a single treated unit, which difference-in-differences handles poorly. Its assumptions: none of the donor markets was treated or affected by spillover from the treatment; the pre-period fit reflects real structure rather than overfitting to noise, which needs a long pre-period relative to the number of donors; and the weights that held before the launch still hold after it. The inference is weak by construction — you get a permutation test that runs the same procedure on each untreated market and asks how unusual the real one looks. With twenty donors, the smallest p-value available to you is about 0.05, and there is no way to sharpen it.
Both methods hand you an estimate you cannot fully defend. That is the price of not randomising, and it should be visible in the writeup: one sentence naming the assumption, sitting next to the number, every time the number is quoted.
A note on how this usually goes wrong, because it is specific. The estimate is computed carefully, the assumption is written into the appendix, and the number travels to a slide on its own. Six months later that number is sizing a budget, and nobody in the room knows it rests on twenty donor markets and a permutation test that could not have gone below 0.05 whatever happened. Put the assumption in the same sentence as the number — in the chart title, if the chart is where the number lives.