Experimentation
A/B tests and the discipline around them: pre-specify, size it, resist peeking.
5 articles, newest first.
- When you cannot randomise users Interference, marketplaces and features that leak between groups. Cluster and geo randomisation, switchback tests, and the observational fallbacks with their assumptions stated.
- Sizing an experiment before you run it Power analysis without the ritual: choosing the smallest effect worth detecting, what drives the sample size, and why an underpowered test is worse than no test.
- Bandits are the reinforcement learning you will actually use Full RL rarely survives contact with a business problem. Multi-armed bandits do: the same explore-exploit trade, without the state, the simulator or the reward engineering.
- Your A/B test is probably lying to you Peeking, underpowered tests, and twenty metrics with one winner: the three failure modes that produce most false experiment wins, and the working defenses.
- P-values: what they say and what they do not What a p-value actually measures, the four things it gets mistaken for, how forking paths produce small p-values from nothing, and what to report instead.