Choosing the unit of analysis

Two analysts are asked the same question — “what share of our traffic converts?” — and hand back 15% and 50%. Neither has made an arithmetic error. Neither has used different data. They counted different things: one counted sessions, the other counted people. Before anyone can say which number is right, someone has to say what a row is supposed to represent.

That is the unit of analysis, and every table you write implies one whether you chose it or not. A row per session, a row per order, a row per user, a row per user-day. The choice sits upstream of the model, the test and the chart, and it decides what all of them mean.

One table, three conversion rates

Take four users over a week. The numbers are small enough to check by hand, which is the point.

Per session: 2 sessions out of 13 ended in an order, so conversion is 15.4%.

Per user: 2 users out of 4 bought something, so conversion is 50%.

Both come from the same five lines of data. The gap is not noise or a bug — it is user A, who browsed ten times and bought once. Counted by session, A is nine failures and one success. Counted by person, A is a customer.

Revenue splits the same way. €220 is €16.92 per session, €55 per user, or €110 per order. Three true statements, three different meanings, and a slide that says “average order value” when it computed revenue per session is simply wrong, no matter how careful the pipeline was.

Averaging averages quietly changes the question

There is a third conversion number hiding in that table, and it is the one that gets produced by accident. Compute each user’s own session conversion rate — A is 1/10 = 10%, B is 1/1 = 100%, C is 0%, D is 0% — then average those four rates. The result is 27.5%.

Nobody asked for 27.5%. It is not the share of sessions that convert, and it is not the share of users who buy. It is the conversion rate of a typical user’s sessions, with every user weighted equally regardless of how many sessions they had. User D’s single visit counts as much as user A’s ten.

The fix is mechanical: to get a rate, add the numerators, add the denominators, then divide. Averaging pre-computed rates only gives the same answer when every group has the same denominator, which real data never does. The failure is easy to miss because it produces a plausible number rather than an error, and it lands between the two legitimate answers, so it survives a sanity check.

It is worth noticing what the per-session figure really is: a weighted blend across users, with the weights set by how often each person visits. If heavy users become a larger share of traffic next month, the per-session rate moves even if not one person changed their behaviour — the same mix effect that reverses trends between the segments and the top line in a metric review. The per-user rate does not have that particular problem, and has others of its own.

The unit must match the decision

Neither number is more correct in the abstract. The decision picks.

If the question is whether to buy more traffic, the unit is the session, or whatever the ad platform charges for. Each new visit costs money and either pays for itself or does not; the per-session rate is the one that divides correctly into the cost per click.

If the question is whether the product is working for people, the unit is the user. “Half our customers buy something in their first week” is a statement about people, and it is what onboarding, pricing and lifecycle email decisions actually hang on.

If the question is about fulfilment cost or basket size, the unit is the order, because that is what gets packed and shipped.

So write the unit into the question before writing the query. “What share of sessions include a purchase, among sessions from logged-in users last week” is answerable and checkable. “What’s our conversion rate?” is not — it is three questions wearing one coat.

Experiments add a hard constraint on top of the decision. The unit of analysis must match the unit of randomisation. If users were assigned to variants, then users are the unit, full stop, because two sessions from the same person always share a variant and cannot be treated as two independent draws. Analysing an A/B test at session level after randomising at user level produces confidence intervals that are too narrow and p-values that are too small — a reliable way to ship changes that do nothing. The same rule holds when the assignment happens higher up: randomise by shop, city or week, and the analysis works at that level too, which is one reason designs that cannot randomise individual users lose so much statistical power.

Repeated rows are not independent observations

Almost every standard test — the t-test, the usual regression standard errors, the confidence interval a dashboard prints next to a lift — assumes each row is an independent observation. The 13-session table has 13 rows and 4 independent sources. Treat it as 13 and you have claimed roughly three times more information than you hold.

The distortion grows with how uneven the activity is. When one user supplies ten rows, their behaviour is being counted ten times, so the estimate is partly a measurement of that person rather than of the population. It also biases what the analysis is even about: session-level tables are dominated by heavy users, so a “typical session” describes the habits of the few people who visit constantly, not the many who visit once.

There are three honest responses. Collapse the data to one row per user before testing, which is the simplest and works well when the per-user summary answers the question. Use clustered standard errors, grouping by user, which keeps the detail and prices the dependence. Or model the structure directly with random effects. What is not available is ignoring it and hoping the sample is large enough — more correlated rows do not converge on the truth, they converge on a tighter interval around the wrong thing.

Fix the unit first, then. It costs one sentence at the top of a query and one line in the write-up. Getting it wrong costs considerably more: an interval two or three times too narrow, a result at p = 0.02 that a per-user test puts at 0.4, a feature shipped on the strength of it, and — the expensive part — a roadmap built on the belief that the effect is real. Undoing the launch takes a day. Undoing the belief takes a quarter.