How to sample data without lying to yourself

At some point every analysis starts with “let me grab some of the data to look at.” How you grab it decides whether the next two weeks of work generalise or quietly describe an accident of ordering. The failure is invisible: a biased sample doesn’t error — it just makes you confidently wrong about the whole.

The default sin: LIMIT is not a sample

SELECT * ... LIMIT 10000 and df.head(1000) return whatever the storage order coughs up first — usually the oldest rows, or one shard, or one customer’s imported batch. Old rows mean old products, old prices, old schema quirks; your “exploration” describes a company that no longer exists. The same trap wears other costumes: “just January” (seasonality), “just Germany” (geography), “users who answered the survey” (self-selection).

Random costs almost nothing more — TABLESAMPLE in the warehouse, DataFrame.sample in pandas. Both SQL queries below run on the sample data, an events table of 40,000 rows belonging to 1,085 customers:

-- Reproducible 1% sample. Postgres writes the size as BERNOULLI (1).
SELECT * FROM events TABLESAMPLE BERNOULLI (1%) REPEATABLE (42);

-- Portable fallback: hash the key (hash in DuckDB, hashtext in Postgres)
SELECT * FROM events
WHERE hash(event_id) % 100 = 0;   -- ~1%

import pandas as pd

events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])
sample = events.sample(n=10_000, random_state=42)

Note the seeds. An irreproducible sample means every rerun is a different dataset, and “the numbers changed” becomes a mystery of your own making.

Sample entities, not rows

The subtler trap: sampling 1% of events when your unit of analysis is users. Row-level sampling shreds every user’s history — sessions have holes, funnels look broken, “average events per user” is nonsense. And it oversamples heavy users: pick random rows and you disproportionately land on people with many rows. Your “typical user” is secretly your most active user.

The size of that distortion on the 40,000-row file:

per_customer = events.groupby("customer_id").size()

rows = events.sample(n=10_000, random_state=42)                 # 25% of rows
ids = pd.Series(per_customer.index).sample(frac=0.25, random_state=42)
entities = events[events.customer_id.isin(ids)]                 # 25% of customers

print(f"population    {len(per_customer):5d} customers, "
      f"mean history {per_customer.mean():6.1f}")
print(f"row sample    {rows.customer_id.nunique():5d} customers, "
      f"mean history {rows.customer_id.map(per_customer).mean():6.1f}")
print(f"entity sample {entities.customer_id.nunique():5d} customers, "
      f"mean history {entities.groupby('customer_id').size().mean():6.1f}")

population     1085 customers, mean history   36.9
row sample      978 customers, mean history  115.5
entity sample   271 customers, mean history   39.1

The customer behind a randomly drawn row has 115.5 events on average. A randomly drawn customer has 36.9. Same file, same 25%, and the row sample describes a customer base three times more active than the one that exists — before any analysis has been run on it. Note the first column too: the row sample touched 978 of the 1,085 customers and gave almost none of them a complete history.

Sample the entity, keep everything belonging to it:

-- All events for a stable 1% of customers
SELECT e.* FROM events e
WHERE hash(e.customer_id) % 100 = 0;

Hash-based selection has a bonus property: it’s stable. The same users fall in the sample every day, so week-over-week comparisons compare the same people, and you can grow the sample (< 5 for 5%) without discarding the old one.

Stratify when the interesting thing is rare

A 1% random sample of ten million transactions contains ~30 fraud cases — too few to learn anything, because the error in an estimate falls only as fast as the square root of the count. Stratified sampling keeps all the rare class and a fraction of the common one — train_test_split(..., stratify=y) does the same job when the split is for modelling:

fraud    = df[df.is_fraud == 1]                      # keep every one
normal   = df[df.is_fraud == 0].sample(len(fraud) * 20, random_state=42)
sample   = pd.concat([fraud, normal])
sample_weight = np.where(sample.is_fraud == 0, actual_ratio / 20, 1.0)

The tax: your sample’s class balance is now fake. Any rate, mean, or model probability computed on it is wrong for the population until you reweight (or recalibrate). Stratified samples are for finding structure; population numbers must be computed with weights, or on unstratified data.

Know the frame you’re sampling from

The most expensive sampling errors happen before any code runs: the frame (what you’re sampling from) doesn’t match the population (what you’re claiming about). Logged-in events can’t tell you about visitors. This month’s active customers can’t tell you about churned ones — every “why do customers stay” analysis run on current customers is survivorship bias by construction. No sampling technique repairs a frame problem; only naming it does. Write one sentence at the top of the analysis: “sampled from X, so conclusions apply to X.”

Four habits, then: random not first, entities not rows, stratify-and-reweight for rare things, and state the frame out loud. Ten extra minutes at the start of the analysis, purchased against weeks of building on a foundation that was never the population you cared about.