The bootstrap: uncertainty for people who hate formulas

Every number you report — a conversion rate, a median, a model’s AUC — was computed on one particular sample. Rerun history with different customers and you’d get a different number. The question “how different?” is the entire subject of uncertainty, and most practitioners either skip it or reach for a t-test formula that doesn’t fit their statistic.

The bootstrap — Efron’s 1979 idea — answers it with brute force instead of formulas: pretend your sample is the population, redraw samples from it, and watch how much the statistic wobbles. Below it is written out longhand with NumPy’s random generator, because seeing the loop is the point:

The amounts come from the public dataset — every purchase event in the file, 2,493 of them.

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)

def bootstrap_ci(data, stat_fn, n_boot=10_000, alpha=0.05):
    n = len(data)
    stats = np.array([
        stat_fn(data[rng.integers(0, n, n)])   # resample WITH replacement
        for _ in range(n_boot)
    ])
    return np.quantile(stats, [alpha / 2, 1 - alpha / 2])

events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])
purchases = events[events.event_type == "purchase"]
revenue = purchases.amount.to_numpy()

print(np.median(revenue), bootstrap_ci(revenue, np.median).round(2))

29.58 [28.17 30.61]

No formula appears anywhere in it. Each resample draws n rows with replacement (some rows repeat, some are absent — that’s the point), the statistic is recomputed, and the middle 95% of the resulting distribution is your confidence interval. SciPy packages the same idea as scipy.stats.bootstrap, with BCa intervals included, once you want the tuned version.

Why this beats the formula you half-remember

The classical mean ± 1.96·se interval borrows its shape from the central limit theorem, and it is fine for means of well-behaved data. But real questions are rarely about means of well-behaved data. They’re about medians, P95 latency, ratios of sums, retention-curve differences, a model’s AUC — statistics with no clean textbook interval, computed on skewed distributions. The bootstrap doesn’t care: if you can compute the statistic, you can bootstrap it. One function replaces a shelf of formulas, and the skewness of your revenue distribution shows up honestly as an asymmetric interval instead of being flattened by a symmetry assumption.

The rules that keep it honest

Resample the unit of independence. The bootstrap assumes rows are independent. If you have 10,000 events from 500 users, resampling events pretends you have 10,000 independent observations — intervals come out flattering and false. Resample users, keeping each user’s events together (the cluster bootstrap). Same for time series: naive row resampling destroys autocorrelation; use block bootstrap variants, or reconsider.

The interval printed above breaks that rule, and the correct version says so:

# resample customers, keeping each customer's purchases together
by_customer = purchases.groupby("customer_id").amount.apply(np.asarray).to_numpy()

def cluster_ci(groups, stat_fn, n_boot=10_000, alpha=0.05):
    m = len(groups)
    stats = np.array([
        stat_fn(np.concatenate(groups[rng.integers(0, m, m)]))
        for _ in range(n_boot)
    ])
    return np.quantile(stats, [alpha / 2, 1 - alpha / 2])

print(len(by_customer), cluster_ci(by_customer, np.median).round(2))

808 [27.37 31.42]

The 2,493 purchases came from 808 customers, and the busiest bought twenty-nine times. Resampling purchases treated those twenty-nine rows as twenty-nine independent draws; resampling customers treats them as one. The honest interval is 4.05 wide against 2.44 — two-thirds wider, from the same data and the same statistic. Nothing about the median changed. What changed is the claim about how much the file knows.

Comparisons: bootstrap the difference. To compare variants, resample each group and record the difference each time. If the interval for B−A excludes zero, you have evidence of a real difference — and, unlike a bare p-value, an interval that says how large it plausibly is.

def diff_of_medians(a, b, n_boot=10_000):
    diffs = [np.median(rng.choice(b, len(b))) - np.median(rng.choice(a, len(a)))
             for _ in range(n_boot)]
    return np.quantile(diffs, [0.025, 0.975])

Model metrics too. Bootstrap the test set (resample rows, recompute AUC) and you’ll discover that your 0.86 is really 0.86 ± 0.02 — which reframes that 0.005 “improvement” you were about to celebrate as noise. Test-set intervals should accompany any model comparison where the decision matters.

What the bootstrap cannot do

It quantifies sampling uncertainty only. It knows nothing about bias in how the sample was collected, confounding, leakage, or the future differing from the past — garbage in, confidently-intervaled garbage out. Tiny samples (a few dozen rows) strain the it’s-the-population pretense. And with heavy tails, use 10,000+ resamples and prefer the percentile-of-differences forms shown here over fancier variants until you need them.

The habit to build: any number that will influence a decision travels with an interval. The bootstrap makes that cheap enough to be a default — thirty seconds of compute to find out whether your finding is a finding.