Data leakage is why your model looks too good

Here is a rule that will save you repeated embarrassment: when a model performs suspiciously well, your first hypothesis should be leakage, not genius. Data leakage means information about the target sneaking into the features — the model isn’t predicting the outcome, it’s reading it off a mirror.

The insidious part is that leakage produces excellent offline metrics. Nothing looks broken. The model then meets reality, where the leaked information doesn’t exist at prediction time, and performance collapses.

The classic leaks

Post-outcome features. Predicting loan default with a feature like number_of_collection_calls. Collection calls happen because the customer defaulted. The feature is a consequence of the target dressed up as a predictor. Any feature whose value is set after the outcome occurs is poison.

This one is easy to reproduce. The frame below is built from the public dataset: one row per subscription, the label is whether it was ever cancelled, and the features count what the customer did in their first thirty days. Then one extra column gets added — days since the customer’s last event, measured today.

import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
y = d.cancelled_at.notna().astype(int)          # 52.8% of 1,198 subscriptions

# The leak: recency measured now, long after any cancellation happened
last = events.groupby("customer_id").occurred_at.max()
d["days_since_last_event"] = (pd.Timestamp("2026-07-31")
                              - d.customer_id.map(last)).dt.days

honest = ["monthly_price", "view", "login", "export", "purchase"]
X = d[honest]

for name, cols in [("honest", honest), ("leaky", honest + ["days_since_last_event"])]:
    X_tr, X_te, y_tr, y_te = train_test_split(d[cols], y, test_size=0.25,
                                              random_state=0, stratify=y)
    m = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
    print(f"{name:7s} AUC {roc_auc_score(y_te, m.predict_proba(X_te)[:, 1]):.3f}")

honest  AUC 0.592
leaky   AUC 0.843

A quarter of a point of AUC from one column, and the column looks entirely innocent — recency is a standard churn feature. The problem is when it was measured. A customer who cancelled eighteen months ago stopped generating events eighteen months ago, so the feature is mostly reading the cancellation date back out. At prediction time, for a customer who has not cancelled yet, that number does not exist in the same form.

Preprocessing on the full dataset. Fitting a scaler, imputer, or feature selector on all the data before splitting into train and test. The test rows influenced the transformation, so the test set is no longer unseen. The fix is mechanical — do every fitted transformation inside the training fold only, which is what a Pipeline exists to guarantee:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])
# The pipeline refits the scaler inside each fold — no leakage
scores = cross_val_score(pipe, X, y, cv=5, scoring="roc_auc")
print(f"{scores.mean():.3f}")        # 0.653

If your preprocessing lives in a pipeline, cross-validation handles this automatically. If it lives in a notebook cell above the split, it probably leaks.

Duplicate or near-duplicate rows across the split. The same customer, the same document with trivial edits, the same image at two crops — one lands in train, one in test. The model memorises instead of generalising, the metric flatters. Split by entity (customer, patient, document family), not by row — that is what GroupKFold is for:

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=cv, groups=d.customer_id,
                         scoring="roc_auc")
print(f"{scores.mean():.3f}")        # 0.654

In the subscription file, 216 of the 1,198 rows belong to a customer who holds another subscription — 103 customers in total. Grouping by customer moves the score from 0.653 to 0.654, which is to say it does nothing measurable here. That is the normal result and it is not an argument against the group split: the cost is one keyword, and the case it protects against is the one where the same entity supplies half the rows.

Temporal leakage. Random splits on time-ordered data let the model train on the future and predict the past. Real deployment only ever runs in one direction. Use a time-based split: train on everything before a cutoff, evaluate after it.

Target encoding done naively. Replacing a category with the mean of the target for that category, computed over all rows, injects the target directly into the features. It must be computed out-of-fold.

Worked leak 1: the target encoding that hands over the label

That last one sounds mild written down. It isn’t, and the size of the effect can be worked out on paper before writing any code.

Take a fraud model with a merchant_id column. Large merchants have thousands of transactions; the long tail has a handful each. Encode each merchant with the mean of the target over all its rows, and consider a merchant with exactly two transactions. Its encoded value is (y_i + y_j) / 2, and that value is attached to both rows. With a 50/50 target it takes three values: 0 when both transactions are clean, 1 when both are fraud, and 0.5 otherwise. So a quarter of the time the feature says “this row is 0”, a quarter of the time it says “this row is 1”, and half the time it says nothing. Predicting the label from the encoding alone gives 0.25 + 0.25 + (0.5 × 0.5) = 0.75 accuracy — on data containing no signal whatsoever.

The demonstration, with a target that is a literal coin flip:

import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
n = 40_000
y = pd.Series((rng.random(n) < 0.5).astype(int))
merchant = pd.Series(np.repeat(np.arange(n // 2), 2))   # exactly 2 rows each

# The bug: encode the category with the target mean over ALL rows.
leaky = merchant.map(y.groupby(merchant).mean())

# The fix: leave-one-out, so a row never sees its own label.
sums = merchant.map(y.groupby(merchant).sum())
honest = sums - y                                        # the other row's label

for name, f in [("leaky", leaky), ("honest", honest)]:
    s = cross_val_score(DecisionTreeClassifier(max_depth=3, random_state=0),
                        f.to_frame("merchant_te"), y, cv=5)
    print(f"{name:7s} {s.mean():.3f}")

The leaky version lands at 0.75. The honest version lands at 0.50, which is the truth — the labels are coin flips and there is nothing to learn. Twenty-five points of accuracy, conjured out of one line of feature engineering, on a dataset with zero signal in it.

Real data hides this better, because the encoding sits among features that do work. The score doesn’t jump to something absurd; it drifts up by a few points and everyone congratulates the person who added it. The rule that catches it: any statistic computed from the target must be computed out-of-fold, without exception, including smoothed and Bayesian variants.

Worked leak 2: a rolling window that includes today

Now a leak that has nothing to do with the target column and everything to do with time.

Predicting whether a machine trips in the next hour. Someone builds a “recent trip rate” feature: the mean of the trip flag over a short window. The intent is the last five readings. The code says rolling(5), and pandas windows are trailing including the current row — so the feature for hour t contains one-fifth of the answer for hour t. In production the answer for hour t is what you are trying to predict, so the feature cannot exist. Offline it exists, and a random K-fold split never notices.

Put the two windows on a timeline and the leak is one cell wide:

Each cell is one hourly reading. The honest window ends where the model runs; the leaky one reaches one cell past it, into the hour whose outcome is the thing being predicted. Nothing in the code looks wrong, and a random split never crosses that line.

import numpy as np, pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(1)
n = 30_000
y = pd.Series((rng.random(n) < 0.2).astype(int))   # trips, independent of the past

leaky = y.rolling(5, min_periods=1).mean()                       # includes today
honest = y.shift(1).rolling(5, min_periods=1).mean().fillna(0.2)  # past only

for name, f in [("leaky", leaky), ("honest", honest)]:
    s = cross_val_score(HistGradientBoostingClassifier(random_state=0),
                        f.to_frame("trip_rate_5"), y, cv=5, scoring="roc_auc")
    print(f"{name:7s} {s.mean():.3f}")

The honest feature scores 0.50, and 0.50 is correct: the trips were generated as independent coin flips, so no past information predicts them. The leaky feature scores clearly above 0.50 — the model is reading a fifth of today’s label out of today’s feature. One character, shift(1), is the entire difference.

Note how the inflation scales. A five-step window leaks 20% of the current label and produces an obvious bump. A ninety-day window leaks about 1% and produces a small, believable improvement — which is worse, because a small believable improvement gets shipped. The same shape appears in df.groupby("machine")["temp"].transform("mean"), a per-entity average over the machine’s whole history including the readings taken after the failure. It looks like an innocent normalisation. It is a time machine.

How to audit for leakage

• Interrogate your best features. Look at feature importances and ask, for each top feature: when does this value get written, and by what process? If the honest answer involves the outcome, it leaks.
• Be suspicious of step changes. A metric jumping from 0.74 to 0.97 after adding one feature is not a breakthrough; it’s a subpoena.
• Re-run with a time-based split. If performance drops sharply versus the random split, something is flowing backwards through time.
• Trace one prediction end-to-end. Take a single row from the test set and reconstruct, by hand, whether every feature value would have been knowable at prediction time.

Where the suspicion should stop

The rule is not “drop any feature that correlates strongly with the target” — that would delete the model. Teams that get burned once often overcorrect and start purging good features on vibes, which costs real accuracy and is just as hard to argue with.

The test is narrower and answerable: for this entity, at the moment the model actually runs in production, is this value already written? A feature can look downstream in the warehouse and still be perfectly legitimate. If scoring runs as an overnight batch, yesterday’s aggregates are available and using them is correct, even though they were computed after some of the events they summarise. If a payment processor genuinely returns a risk flag before authorization, that flag is a fair feature regardless of how much it correlates with fraud.

Two more places the alarm should stay quiet. Using future data to construct labels is not leakage — deciding today that a loan issued last year defaulted is how the label gets made. And a deliberately optimistic upper-bound experiment is fine, as long as the number is reported as an upper bound and never as the model’s score.

The cultural fix

Leakage isn’t a rookie mistake; it’s the default outcome of assembling training data casually, and it happens in experienced teams. The protection is procedural: pipelines for anything fitted, entity-aware and time-aware splits, and a standing team norm that great results get audited before they get celebrated. Give that last one an owner rather than leaving it to culture: a named reviewer, the leakage types above as a checklist, and a rule that no headline number reaches a deck until someone other than its author has re-run the split.