Cross-validation without fooling yourself

Cross-validation has one purpose: to estimate how your model will perform on data it hasn’t seen, under the conditions it will actually face. The second half of that sentence is where people go wrong. Vanilla k-fold answers “how well does this model do on a random sample of the same data?” — which is only the right question if production will also hand you random samples of the same data. It usually won’t.

The mechanics are simple enough to draw. Cut the rows into k parts, then train k times, holding out a different part each round.

Five rounds, five scores, one average. The teal block never appears in the data the model saw for that round — which is the entire guarantee cross-validation offers, and it only holds if the rows really are independent.

Match the split to the deployment reality

Rows that share an entity must share a fold. If the same customer appears in train and validation, the model partly memorises customers instead of learning patterns, and your estimate inflates. Same for repeated measurements of one patient, multiple photos of one product, chunks of one document. GroupKFold keeps each entity whole:

The three splits below run against the public dataset — 1,198 subscriptions, one row each, predicting whether the subscription was ever cancelled from what the customer did in their first thirty days.

import numpy as np
import pandas as pd
from sklearn.model_selection import GroupKFold, TimeSeriesSplit, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

df = subs.set_index("subscription_id").join(counts).sort_values("signed_up_at")
df[counts.columns] = df[counts.columns].fillna(0)

y = df.cancelled_at.notna().astype(int)
X = df[["monthly_price", "view", "login", "export", "purchase"]]
pipe = Pipeline([("scale", StandardScaler()),
                 ("clf", LogisticRegression(max_iter=1000))])

# Grouped: no customer straddles train and validation
gkf = GroupKFold(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=gkf, groups=df["customer_id"],
                         scoring="roc_auc")
print(np.round(scores, 3), f"{scores.mean():.3f}")

[0.652 0.664 0.66  0.646 0.65 ] 0.654

103 of those customers hold more than one subscription, covering 216 rows. Grouping them costs one keyword and, on this file, changes the mean by a thousandth. That is the usual outcome — the split is insurance, and insurance that pays out is the bad case, not the good one.

Time flows one way. If the model will predict the future, validate it predicting the future. TimeSeriesSplit trains on an expanding past window and tests on the next slice. A random shuffle on temporal data is quietly training on tomorrow to predict yesterday — scores go up, meaning goes away.

tscv = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=tscv, scoring="roc_auc")
print(np.round(scores, 3), f"{scores.mean():.3f}")

for i, (train, test) in enumerate(tscv.split(X), start=1):
    print(f"fold {i}  test rate {y.iloc[test].mean():.3f}  "
          f"signed up to {df.signed_up_at.iloc[test].max().date()}")

[0.725 0.667 0.687 0.702 0.591] 0.675
fold 1  test rate 0.729  signed up to 2025-04-08
fold 2  test rate 0.533  signed up to 2025-09-02
fold 3  test rate 0.497  signed up to 2025-12-23
fold 4  test rate 0.452  signed up to 2026-03-28
fold 5  test rate 0.221  signed up to 2026-07-31

Look at the second column before the first. The share of cancelled subscriptions falls from 72.9% to 22.1% across the folds, and almost none of that is the business improving — the last cohort signed up weeks before the file ends, so most of them have not had time to cancel yet. The label is censored, and it gets more censored the later the fold. The scores in the first column are being computed against five different questions.

The general principle: your validation split should simulate the gap between training and serving. New customers? Group by customer. Next quarter? Split by time. New hospitals? Leave whole hospitals out. If the estimate matters, the split design matters more than the model choice.

The selection trap: reusing the same folds too many times

Run 200 configurations through the same 5-fold CV and pick the best score, and that best score is biased upward — you’ve partly selected for configurations that got lucky on those particular folds. This is overfitting one level up, and it’s why “CV said 0.87, production says 0.82” is such a common story.

Practical defenses, in increasing order of rigor:

• Keep a final holdout set that is touched exactly once, after all selection is finished. Its number is the one you report.
• Use nested CV when you need an honest estimate of the whole select-and-tune procedure, not just the winning model.
• Simply run fewer experiments with more intention. The bias grows with the number of things you tried.

Read the variance, not just the mean

Five fold scores are five numbers, and the spread is telling you something.

plain = cross_val_score(pipe, X, y, cv=5, scoring="roc_auc")
group = cross_val_score(pipe, X, y, cv=gkf, groups=df["customer_id"],
                        scoring="roc_auc")
for name, s in [("plain", plain), ("grouped", group)]:
    print(f"{name:8s} {s.mean():.3f} ± {s.std():.3f}  {np.round(s, 3)}")

plain    0.653 ± 0.053  [0.747 0.651 0.619 0.657 0.588]
grouped  0.654 ± 0.007  [0.652 0.664 0.66  0.646 0.65 ]

Same model, same rows, same mean to a thousandth — and eight times the spread. cross_val_score uses StratifiedKFold for classifiers and does not shuffle, so on a frame sorted by signup date the plain folds are contiguous cohorts, and the first one scores 0.747 because it is the oldest customers. The grouped split shuffles the entities, so its folds look alike. Quoting “0.65” from either run is fine; quoting the spread from the wrong one is how a stable model gets called unstable.

A model at 0.85 ± 0.01 and a model at 0.86 ± 0.06 are not “0.85 vs 0.86”. The second one’s advantage may be fold luck, and its instability is itself a defect — it suggests performance depends heavily on which slice of data it sees. When two models are within each other’s noise, prefer the simpler, stabler one and stop burning weeks on the difference.

Also look at which folds are bad. One catastrophic fold often means a data problem — a segment the model can’t handle, a time period with a schema change — and that’s more actionable than any average.

A short checklist

1. Does any entity appear on both sides of a split? Fix with groups.
2. Could any feature or transformation see the future or the target? Fix with pipelines and time-aware splits.
3. How many times have these folds judged a decision? Keep an untouched holdout.
4. Report mean and spread; investigate the worst fold.

The cheapest place to apply that checklist is before the model, not after. Write the split rule and the reason for it into the same commit as the first cross_val_score call — one sentence, in a comment — so the next person can see which deployment condition each fold was built to imitate.