Backtesting a forecast honestly

A demand model reports a mean absolute error of about four units in the notebook. It ships. Three weeks later the planners say it is worse than the spreadsheet it replaced, and nobody has touched the code. Nothing broke. The backtest was scoring a task the model will never be asked to do.

Backtesting is the part of forecasting where the work is not the model. It is the simulation: rebuilding, at every point in the past, the exact situation the model will be in on a Monday morning, and refusing to give it anything it would not have had.

A random split is meaningless on ordered data

train_test_split with shuffle=True puts March and May in training and April in test. The model interpolates. It has seen both sides of the gap and the series is autocorrelated, so filling the middle is close to trivial. In production there is no other side. Every prediction is extrapolation from a hard edge.

The split defines the question being asked. A shuffled split asks: given the weeks around it, what happened in this week? Nobody ever needs that answer. It also quietly duplicates information across the boundary — the same promotion, the same outage, the same seasonal peak sitting in training and test at once.

The fix is not a better metric. It is a split with the arrow of time intact.

Rolling origin, and the four choices inside it

Train on everything up to a cut point, forecast the next h steps, score against what actually happened, slide the cut point forward, repeat. That is rolling-origin evaluation, also called walk-forward; scikit-learn’s TimeSeriesSplit is the same idea with the origins chosen for you.

import numpy as np
import pandas as pd

def origins(n, first, horizon, step):
    """Cut points, arrow of time intact: train [0, t), score [t, t+h)."""
    t = first
    while t + horizon <= n:
        yield t
        t += step

rows = []
for t in origins(len(y), first=365, horizon=14, step=14):
    history, actual = y.iloc[:t], y.iloc[t:t + 14]
    forecast = fit_and_forecast(history, horizon=14)   # refit at each origin
    rows.append(np.abs(actual.to_numpy() - forecast))

errors = pd.DataFrame(rows, columns=range(1, 15))
print(errors.mean())          # error per step of the horizon
print(errors.mean().mean())   # the single number — report it last

Four decisions are hiding in those arguments.

Horizon. Covered below; it is not a modelling choice.

Step. A step equal to the horizon gives non-overlapping test windows and roughly independent errors. A step of one gives many more origins, but their test windows overlap heavily, so the extra origins are not extra evidence — the average gets smoother without getting more trustworthy.

Number of origins. Each origin is one noisy sample of a bad week or a good one. A handful of origins tells you almost nothing about a seasonal series. Enough origins to cover at least a full seasonal cycle of cut points is the minimum worth reporting, and the spread across origins matters as much as the mean: a model that is excellent at eight origins and catastrophic at two is not a good model.

Expanding or sliding window. Expanding uses all history at every origin. Sliding keeps the training window a fixed length, so every origin trains on the same amount of data and the early origins are not handicapped. Sliding is also closer to what a retraining job does in production.

Which points at the choice people forget: the backtest should retrain on the same cadence production does. Refitting at every origin when the real job refits monthly measures a model nobody will run.

The horizon belongs to the decision, not the model

Ask what the forecast is for. If it feeds a purchase order placed with a fourteen-day lead time, the honest score is the error at day fourteen. One-step accuracy is irrelevant to that decision, and it is the number that flatters most, because tomorrow is mostly today.

So score the whole horizon, not the first step. errors.mean() above is the useful output: day-1 error and day-14 error are different products, and a model can win at one and lose at the other. Averaging them into a single figure hides exactly the comparison the planner needs.

If the decision is asymmetric — running out costs more than holding stock — a point forecast is under-specified anyway, and the backtest should score quantiles with pinball loss rather than a mean.

The baselines that must be beaten

Three forecasts cost nothing and must be beaten before a model earns a review:

def naive(history, h):
    return np.repeat(history.iloc[-1], h)

def seasonal_naive(history, h, period=7):
    return np.resize(history.iloc[-period:].to_numpy(), h)

def drift(history, h):
    v = history.to_numpy()
    slope = (v[-1] - v[0]) / (len(v) - 1)
    return v[-1] + slope * np.arange(1, h + 1)

Run them through the same loop, at the same origins, over the same horizon. Then report a skill score — 1 - mae_model / mae_seasonal_naive — so the headline is how much the model adds, not how large the numbers are.

Seasonal naive wins more often than anyone expects on business series with a strong weekly shape, and it wins outright at short horizons often enough that skipping it, which is the easiest of the classic forecasting self-deceptions to fix, is how teams end up maintaining a gradient-boosted pipeline that adds a rounding error. When the margin is thin, that is the finding. Shipping seasonal naive with a monitoring page is a legitimate outcome.

What the backtest knew and the model would not have

A rolling origin gets the timing right and can still leak, because leakage enters through the features rather than the split. Three routes:

Anything fitted on the whole series. A scaler, a target encoding, a global mean, a trend fitted on all the data, an outlier filter tuned on the full range. Each writes the future into every training row. Every fitted object belongs inside the loop, fitted on history only — the general case of why a model that looks great in evaluation is usually reading its own answers.

Aggregates derived from the target. A “customer lifetime value” column, a segment label built from full-period revenue, a rolling mean with default centring. Centred windows average future values into the present, and the default is easy to miss.

Revised data. This is the quiet one. Warehouses restate: sales get corrected, refunds land late, a customer is marked churned with a backdated timestamp, an external index is republished. The backtest reads today’s corrected table. The model in production reads the first print. Where the warehouse keeps snapshots, train on the vintage as it stood at each origin. Where it does not, the backtest is optimistic by an unknown amount and the report should say so rather than quote a number to two decimals.

The test that catches all three is one question, asked at every origin: on that morning, what was actually in the database? Not what is in it now. A backtest is worth its number only when the answer is the same both times.