Time series forecasting: the five classic self-deceptions

Forecasting is the branch of data science where it’s easiest to accidentally cheat, because the thing you must not touch — the future — is sitting right there in your dataframe. Five self-deceptions account for most inflated forecasting results.

1. Validating on shuffled time

A random train/test split on time series lets the model interpolate: it has seen March and May, so April is easy. Deployment is pure extrapolation — the model has seen through today and must predict tomorrow. Always evaluate with the arrow of time intact, ideally as a rolling origin: train up to time t, forecast the next horizon, slide forward, repeat — which is exactly what TimeSeriesSplit produces.

from sklearn.model_selection import TimeSeriesSplit
for train_idx, test_idx in TimeSeriesSplit(n_splits=5).split(X):
    model.fit(X.iloc[train_idx], y.iloc[train_idx])
    evaluate(model, X.iloc[test_idx], y.iloc[test_idx])

Averaging across several origins matters too: any single split may straddle a promotion, an outage, or a season and mislead in either direction.

2. Skipping the embarrassing baselines

Two forecasters must be beaten before any model earns attention:

• Naive: tomorrow equals today — the forecast you get from a memoryless random walk.
• Seasonal naive: next Monday equals last Monday.

On many business series — demand, traffic, revenue — seasonal naive is brutally competitive. Both baselines below run on the public dataset, as daily event counts, over twelve rolling origins and a 28-day horizon.

import numpy as np
import pandas as pd

events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])
y = events.set_index("occurred_at").resample("D").size()

h, n_origins = 28, 12
rows = []
for i in range(n_origins):
    cut = len(y) - (n_origins - i) * h
    train, test = y.iloc[:cut], y.iloc[cut:cut + h]
    naive = np.repeat(train.iloc[-1], h)                 # tomorrow = today
    snaive = np.resize(train.iloc[-7:].to_numpy(), h)    # = same weekday
    rows.append((np.abs(test.to_numpy() - naive).mean(),
                 np.abs(test.to_numpy() - snaive).mean()))

mae = pd.DataFrame(rows, columns=["naive", "seasonal_naive"])
print(mae.mean().round(2))
print("seasonal naive wins at", (mae.seasonal_naive < mae.naive).sum(),
      "of", n_origins, "origins")

naive             32.53
seasonal_naive    21.39
dtype: float64
seasonal naive wins at 10 of 12 origins

Copying last week beats copying yesterday by a third of the error, on eight lines of code and no features. That 21.39 is the number a model has to beat, and it is the one nobody computes. If your gradient-boosted, feature-rich model beats it by 2%, that’s your honest headline, and sometimes the correct decision is to ship the naive forecast with a monitoring page. Skill scores (error relative to the naive baseline) keep everyone honest about how much the model is actually contributing.

3. Features that quietly contain the future

Lag features are safe. Rolling means are safe if the window ends strictly before the forecast time. The traps:

• A rolling window computed with center=True, which averages future values into the feature.
• Joining external data (weather, prices, holidays-with-adjustments) using its event date instead of its availability date. Actual weather for Tuesday is only known on Wednesday; if the model needs Tuesday’s weather to forecast Tuesday, production must feed it a weather forecast, and training should use archived forecasts, not observed actuals.
• Any feature derived from the whole series — a global mean, a trend fitted on all data — which leaks the future into every training row.

The audit question is always the same: on the morning I make this forecast, what exactly is in the database?

4. Metrics that flatter

MAPE explodes near zero actuals and rewards under-forecasting (an error capped at 100% on the downside, unbounded on the upside). Training on log(y) and reporting errors in log space makes everything look small; exponentiating back without care biases forecasts low. Prefer MAE or RMSE in original units, supplemented by a skill score against seasonal naive, and report performance by horizon — day-1 error and day-14 error are different products.

5. One-step scores for multi-step decisions

A model that looks excellent at one-step-ahead can be useless at the horizon the business plans on. If you forecast fourteen days out by feeding predictions back in as inputs, errors compound — evaluate the whole fourteen-day trajectory, not the first step. And if the decision is inventory or staffing, a point forecast alone is under-specified: the cost of forecasting 10 too high is not the cost of 10 too low. Quantile forecasts (P10/P50/P90) map directly onto those asymmetric costs, and pinball loss evaluates them properly.

The theme

Every one of these pitfalls is a version of the same error: evaluating under conditions kinder than deployment. The fix is a standing rule — simulate the deployment loop in the evaluation: real cutoffs, only-then-available data, the real horizon, costs in real units, always compared to the naive forecast. Written as code, that is a loop over forecast origins with an as-of filter applied at each one. Most teams already have the loop. The filter is the part that has to be added.