Always build the dumb baseline first

Before any model earns a place in your project, spend an hour building something deliberately dumb: predict the majority class, predict yesterday’s value, predict the per-segment average, or encode the one rule the domain expert recites when asked how they’d guess. Then measure it exactly the way you’ll measure the real model.

This feels like a waste of an hour. It’s the highest-ROI hour in the project.

What the dumb baseline buys you

A denominator for every future claim. “Our model achieves 0.83 AUC” means nothing in isolation. “The always-guess-the-average baseline scores 0.79, we score 0.83” is an honest sentence — and often an uncomfortable one, which is precisely its value. Improvement over trivial is the only number that measures what your modelling work actually contributed.

An end-to-end skeleton on day one. Building even a constant-prediction baseline forces every unglamorous decision early: what exactly is the target, at what moment is the prediction made, which rows count, how is the metric computed, where do results get written. Teams that skip this discover in week six that they disagree about the definition of churn.

A leakage tripwire. If a “dumb” baseline scores suspiciously well, you’ve learned something important about your setup before wasting a month: maybe the target leaks into a feature, maybe the split is wrong, maybe the problem is easier than anyone thought and doesn’t need ML at all. All three are things you want to know on day two, not at the launch review.

A shipping fallback. Sometimes the baseline is good enough to deploy while the real model is developed — and occasionally it’s good enough that the real model never needs to exist. That’s not failure. That’s a cheap win and an honest resume of the problem’s difficulty.

What “dumb” looks like in practice

For classification and regression, scikit-learn ships them as DummyClassifier and DummyRegressor:

from sklearn.dummy import DummyClassifier, DummyRegressor

DummyClassifier(strategy="most_frequent")   # majority class
DummyClassifier(strategy="stratified")      # guess by class frequency
DummyRegressor(strategy="median")           # constant median

For forecasting: predict the last observed value, or the value from one season ago. For ranking and recommendation: most-popular-item. For churn: “everyone who did X in the last 30 days stays”. And always include the strongest non-ML candidate — the current human process or rule of thumb, because that’s the real incumbent your model must beat to justify existing.

One level up sits the simple model baseline — logistic regression on five obvious features — which tells you how much signal is easy. The gap between dumb → simple → complex is a map of where the value is.

Here is that whole ladder on the public dataset, predicting whether a subscription is ever cancelled from what the customer did in their first month:

import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, accuracy_score

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
y = d.cancelled_at.notna().astype(int)
X = d[["monthly_price", "view", "login", "export", "purchase"]]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
                                          random_state=0, stratify=y)

logreg = lambda: Pipeline([("scale", StandardScaler()),
                           ("clf", LogisticRegression(max_iter=1000))])

ladder = [("dumb    (majority)", DummyClassifier(strategy="most_frequent"), X.columns),
          ("rule    (price only)", logreg(), ["monthly_price"]),
          ("simple  (logistic)", logreg(), X.columns),
          ("complex (boosting)", HistGradientBoostingClassifier(random_state=0), X.columns)]

for name, model, cols in ladder:
    model.fit(X_tr[list(cols)], y_tr)
    probs = model.predict_proba(X_te[list(cols)])[:, 1]
    preds = model.predict(X_te[list(cols)])
    print(f"{name:22s} AUC {roc_auc_score(y_te, probs):.3f}   "
          f"accuracy {accuracy_score(y_te, preds):.3f}")

dumb    (majority)     AUC 0.500   accuracy 0.528
rule    (price only)   AUC 0.640   accuracy 0.625
simple  (logistic)     AUC 0.690   accuracy 0.619
complex (boosting)     AUC 0.589   accuracy 0.567

Read it top to bottom. The plan price alone carries 0.14 of the 0.19 total AUC gain — most of the signal is “cheap plans churn”. Four behavioural counts add 0.05 more. And the boosted model, given the same five columns, gives back 0.10 against the logistic regression, because 838 training rows and five weak features is not a regime where extra capacity has anything to spend itself on.

Without the first row, “AUC 0.589” would have sounded like a result. Without the second, the four event-count features would have looked like the reason the model works. Both readings are wrong, and one afternoon of baselines is what separates them. Past the point where the ladder flattens, extra capacity buys variance, not accuracy.

The cultural part

The hard part isn’t technical — a baseline takes an hour. The hard part is that baselines are deflating. Nobody gets promoted for “seasonal naive was nearly as good.” So teams skip them, and then nobody in the room can answer the only question that matters: better than what?

Make it a norm instead: no model result gets presented without its baseline in the same table. It keeps the team honest, catches broken setups early, and occasionally saves months by revealing that the problem didn’t need the machinery at all. Most teams that skip it have a story about the quarter they spent beating a number the mean could have produced.

One caveat, because it comes up often: a baseline that beats the model is not always a verdict on the model. It can mean the evaluation is wrong — a leaky split, a metric that rewards the wrong thing, a test set drawn from a different period. Check the setup before you check the algorithm. In practice that means three things, in order: re-run the split and confirm no row appears on both sides, re-read the metric definition against what the business cares about, and check that the test period sits after the training period on the calendar. If all three come back clean, the model is the problem.