Start with the simplest model that could possibly work

The rule is short: start with the simplest thing that could possibly work, and add complexity only when the simple thing demonstrably isn’t enough. The exception is real but narrow — problems where the raw inputs carry structure a simple model cannot see, such as images, audio, or free text. Almost everything that arrives as a business table falls under the rule, and the pull to skip it comes from tutorials, job postings and portfolios rather than from the problem.

What simple-first actually buys you

A baseline that keeps everyone honest. If logistic regression on five obvious features already gets you 90% of the way, every fancier model must justify itself against that number — in accuracy, but also in training cost, serving cost, latency, and explainability. Without the simple model, “the neural network achieves 0.84” floats free of any context. With it, the conversation becomes “0.84 versus 0.82 for a model we can deploy in an afternoon and explain to compliance” — which is the actual decision.

Debuggability when things go wrong. When a linear model is wrong, you can inspect the coefficients, find the feature dragging predictions sideways, and usually trace it to a data problem in an hour. When a deep model is wrong, the investigation starts with “well…” — and data problems are the usual culprit either way. Simple models are transparent windows onto your data; complex models are mirrors that reflect your assumptions back at you with confidence.

Speed of iteration where it matters. Early in a project, the bottleneck is never model capacity — it’s understanding: of the target definition, the leakage risks, the features worth building. A model that retrains in ten seconds lets you test ten hypotheses about the data before lunch. A model that trains in four hours converts every little question into a scheduling problem.

A realistic estimate of the problem’s difficulty. The gap between the dumb baseline, the simple model, and the complex model is a map. If simple gets 0.82 and complex gets 0.84, the problem’s signal was mostly easy — invest in features and data quality. If simple gets 0.60 and complex 0.83, there’s real structure that needs capacity. You can’t read that map if you never ran the early points.

What “simple” means in practice

Not naive — transparent and cheap. Reasonable first models: a rules baseline written with the domain expert, then linear or logistic regression with sensible regularisation on a handful of interpretable features. For most tabular problems the escalation path after that is short: gradient-boosted trees, which are still cheap and nearly as inspectable with modern tooling. The leap to deep learning is justified when the data stops being tabular — text, images, audio, sequences — or when a measured gap says the simpler family has hit its ceiling.

Both rungs below run on the public dataset — 1,198 subscriptions, predicting whether each was ever cancelled from its price and four counts of what the customer did in their first thirty days.

import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
y = d.cancelled_at.notna().astype(int)
X = d[["monthly_price", "view", "login", "export", "purchase"]]

simple = make_pipeline(StandardScaler(),
                       LogisticRegression(max_iter=1000))
complex_ = HistGradientBoostingClassifier(random_state=0)

for name, model in [("simple  (logistic)", simple),
                    ("complex (boosting)", complex_)]:
    auc = cross_val_score(model, X, y, cv=5, scoring="roc_auc").mean()
    print(f"{name}  AUC {auc:.3f}")

simple  (logistic)  AUC 0.653
complex (boosting)  AUC 0.574

Five features, three lines of modelling, and the boring model wins by eight points of AUC. Boosting has capacity to spare and spends it memorising a 1,198-row table; there is not enough signal here to need it. That is not a rare result on a small business table, and it is only visible because the simple rung was run. Whatever ships in the end, the 0.653 is the denominator that makes its story honest — and the dumb baseline below it is the denominator for that.

The escalation discipline

Complexity must be pulled by evidence, never pushed by enthusiasm. The checklist before each escalation is short: the simpler model’s errors have been examined (are they even fixable by capacity, or are they label noise?); the metric gap is larger than the seed-to-seed noise; and the operational price — latency, memory, retraining cost, explainability — has been priced in. If those three don’t clearly pass, the honest conclusion is that the boring model won.

A lot of senior data science is just this discipline: not using a sledgehammer until you’ve confirmed the problem is actually a wall. Boring, and almost always right — and on the occasions the wall is real, you’ll swing the sledgehammer knowing exactly why.