Class imbalance: move the threshold before you reach for SMOTE

Ask the internet about class imbalance and you’ll be handed SMOTE within two paragraphs. Synthetic minority oversampling is the most cited and least necessary tool in the imbalance toolbox. Most “imbalance problems” are actually two simpler problems wearing a trench coat: the wrong metric and the default threshold.

First: is there a problem at all?

Train your model on the raw imbalanced data and look at the precision-recall curve. Modern gradient boosting and logistic regression tolerate imbalance far better than the folklore suggests, provided they see enough absolute positive examples. Rare-but-plentiful (2% of ten million rows = 200k positives) is a comfortable regime. Rare-and-few (2% of five thousand rows) is a genuinely hard small-data problem, and no resampling trick manufactures information you don’t have.

The default threshold is the usual culprit

model.predict() applies a 0.5 cutoff. On a low base rate, a well-calibrated model may never emit a probability above 0.5 — so it “predicts all negative” and someone concludes the model failed. The model is fine. The cutoff is absurd.

Watch it happen. The frame below comes from the public dataset; the label is “cancelled within 30 days of signing up”, which 17.9% of customers do.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_recall_curve

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

first = subs.sort_values("signed_up_at").drop_duplicates("customer_id")
ev = events.merge(first[["customer_id", "signed_up_at"]], on="customer_id")
week1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days < 7]
counts = week1.pivot_table(index="customer_id", columns="event_type",
                           values="event_id", aggfunc="count", fill_value=0)

d = first.set_index("customer_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)

y = ((d.cancelled_at - d.signed_up_at).dt.days <= 30).fillna(False).astype(int)
X = d[["monthly_price", "view", "login", "export", "purchase"]]
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.3,
                                            random_state=0, stratify=y)

model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
probs = model.predict_proba(X_val)[:, 1]

print(f"highest probability {probs.max():.3f}")
print(f"flagged at 0.5      {(probs >= 0.5).sum()} of {len(probs)}")

highest probability 0.368
flagged at 0.5      0 of 326

Not one customer clears 0.5, so predict() returns all-negative and the model looks broken. It isn’t: it ranks well enough to score 0.725 ROC-AUC. Choose the threshold from the precision-recall curve and the operating constraint instead:

prec, rec, thr = precision_recall_curve(y_val, probs)

# The retention team can call 40 customers -> pick the threshold by volume
budget = 40 / len(X_val)
threshold = np.quantile(probs, 1 - budget)
print(f"threshold {threshold:.3f}, flagged {(probs >= threshold).sum()}")

threshold 0.280, flagged 47

Forty-seven rather than the forty asked for, because the features are coarse counts and 223 distinct scores serve 326 customers — several customers sit on the same probability and a quantile cannot split them. Worth knowing before someone promises an exact alert volume.

Threshold selection is a free action: no retraining, no synthetic data, and it directly expresses the business constraint. It’s step one, not a footnote.

Picking the threshold from what the mistakes cost

Capacity is one constraint. Cost is the other, and it has an exact answer.

Stay with the same model. The average subscription in that file bills €47.57 a month, so a customer lost in month one who could have been kept is worth roughly three more months of revenue — call the miss €144. The retention offer costs a month of revenue given away plus the call, call it €48. Those two numbers are the whole decision. Calling a customer is worth it when the expected saving beats the expected giveaway:

p × 144 > (1 − p) × 48

which rearranges to a threshold that depends only on the costs, not on the base rate:

t = C_FP / (C_FP + C_FN) = 48 / 192 = 0.25

So the cutoff is 25%, not 50%. Nothing about the model changed; the 0.5 default was simply an assertion that a wasted offer and a lost customer cost the same amount, which nobody would say out loud.

Check it empirically rather than trusting the algebra, because the algebra assumes the model is calibrated:

C_FN, C_FP = 144.0, 48.0
grid = np.linspace(0.001, 0.5, 500)

cost = [((probs < t) & (y_val == 1)).sum() * C_FN +
        ((probs >= t) & (y_val == 0)).sum() * C_FP for t in grid]

best = grid[int(np.argmin(cost))]
print(f"empirical {best:.3f} vs analytic {C_FP / (C_FP + C_FN):.3f}")
print(f"flagged {(probs >= best).sum()} of {len(probs)}")

empirical 0.251 vs analytic 0.250
flagged 78 of 326

The two agree to a thousandth, so this model is calibrated well enough to reason about cost analytically from here on. Had the empirical minimum sat well away from 0.25, that gap would be a calibration report, not a threshold problem — fix the calibration (CalibratedClassifierCV, or isotonic regression on a held-out slice) before tuning anything else.

The cost-optimal threshold and the capacity threshold usually disagree, and they do here: the economics say call 78 customers, the rota says 47. The binding constraint is headcount, not economics, and the right move is to say so — “we call the top 40 because that is what we can staff” — rather than presenting a capacity limit as an optimum.

What actually helps, in order

1. Right metric, right threshold. PR curves, cost-weighted evaluation, and an explicitly chosen operating point. Half of “imbalance” dissolves here.
2. Class weights. class_weight="balanced" (or scale_pos_weight in XGBoost/LightGBM) reweights the loss instead of duplicating rows. Same intent as oversampling, cheaper, nothing synthetic, one line.
3. More real positives. Widen the time window, relax the label definition at the edges, invest labelling hours in minority cases specifically. Boring, and it beats every algorithmic remedy, because it adds information instead of rearranging it.
4. Undersampling the majority — as a pragmatic speed tool when you have millions of rows: train on all positives plus a sample of negatives. Note that resampling distorts predicted probabilities; recalibrate afterwards or correct for the sampling rate if anyone consumes the scores as probabilities.

Rebalancing changes the base rate the model outputs

That last warning deserves the arithmetic, because it is the failure that survives review.

Keep every positive and 5% of the negatives. The training base rate goes from 17.9% to

0.179 / (0.179 + 0.821 × 0.05) = 0.813

Eighty-one percent. The model now emits probabilities around an 81.3% prior instead of a 17.9% one — four and a half times too high. Every score it produces is wrong as a probability, and the 0.5 default suddenly looks reasonable, which is exactly why nobody notices.

The correction is one line, because undersampling multiplies the prior odds by a known constant. If you kept negatives at rate r, then true odds = training odds × r:

r = 0.05                       # fraction of negatives kept
odds = p_train / (1 - p_train)
p_true = odds * r / (1 + odds * r)

Run the cost threshold through the same map in reverse and the €48/€144 cutoff of 0.25 becomes 0.87 on the raw undersampled output. Same decision, same rows, same economics — the number moved from 0.25 to 0.87 purely because of how the training set was sampled. A team that tunes the threshold on undersampled scores and then deploys against a corrected pipeline, or the reverse, ships a silent error in its operating point.

Class weights do the same thing. class_weight="balanced" is equivalent to resampling to a 50/50 prior, so the same correction applies with r = n_pos / n_neg. The scores come back miscalibrated either way. If anything downstream multiplies your probability by a euro amount — expected loss, expected lifetime value, a reserve calculation — recalibrate, or skip the rebalancing entirely and just move the threshold.

Where SMOTE fits

SMOTE — synthetic minority oversampling, from Chawla, Bowyer, Hall and Kegelmeyer’s 2002 paper in the Journal of Artificial Intelligence Research — interpolates new minority points between existing neighbors. On tabular data with mixed types and meaningful category boundaries, those interpolated rows are frequently impossible customers — a synthetic point halfway between a student in Lisbon and a retiree in Porto. Empirically, on strong learners like gradient boosting, SMOTE plus threshold tuning rarely beats class weights plus threshold tuning — and it adds a preprocessing stage that must never touch validation folds (oversampling before the split is a classic leakage bug that inflates scores impressively).

That leaves SMOTE a narrow legitimate niche: small datasets, weak learners, purely continuous features. If you’re there, fine — apply it inside the CV loop only. If you’re not, it’s mostly ritual.

Where this advice runs out

Threshold-first assumes one threshold serves everyone. It often doesn’t. When the cost of a mistake varies row by row — losing a €12 starter subscription and losing a €249 enterprise one are not the same event, and both prices are in that file — stop tuning a global cutoff and rank by expected cost instead: p × monthly_price. The threshold then applies to euros at risk, not to probability.

It also assumes the deployment base rate matches the validation base rate. If cancellations move seasonally, or the model is reused in a market with different prevalence, a threshold frozen from last quarter is quietly wrong. Recheck it on recent data on a schedule.

And in the rare-and-few regime — a few dozen positives in total — none of this saves you. A threshold picked from thirty positive validation examples has enormous variance, and the honest answer is that the evaluation set is too small to support an operating point at all.

The summary that fits on a sticky note

Imbalance is rarely a data-quantity problem and usually an evaluation problem. Metric first, threshold second, class weights third, real data fourth, resampling last. Start at the top of that list on the next one: pick the metric that matches what each mistake costs, plot precision and recall against the threshold, and choose an operating point. Most “imbalance problems” close right there, before any resampling library gets installed.