Your model says 90%. Should anyone believe it?

A classifier can rank perfectly — every fraud scored above every non-fraud — and still be lying every time it says “90%”. Ranking and probability are different promises. AUC certifies the first. The moment anyone does arithmetic with your scores — expected loss, ranking by risk × amount, an automated action above a stated confidence — they’re trusting the second, and it needs its own audit.

Seeing the lie: the reliability curve

Bucket predictions by stated probability and check each bucket’s actual outcome rate. Among everything the model called “70%”, did roughly 70% happen?

Here it is on the public dataset — a random forest predicting whether a subscription is ever cancelled, from what the customer did in their first thirty days.

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, roc_auc_score

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
y = d.cancelled_at.notna().astype(int)
X = d[["monthly_price", "view", "login", "export", "purchase"]]
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3,
                                                  random_state=0, stratify=y)

model = RandomForestClassifier(n_estimators=300, random_state=0)
model.fit(X_train, y_train)
probs = model.predict_proba(X_val)[:, 1]

print(f"AUC {roc_auc_score(y_val, probs):.3f}  "
      f"Brier {brier_score_loss(y_val, probs):.4f}")

observed, stated = calibration_curve(y_val, probs, n_bins=5,
                                     strategy="quantile")
for s, o in zip(stated, observed):
    print(f"  said {s:.2f}   happened {o:.2f}")

AUC 0.567  Brier 0.2808
  said 0.17   happened 0.42
  said 0.37   happened 0.50
  said 0.52   happened 0.57
  said 0.68   happened 0.53
  said 0.84   happened 0.62

Every bin is pulled toward the middle from the outside: the confident-cancel bucket says 84% and delivers 62%, the confident-stay bucket says 17% and delivers 42%. Sixteen subscriptions scored above 0.9, and 31% of them were cancelled. The forest is stretching a weak signal across the full zero-to-one range because averaging 300 trees that each vote hard produces spread that the evidence does not support.

Five quantile bins of 72 rows, not ten bins of 36 — with a validation set this small, ten bins is noise with a shape. CalibrationDisplay draws the same thing as a chart.

A calibrated model tracks the diagonal. The typical failures:

• Overconfidence — “99%” events that happen 80% of the time. Random forests and boosted trees drift this way; deep nets are notorious for it.
• Underconfidence — the S-shaped curve of SVMs and some averaging ensembles, where predictions huddle toward the middle.

Both are easy to spot once you know which side of the diagonal to look at.

Read it vertically. Pick a stated probability on the bottom axis and see what actually happened. The overconfident model claims 90% for events that occur 73% of the time. The underconfident one hedges at 70% for events that occur 83% of the time. Both curves cross the diagonal at 0.5, so an accuracy check at the default threshold notices neither.

One number to track alongside the plot: the Brier score (mean squared error of the probabilities). And a caution about the plot itself: with few samples per bin, reliability curves get noisy — don’t diagnose from ten points and eight bins of thirty samples each.

Where miscalibration comes from

Some of it is the loss function — hinge loss never promised probabilities, the way logistic regression’s cross-entropy does. Some is the algorithm’s structure (bagging pulls votes toward the middle). But often you did it yourself: undersampling or oversampling the training data changes the base rate the model learned. Train on balanced classes for a problem with 2% positives and the output scores live in a fantasy world where positives are half the universe. The ranking may survive; every stated probability is wrong until corrected for the true prior.

Fixing it

Calibration is a post-processing step: learn a mapping from the model’s scores to honest probabilities, using data the model didn’t train on.

• Platt scaling — fit a logistic function on the scores. One parameter pair, works with small calibration sets, assumes the distortion is sigmoid-shaped.
• Isotonic regression — fit any monotonic mapping. More flexible, needs more data (roughly a thousand-plus examples) or it overfits the calibration set.

Both are wrapped by CalibratedClassifierCV:

from sklearn.calibration import CalibratedClassifierCV

calibrated = CalibratedClassifierCV(
    RandomForestClassifier(n_estimators=300, random_state=0),
    method="isotonic", cv=5,
)
calibrated.fit(X_train, y_train)   # internally keeps calibration data separate
fixed = calibrated.predict_proba(X_val)[:, 1]

print(f"Brier {brier_score_loss(y_val, fixed):.4f}")
observed, stated = calibration_curve(y_val, fixed, n_bins=5,
                                     strategy="quantile")
for s, o in zip(stated, observed):
    print(f"  said {s:.2f}   happened {o:.2f}")

Brier 0.2470
  said 0.42   happened 0.41
  said 0.49   happened 0.51
  said 0.54   happened 0.53
  said 0.57   happened 0.60
  said 0.62   happened 0.60

Brier drops from 0.2808 to 0.2470 and every bin now lands within three points of what it claims. Note what else happened: the range collapsed from 0.17–0.84 to 0.42–0.62. That is the correction working. The model never had the evidence to say 84%, and now it doesn’t say it.

The non-negotiable: calibrate on data the underlying model never fit, via the built-in CV or a held-out slice. Calibrating on training scores yields the same overconfidence with better paperwork.

The claim that calibration leaves the ranking untouched needs one qualifier. Platt scaling is strictly monotonic, so it preserves AUC exactly. Isotonic regression is a step function: it is monotonic but flat in places, and flat means ties. Fitting both on a held-out slice of the same data, Platt returns the identical 0.5804 AUC on 260 distinct scores, while isotonic maps those 260 scores onto 8 distinct values and AUC falls to 0.5667. The cv=5 form above moves it a third way, because it averages five separately-fitted models rather than post-processing one. If the ranking is what a downstream system consumes, measure AUC after calibrating instead of assuming it survived.

When it matters, and when it doesn’t

Skip the ceremony when the output feeds a single fixed threshold chosen empirically on validation data — “flag the top 1%” doesn’t care whether scores are honest probabilities. Insist on it when:

• probabilities enter arithmetic: expected value, pricing, risk-weighted prioritisation;
• humans read the number and act on its magnitude — a surgeon reads “90%” as ninety percent, not as “high-ish rank”;
• scores from different models get compared or combined;
• a threshold must be chosen in probability terms (“act above 80% confidence”) rather than tuned empirically.

One more thing: calibration decays. A model calibrated on last year’s base rate drifts as the world moves — recheck the reliability curve on recent data periodically, not once at launch.

The whole check is smaller than the explanation. Take the model already in production, score a recent held-out sample, bin the scores, and plot mean predicted against observed rate. If the curve sits below the diagonal in the middle, the model is over-confident and CalibratedClassifierCV with cv=5 will pull it back. Note the base rate of the data the correction was fitted on, because that is the number the curve will drift away from.