Your model started dying the day you deployed it

Software mostly keeps working after deployment. Models don’t — they’re snapshots of a world that keeps moving. Products launch, competitors react, fraudsters adapt, an upstream team renames a field. The model’s accuracy decays while its confidence stays exactly the same, which is what makes the failure mode so quiet: no exception, no crash, just predictions drifting from reality until a business metric finally complains, weeks late.

Monitoring is how you find out first. The good news: a useful setup is small.

The three layers, cheapest first

1. Input monitoring — is the data still shaped like the training data? Catches the most common failures (schema changes, broken upstream jobs, new categories, unit changes) and needs no labels at all. Track per-feature: null rate, mean/median, min/max, and category frequencies, compared against a frozen training-time snapshot.

The reference below is the 2025 signup cohort from the public dataset, and the batch is the 2026 cohort — same features, a year later.

import pandas as pd

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
# Keep only subscriptions old enough to have a complete 30-day window
d = d[d.signed_up_at <= events.occurred_at.max() - pd.Timedelta(days=30)]

numeric_cols = ["monthly_price", "view", "login", "export", "purchase"]
X = d[d.signed_up_at.dt.year == 2025]        # what the model trained on
batch = d[d.signed_up_at.dt.year == 2026]    # what turned up later

# At training time, freeze reference stats
reference = {
    col: {"null": X[col].isna().mean(),
          "p05": X[col].quantile(.05), "p95": X[col].quantile(.95)}
    for col in numeric_cols
}

# Daily in production: compare and alert
def check(batch, reference, tol=3.0):
    alerts = []
    for col, ref in reference.items():
        if batch[col].isna().mean() > 5 * max(ref["null"], 0.002):
            alerts.append(f"{col}: null-rate spike")
        if batch[col].median() > ref["p95"] or batch[col].median() < ref["p05"]:
            alerts.append(f"{col}: distribution shift")
    return alerts

print("alerts:", check(batch, reference))

alerts: []

Silence — and there is something to find. Look at the same features by quarter:

print(d.groupby(d.signed_up_at.dt.to_period("Q"))[numeric_cols].mean().round(2))

              monthly_price  view  login  export  purchase
signed_up_at
2024Q3                32.24  9.48   3.25    1.73      1.16
2024Q4                45.51  9.37   3.29    1.84      0.96
2025Q1                46.47  9.15   3.37    1.72      1.08
2025Q2                54.41  8.67   3.12    1.59      1.03
2025Q3                50.35  8.24   2.81    1.23      1.02
2025Q4                48.01  8.37   3.07    1.72      1.03
2026Q1                52.40  6.80   2.38    1.31      0.87
2026Q2                48.18  6.79   2.23    1.18      0.87
2026Q3                29.00  4.00   2.00    0.00      0.00

(2026Q3 is a single subscription — ignore that row.) Every event count drops between 2025Q4 and 2026Q1 and then sits flat: views down 21%, logins 26%, exports 21%, purchases 17%. The median never leaves the reference band, so the check above never fires, because these are counts with long tails and a median of 6 becoming a median of 4 clears no percentile threshold.

Note the shape before diagnosing the cause. A step at one quarter boundary followed by a flat line is not how customer behaviour usually moves — it is how a tracking change looks. The first question is whether an event type stopped being logged, not whether the users changed.

Crude thresholds like these catch the catastrophic cases — the null-rate going from 0.1% to 60% because a join broke — which are the cases that matter most, and miss a quarter of the traffic disappearing quietly. Population stability index (PSI) or KS tests are refinements, and a mean-per-period table costs one line and catches what the percentile check above does not.

2. Prediction monitoring — is the model’s output distribution stable? Also label-free. Plot the daily distribution of predicted scores and the rate of positive decisions. A fraud model that flagged 1.1% of transactions for months and suddenly flags 4% is telling you something changed — inputs, users, or an attack — before any label confirms it.

3. Outcome monitoring — is the model still right? The real thing, and the hard one, because labels arrive late: churn takes a quarter, defaults take a year. Two practical moves. First, compute performance on labels as they mature — today you can score predictions made 90 days ago; that’s your freshest true accuracy, so plot it monthly. Second, find a leading proxy that correlates with the eventual label (early delinquency for default, 7-day inactivity for churn) and monitor that at higher frequency.

Drift diagnosis in one sentence each

• Data drift: inputs changed (new country launched, traffic mix shifted). Layer 1 fires. Model may still be fine — investigate before retraining.
• Concept drift: the input→outcome relationship itself changed (fraudsters adapted, a pandemic rewrote demand). Layers 1–2 may stay silent while layer 3 sinks. This is why outcome monitoring is non-negotiable, however delayed.
• Upstream breakage: not drift at all — a bug wearing drift’s clothes. Sudden step changes are almost always this; gradual slopes are the real thing.

Retraining is a decision, not a reflex

Scheduled retraining (“every month, on the last N months”) is a fine default — but every retrain deserves the same gate as the first deployment: compared against the incumbent on a holdout, checked for calibration, and rolled out behind a comparison window. Retraining on drifted-but-broken data (layer 1 alert unresolved) bakes the breakage into the weights, converting an incident into a permanent regression.

Start with a daily job and a one-page dashboard: input checks, score distribution, decision rate, and matured accuracy. That’s an afternoon of work, and the platforms can come later. Weigh the afternoon against what the unmonitored version usually costs: one feature pinned at its default for eleven weeks after an upstream rename, noticed because a partner team asked why approvals had climbed, and then a quarter of decisions reviewed by hand because nobody can say which day the model stopped working.