Accuracy is almost never the metric you want

A fraud model that flags nothing is 99.7% accurate, because 99.7% of transactions aren’t fraud. That single observation should permanently end your relationship with accuracy as a default metric. Yet accuracy keeps showing up in reports, because it’s the metric everyone thinks they understand.

The right question is never “how often is the model correct?” It’s “what does each kind of mistake cost, and how does this model trade them off?”

The two mistakes and their names

Every binary classifier makes two kinds of errors: false positives (flagging the innocent) and false negatives (missing the guilty). Two metrics describe them:

• Precision — of the things you flagged, how many were real? Low precision means alert fatigue: humans reviewing your flags stop trusting them.
• Recall — of the real cases, how many did you catch? Low recall means the thing you were built to catch is sailing through.

These trade off against each other via the decision threshold. Any classifier can hit 100% recall by flagging everything — with dismal precision. The model doesn’t have a precision and a recall; it has a curve of pairs, one per threshold.

Here is what that looks like on real numbers, using the subscription and event files from the public dataset. The target is “cancelled within 30 days of signing up”, read off cancelled_at; the features are counts of what each customer did in their first week.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_recall_curve

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

first = subs.sort_values("signed_up_at").drop_duplicates("customer_id")
ev = events.merge(first[["customer_id", "signed_up_at"]], on="customer_id")
week1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days < 7]
counts = week1.pivot_table(index="customer_id", columns="event_type",
                           values="event_id", aggfunc="count", fill_value=0)

d = first.set_index("customer_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)

y = ((d.cancelled_at - d.signed_up_at).dt.days <= 30).fillna(False).astype(int)
X = d[["monthly_price", "view", "login", "export", "purchase"]]
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.3,
                                            random_state=0, stratify=y)

model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
probs = model.predict_proba(X_val)[:, 1]

print(f"base rate       {y_val.mean():.3f}")
print(f"accuracy at 0.5 {accuracy_score(y_val, model.predict(X_val)):.3f}")
print(f"flagged at 0.5  {model.predict(X_val).sum()}")

base rate       0.178
accuracy at 0.5 0.822
flagged at 0.5  0

The model is 82.2% accurate and it flagged nobody. Its highest probability on any validation customer is 0.368, so a 0.5 cutoff never fires once. The model is not useless — its ROC-AUC is 0.725, so it does rank early cancellations above the rest — but accuracy cannot see that, because the base rate scores 82.2% on its own.

Read the same model as a curve and it starts doing work:

precision, recall, thresholds = precision_recall_curve(y_val, probs)

# Most recall available while at least a third of the flags are right
viable = precision[:-1] >= 0.35
best = np.argmax(recall[:-1] * viable)
print(f"threshold={thresholds[best]:.3f}, "
      f"precision={precision[best]:.2f}, recall={recall[best]:.2f}")

threshold=0.251, precision=0.35, recall=0.48

At a cutoff of 0.251 the model catches 48% of the customers who cancel in their first month, at 35% precision: 80 flags instead of zero. Nothing about the model changed. Only the number it was judged against.

That constraint — “the retention team can act on one flag in three” — is a business fact, not a modelling fact. Getting it out of a stakeholder’s head and into the evaluation is more valuable than a month of hyperparameter tuning.

ROC-AUC vs PR-AUC

ROC-AUC is a fine measure of overall ranking ability — “does the model score positives above negatives?” — and is threshold-free. But on heavily imbalanced data it flatters: the enormous pool of true negatives makes the false positive rate look tiny even when your flags are mostly wrong. A model can have 0.95 ROC-AUC and 8% precision at any useful operating point.

When positives are rare and finding them is the job — fraud, defects, rare disease — look at the precision-recall curve, which stays honest because it never rewards you for the true negatives you didn’t flag.

F1 is a compromise, not a law

F1 is the harmonic mean of precision and recall — an opinionated summary that weights both mistakes equally. Sometimes that’s right. Often it isn’t: missing a cancer case is not the same cost as an unnecessary follow-up test. If the costs are asymmetric, either use F-beta with a deliberate beta, or better, skip the single-number summary and report the operating point: “at the threshold we ship, precision is X and recall is Y, which means Z missed cases and W false alarms per week.” Concrete units beat abstract scores in every stakeholder conversation.

A decision procedure

1. Write down what a false positive costs and what a false negative costs, in money, hours, or harm. Rough numbers are fine.
2. If one error type has a hard operational budget (review capacity, alert volume), fix that as a constraint and optimise the other metric.
3. Evaluate the model as a curve, choose a threshold as a business decision, and report performance at that threshold in real-world units.
4. Keep ROC-AUC or PR-AUC as a model-comparison metric during development, always on data the model never trained on — a training-set score measures memorisation, not skill. Just never make it the shipped success criterion.

None of which makes accuracy a forbidden number. On a balanced problem where both mistakes cost about the same — is this photo a cat or a dog — it is a fair summary and every reader interprets it correctly. The damage comes from reaching for it by default, on problems where the base rate does the work and nobody checks what the base rate is.