Feature importance is not explanation

“As you can see, tenure drives churn.” Behind that sentence is a horizontal bar chart of feature importances with tenure_days at the top, and a room nodding along. Three misunderstandings are packed into the sentence — about what was measured, about correlated features, and about causality. All three are worth unpacking, because decisions get made off that slide.

Misunderstanding 1: “importance” is one thing

The bar chart could be any of several quantities that don’t agree:

Impurity importance (the default feature_importances_ in tree ensembles) sums how much each feature reduced training loss across splits. It’s biased toward high-cardinality features — a random ID column can rank near the top, because with enough distinct values something always looks splittable. It also describes the training set, not generalisation.

Permutation importance shuffles one column in validation data and measures how much the score drops. Model-agnostic and tied to actual predictive performance — a better default:

Both, on the churn table from the public dataset, with the raw customer_id left in the feature set on purpose:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
y = d.cancelled_at.notna().astype(int)
X = d[["monthly_price", "view", "login", "export", "purchase", "customer_id"]]
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.3,
                                            random_state=0, stratify=y)

model = RandomForestClassifier(n_estimators=300, random_state=0)
model.fit(X_tr, y_tr)

print(pd.Series(model.feature_importances_,
                index=X.columns).sort_values(ascending=False).round(4))

r = permutation_importance(model, X_val, y_val, n_repeats=10,
                           random_state=0, scoring="roc_auc")
for i in r.importances_mean.argsort()[::-1][:10]:
    print(f"{X_val.columns[i]:16s} {r.importances_mean[i]:+.4f} "
          f"± {r.importances_std[i]:.4f}")

customer_id      0.4732
view             0.1778
login            0.0980
export           0.0915
monthly_price    0.0903
purchase         0.0693

customer_id      +0.1202 ± 0.0154
monthly_price    +0.0525 ± 0.0162
login            +0.0283 ± 0.0114
export           -0.0057 ± 0.0115
purchase         -0.0098 ± 0.0075
view             -0.0099 ± 0.0093

Impurity importance hands customer_id 47% of the total, and view second place. Permutation importance reorders almost everything: view drops to last with a negative score — shuffling it improves validation AUC slightly, which means the model was using it to fit training noise — and monthly_price, fifth on the first list, moves to second on the second.

customer_id stays on top of both, which is not the answer the usual story predicts, and it is worth following. It is not surviving as an in-sample artefact; it genuinely predicts on data the model never saw, because IDs were issued in signup order and correlate 0.915 with the signup date. Later customers have had less time to cancel — 74% of the 2024 cohort has cancelled against 33% of the 2026 cohort — so the ID is a clock, and the label is censored by that clock. Permutation importance is right that the column predicts. It cannot tell you the column is a calendar.

SHAP values decompose each individual prediction into per-feature contributions with a solid game-theoretic contract. Excellent for “why did customer 4127 get this score?” — a different question from “what matters globally?”, though SHAP aggregates can serve both.

If your conclusion survives only one of these lenses, it’s fragile. Check two.

Misunderstanding 2: correlated features split the credit

When tenure_days and n_orders are correlated at 0.9, a tree ensemble uses them interchangeably. Impurity importance splits credit between them — sometimes 70/30, sometimes 30/70 across retrained seeds. Permutation importance has the opposite pathology: shuffle one and the model leans on its twin, so both can look unimportant while jointly essential.

So the ranking between correlated features is noise, and “feature X doesn’t matter, drop it” is not a conclusion you can read off either chart. If it matters, test it properly: retrain without the feature and compare validation scores. For groups of related features, permute or drop the group together.

A case where the three lenses disagree

Build a dataset where the answer is known in advance, then see what each method reports. Four features. tenure_days genuinely affects churn. n_orders is tenure plus noise, correlated with it around 0.9, and has no effect of its own. support_tickets is a pure symptom: a latent frustration variable drives both the tickets and the churn, so tickets cause nothing. account_id is a random integer with 20,000 distinct values and no relationship to anything.

import numpy as np, pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance

rng = np.random.default_rng(0)
n = 20_000

tenure      = rng.gamma(2.0, 300.0, n)
n_orders    = tenure / 30.0 + rng.normal(0, 1.5, n)
frustration = rng.normal(0, 1, n)
tickets     = rng.poisson(np.exp(0.2 + 0.8 * frustration))
account_id  = rng.permutation(n)

logit = -1.2 - 0.003 * tenure + 1.1 * frustration
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

X = pd.DataFrame({"tenure_days": tenure, "n_orders": n_orders,
                  "support_tickets": tickets, "account_id": account_id})
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=0.3, random_state=0)

rf = RandomForestClassifier(n_estimators=300, min_samples_leaf=5,
                            random_state=0).fit(Xtr, ytr)

print(pd.Series(rf.feature_importances_, index=X.columns).sort_values())
r = permutation_importance(rf, Xva, yva, n_repeats=10, random_state=0,
                           scoring="roc_auc")
print(pd.Series(r.importances_mean, index=X.columns).sort_values())

Run it and read the two lists side by side. Impurity importance gives account_id a real share of the total, above at least one feature that matters, because a continuous column with 20,000 distinct values offers 20,000 candidate split points and some of them fit training noise. Permutation importance on validation data sends account_id to zero, where it belongs — shuffling a column the model shouldn’t rely on costs nothing out of sample. That is the first disagreement, and permutation is right.

The second disagreement is the correlated pair. Permuting tenure_days alone barely moves validation AUC, because n_orders carries almost the same information and the forest falls back on it. Permuting n_orders alone barely moves it either, for the mirror-image reason. Read naively, both features look disposable. Permute the two together and the score drops sharply:

Xp = Xva.copy()
idx = rng.permutation(len(Xp))
Xp[["tenure_days", "n_orders"]] = Xp[["tenure_days", "n_orders"]].values[idx]

Same rows, same model, one grouped shuffle, and the pair goes from “neither matters” to “this is most of the signal”. Nothing changed except the question.

SHAP adds a third reading. Aggregate mean absolute SHAP values and the correlated pair gets its credit split again — arbitrarily, and differently across seeds — while support_tickets sits high, because it is a strong predictor. It is also a symptom that no intervention can move, which no importance method of any kind will tell you.

Misunderstanding 3: importance is causal

The chart says “the model found this feature useful for predicting.” It does not say “changing this feature changes the outcome.” Support tickets may be the top churn predictor — as a symptom of frustration, not a cause. “Reduce ticket volume” (say, by hiding the contact button) would optimise the symptom and worsen the disease.

Predictive models learn correlations, and are entirely happy to lean on proxies: neighbourhood as a proxy for income, device type as a proxy for age. Any sentence of the form “to improve the outcome, change feature X” is a causal claim and needs causal evidence — an experiment, or careful quasi-experimental analysis. The model’s importance chart is, at best, a list of hypotheses worth testing.

When the caution stops earning its keep

Not every use of the chart needs this much care. If features are close to independent — a handful of engineered columns, correlations under about 0.3 — the three methods usually agree, and the ranking is stable enough to quote. If the question is monitoring rather than explanation (“did the model’s reliance on this feature shift after the retrain?”), a consistent method compared against itself over time is fine, even a biased one, because the bias is the same on both sides of the comparison. And if you only want a smell test — is a timestamp, a row index, or a near-copy of the label at the top? — impurity importance costs nothing and catches it.

The care matters when someone will act on the ranking: dropping features, reallocating budget, briefing an executive, or arguing that a group is or isn’t being treated unfairly. Those are decisions, and decisions deserve the second lens and the retrain.

Using importance well

It’s genuinely useful for: sanity-checking (an ID or timestamp near the top means leakage or bias — investigate), debugging surprising predictions with SHAP, communicating roughly what the model attends to, and generating hypotheses for experiments. A reasonable default workflow: permutation importance on validation data for the global view, grouped by correlated clusters; SHAP for individual cases; retrain-without-it for any drop/keep decision; an experiment for any causal claim.

One question is worth asking before the next review meeting: what would have to be true for this bar chart to support the decision it is about to support? The answer usually names an experiment nobody has run yet. Saying that out loud takes ten seconds and settles the argument faster than the chart ever will.