Reading SHAP values correctly

A loan application scores 0.71 and gets declined. The review meeting opens on a SHAP waterfall plot for that single applicant: n_open_accounts +0.22, income −0.18, months_at_address −0.05. Someone says “so the open accounts caused the decline — close two and they’d be through.” Both halves of that sentence are wrong, and nothing on the plot warned anybody.

A SHAP value is a share of a difference

Shapley values come from cooperative game theory; Lundberg and Lee’s 2017 paper carried them into machine learning, and the shap library is the reference implementation. A payout has to be divided among players who contributed unequally, and the division should not depend on who does the dividing. Here the payout is one prediction, the players are the features of one row, and the question is how much each feature moved that prediction away from what the model says when it knows nothing.

The recipe is mechanical. Reveal the features one at a time, in some order, and record how far the prediction moves as each arrives. income revealed third, after n_open_accounts and months_at_address, moves the prediction by a different amount than income revealed first. So the contribution is averaged over every ordering. That averaging is the whole trick: it makes the split unique under a short list of reasonable axioms, and it makes the parts add up exactly.

baseline + sum of the SHAP values = the prediction for this row

Drawn for the declined applicant, that is the waterfall the meeting was looking at — every bar a share of the distance from the baseline to 0.71:

Each bar starts where the one above it finished, so the plot is an accounting of one number. Note how much of it is furniture: the base value 0.28 is a choice of background set, and the small bars near the bottom would reorder on a different random seed.

That identity holds for every row, every time. No other attribution method in common use offers a contract that tight, and it is the reason SHAP is worth the compute. It also marks out the limits, because every term in it is measured against the baseline.

Change the baseline and every number changes

“What the model says when it knows nothing” is not a property of the model. It is the mean prediction over a background set that you pick. Use the full training data and the baseline is the average applicant. Use only approved applicants and the baseline is a typical approval. Same row, same model, different numbers on every bar — because the plot now answers a different question. Not “why 0.71 rather than average?” but “why 0.71 rather than someone who got through?”

Both questions are fair. They are not interchangeable, and the chart does not say which one it drew. Two habits follow. Print the base value beside the values, because a waterfall without it is a set of differences from a number nobody stated. And freeze the background set for the life of the project: moving from a 100-row sample to a 1,000-row sample rewrites every explanation already in circulation, and it looks to everyone else like the model changed.

Fix the scale once, too. For a classifier you can explain the raw margin, where the values sum exactly to the log-odds, or the probability, which reads naturally but where the sum no longer lands on the prediction — the sigmoid in between is not linear, so the additivity you were sold does not survive the transformation. Pick one, say which, and keep it.

The global bar chart is a different, weaker object

Take the absolute SHAP value of each feature on each row, average down the column, sort, and you have the global importance chart everyone recognises. It is useful. It is also much less trustworthy than the local values it came from, for three reasons.

Averaging absolute values throws away direction. A feature that pushes half the population up by 0.2 and the other half down by 0.2 scores exactly the same as one that pushes everybody up by 0.2, and those are not remotely the same feature. The beeswarm plot shows the difference; the bar chart hides it.

A churn model on the public dataset shows the gap in five lines — print the share of rows where each feature pushed upward next to the usual average.

import pandas as pd
import shap
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
y = d.cancelled_at.notna().astype(int)
X = d[["monthly_price", "view", "login", "export", "purchase"]]
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3,
                                                  random_state=0, stratify=y)
model = GradientBoostingClassifier(random_state=0).fit(X_train, y_train)

background = shap.sample(X_train, 100, random_state=0)
explainer = shap.TreeExplainer(model, data=background,
                               feature_perturbation="interventional")
sv = explainer(X_val)

v = pd.DataFrame(sv.values, columns=X.columns)
out = pd.DataFrame({"mean_abs": v.abs().mean(), "share_positive": (v > 0).mean()})
print(out.sort_values("mean_abs", ascending=False).round(3))

               mean_abs  share_positive
login             0.578           0.467
monthly_price     0.387           0.708
view              0.308           0.611
export            0.282           0.531
purchase          0.134           0.597

login tops the bar chart at 0.578, nearly half again the next feature. It also pushed 46.7% of rows up and the other 53.3% down — so the model has learnt two opposite things about logging in and the ranking says nothing about either. monthly_price, a third of the way down the chart, pushed 70.8% of rows the same way and is the one feature whose bar can be read as a direction at all.

Correlated features split the credit arbitrarily. When two columns carry nearly the same information, the ordering average hands each of them part of the contribution, and the exact split depends on which one the fitted trees happened to use. Retrain with a different seed and the ranking between the twins reshuffles while the model performs identically. This is the same weakness that afflicts every global importance method, and SHAP’s mathematical pedigree does not exempt it.

And the correlation problem has no clean setting. Break the correlations when you perturb the background — the interventional option — and you score the model on applicants who cannot exist, with a graduate’s income and a retiree’s credit history. Respect them instead, by following the paths the trees actually take, and you stay on real data but lose the axioms that made the decomposition principled in the first place. There is no third option that avoids the trade.

What it costs to compute

TreeSHAP is exact and fast for tree ensembles: polynomial in trees, depth and leaves, milliseconds per row, so explaining a whole validation set is routine. KernelSHAP is model-agnostic and approximate — it samples coalitions of features, replaces the missing ones from the background, and fits a weighted linear model to the results. That costs hundreds or thousands of model evaluations per explained row, and it assumes the features are independent when it does the replacing, which sharpens the correlation problem rather than avoiding it. For deep networks, the gradient-based variants sit in between.

The practical rule: tree model on tabular data, explain everything; anything else, explain a sample and budget the time. And whatever the explainer, check the contract once on real output rather than trusting it:

# base value plus contributions must equal the raw model output
print(f"{sv.base_values[0] + sv.values[0].sum():.4f}")
print(f"{model.decision_function(X_val[:1])[0]:.4f}")

0.6481
0.6481

Note which output the check used. decision_function returns the log-odds margin, and that is the scale the identity holds on. The same row’s predict_proba is 0.657, and no set of contributions on the waterfall adds up to it. If the two lines disagree, something upstream is wrong — usually the wrong scale, or a pipeline step the explainer never saw.

The two readings that do damage

“This feature caused the outcome.” SHAP explains the model, faithfully. If the model leant on a proxy, SHAP reports the proxy, correctly and with confidence. An applicant’s postcode standing in for income earns an honest attribution and attracts a dishonest story. The decomposition is a statement about a function you fitted, not about the borrower, the market, or what would happen if anything changed.

“Move the feature by X and the prediction moves by the SHAP value.” The value is not a derivative and not a counterfactual. It is one feature’s share of the gap between this prediction and a baseline, averaged over orderings that never happen in reality. Closing two accounts changes several correlated inputs at once and lands the applicant somewhere the attribution never described. If you want the answer to that question, edit the row and score it again. One prediction, exactly the question asked, no game theory required.

Used narrowly, SHAP is excellent at the job regulators actually demand: naming the factors that moved one decision, on the record, for one person. That is a real question with a defensible answer. The trouble is that the same plot, read one clause too far, becomes a claim about cause and a promise about the future — and the numbers on it stay just as exact while the sentence stops being true.