Mutual information for feature selection, and where it misleads

A fraud model has hour_of_day in it. The correlation table gives that column a Pearson coefficient of roughly zero, so it goes on the drop list. Then somebody plots the fraud rate by hour: it peaks at three in the morning, sinks through the working day, and rises again after midnight. The feature is one of the strongest signals in the dataset. Pearson scored it at zero because the relationship is not a line, and a line is the only thing Pearson can see.

Mutual information sees it. That is why it ends up in feature selection pipelines — and why the scores it returns get trusted further than they should.

Surprise removed

Mutual information comes straight out of entropy, the measure of surprise. The entropy of the target, H(Y), is how surprised you expect to be by an outcome before you know anything. The conditional entropy H(Y|X) is how surprised you still expect to be once you know the feature. Mutual information is the difference:

I(X;Y) = H(Y) − H(Y|X)

That is it. How much of the uncertainty about the target does knowing this feature remove. It is symmetric — the surprise the target removes about the feature is the same number — and it is zero exactly when the two are independent. Not “zero when they are linearly related”, zero when the feature carries nothing about the target at all, in any form.

The units follow the logarithm. scikit-learn uses natural logs, so its output is in nats; divide by np.log(2) for bits. And the quantity has a ceiling for classification: a feature cannot remove more uncertainty than there was, so I(X;Y) can never exceed H(Y). For a balanced binary target, H(Y) is ln 2, about 0.693 nats. Keeping that number in view is the single most useful habit here, for reasons that arrive at the end of this article.

In practice, scikit-learn’s mutual_info_classif does the estimating:

The frame below is the churn table from the public dataset, with two columns added on purpose: the raw customer_id, and days_since_last_event, a recency measured today — which for a cancelled subscription is mostly the cancellation date in disguise.

import numpy as np
import pandas as pd
from sklearn.feature_selection import mutual_info_classif

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

ev = events.merge(subs[["subscription_id", "customer_id", "signed_up_at"]],
                  on="customer_id")
month1 = ev[(ev.occurred_at - ev.signed_up_at).dt.days.between(0, 29)]
counts = month1.pivot_table(index="subscription_id", columns="event_type",
                            values="event_id", aggfunc="count", fill_value=0)

d = subs.set_index("subscription_id").join(counts)
d[counts.columns] = d[counts.columns].fillna(0)
y = d.cancelled_at.notna().astype(int)

last = events.groupby("customer_id").occurred_at.max()
X = pd.DataFrame({
    "monthly_price": d.monthly_price,
    "view": d.view, "login": d.login, "export": d.export, "purchase": d.purchase,
    "country": d.country.astype("category").cat.codes,
    "channel": d.channel.astype("category").cat.codes,
    "plan": d.plan.astype("category").cat.codes,
    "customer_id": d.customer_id,
    "days_since_last_event": (pd.Timestamp("2026-07-31")
                              - d.customer_id.map(last)).dt.days,
})

p = y.mean()
print(f"H(Y) = {-(p * np.log(p) + (1 - p) * np.log(1 - p)):.3f} nats")

cat_idx = [X.columns.get_loc(c)
           for c in ["country", "channel", "plan", "customer_id"]]

mi = mutual_info_classif(
    X, y,
    discrete_features=cat_idx,   # indices of the categorical columns
    n_neighbors=3,
    random_state=0,
)

for name, score in sorted(zip(X.columns, mi), key=lambda t: -t[1])[:10]:
    print(f"{name:24s} {score:.4f}")

H(Y) = 0.692 nats
customer_id              0.6246
days_since_last_event    0.2191
plan                     0.0248
monthly_price            0.0213
export                   0.0191
view                     0.0165
login                    0.0065
channel                  0.0023
country                  0.0023
purchase                 0.0005

Hold that ranking in mind. The two columns added on purpose took first and second place, and everything a churn model would actually use is clustered near zero. Both halves of that sentence are explained below.

mutual_info_regression has the same signature for a continuous target. The discrete_features argument matters: leave it at 'auto' and scikit-learn guesses from whether the input is sparse, which is usually wrong for a dataframe holding both kinds of column.

Every score is an estimate, and the estimate moves

The definition above needs the joint distribution of X and Y, which nobody has. What gets computed is an estimate, and different estimators disagree about the same data.

For discrete features, the counting estimator is straightforward but biased upward on small samples — with enough distinct values, random noise looks like structure. For continuous features, scikit-learn uses a k-nearest-neighbour estimator, and n_neighbors is a smoothing dial: small values keep detail and add variance, large values do the opposite. Change it and the ranking changes. The estimator also breaks ties with random jitter, so two runs with different seeds return different numbers. Measure that before reading anything into the order:

runs = np.vstack([
    mutual_info_classif(X, y, discrete_features=cat_idx, random_state=s)
    for s in range(10)
])

spread = pd.DataFrame(
    {"mean": runs.mean(axis=0), "sd": runs.std(axis=0)}, index=X.columns
).sort_values("mean", ascending=False)
print(spread.round(4))

                         mean      sd
customer_id            0.6246  0.0000
days_since_last_event  0.2210  0.0071
monthly_price          0.0304  0.0101
plan                   0.0248  0.0000
login                  0.0158  0.0107
export                 0.0137  0.0084
view                   0.0110  0.0094
purchase               0.0101  0.0081
channel                0.0023  0.0000
country                0.0023  0.0000

The discrete columns have zero standard deviation — no jitter is added to counts. Every continuous column has a standard deviation of roughly the same size as its mean. login, export, view and purchase occupy ranks five to eight in this run and cannot be distinguished from each other at all; on the single seed used earlier, export and view sat above login, and purchase came last at 0.0005 rather than 0.0101. Same data, same estimator, different seed.

If the gap between rank three and rank eight is smaller than their standard deviations, there is no ranking there — only noise with an ordering imposed on it.

The bias has a direction, and it is the one that hurts. Cardinality inflates the score. Split a continuous feature into more bins, or hand over a categorical column with two thousand levels, and the estimator finds more apparent structure simply because each cell holds fewer rows and fewer rows are easier to make look pure. A customer ID is the extreme case: near-unique per row, so it “determines” the target in-sample while carrying no generalisable information whatsoever. That is customer_id at the top of both tables above, scoring 0.6246 against a ceiling of 0.692 — ninety percent of everything there is to know about churn, apparently held by an arbitrary integer. Any high-cardinality identifier near the top of an MI ranking is an artefact until proven otherwise.

It looks at one column at a time

mutual_info_classif computes I(X_j;Y) for each feature independently. Two consequences, both bad if unnoticed.

It cannot see interactions. The clearest demonstration is exclusive-or, where each feature alone is worthless and the pair determines the target exactly:

rng = np.random.default_rng(0)
a = rng.integers(0, 2, 20_000)
b = rng.integers(0, 2, 20_000)
y_xor = a ^ b

mutual_info_classif(np.c_[a, b], y_xor, discrete_features=True, random_state=0)
# both scores ≈ 0 — neither column tells you anything on its own

Drop features on univariate MI and this pair goes first. A tree ensemble would have found it in two splits.

It also double-counts. Ten variants of the same underlying quantity — order count, order count last 90 days, log order count — all score highly and all say the same thing. Take the top ten by MI and you may have taken one signal ten times over while leaving out the second real signal at rank fourteen. The usual correction is a redundancy penalty: pick greedily, and at each step subtract the mutual information between the candidate and what is already selected (mRMR does exactly this). At that point, though, a model with built-in feature selection is often the shorter route.

A very high score is usually bad news

Here is where the entropy ceiling earns its keep. On a balanced binary target with a ceiling of 0.693 nats, real features score modestly — they shift the odds, they do not settle the question. A single feature returning something like 0.5 or 0.6 nats is claiming to remove most of the uncertainty about the outcome on its own, and honest features almost never can.

So treat a standout score as an accusation rather than a discovery. Nearly every time, the column contains the answer: a status field written after the event, a timestamp that only exists for one class, an amount reconciled after the case closed. days_since_last_event in the tables above is the third kind. It scored 0.2210 — ten times the best honest feature — because a customer who cancelled stopped generating events when they cancelled, so the column is partly a copy of the label’s timestamp. It reads as a perfectly ordinary recency feature, which is precisely the problem. That is leakage, and it looks like a brilliant model right up until deployment. Mutual information is unusually good at surfacing it, because leaked columns are exactly the ones that determine the target, and MI measures determination without caring whether the relationship is linear, monotonic, or even sensible.

Which suggests the better use for the whole technique. As a filter for what to keep, MI is mediocre — blind to interactions, inflated by cardinality, unstable across seeds. As a smoke alarm, it is excellent. Run it early, sort descending, and look hard at whatever sits on top.