Missing data is information, not an inconvenience

The reflexive treatments of missing data — dropna() or fill-with-the-mean — share a hidden assumption: that missingness is boring noise. It rarely is. Why a value is missing is often as informative as the values you kept, and the reflexive treatments both destroy that signal and quietly bias what remains.

The three mechanisms

Statisticians distinguish three regimes, and the distinction is practical, not academic:

• MCAR (completely at random): a sensor dropped packets randomly. Deleting rows loses power but adds no bias. This is the only regime where dropna() is innocent — and the least common in business data.
• MAR (at random, given other columns): older customers skip the app-rating question. Within an age group, missingness is random. Imputation that conditions on the other columns can work well.
• MNAR (not at random): the value itself drives its absence. People with high debt decline to answer the debt question; a device fails to report exactly when it overheats. No imputation can fully fix this — the information isn’t in your dataset. You mitigate: model the missingness explicitly, chase an external data source, or bound the damage with sensitivity analysis.

You can’t prove which regime you’re in from the data alone, but you can interrogate it: compare rows with and without the value across every other column. If the two populations differ, deletion is not neutral — it’s silently reshaping your dataset toward the kind of rows that answer questions.

The interrogation is a crosstab. Run it on the event file from the public dataset, where the amount column is blank on 93.8% of rows:

import pandas as pd

events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])

print(f"blank: {events.amount.isna().mean():.3f}")
print(pd.crosstab(events.event_type, events.amount.notna()))

blank: 0.938
amount      False  True
event_type
export       4675      0
login        8597      0
purchase        0   2493
view        24235      0

Not one of the three mechanisms. amount is blank on every login, view and export because a login has no amount — the value does not exist, rather than existing and going unrecorded. Call it structurally missing, and treat it as a fourth case, because both standard treatments are catastrophic here. dropna() on this column keeps 2,493 rows out of 40,000 and silently redefines the dataset as “purchases”. Mean imputation is worse:

paid = events.amount.dropna()
print(f"true       {paid.sum():>12,.2f}")
print(f"mean-fill  {events.amount.fillna(paid.mean()).sum():>12,.2f}")

true         187,350.43
mean-fill  3,006,023.75

Sixteen times the real revenue, from one defensible-sounding line. The crosstab above takes five seconds and prevents it.

Practical patterns

Always add the indicator. Before any imputation, record that the value was missing. It costs one column and repeatedly earns its keep as a predictive feature — “didn’t provide income” is a behaviour, and models exploit it.

Sometimes the indicator is the only thing in the column that matters. In the subscription file, cancelled_at is blank for 566 of 1,198 rows, and blank means the subscription is still running:

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])

subs["is_active"] = subs.cancelled_at.isna().astype(int)
print(subs.groupby("is_active").monthly_price.agg(["size", "mean"]).round(2))

           size   mean
is_active
0           632  35.79
1           566  61.71

The two groups pay very different prices, so the missingness is nowhere near random — and any churn model built on this file has the indicator as its target. A column can be 47% blank and still be the most valuable one you have.

Impute inside the pipeline, not before the split. An imputer fitted on the full dataset leaks test-set statistics into training. Same rule as every fitted transformation.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("model", LogisticRegression(max_iter=1000)),
])

Match effort to the column’s importance. Median imputation plus an indicator is a fine default for weak features. For a feature that carries the model, consider model-based imputation (IterativeImputer, KNN) — but measure whether it actually beats the simple approach on your validation metric; often it doesn’t. For tree ensembles, note that modern gradient boosting libraries handle NaN natively and learn which side of a split to send missing values — frequently the strongest and simplest option of all.

Mind the fake missing values. Real datasets encode absence as -999, 0, empty strings, "N/A", or a date in 1970. A zero blood pressure is not a measurement. Audit minimums, value counts, and suspicious spikes before trusting isna() to find the gaps — the most dangerous missing values are the ones wearing a number.

For analysis, not just models

Everything above applies double to statistics you report. A churn rate computed on customers who answered the survey is a churn rate for survey-answerers — a population that self-selected. Every aggregate you publish should be able to answer the question: who is missing from this denominator, and did they leave at random?

Treat missingness as a first-class feature of the dataset: detect the fake zeros, test whether absence correlates with anything, keep indicators, impute inside pipelines, and stay humble about MNAR. Of those, the indicator column is the cheapest and the one most often skipped. One boolean per column with a meaningful gap, created before any imputation runs, kept in the model even when its importance looks small.