Encoding categorical variables without shooting yourself

Models eat numbers, and much of your data is words: country, product category, acquisition channel, device type. How you turn those words into numbers changes model performance more than most hyperparameters — and one popular technique comes with a leakage trap that has burned nearly everyone who’s used it.

The straightforward cases

One-hot encoding — one binary column per category — is the honest default for low-cardinality features (up to a few dozen values). It imposes no fake ordering and every model type handles it.

The subscription file in the public dataset has three of these: plan (4 values), channel (5) and country (10).

import pandas as pd

subs = pd.read_csv("https://dataacademy.ai/data/subscriptions.csv",
                   parse_dates=["signed_up_at", "cancelled_at"])
df = subs.assign(churned=subs.cancelled_at.notna().astype(int))

X = pd.get_dummies(df[["monthly_price", "channel", "country"]],
                   columns=["channel", "country"], drop_first=False)
print(X.shape)

(1198, 16)

Fifteen binary columns and one numeric, from 1,198 rows. Nothing to think about — that is what low cardinality buys you.

Two footnotes: for unregularised linear models, drop one column to avoid perfect collinearity; and handle unseen categories deliberately — OneHotEncoder(handle_unknown="ignore") in a pipeline beats get_dummies for anything that will ever score new data, because production will send you a category you didn’t train on.

Ordinal encoding — mapping categories to integers — is correct only when the order is real: S < M < L < XL, education levels, credit ratings. Applying it to unordered categories tells linear models a lie (“France is twice Germany”). Tree models split on order, not distance, so they mostly shrug it off, which is why sloppy ordinal encoding often survives unnoticed under gradient boosting.

The high-cardinality problem

Zip code, merchant id, job title: thousands of values. One-hot explodes into a sparse desert where most columns almost never fire. The workable options:

• Group then encode. Keep the top-N categories, bucket the tail into OTHER. Crude, robust, often sufficient.
• Count/frequency encoding. Replace the category with how often it occurs. Cheap, no leakage risk, surprisingly effective when popularity itself is predictive.
• Native categorical support. LightGBM and CatBoost accept categorical columns directly and handle the splitting internally — frequently the best answer and the least code.
• Target encoding. The powerful one, and the dangerous one.

Target encoding: powerful, booby-trapped

Target encoding replaces each category with the mean of the target for that category — merchant_id becomes “fraud rate of this merchant”. Done naively, each row’s own target contributes to its feature: you’ve piped the answer into the inputs. Validation scores soar; production performance doesn’t. For rare categories the effect is grotesque — a category appearing once encodes to exactly its own target value.

The two rules that make it safe:

1. Compute out-of-fold: each row’s encoding must come from data that excludes that row (and, for grouped data, excludes its entity entirely).
2. Smooth toward the global mean so rare categories aren’t taken literally:

def smoothed_target_encode(train, col, target, alpha=20):
    global_mean = train[target].mean()
    stats = train.groupby(col)[target].agg(["mean", "count"])
    smooth = (stats["count"] * stats["mean"] + alpha * global_mean) \
             / (stats["count"] + alpha)
    return smooth  # map onto validation/production data; unseen -> global_mean

raw = df.groupby("country").churned.agg(["mean", "count"])
print(raw.join(smoothed_target_encode(df, "country", "churned")
               .rename("smoothed")).round(3).sort_values("mean"))

          mean  count  smoothed
country
FR       0.466    103     0.476
DE       0.469    147     0.476
GB       0.523    218     0.523
SG       0.536    110     0.535
US       0.541    290     0.540
CA       0.547     86     0.543
ES       0.558     52     0.549
PT       0.558     52     0.549
AE       0.567     67     0.558
NL       0.575     73     0.565

The global churn rate is 0.528, and alpha=20 pulls each country toward it in proportion to how little evidence it has: Portugal’s 52 rows move from 0.558 to 0.549, Germany’s 147 barely move at all.

It is also, on this file, a waste of effort. Ten countries spanning eleven points of churn is exactly the case one-hot encoding handles for free. Fed to a logistic regression on its own, the one-hot country column cross-validates at 0.48 ROC-AUC — under chance — while monthly_price alone reaches 0.61. There is no signal here for a clever encoding to extract. Target encoding earns its risk at thousands of categories, not ten.

In practice, use a maintained implementation (CatBoost does ordered target encoding internally; category_encoders offers CV-safe wrappers) rather than hand-rolling the fold logic under deadline pressure. And after fitting, audit it like a suspect: if target-encoded features dominate importance and your score jumped suspiciously, re-validate with grouped, time-aware splits before believing anything.

Choosing quickly

• Few categories → one-hot.
• True order → ordinal, with an explicit mapping you wrote down.
• Many categories, tree model → native categorical support or count encoding.
• Many categories, strong signal suspected → target encoding, out-of-fold, smoothed, then audited.
• Any encoding, any model → fit it inside the pipeline, on training folds only, with a defined behaviour for unseen values.

A concrete next step: open the last project that used target encoding and find where the encoder was fitted. If it was fitted on the whole training set before the folds were made, the validation score in that write-up is too high. The number worth reporting is the one that comes back after the encoder moves inside the pipeline.