Feature engineering that survives contact with production

Most feature engineering tutorials end at the validation score. In practice, that’s the halfway point. The real test of a feature is whether the exact same value can be computed at the moment the model is asked for a prediction, using only information that exists at that moment. A surprising number of features fail this test, and the failure is invisible offline.

The time-travel test

For every feature, ask: at prediction time, what will I actually know?

Suppose you’re predicting whether a customer will churn this month, and you build a feature orders_this_month. In your training data — assembled after the month ended — that count is complete. In production, on the 4th of the month, it’s a partial count. Your model learned from a version of the feature that will never exist when it matters. Offline metrics look great; production performance quietly sags, and nothing errors out to warn you.

The disciplined way to build training data is a point-in-time join: pick the moment the prediction would have been made, and compute every feature using only data stamped before that moment. pandas spells this merge_asof when the join is against a timestamped table.

The event file in the public dataset is a timestamped table of exactly this kind — 40,000 rows, of which 2,493 are purchases with an amount.

import pandas as pd

events = pd.read_csv("https://dataacademy.ai/data/events.csv",
                     parse_dates=["occurred_at"])
purchases = events[events.event_type == "purchase"]

# Build features as-of the prediction date, not as-of "now"
def features_asof(customer_id, asof, purchases):
    past = purchases[
        (purchases.customer_id == customer_id) & (purchases.occurred_at < asof)
    ]
    recent = past[past.occurred_at >= asof - pd.Timedelta(days=30)]
    return {
        "purchases_30d": len(recent),
        "avg_value_30d": recent.amount.mean(),
        "days_since_last": (asof - past.occurred_at.max()).days,
    }

april = purchases[(purchases.customer_id == 1333)
                  & (purchases.occurred_at.dt.to_period("M") == "2025-04")]
print(f"purchases_this_month on 15 Apr : {(april.occurred_at < '2025-04-15').sum()}")
print(f"purchases_this_month after close: {len(april)}")

for asof in ["2025-04-15", "2025-05-01"]:
    f = features_asof(1333, pd.Timestamp(asof), purchases)
    print(f"{asof}  30d={f['purchases_30d']}  "
          f"avg={f['avg_value_30d']:.2f}  since={f['days_since_last']}")

purchases_this_month on 15 Apr : 4
purchases_this_month after close: 6
2025-04-15  30d=7  avg=102.77  since=6
2025-05-01  30d=6  avg=108.92  since=9

Customer 1333 bought four times by the middle of April and six times by the end of it. A model trained on the month-end figure and served on the fifteenth is being handed a number two thirds the size of the one it learned from, every time, for every customer — and nothing errors. The trailing 30-day version underneath has no such gap: it means the same thing on any date it is computed.

It’s slower and more annoying than a single groupby over the whole table. It’s also the difference between a model that works and one that only demos well.

Two implementations of every feature is one too many

The classic production failure: features are computed one way in the training pipeline (pandas, batch) and reimplemented in the serving path (SQL, Java, whatever the application runs). The two versions drift — a different null default, a different rounding, a different definition of “active” — and the model receives inputs from a distribution it never saw.

You don’t need a feature store to avoid this. You need one of:

• A single implementation called from both paths (a shared function or SQL view).
• Logged features: at serving time, log the exact feature vector used; train the next model version on those logs instead of recomputing history.

The second option is underrated. It guarantees training data matches serving reality, because it is serving reality.

Prefer features that degrade gracefully

Some features are accurate but fragile: they depend on a third-party enrichment API, a table that lands at 6 a.m. sometimes-ish, or a schema another team can change without telling you. When the dependency hiccups, what does your feature become — null, zero, or stale? Each of those pushes the model in a different direction, and you should decide which one deliberately rather than discover it during an incident.

A good habit is to encode “missingness” explicitly. The feature built above needs this immediately: avg_value_30d is the mean of an empty slice for any customer who has not bought recently, and most of them have not.

asof = pd.Timestamp("2026-01-01")
window = purchases[(purchases.occurred_at < asof)
                   & (purchases.occurred_at >= asof - pd.Timedelta(days=30))]

X = pd.DataFrame(index=sorted(events.customer_id.unique()))
X["avg_value_30d"] = window.groupby("customer_id").amount.mean()

X["has_recent_purchase"] = X.avg_value_30d.notna().astype(int)
X["avg_value_30d"] = X.avg_value_30d.fillna(0.0)

print(f"{X.has_recent_purchase.sum()} of {len(X)} customers")

91 of 1085 customers

Ninety-one. Without the flag, 994 customers get an average purchase value of zero and the model cannot tell “bought nothing” from “bought something worth nothing” — and worse, if someone had filled the gap with the column mean instead, every dormant customer would arrive looking like an active one.

Now an upstream outage moves predictions along a path the model actually trained on, instead of feeding it a fabricated default it has never seen.

Boring beats clever, again

The features that survive years in production are rarely exotic. They’re counts, recencies, ratios, and flags — computed correctly as-of prediction time, from tables someone actually maintains. A clever embedding of session clickstreams that only one ex-employee’s notebook can reproduce is a liability, not an asset.

A reasonable priority order when inventing features:

1. Can it be computed at prediction time, from data that will exist then?
2. Will it still be computable in six months, by someone who isn’t you?
3. Does it move the validation metric — measured with a point-in-time-correct split?

Note that “does it help the score” comes last. That ordering usually gets settled in a single meeting. Someone presents a feature worth two points of AUC. Someone else asks which table it comes from. The answer is a nightly export that a partner team rebuilds whenever their upstream changes, with no owner and no schedule anyone outside that team can see. The feature gets dropped, the model ships two points worse, and nobody is woken at 3am in March.