TF-IDF still works, and it should be your first NLP model

A ticket-classification system took a fine-tuned transformer, a GPU box and three weeks of work. Four months after it shipped, someone ran TF-IDF and logistic regression on the same split out of curiosity. That model trained in eight seconds on a laptop and landed roughly two points behind — at a thousandth of the cost, with none of the latency, and with coefficients you can read. The comparison belonged in week one.

The recipe

TF-IDF turns each document into a sparse vector: each word (or character n-gram) weighted by how frequent it is in this document and how rare across documents. Common-everywhere words score low; distinctive words score high. Feed those vectors to a linear model and you have a text classifier:

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

clf = Pipeline([
    ("tfidf", TfidfVectorizer(
        ngram_range=(1, 2),      # words and word pairs
        min_df=3,                # ignore near-unique tokens
        sublinear_tf=True,
    )),
    ("model", LogisticRegression(max_iter=2000, C=1.0)),
])
print(cross_val_score(clf, texts, labels, cv=5, scoring="f1_macro").mean())

For noisy, misspelled, multilingual, or short text (tickets, search queries, product titles), swap in character n-grams — analyzer="char_wb", ngram_range=(3, 5) — which shrug at typos that break word-level features.

Why it holds up

Many commercially valuable text tasks are, at bottom, keyword-ish: routing (“invoice”, “refund”, “password”), spam and abuse filtering, topic tagging, intent detection with distinctive phrasings. When the signal lives in which words appear, a linear model over n-grams captures most of it — transformers add value mainly where meaning hides in composition, word order, or context (“not bad at all”, sarcasm, coreference). Plenty of production tasks simply don’t need that, and paying transformer costs for keyword problems is buying a crane to lift a chair.

The operational advantages compound: training in seconds means you iterate on data and labels instead of infrastructure; inference is microseconds on a CPU, deployable inside any service with no GPU line item; and explanations come free — the model’s coefficients are literally a ranked vocabulary per class:

import numpy as np
vec, model = clf.named_steps["tfidf"], clf.named_steps["model"]
names = np.array(vec.get_feature_names_out())
for i, cls in enumerate(model.classes_):
    top = np.argsort(model.coef_[i])[-8:]
    print(cls, "→", ", ".join(names[top]))

That printout has ended more than one “why did it classify this as billing?” debate in thirty seconds — and it’s a data-quality audit in disguise: when a class’s top features include an agent’s name or the word “Tuesday”, you’ve found leakage or a labelling artifact before it hides inside a fine-tune.

One run, start to finish

The abstract case is less useful than watching the baseline catch its own mistakes. Take a support inbox: five queues (billing, technical, account, shipping, other), a few tens of thousands of labelled tickets, badly balanced — “other” is roughly half the data and “shipping” is a few percent of it. The numbers below are illustrative of the shape of this work, not a benchmark.

The first run scores macro-F1 in the high 0.70s. Encouraging, so print the per-class report rather than the average:

from sklearn.model_selection import cross_val_predict
from sklearn.metrics import classification_report

pred = cross_val_predict(clf, texts, labels, cv=5)
print(classification_report(labels, pred, digits=3))

Two problems show up immediately. Billing scores near-perfect — far better than any other class. Shipping scores close to zero: the model almost never predicts it.

The billing result is the suspicious one. A model does not get 0.99 on messy human text. The coefficient printout explains it in one line: the top feature for billing is bil, because every ticket forwarded from the billing system starts with the string Ref: BIL-2026-. The classifier learned the routing system’s own stamp, which is present in training data and absent (or present for different reasons) at prediction time. That is textbook leakage, and it is the single most common way a text baseline lies to you. Strip the template before vectorizing — a regex over the known prefixes — and billing drops back into the 0.80s, which is the real number.

Shipping is a threshold problem, not a text problem. With a rare class, the default decision rule almost never fires. class_weight="balanced" reweights the loss by inverse class frequency and costs nothing:

LogisticRegression(max_iter=2000, C=1.0, class_weight="balanced")

Shipping recall moves from almost nothing to usable, precision drops some, and macro-F1 rises because the average was being dragged down by one dead class. Then swap the word analyzer for char_wb n-grams, because tickets are short and full of misspelled product names, and it moves again. Total elapsed time for all three fixes: an afternoon, most of it reading the coefficient dumps.

That afternoon is the argument. Two of those three problems — the template prefix and the dead rare class — would have existed in the transformer too, and would have been much harder to see.

The mistake that inflates every score

One detail in the first code block is doing real work: the vectorizer lives inside the Pipeline. The common shortcut is to fit it first:

# Wrong. Do not do this.
X = TfidfVectorizer(min_df=3).fit_transform(texts)
scores = cross_val_score(LogisticRegression(), X, labels, cv=5)

That vectorizer computed document frequencies — and the surviving vocabulary — over every fold, including the ones it is about to be tested on. The leak is small per feature and reliable in direction: cross-validation reports a score the deployed model will never reproduce. Inside a Pipeline, cross_val_score refits the vectorizer on each training fold and the number becomes honest.

The same rule applies to any preprocessing that reads the label column or the whole corpus: fit on train, apply to test, always, and let the pipeline enforce it so nobody has to remember.

Where the ceiling is

Be honest about the failure modes: bag-of-words has no notion of similarity between “refund” and “reimbursement”, which is exactly what embeddings are built to capture; it degrades gracefully but noticeably on tasks needing context windows, and offers nothing generative.

Three more limits show up in production. Very short text — two or three tokens, like search queries — leaves almost nothing for IDF to weight, and character n-grams help more than word features there. Vocabulary drifts: a new product line ships, its name is out-of-vocabulary, and accuracy on that segment falls without any alert firing, so the refit cadence is part of the design, not an afterthought. And character n-grams on a large corpus can produce well over a million features; if memory becomes the constraint, max_features or a HashingVectorizer trades a little accuracy for a fixed footprint.

When the baseline plateaus short of the requirement, escalate deliberately: add embedding features to the same linear model, or move to a small fine-tuned transformer — measured against the baseline you now have.

That’s the real point. The TF-IDF baseline isn’t nostalgia; it’s the denominator. “The transformer scores 0.91” is marketing until you can say “…versus 0.88 for a model that costs nothing.” Sometimes those three points justify the GPUs. Surprisingly often, they don’t — and you can only know which story you’re in if the eight-second model ran first.