Gradient boosting is still the default for tabular data

The rule for tabular data is short: on a business table — a few thousand to a few million rows, mixed numeric and categorical columns — start with gradient-boosted trees. The exception is real but narrow, and it arrives when the data stops behaving like a table: pixels next to pixels, samples in time, tokens in a sentence. Short of that, the boring model is also the strongest, cheapest and most forgiving one available.

Why trees fit tables

Tabular data is full of things trees handle natively and networks handle grudgingly: irregular, non-smooth relationships (thresholds, ceilings, weird kinks), features on wildly different scales (no normalisation needed), missing values (modern libraries route NaNs down a learned branch), and low-sample-size regimes where a network’s flexibility is a liability. A boosted ensemble builds trees sequentially, each correcting the residuals of the last — bias reduction by installments, with regularisation knobs at every step.

Practical consequence: your default stack for a new tabular problem is a dumb baseline, then logistic/linear regression, then LightGBM (or XGBoost, CatBoost, or scikit-learn’s own HistGradientBoostingClassifier — differences are marginal; CatBoost pulls ahead when high-cardinality categoricals dominate). Deep learning enters when the data stops being tabular: images, text, audio, sequences — or as embeddings feeding the boosted model.

The five knobs that matter

The libraries expose dozens of parameters; five carry nearly all the value:

• n_estimators — don’t tune it; set it high and let early stopping choose.
• learning_rate — lower is better but slower; 0.05 is a sane start.
• num_leaves / max_depth — model capacity, the main overfitting lever.
• min_child_samples — raises the evidence bar per leaf; increase on noisy data.
• subsampling (feature_fraction, bagging_fraction) — decorrelates trees, cheap regularisation.

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=5000,
    learning_rate=0.05,
    num_leaves=63,
    min_child_samples=50,
    feature_fraction=0.8,
    bagging_fraction=0.8, bagging_freq=1,
)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(200)],
)

Random search or Optuna over those ranges for an evening beats hand-fiddling for a week. The gains from tuning are real but modest — typically far smaller than the gains from one good feature.

The standard mistakes

Early stopping on the test set. The rounds count is a fitted parameter; if it’s chosen on the same data that produces your headline metric, the metric is biased up. Use three splits (train / early-stop validation / untouched test) or choose the round count inside cross-validation.

Trusting the probabilities raw. Boosted models rank beautifully and calibrate poorly. If anyone consumes the scores as probabilities, check the reliability curve and calibrate.

Ignoring native categorical support. One-hot encoding a 5,000-value category into a sparse cliff is self-harm; LightGBM and CatBoost split on categories directly — pass them as category dtype and delete the encoder.

Explaining with default importances. Impurity importance is biased toward high-cardinality features; use permutation importance or SHAP (fast for trees) when the “what drives this?” conversation starts.

Assuming extrapolation. Trees predict constants outside the training range. A boosted model trained on 2023 prices does not gracefully extend a trend into 2026 — it flatlines. For trending targets, model the trend separately or predict changes rather than levels.

The boring conclusion

The correct amount of excitement about gradient boosting is none — and that’s the compliment. It’s a mature tool that converts feature-engineering effort into accuracy at a better exchange rate than anything else on tables. Spend your innovation budget on the data — point-in-time correctness, better labels, one genuinely informative feature — and let the boring model bank it.