Evaluation
Measuring a model honestly — the metric that matches the decision, the leaks that flatter it, the score you can defend.
14 articles, newest first.
- Your model started dying the day you deployed it Data drift, concept drift, and delayed labels: a minimal monitoring setup that catches decay before the business does, without buying a platform.
- MAE, RMSE, or MAPE: what your regression metric secretly optimises Each error metric implies a different "best" prediction — median, mean, or something biased low. Choose by the cost of being wrong, not by habit.
- Cross-entropy, log loss and perplexity are the same idea Three names from three fields for one quantity: how surprised your model was. Where each shows up, how to read the units, and what a good value looks like.
- Are your clusters real? k-means always returns clusters, including on noise. How to tell structure from partition: stability, internal indices, and the check most people skip.
- Class imbalance: move the threshold before you reach for SMOTE Imbalanced classes are usually a thresholding and metrics problem, not a data problem. What actually helps, what mostly does not, how to pick a threshold from the cost of each mistake, and in which order to try things.
- Always build the dumb baseline first Majority class, yesterday-equals-today, one rule of thumb: an hour spent on a deliberately dumb baseline is the best ROI in a data project.
- Backtesting a forecast honestly Rolling-origin evaluation, the baselines a forecast must beat, and the quiet ways a backtest ends up scoring information the model would never have had.
- Your model says 90%. Should anyone believe it? Probability calibration: why good classifiers output bad probabilities, how to see it with a reliability curve, and when Platt scaling or isotonic regression fixes it.
- Time series forecasting: the five classic self-deceptions Random splits, ignored baselines, leaking features, log-space metrics, and one-step myopia — the mistakes that make forecasts look better than they are.
- Cross-validation without fooling yourself K-fold is the easy part. Grouped data, temporal data, and the quiet ways a validation scheme overstates how good your model really is.
- Accuracy is almost never the metric you want On imbalanced problems, a model that does nothing scores 99%. Precision, recall, PR curves, and how to pick a metric that matches the cost of being wrong.
- Data leakage is why your model looks too good If your validation score seems miraculous, the most likely explanation is that the answer leaked into the features. The common leaks, two worked through end to end, and how to audit for them.
- Why language models make things up Hallucination is not a bug in the usual sense. It follows from what the model is trained to do, and only some of the popular fixes address the actual cause.
- Overfitting and the bias-variance tradeoff What overfitting really is, what bias and variance each mean in concrete terms, how the tradeoff shows up in model choice, and where the classic curve stops holding.