MAE, RMSE, or MAPE: what your regression metric secretly optimises

Regression metrics look interchangeable — they all summarise “how far off were we?” — so most people pick by habit. But each metric implies a different ideal prediction, and a model tuned to the wrong one is systematically aimed at the wrong target. This isn’t a reporting nuance; it changes what the model learns to say.

The core fact: metrics have optimal predictions

For a given uncertain outcome, the prediction that minimises each metric is different:

• RMSE (root mean squared error) is minimised by the mean of the outcome distribution. Squaring makes large errors dominate, so RMSE-trained models stretch to accommodate extremes.
• MAE (mean absolute error) is minimised by the median. Errors count linearly; the model happily ignores the tail’s magnitude as long as it’s on the right side half the time.
• MAPE (mean absolute percentage error) is minimised by something below the median. The asymmetry: an under-forecast can be wrong by at most 100%, while an over-forecast on a small actual explodes the percentage without bound — so MAPE rewards predicting low. A MAPE-tuned demand forecast is a built-in understocking machine.

That is not a derivation, it is arithmetic, and it takes six lines to check. The amounts below are the 2,493 purchases in the public dataset; the task is to find the single constant each metric prefers.

import numpy as np
import pandas as pd

events = pd.read_csv("https://dataacademy.ai/data/events.csv")
y = events[events.event_type == "purchase"].amount.to_numpy()

grid = np.linspace(y.min(), y.max(), 20_001)
for name, loss in [("RMSE", lambda c: np.sqrt(((y - c) ** 2).mean())),
                   ("MAE",  lambda c: np.abs(y - c).mean()),
                   ("MAPE", lambda c: (np.abs(y - c) / y).mean())]:
    best = grid[np.array([loss(c) for c in grid]).argmin()]
    print(f"{name:5s} minimised at {best:7.2f}")
print(f"mean {y.mean():.2f}   median {np.median(y):.2f}")

RMSE  minimised at   75.15
MAE   minimised at   29.58
MAPE  minimised at    7.92
mean 75.15   median 29.58

RMSE lands on the mean to the cent, MAE on the median to the cent, and MAPE on 7.92 — a quarter of the median and a tenth of the mean. Three metrics, one column of data, and the “best” answer ranges over a factor of nine. If those were demand forecasts, the MAPE-tuned one would stock a ninth of what the RMSE-tuned one stocks, and both teams would report that their model is performing well.

On skewed data — revenue, demand, latency — mean and median differ a lot, so “MAE model” and “RMSE model” genuinely predict different numbers for the same customer. Neither is wrong; they answer different questions. The question is which error your business actually pays for.

Choosing by consequence

Are big misses disproportionately bad? Capacity planning, safety stock, anything with a cliff: RMSE’s squaring mirrors that reality. If a 10× miss is merely 10× as bad, MAE’s linearity is the honest choice and is far less hijacked by a few outliers.

Do you need percentages for communication? Stakeholders love MAPE because “we’re off by 12%” feels intuitive. Report it if you must, but know its pathologies: undefined at zero actuals, explosive near them, asymmetric between over- and under-forecasting. WMAPE (total absolute error / total actuals) keeps the percentage flavor while fixing the worst behaviour:

wmape = np.abs(y_true - y_pred).sum() / np.abs(y_true).sum()

Are the costs asymmetric? Usually yes: understocking costs a lost sale, overstocking costs storage. Then no symmetric metric is right — use quantile (pinball) loss and predict the quantile matching the cost ratio. If a lost sale hurts 3× as much as an overstock, predict the 75th percentile of demand, not the middle of anything:

import lightgbm as lgb
model = lgb.LGBMRegressor(objective="quantile", alpha=0.75)

The evaluation hygiene that goes with it

• Always report against a baseline — error relative to “predict the historical average” (that’s what R² does) or the seasonal-naive forecast. “MAE = 4.2” means nothing; “MAE 4.2 vs 6.8 naive” is a contribution. Both are sample means, so they wobble from one test set to the next.
• Slice it. One aggregate hides everything: error by segment, by size decile, by horizon. Models are routinely great on the fat middle and useless on the expensive tail — which the average obligingly conceals.
• Check the bias, not just the spread. Mean error (signed) near zero tells you the model isn’t systematically high or low overall; slicing the signed error reveals where it is. A model can have great MAE and still under-predict every large customer — the ones where it matters.
• Train and evaluate on the same loss when you can. Training on RMSE and judging on MAE leaves free accuracy on the table; every library above lets you set the objective.

The hard part is that the cost structure is usually unwritten, and the person who knows it does not think in metrics. Two questions get further than asking which metric they prefer: what happens when the forecast is 20% high, and what happens when it is 20% low. If those answers differ, no symmetric metric fits the problem, and the choice is between an asymmetric loss and saying plainly that the reported number understates one side.