Cross-entropy, log loss and perplexity are the same idea

Three numbers land in the same week. The fraud team reports a log loss of 0.42. The search team reports a perplexity of 18.7. A paper everyone is passing around reports 2.6 bits per character. Someone asks which model is doing best, and the room goes quiet — not because the answer is subtle, but because nobody has noticed that all three are measurements of the same thing.

One quantity, measured on the true outcome

Take an example whose real answer you know. Ask the model what probability it gave that answer. Take the logarithm, flip the sign. Average over the dataset.

That is the whole quantity: the mean negative log probability of what actually happened. If the model gave the truth a probability of 1, the term is 0 — no surprise. As the probability falls, the term grows without limit.

The textbook formula sums over all classes, weighting each by the true label’s probability, but with ordinary one-hot labels every term except one is multiplied by zero. What survives is -log p(true class). The general form matters when the targets are themselves distributions — soft labels, distilled teachers, smoothed targets — and otherwise it is the same arithmetic dressed formally. That sum is what information theory calls cross-entropy: the average cost of describing outcomes drawn from one distribution using a code built for another. When the code fits, the cost drops to the entropy. When it does not, the excess is exactly the model’s error.

The same number in three sets of clothes

Machine learning calls it log loss, computes it in nats because np.log is natural, and reports it per example. Deep learning frameworks call it cross-entropy loss and fold the softmax into it. Information theory calls it cross-entropy, prefers bits, and reads it as description length. Language modelling calls it perplexity, exponentiates it, and reports one number per token.

The conversions are exact:

import numpy as np

nll  = -np.log(p_true).mean()   # nats per example — this is log loss
bits = nll / np.log(2)          # cross-entropy in bits
ppl  = np.exp(nll)              # perplexity == 2 ** bits

One nat is about 1.443 bits. Perplexity is the exponential of the mean cross-entropy in nats, or two raised to it in bits — the same climb, expressed in whichever base was used on the way down. A log loss of 0.693 nats is 1 bit is a perplexity of 2, and that triple is worth memorising because it is the score of an honest coin.

Perplexity is a branching factor

Exponentiating turns the average surprise back into a count of options. A model with a perplexity of 18.7 is, on average, as uncertain as if it were picking uniformly among 18.7 equally likely choices at each step. Uniform over 50,000 tokens gives a perplexity of exactly 50,000, which is the number an untrained language model starts from. That is why the metric caught on: everyone can hold “about nineteen plausible next words” in their head, and nobody has intuition for 2.93 nats.

The catch is that the count depends on what you are counting. A model with a large vocabulary makes fewer, harder predictions per sentence; a character-level model makes many easy ones. Their perplexities are not comparable, and neither are two subword models with different tokenisers. Bits per character or per byte normalise this away by fixing the unit of text rather than the unit of prediction, which is why serious comparisons are quoted that way. Perplexity across tokenisers is a marketing number.

Confident and wrong is punished hard, on purpose

Assign 0.01 to the outcome that happens and the loss for that example is 4.6 nats. Assign 0.5 and it is 0.69. Assign 0.001 and it is 6.9. One badly overconfident example can outweigh a hundred mildly uncertain ones, and there is no ceiling.

This is deliberate. Log loss is a proper scoring rule: the way to minimise it is to report the probability you actually believe, and any attempt to game it by shading predictions toward the extremes loses on average. That property is what makes it the right training objective for anything where the probability itself has to mean something, rather than just the ranking.

The same property turns hostile when the labels are wrong. A mislabelled example that the model confidently and correctly disbelieves produces an enormous loss and an enormous gradient, and training will bend the model toward the mistake. With a few percent label noise this shows up as a model that seems to be learning something strange late in training.

Two things help. Label smoothing caps the target at, say, 0.95 instead of 1.0, which caps the reward for extreme confidence and blunts the pull of bad labels. And sorting the training set by per-example loss and reading the top hundred rows is the fastest label audit available — the worst offenders are usually not hard cases but plain errors, and finding them costs one scoring pass.

Is 0.42 any good?

There is no universal answer, because log loss has no fixed range. The only honest read is against a baseline: what does a model score if it predicts the base rate for every row and knows nothing else?

For a 3% positive rate, that constant model scores about 0.135 nats. So a log loss of 0.42 on that problem is not mediocre — it is far worse than predicting the same number every time, which usually means the probabilities are on the wrong scale or the classes got swapped. Meanwhile a log loss of 0.42 on a balanced problem, where the coin-flip baseline is 0.693, is a genuine improvement. The identical number means opposite things one dataset apart, which is why quoting log loss alone tells a reader nothing.

Report the pair, or report the ratio. One minus your loss over the baseline loss gives the fraction of the baseline’s uncertainty the model removed, which is comparable across problems in a way the raw figure never is. Language modelling does the same thing informally when it compares against a unigram model built from the same corpus.

Three fields measured how surprised a model was, invented a name each, and never compared notes. The practical gain from knowing that is portable intuition: the sharp penalty a language modeller feels for a confident wrong token is the same penalty sitting in a classifier’s training loop, and the baseline discipline from one transfers straight to the other. A perplexity of 18.7 and a log loss of 2.93 are one sentence in two languages, and so is “the model was surprised.”