Entropy, or how to measure surprise

Two colleagues send a message. One writes “the sun came up”. The other writes “the warehouse burned down”. Both are one short sentence. Only one of them told you anything.

That gap is what information theory measures. The content of a message has nothing to do with its length or its importance to you personally. It has to do with how unlikely it was. A message that says something you already expected carries almost nothing. A message that says something you thought improbable carries a lot. Information is surprise.

Once surprise is the quantity, the rest follows almost by force. Ask what a sensible measure of surprise must do, and there is essentially one answer — and that answer turns out to be the loss function you already use to train classifiers, and the rule a decision tree uses to pick its splits. Same idea, three costumes.

What surprise has to look like

Two requirements pin the formula down.

First, surprise must fall as probability rises. Something certain (p = 1) must carry zero surprise. Something nearly impossible must carry a lot.

Second, surprise from independent events must add up. If you learn one coin came up heads and, separately, that a second coin came up heads, the total surprise should be the sum of the two. But probabilities multiply when events are independent: 1/2 times 1/2 is 1/4.

So you need a function that turns multiplication into addition and decreases with p. That is a logarithm with a minus sign in front:

surprise(x) = -log p(x)

Not a convention, not a choice someone made for tidiness. Those two demands leave no other option, up to the base of the log. Shannon ran the argument the same way in A Mathematical Theory of Communication in 1948, deriving the formula from a short list of properties any honest measure of uncertainty would have to satisfy.

Why bits

The base sets the unit. Base 2 gives bits, and a bit has a concrete meaning: the answer to one yes/no question, when both answers were equally likely.

Check it. A fair coin flip has p = 1/2, so -log₂(1/2) = 1 bit. Exactly one yes/no question’s worth. A fair die roll has p = 1/6, giving about 2.58 bits — you cannot pin down a die with two yes/no questions, and three is slightly more than enough. The number is the length of the shortest yes/no interrogation that identifies the outcome.

Base e gives nats instead. Nothing changes but the scale, which is why libraries switch between them without much ceremony. sklearn’s log_loss uses natural log; a paper reporting “bits per character” uses base 2. Same quantity, different ruler.

Entropy is the average

Surprise applies to a single outcome. Entropy applies to the whole distribution: it is the average surprise you expect, weighting each outcome by how often it happens.

H(p) = -Σ p(x) log p(x)

Read it as: how uncertain is this thing, in bits, before you look?

import numpy as np

def entropy(p):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]                       # 0 log 0 counts as 0
    return -(p * np.log2(p)).sum()

entropy([0.5, 0.5])          # 1.0   — a fair coin
entropy([0.99, 0.01])        # 0.081 — a coin that is nearly always heads
entropy([0.25] * 4)          # 2.0   — four equally likely outcomes

Uniform is the maximum. Anything lopsided is lower. A distribution with all its mass on one outcome has zero entropy — nothing to learn, because you already know the answer. A single coin shows the whole shape.

Entropy of one coin against the probability of heads. It is zero at both ends, where the answer is known before you look, and it peaks at exactly 1 bit when the coin is fair. Notice how flat the top is: a 60/40 coin still carries 0.97 bits.

This gives you an immediate, practical read on a dataset. The entropy of your label column tells you how much uncertainty a perfect model would have to remove. At 99/1, that is 0.08 bits, and a model that always predicts the majority class already captures nearly all of it. Which is one more way of saying what accuracy hides on imbalanced problems.

Cross-entropy: the loss you already train with

Now the useful twist. Entropy assumes you know the true distribution. In practice you have a model, q, that is guessing at a truth, p.

Cross-entropy asks: if you build your expectations around q, but the world actually runs on p, what is your average surprise?

H(p, q) = -Σ p(x) log q(x)

It is never smaller than H(p). The excess — the extra bits you pay for believing the wrong distribution — is the KL divergence. Minimising cross-entropy is exactly minimising that gap.

For a classifier, p is the observed label: 1 for the true class, 0 elsewhere. The sum collapses to a single term, -log q(true class), averaged over the data. That is log loss. That is what torch.nn.CrossEntropyLoss computes, and what every neural network classifier is minimising.

Which explains a behaviour that puzzles people the first time: log loss punishes confident mistakes viciously. Predict 0.01 for something that happens and you pay -log₂(0.01) ≈ 6.6 bits. Predict 0.4 and you pay 1.3. The loss is not measuring whether you were on the right side of a threshold. It is measuring how astonished the world made you. A model tuned on this loss has a reason to report honest probabilities — and a reason to be checked for calibration once someone starts making decisions from those numbers.

Information gain: why a tree splits where it does

A decision tree faces one question at every node: of all available splits, which one helps most?

Entropy answers it directly. Compute the entropy of the labels at the node. Compute the entropy in each child after a candidate split, weighted by how many rows land there. The drop is the information gain — literally, the bits of uncertainty that split removed.

Start at 1.0 bit with a balanced node. Find a split that produces two pure children, and both are at 0. Gain: 1.0 bit, all the uncertainty gone. A split that leaves both children as mixed as the parent gains nothing, and the tree passes it over.

ID3 and C4.5 do nothing beyond this. CART, which is what most people actually run, defaults to Gini impurity instead — a cheaper measure that ranks splits almost identically and skips the logarithms. In sklearn, criterion= "entropy" gives you the information-theoretic version. The trees you get are rarely very different, which is itself informative about how much the exact measure matters.

One last thing about the word

Entropy carries baggage from physics, where it is usually explained with heat, disorder and a broken cup that will not reassemble. The physical and the information versions are genuinely the same mathematics, applied to the number of arrangements a system might be in. If you want that link drawn properly, the best popular treatment is The Most Misunderstood Concept in Physics by Veritasium.

For daily work, though, the plain version is enough and more useful: entropy is how many yes/no questions the answer is worth. Your loss function counts the ones your model still has to ask.