Bayes theorem from first principles

A test for a disease is 99% accurate. You test positive. What is the chance you are ill?

Most people say 99%. Most doctors say something close to it. The right answer, for a disease that affects one person in a hundred, is about 50%. Nothing is wrong with the test. What is wrong is the instinct to judge the evidence without asking how many people had the disease to begin with.

Bayes’ theorem is the correction for that instinct. It is usually taught as a formula with four terms, which is why it feels slippery. Taught as a picture — counts of people, areas of a rectangle — it stops being slippery almost immediately.

Count people, not probabilities

Take 10,000 people. The disease affects 1%, so 100 of them are ill and 9,900 are healthy. That split is the base rate, and it exists before any test is run.

Now apply the test. Say it catches 99% of ill people and wrongly flags 1% of healthy people.

Look at the positive row. There are 198 positive results, and only 99 of them belong to ill people. Your positive result is one of 198 tickets in a hat, half of which are false alarms. Hence 50%.

The reason is not subtle once the table is in front of you. The healthy group is 99 times larger than the ill group. A 1% error rate on a huge group produces about as many positives as a 99% hit rate on a tiny one. Rare things stay fairly rare even after evidence that points at them.

Drawn as areas, with every region in true proportion, the argument needs no arithmetic at all:

The square is 10,000 people, split left to right by the base rate. The ill are the narrow strip on the left, and nearly all of it tests positive. The healthy are everything to the right of it, and 1% of that large area is the thin band across the top. The two shaded blocks are the same size — 99 people each — which is why a positive result is a coin flip.

The formula is just that table

The question “given a positive test, what fraction is ill?” is a ratio of two cells:

P(ill | positive) = (ill and positive) / (all positive)

Expand each piece into base rate times test behaviour, and Bayes’ theorem falls out with no invention required:

P(ill | positive) = P(ill) · P(positive | ill) / P(positive)

Every term is a piece of the table. P(ill) is the base rate, the size of the row. P(positive | ill) is how the test behaves inside that row. P(positive) is the whole positive column — the total, which is what makes the answer a proportion rather than a raw count.

The square above is that same table drawn as area, and it reads the same way: split left-right by the base rate, split each side top-bottom by the test’s behaviour, and the answer is the shaded strip on the left divided by both shaded regions together. That square is drawn and re-drawn in Bayes theorem, the geometry of changing beliefs by 3Blue1Brown.

Odds make it a one-liner

There is a form of Bayes worth memorising, because it can be done in your head:

posterior odds = prior odds × likelihood ratio

The likelihood ratio is how much more often the evidence shows up when the hypothesis is true than when it is false. For our test: 99% versus 1%, a ratio of 99. Prior odds of being ill are 1:99. Multiply: 99:99, which is 1:1, which is 50%. Same answer, no table.

This form makes the structure obvious. Evidence does not set your belief. It multiplies it. Strong evidence applied to a rare hypothesis still leaves you with a rare hypothesis — just less rare. Two independent positive tests would multiply again: 99:1 odds, or 99%.

It also shows what makes evidence strong. Not that it is likely under your hypothesis, but that it is unlikely under the alternatives. A symptom that every illness produces tells you nothing, however reliably it appears.

Where this shows up in the work

You do not need to run a Bayesian model to use any of this. The base-rate mistake turns up in ordinary machine learning work constantly, wearing different clothes.

Precision is Bayes. When a fraud model flags a transaction, the question “what fraction of flags are real fraud?” is exactly the table above. Recall is P(flag | fraud). Precision is P(fraud | flag). Teams quote the first and mean the second, then are surprised when analysts drown in false positives. With a 0.1% base rate, a model would need a false-positive rate near zero to make flags worth reading. This is the real content of the class imbalance problem — see class imbalance: thresholds, not SMOTE for what to do about it.

Alerting is Bayes. A monitoring rule that fires on a genuinely broken pipeline 95% of the time still produces mostly false alarms if pipelines break once a month and the rule runs hourly. Alert fatigue is a base-rate calculation that nobody did.

Model scores are Bayes. A model that outputs 0.9 is claiming a posterior. If it was trained on a rebalanced sample, its base rate is wrong and its probabilities are wrong with it, even when its ranking is fine. That mismatch is the subject of calibration: when probabilities lie.

Judging a surprising result is Bayes. A colleague reports a feature that lifts revenue 40%. Before checking the analysis, ask how often that is true. Rarely. So the likelihood ratio has to be enormous before the claim beats the alternative that something leaked. Usually it is leakage.

Full Bayesian inference — priors over parameters, posterior distributions, MCMC — is a larger commitment and often not worth it for a first model. The habit of thinking is worth it every day, and costs nothing.

What none of this settles is where the base rate itself comes from. Sometimes there is a register, a published prevalence, a clean count from last year. Often there is only an estimate with a wide range on it, and the posterior moves with the estimate — a prevalence of 0.5% instead of 1% halves the positive predictive value. Working out how much a conclusion rests on a base rate nobody has measured properly is a separate piece of work, and the arithmetic above will not do it for you.