Logistic regression is the first model most people meet after linear regression, and it is usually taught backwards. Here is a formula with an exponential in it, here is the sklearn call, off you go. That order hides the only question worth asking first: why can’t you fit a straight line to a yes/no outcome and be done with it?
Answer that properly and the model builds itself. The sigmoid, the log-odds, the strange-looking loss — each one turns out to be the only sensible fix for a problem the straight line creates. None of it is arbitrary, and none of it needs to be memorised.
It is worth the hour. Logistic regression is still the default first classifier in most serious shops, for the same reasons it always was: it trains in seconds, it produces probabilities rather than labels, and you can read what it learned. That is the simplest model that could possibly work for a large share of real problems.
The straight line breaks on the first day
Say you want to predict whether a customer cancels. The outcome is 0 or 1. Fit ordinary least squares of cancellation on tenure in months and look at what comes out.
Some predictions are below 0. Some are above 1. You cannot clip your way out of that. Clipping is an admission the model is wrong, and the bad fit at the edges drags the line in the middle too, so the predictions you did believe get worse.
The deeper problem is the shape of the claim. A linear model says every extra month of tenure changes the chance of cancelling by the same fixed amount. That cannot be true. Moving a customer from 0.50 to 0.55 is a real change; moving one from 0.98 to 1.03 is not a change, it is nonsense. Probabilities live in a box, and effects have to shrink as you approach the walls.
Least squares has a second complaint. It assumes the noise has constant
spread. For a 0/1 outcome the variance is p(1 - p), which depends on the
prediction itself. The assumption is broken before you start.
Stretch the target, don’t bend the model
The fix is not to force a linear output into [0, 1]. It is to stretch
[0, 1] out over the whole number line, fit the line there, and map back.
Two steps do it. First, odds: p / (1 - p). A probability of 0.5 becomes
odds of 1, and the range opens up to everything from 0 to infinity. Second,
take the log. Now the range is the entire line, and 0 sits at the halfway
point, with symmetric behaviour either side.
So the model is a straight line on the log-odds:
log(p / (1 - p)) = b0 + b1·x1 + b2·x2 + ...
Solve that for p and you get p = 1 / (1 + exp(-z)), where z is the
right-hand side. That is the sigmoid, drawn below. It was not picked from a
catalogue because the S-curve looked nice. It is simply what falls out when
you undo the log and the odds.
The line lives on the log-odds axis and runs the whole width of the number line; the sigmoid folds it into the 0-to-1 box. Near z = 0 a unit of z moves the probability a lot. Out at the edges it moves almost nothing.
Everything else follows. Effects are additive in log-odds, multiplicative in odds, and squashed in probability. The same coefficient moves the probability a lot near 0.5 and almost nothing near 0 or 1 — exactly the shrinking-at-the-walls behaviour the straight line could not produce.
Reading a coefficient
Exponentiate it. exp(b) is the odds ratio for a one-unit increase in that
feature, holding the others fixed.
| Coefficient | exp(b) | Plain reading |
|---|---|---|
| -0.69 | 0.50 | one unit halves the odds |
| -0.10 | 0.90 | one unit cuts the odds ~10% |
| 0.00 | 1.00 | no effect |
| 0.10 | 1.11 | one unit raises the odds ~11% |
| 0.69 | 2.00 | one unit doubles the odds |
| 1.10 | 3.00 | one unit triples the odds |
For a rough probability reading, divide the coefficient by 4. That is the steepest slope the sigmoid ever reaches, so a coefficient of 0.2 moves the probability by at most about 5 percentage points per unit — and less than that everywhere except near 0.5.
import numpy as npfrom sklearn.linear_model import LogisticRegression
X = np.array([[0.0], [1.0], [2.0], [3.0], [4.0], [5.0]])y = np.array([0, 0, 0, 1, 1, 1])
model = LogisticRegression().fit(X, y)b = model.coef_[0][0]
print(b) # change in log-odds per unitprint(np.exp(b)) # odds ratio per unitprint(model.predict_proba([[2.5]])[0, 1]) # probability at x = 2.5Trust the sign. Be careful with the size. It depends on the units of the feature and on what else is in the model — two correlated features split the credit between them in a way that has nothing to do with which one matters.
Why the loss is cross-entropy
Each observation is a coin flip with its own bias p_i. The chance of
seeing the data you saw is the product of p_i for the ones and
(1 - p_i) for the zeros. Take the log to turn the product into a sum, and
the negative to make it something to minimise:
-Σ [ y·log(p) + (1 - y)·log(1 - p) ]
That is cross-entropy. Nobody designed it; it is the likelihood, written down. Three things follow from that.
It punishes confident mistakes without limit. As p goes to 0 for a case
that was actually 1, -log(p) goes to infinity. Squared error caps the
penalty at 1, so a model can be catastrophically sure and wrong for a
bounded price. Cross-entropy makes that unaffordable, which is what forces
the numbers to behave like probabilities.
It is well behaved to optimise. Paired with the sigmoid, squared error is
not convex in the weights, so the fit depends on where you started.
Cross-entropy is convex: one optimum, found reliably. Its gradient is also
about as simple as gradients get — (p - y)·x, the error times the
feature.
And it has a clean meaning. Cross-entropy is the average number of bits needed to encode what actually happened using the probabilities you predicted. Wrong confidence literally costs bits. It is a proper scoring rule, which means your best move is to report what you truly believe — shading the number toward a nicer-looking answer always scores worse.
Where the intuition breaks
Coefficients are not causes. “Holding the others fixed” is an arithmetic statement about the fitted equation, not a claim about the world.
If a feature separates the classes perfectly, the likelihood keeps improving as its coefficient runs to infinity. The fit will not converge sensibly. Regularisation is the standard cure, and sklearn applies it by default — which is also why the raw coefficients are slightly shrunk.
“Linear” means linear in the log-odds, not in your features. Interactions, splines, and sensible transformations are still your job.
Calibration is conditional, not guaranteed. Fit the model, then resample the classes or apply class weights, and the intercept shifts: ranking survives, the probability level does not. That is a common way good models start lying with their probabilities.
For the fitting step drawn out slowly, watch StatQuest: Logistic Regression by StatQuest with Josh Starmer.
Once the link function clicks, a lot of machine learning stops looking like a pile of separate techniques. Softmax is this same argument with more than two outcomes. The last layer of a neural network classifier is logistic regression, run on features the earlier layers invented. What travels with the model is its constraint: it is linear in the log-odds, so the boundary it can draw is a straight one in whatever space it is handed. Interactions, splines and learned representations all work by changing that space. None of them change the model.