This note answers the "why," not the "what": why linear regression fails at classification, why the sigmoid function specifically fixes it, and what the log-odds interpretation actually means.
Why Not Just Use Linear Regression for Classification?
A straight line fit to 0/1 data has no reason to stay bounded — it can predict "-0.3" or "1.4," which are meaningless as probabilities.
Linear regression's output is unbounded — it can be any real number, positive or negative, of any size. A "probability" of -0.3 or 1.4 is nonsensical. Linear regression also assumes a constant-rate linear relationship, but the actual relationship between a feature and a class probability is naturally S-shaped: probability changes slowly near the extremes (very unlikely stays very unlikely) and fastest near the middle (where the outcome is genuinely uncertain) — exactly the shape the sigmoid function has, and a straight line doesn't.
The Log-Odds (Logit) Interpretation
Logistic regression is secretly still linear — just not in probability space. Rearranging the sigmoid equation:
The left side, \(\ln(\hat{y}/(1-\hat{y}))\), is the log-odds (or "logit") of the positive class. This equation says: logistic regression is a perfectly ordinary linear regression — but on the log-odds of the outcome, not the probability itself. This is exactly why the algorithm is called "logistic regression," not "logistic classification."
What a Coefficient Actually Means, in Log-Odds Terms
In linear regression, a coefficient \(b_1\) means "a one-unit increase in \(x\) changes \(\hat{y}\) by \(b_1\)." In logistic regression, a coefficient \(b_1\) means "a one-unit increase in \(x\) changes the log-odds of the outcome by \(b_1\)" — equivalently, it multiplies the odds by \(e^{b_1}\).
import numpy as np
b1 = 0.8 # a fitted coefficient, in log-odds units
odds_multiplier = np.exp(b1)
print(odds_multiplier) # 2.23 -- each unit increase in x roughly DOUBLES the odds of the positive classWhy the Cost Function Also Had to Change
Linear regression's MSE cost function, if applied directly to sigmoid outputs, produces a non-convex surface — full of local minima that gradient descent can get stuck in. Logistic regression instead uses log-loss, chosen specifically because it stays convex (bowl-shaped) even with the sigmoid nonlinearity in the mix.
Common Mistakes
- Believing "logistic regression" means it's somehow a regression algorithm used for continuous targets — the name refers to the log-odds linearity, not the type of prediction task.
- Interpreting a coefficient directly as a probability change rather than a log-odds (or odds-multiplier) change — a very common source of misstatement.
Interview Relevance
Q: "In what sense is logistic regression still 'linear'?" It's linear in the log-odds of the outcome — \(\ln(\hat{y}/(1-\hat{y})) = w^Tx+b\) is a perfectly ordinary linear equation; the sigmoid function is just the transformation that converts those log-odds back into a bounded [0,1] probability.
Practice Question
A logistic regression coefficient for "years_of_experience" is \(b_1 = 0.4\). Compute \(e^{0.4}\) and explain what it means about how the odds of the outcome change per additional year.
