L2 regularization (Ridge) penalizes the sum of squared coefficients — it shrinks every coefficient smoothly toward zero, without typically reaching it exactly, making it the standard choice when you want a simpler model without discarding any features entirely.
Formula
Worked Example — Shrinkage in a Simplified Case
For a single feature \(x=2\), target \(y=10\) (so the unregularized OLS coefficient is \(w_{\text{OLS}}=xy/x^2=5\)), the Ridge-regularized solution for this simplified case works out to:
| λ | Calculation | w |
|---|---|---|
| 0 (unregularized) | \(20/4\) | 5.0 |
| 4 | \(20/(4+4)\) | 2.5 (shrunk by half) |
| 16 | \(20/(4+16)\) | 1.0 (shrunk further, still nonzero) |
Notice the coefficient keeps shrinking smoothly as \(\lambda\) grows, but — unlike L1 — never actually reaches exactly zero, no matter how large \(\lambda\) gets.
from sklearn.linear_model import Ridge
import numpy as np
X = np.random.rand(100, 20)
y = 3*X[:,0] - 2*X[:,1] + 5*X[:,2] + np.random.normal(0, 0.1, 100)
for alpha in [0, 1, 10, 100]:
model = Ridge(alpha=alpha).fit(X, y)
print(f"alpha={alpha}: max|coef|={np.max(np.abs(model.coef_)):.3f}, "
f"min|coef|={np.min(np.abs(model.coef_)):.3f}")
# All coefficients shrink together as alpha grows -- none typically hit exactly zeroL1 vs L2 — When to Choose Which
| L1 (Lasso) | L2 (Ridge) | |
|---|---|---|
| Produces sparse (zero) coefficients? | Yes | No — shrinks smoothly, rarely exactly zero |
| Handles correlated features well? | Arbitrarily picks one, can be unstable | Shares weight smoothly across correlated features — more stable |
| Best for | Suspecting many irrelevant features exist | Believing most/all features contribute somewhat |
| Interpretability benefit | Automatic feature selection built in | None directly — all features remain, just shrunk |
Why L2 Handles Multicollinearity Better Than L1
When two features are highly correlated, L2's squared penalty distributes the coefficient "weight" between them roughly evenly, since splitting a large coefficient into two smaller ones reduces the sum-of-squares penalty. L1's absolute-value penalty has no such incentive to split — it's often just as happy to put all the weight on one of the correlated features and zero out the other, which can be an arbitrary, unstable choice.
Practical Use Cases
- The standard default regularization choice for linear/logistic regression when feature selection specifically isn't the goal
- Situations with meaningfully correlated features, where L2's smoother, more stable shrinkage is preferable to L1's sometimes-arbitrary sparsity
Common Mistakes
- Choosing L2 when the actual goal is feature selection — L2 alone won't zero out any coefficients, however small \(\lambda\) makes them.
- Not tuning \(\lambda\)/alpha via cross-validation, same caveat as any regularization technique.
Interview Relevance
Q: "Why might Ridge be preferable to Lasso when features are highly correlated?" Ridge's squared penalty distributes weight relatively evenly across correlated features rather than arbitrarily favoring one and zeroing the rest — producing a more stable model whose coefficients don't swing unpredictably with small changes in the training data.
Practice Question
Using the shrinkage formula, compute \(w_{\text{Ridge}}\) for \(x=3\), \(y=12\), \(\lambda=9\).
