L2 regularization adds a penalty proportional to the sum of squared weights — the most commonly used regularization technique in deep learning, and the direct mathematical basis for weight decay, covered next.
Formula
This is \(\frac{\lambda}{2}\|\mathbf{w}\|_2^2\) — the squared L2 norm, scaled by \(\lambda\) (the \(\frac{1}{2}\) is a common convention that simply cancels neatly with the squared term's derivative, and doesn't change what minimizes the loss).
Why L2 Shrinks Without Zeroing Out
The gradient of \(\frac{1}{2}w_i^2\) with respect to \(w_i\) is exactly \(w_i\) — proportional to the weight's own current value. This means larger weights get pulled down more aggressively (in absolute terms) than smaller weights, but a weight's shrinkage is always proportional to itself — it approaches zero geometrically, getting smaller and smaller, but essentially never landing exactly on zero the way L1's constant-magnitude pull can.
Numerical Example
With \(\lambda=0.1\), \(\eta=0.5\): a weight \(w=2.0\) receives an L2 penalty gradient contribution of \(\lambda w = 0.2\), shrinking it by \(0.5\times0.2=0.1\) this step, to \(1.9\). A smaller weight \(w=0.2\) receives a gradient contribution of \(\lambda w=0.02\), shrinking it by only \(0.5\times0.02=0.01\), to \(0.19\) — a proportionally similar, but absolutely much smaller, correction.
Code
import torch
def l2_penalty(model, lam=0.01):
return lam * sum((p ** 2).sum() for p in model.parameters()) / 2
loss = loss_fn(y_pred, y_true) + l2_penalty(model)
loss.backward()
# In practice, PyTorch optimizers implement L2 regularization directly via weight_decay
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.01)Why L2 Is the More Common Default
Its smooth, everywhere-differentiable penalty (unlike L1's non-differentiable kink at exactly zero) makes it slightly simpler to optimize, and its effect — a general, proportional discouragement of overly large weights, rather than aggressive feature elimination — is usually what's wanted when the goal is simply "keep the model from relying too heavily on any single weight," without a specific need for feature selection or sparsity.
Common Mistakes
- Using PyTorch optimizers' built-in
weight_decayargument and assuming it's mathematically identical to L2 regularization for every optimizer — as covered in AdamW, this equivalence holds cleanly for plain SGD but breaks subtly under Adam's adaptive per-parameter scaling, which is exactly why AdamW exists. - Choosing \(\lambda\) without tuning — too large a value can push the model toward underfitting by over-constraining every weight, regardless of whether overfitting was actually a problem to begin with.
Interview Relevance
Q: "Why is L2 regularization's gradient proportional to the weight itself, and what effect does that have?" Because the derivative of \(\frac{1}{2}w^2\) is \(w\) — larger weights are pulled down more in absolute terms, but every weight shrinks by a fraction of its own current size. This produces smooth, proportional shrinkage toward zero across all weights, rather than the sparsity (exact zeros) that L1's constant-magnitude penalty gradient produces.
Practice Question
With \(\lambda=0.2\) and a weight currently at \(w=1.5\), what is the L2 penalty's gradient contribution for this weight?
