Weight decay directly shrinks every weight by a fixed proportion at each update step — a simple, direct-acting rule that turns out to be mathematically identical to L2 regularization under plain gradient descent, though the two subtly diverge under adaptive optimizers like Adam.
Formula
Compare this directly to plain gradient descent's update, \(\mathbf{w}_{t+1}=\mathbf{w}_t-\eta\nabla L(\mathbf{w}_t)\): weight decay multiplies the current weight by a factor slightly less than 1 (\(1-\eta\lambda\)) before applying the usual gradient step — an explicit, direct shrinkage applied every single update, independent of what the loss gradient itself says.
Proving the SGD Equivalence to L2 Regularization
Starting from L2-regularized loss \(L_{\text{reg}} = L + \frac{\lambda}{2}\|\mathbf{w}\|^2\), its gradient is \(\nabla L_{\text{reg}} = \nabla L + \lambda\mathbf{w}\). Plugging this into the standard gradient descent update:
This is exactly the weight decay formula above — under plain SGD, "add an L2 penalty to the loss" and "directly shrink every weight by a fixed proportion each step" are algebraically the same operation, just two different ways of describing it.
Where the Equivalence Breaks: Adam
As covered in full in AdamW, Adam's adaptive per-parameter scaling divides the gradient (including any L2 penalty folded into it) by \(\sqrt{\hat v_t}\) — a quantity that varies per parameter based on gradient history. This means parameters with large \(\hat v_t\) get their L2-derived shrinkage effectively weakened, and vice versa — the clean SGD-era equivalence between "L2 penalty" and "direct weight shrinkage" no longer holds. AdamW fixes this specifically by applying weight decay as a truly separate, direct shrinkage term, decoupled from the adaptive gradient scaling — restoring the clean behavior this note describes.
Numerical Example
With \(\eta=0.1\), \(\lambda=0.01\): \(1-\eta\lambda = 1-0.001=0.999\). A weight \(w=2.0\), with gradient contribution 0 for this illustration: \(w_{t+1} = 0.999\times2.0 = 1.998\) — a small, direct shrinkage applied regardless of any other gradient signal.
Code
import torch.optim as optim
# For plain SGD, weight_decay IS equivalent to L2 regularization
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.01)
# For Adam, weight_decay does NOT behave as true, decoupled weight decay --
# use AdamW instead if that's actually what you want
optimizer_correct = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)Common Mistakes
- Using the terms "L2 regularization" and "weight decay" completely interchangeably in every context — they're equivalent under SGD, but this equivalence is specifically what breaks under Adam, making the distinction genuinely important to know.
- Choosing a weight decay value tuned for SGD and reusing it unchanged after switching to Adam/AdamW — since the underlying update mechanics differ, previously-tuned values aren't guaranteed to transfer directly.
Interview Relevance
Q: "Are L2 regularization and weight decay the same thing?" They're mathematically equivalent specifically under plain SGD — adding an L2 penalty to the loss produces an update rule algebraically identical to directly shrinking each weight by a fixed proportion. Under adaptive optimizers like Adam, this equivalence breaks because the L2 penalty gets folded into the gradient before adaptive per-parameter scaling is applied — which is exactly the problem AdamW's decoupled weight decay fixes.
Practice Question
Starting from the L2-regularized gradient \(\nabla L + \lambda\mathbf{w}\), re-derive the weight decay update rule step by step, as shown above, using \(\eta=0.2\) and \(\lambda=0.05\) to compute the resulting shrinkage factor.
