AdamW fixes a subtle but real bug in how standard Adam interacts with L2 regularization — and has become the more commonly recommended default over plain Adam for exactly this reason, especially for training Transformers.
The Problem AdamW Fixes
In plain SGD, adding L2 regularization to the loss (\(L + \frac{\lambda}{2}\|\mathbf{w}\|^2\)) is mathematically equivalent to directly shrinking each weight by a fixed proportion every step ("weight decay") — the two are the same thing. But in Adam, L2 regularization is applied by adding \(\lambda w\) to the gradient before it gets divided by \(\sqrt{\hat v_t}\) — which means parameters with a large accumulated \(\hat v_t\) (frequently, strongly updated parameters) end up with their regularization effectively weakened, while parameters with small \(\hat v_t\) get regularized more strongly than intended. The clean SGD-era equivalence between "L2 regularization" and "weight decay" silently breaks under Adam's adaptive scaling.
Formula — Decoupling Weight Decay from the Gradient
The key change: the weight decay term \(\lambda w_t\) is added directly to the final update, completely separate from the adaptive-scaling machinery (\(\hat m_t\), \(\hat v_t\)) — instead of being folded into the gradient before it gets adaptively scaled. This restores the clean, direct "shrink every weight by a fixed proportion" behavior that true weight decay is supposed to have, regardless of any individual parameter's gradient history.
Code
import torch.optim as optim
# AdamW takes weight_decay as its own explicit, decoupled argument
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)
# Contrast with plain Adam's weight_decay, which folds decay into the
# gradient BEFORE adaptive scaling -- exactly the behavior AdamW avoids
optimizer_plain_adam = optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)Complete Optimizer Comparison
| Optimizer | Per-Parameter Adaptive LR? | Momentum? | Key Idea | Typical Use |
|---|---|---|---|---|
| Batch GD | No | No | Exact gradient over the full dataset | Rare in deep learning; mostly theoretical/small-scale |
| SGD (mini-batch) | No | No | Noisy but fast, frequent updates | Simple baselines; still competitive for some CV tasks with tuning |
| SGD + Momentum | No | Yes | Smooths oscillation, accelerates consistent directions | Classic CNN training recipes (e.g. ResNet on ImageNet) |
| Nesterov Momentum | No | Yes (look-ahead) | Anticipates momentum's next position before computing gradient | Same as SGD+momentum, slightly improved |
| AdaGrad | Yes (ever-shrinking) | No | Per-parameter rate from cumulative squared gradients | Sparse features; rarely used for long training runs |
| RMSProp | Yes (adaptive) | No | Decaying average fixes AdaGrad's shrinking-rate problem | RNNs, historically |
| Adam | Yes | Yes | Combines momentum + RMSProp + bias correction | General-purpose default across most architectures |
| AdamW | Yes | Yes | Adam with correctly decoupled weight decay | Modern default, especially for Transformers/LLMs |
Common Mistakes
- Using plain Adam with a non-trivial
weight_decayvalue, unaware that it's not behaving as true weight decay — for any project relying meaningfully on regularization strength, AdamW is the more predictable, theoretically correct choice. - Assuming AdamW requires a completely different learning rate range from Adam — in practice, similar learning rates work for both; the difference is specifically in how weight decay is applied, not the core adaptive update.
Interview Relevance
Q: "Why is AdamW generally preferred over plain Adam with weight_decay set, especially for training Transformers?" Plain Adam applies L2 regularization by folding it into the gradient before adaptive per-parameter scaling, which breaks the clean equivalence between L2 regularization and true weight decay that holds for SGD — parameters get inconsistently regularized depending on their gradient history. AdamW decouples weight decay from the adaptive scaling entirely, applying it as a direct, consistent shrinkage to every weight — restoring predictable regularization behavior, which has proven especially important for large-scale Transformer training.
Practice Question
Explain, in your own words, why a parameter with a large \(\hat v_t\) (heavily, frequently updated) ends up under-regularized when weight decay is folded into the gradient the way plain Adam does it.
