Adam (Adaptive Moment Estimation) combines momentum's smoothed gradient direction with RMSProp's per-parameter adaptive scaling, plus a bias-correction step neither of those alone includes. It's the most widely used optimizer in deep learning today, and a very reasonable default for most new projects.
Formula
Default hyperparameters (rarely need changing): \(\beta_1=0.9\), \(\beta_2=0.999\), \(\epsilon=10^{-8}\).
Why Bias Correction Is Needed
Both \(m_0\) and \(v_0\) start at exactly 0. In the very early steps of training, this initialization biases \(m_t\) and \(v_t\) toward zero — they haven't had enough steps to "warm up" to their true running-average values yet. Dividing by \((1-\beta_1^t)\) and \((1-\beta_2^t)\) — both very small for small \(t\), close to 1 for large \(t\) — exactly corrects for this early-step bias, boosting the estimates back to their proper scale early in training and having almost no effect once \(t\) is reasonably large.
Numerical Example — First Two Steps
With \(\beta_1=0.9\), \(\beta_2=0.999\), \(g_1=2\): \(m_1 = 0.9(0)+0.1(2)=0.2\), \(v_1=0.999(0)+0.001(4)=0.004\). Bias-corrected: \(\hat m_1 = \frac{0.2}{1-0.9}=2.0\), \(\hat v_1=\frac{0.004}{1-0.999}=4.0\) — notice the correction restores \(\hat m_1\) and \(\hat v_1\) to values matching the raw gradient's own scale (\(g_1=2\), \(g_1^2=4\)), exactly compensating for the zero-initialization bias at this very first step.
Code
import torch.optim as optim
optimizer = optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-8)
# A full training step, exactly as with any other optimizer
loss.backward()
optimizer.step()
optimizer.zero_grad()Why Adam Became the Default
- Combines the benefits of momentum (smoothed direction, faster convergence in consistent directions) and RMSProp (adaptive per-parameter scaling, robust to varying gradient magnitudes across parameters) in one optimizer.
- Works reasonably well "out of the box" across a very wide range of architectures and tasks, with minimal hyperparameter tuning required — a major practical convenience.
- Handles sparse gradients well, similar to AdaGrad/RMSProp's original motivation.
Common Mistakes
- Assuming Adam always outperforms plain SGD with momentum — for some tasks (notably, some computer vision benchmarks), well-tuned SGD with momentum and a good learning rate schedule can generalize slightly better than Adam, though Adam usually converges faster and needs less tuning to get a solid result.
- Forgetting Adam's interaction with L2 regularization is subtly broken — this specific issue, and its fix, is the entire subject of the next note, AdamW.
Interview Relevance
Q: "What two ideas does Adam combine, and why does it also need bias correction?" It combines momentum (an exponentially weighted average of gradients, the "first moment") and RMSProp-style per-parameter adaptive scaling (an exponentially weighted average of squared gradients, the "second moment"). Bias correction is needed because both moving averages are initialized at zero, which biases their early estimates toward zero until enough steps have accumulated — the correction terms compensate for exactly this early-training bias.
Practice Question
Why does the bias-correction term \(\frac{1}{1-\beta_1^t}\) have almost no effect once \(t\) becomes large (e.g. \(t=1000\)), compared to its large effect at \(t=1\)?
