RMSProp keeps AdaGrad's core idea — per-parameter adaptive learning rates based on gradient history — but fixes its fatal flaw with one small change: replacing the ever-growing sum of squared gradients with a decaying average that "forgets" old gradients over time.
Formula
\(\beta\) (commonly 0.9) controls how much weight recent squared gradients get versus older ones — this is an exponentially weighted moving average, not a running sum. Because it's an average rather than an ever-growing sum, \(E[g^2]_t\) can go back down if recent gradients happen to be small, letting the effective learning rate recover instead of shrinking forever.
Directly Fixing AdaGrad's Problem
| AdaGrad | RMSProp | |
|---|---|---|
| Accumulator | Sum of ALL past squared gradients (\(G_t = G_{t-1}+g_t^2\)) | Exponentially decaying average (\(E[g^2]_t = \beta E[g^2]_{t-1}+(1-\beta)g_t^2\)) |
| Behavior over long training | Effective learning rate shrinks monotonically toward zero | Effective learning rate adapts continuously to recent gradient magnitude, never permanently vanishing |
| Suited to long training runs? | Poorly | Well |
Numerical Example
With \(\beta=0.9\), \(E[g^2]_0=0\): gradient \(g_1=4\) gives \(E[g^2]_1 = 0.9(0)+0.1(16)=1.6\). If gradients then shrink to \(g=1\) for several steps, \(E[g^2]\) gradually decays back down toward \(0.1(1)=0.1\) — the effective learning rate recovers, something AdaGrad's ever-growing sum could never do.
Code
import torch.optim as optim
optimizer = optim.RMSprop([w], lr=0.001, alpha=0.9) # alpha is PyTorch's name for beta hereWhere It's Used Today
RMSProp was widely used and effective for training recurrent neural networks (see the RNN category) before Adam's popularity grew, and it remains a solid, reasonable choice — Adam, covered next, essentially combines RMSProp's per-parameter adaptive scaling with momentum, making Adam the more common default today.
Common Mistakes
- Confusing RMSProp's \(\beta\) (controlling the squared-gradient average's decay) with momentum's \(\beta\) (controlling the raw-gradient average's decay) — they serve structurally similar but distinct roles, and Adam (next) uses both simultaneously with separate hyperparameters for each.
Interview Relevance
Q: "How does RMSProp solve AdaGrad's diminishing learning rate problem?" By replacing AdaGrad's ever-growing sum of squared gradients with an exponentially decaying average, which can decrease as well as increase depending on recent gradient magnitudes — this lets the effective per-parameter learning rate adapt continuously throughout training instead of shrinking irreversibly toward zero.
Practice Question
If a parameter's gradients suddenly become much larger after a long period of small gradients, how would RMSProp's effective learning rate for that parameter respond, compared to how AdaGrad's would respond?
