Practical guidance for tuning the single most consequential hyperparameter in deep learning — building on the conceptual foundation from Learning Rate.
Typical Starting Ranges, by Optimizer
| Optimizer | Typical Starting Range |
|---|---|
| SGD (with momentum) | 0.01 – 0.1 |
| Adam / AdamW | 1e-4 – 1e-3 |
| Fine-tuning a pretrained model | 1e-5 – 1e-4 (much smaller — see Fine-Tuning) |
The Learning Rate Range Test — A Practical Technique
Rather than guessing, a systematic approach: start training with a very small learning rate, and gradually increase it (often exponentially) over a short number of steps, plotting loss against learning rate. The loss typically decreases as the learning rate rises to a useful range, then sharply increases once the learning rate becomes too large for stable training — the ideal learning rate sits just before that sharp increase.
Code — A Simple Learning Rate Range Test
import torch
import matplotlib.pyplot as plt
def lr_range_test(model, train_loader, loss_fn, start_lr=1e-7, end_lr=1, num_steps=100):
optimizer = torch.optim.Adam(model.parameters(), lr=start_lr)
lr_mult = (end_lr / start_lr) ** (1 / num_steps)
lrs, losses = [], []
for i, (x, y) in enumerate(train_loader):
if i >= num_steps:
break
optimizer.zero_grad()
loss = loss_fn(model(x), y)
loss.backward()
optimizer.step()
lrs.append(optimizer.param_groups[0]['lr'])
losses.append(loss.item())
optimizer.param_groups[0]['lr'] *= lr_mult # exponentially increase LR each step
plt.plot(lrs, losses)
plt.xscale('log')
plt.xlabel('Learning Rate'); plt.ylabel('Loss')
# pick a learning rate from just before the loss starts sharply increasingDiagnosing From Symptoms
| Symptom | Likely Cause |
|---|---|
| Loss decreases extremely slowly | Learning rate too low |
| Loss oscillates wildly or diverges (becomes NaN) | Learning rate too high |
| Loss plateaus early, higher than expected | Could be too high (stuck oscillating near a minimum) or too low (hasn't reached it yet) — try both directions |
Common Mistakes
- Never adjusting the learning rate from a copied default, regardless of the specific model/dataset — while the typical ranges above are reasonable starting points, the truly optimal value genuinely varies by task and is worth verifying.
- Confusing a too-high learning rate's symptoms with a fundamentally broken model architecture — always rule out learning rate first, since its symptoms (oscillating or diverging loss) can look similar to other, more structural problems.
Interview Relevance
Q: "How would you systematically find a good learning rate for a new model, rather than guessing?" A learning rate range test — gradually increasing the learning rate over a short warmup run while tracking loss, then plotting loss against learning rate on a log scale. The loss typically decreases through a useful range, then sharply increases once the rate becomes destabilizingly large; picking a rate just before that sharp increase is a reliable, systematic starting point.
Practice Question
If a model's loss decreases very slowly over many epochs but never oscillates or spikes, is the learning rate more likely too high or too low?
