A single fixed learning rate for an entire training run is rarely optimal. Learning rate scheduling changes \(\eta\) systematically over the course of training — typically starting higher for fast early progress, and decreasing later for fine, stable convergence. This note introduces the general idea before the next five notes cover specific schedules.
Why a Fixed Rate Is a Compromise
| Training Stage | What's Needed | Problem with a Fixed Rate |
|---|---|---|
| Early training | Large steps to make fast progress from a random initialization | Too small a fixed rate wastes early training time |
| Late training | Small, precise steps to settle into a good minimum without overshooting | Too large a fixed rate causes oscillation right when precision matters most |
A schedule resolves this tension directly: use a larger rate early, and shrink it later — getting the benefits of both regimes across a single training run.
General Notation
\(\eta_0\) is the initial (or peak) learning rate; \(t\) is the current training step or epoch; "schedule" is one of several functions covered in the next five notes.
Code — The General Pattern in PyTorch
import torch.optim as optim
from torch.optim.lr_scheduler import StepLR
optimizer = optim.Adam(model.parameters(), lr=0.001)
scheduler = StepLR(optimizer, step_size=10, gamma=0.5) # halves lr every 10 epochs
for epoch in range(30):
for batch in dataloader:
# ... forward pass, loss, backward pass ...
optimizer.step()
optimizer.zero_grad()
scheduler.step() # called once per epoch, AFTER the epoch's optimizer.step() calls
print(f"Epoch {epoch}: lr={scheduler.get_last_lr()}")Every PyTorch scheduler follows this same pattern: it wraps an existing optimizer and adjusts its learning rate according to a rule, called once per epoch (or sometimes per step, depending on the schedule).
A Quick Map of What's Coming
| Schedule | Shape |
|---|---|
| Step Decay | Sudden drops at fixed intervals |
| Exponential Decay | Smooth, continuous exponential shrinkage every step |
| Cosine Annealing | Smooth decay following a cosine curve, popular for its gentle, non-linear shape |
| Warmup | Gradual increase at the very start, before the main schedule takes over |
| One-Cycle | Rises then falls within a single training run — a full cycle of increase and decrease |
Common Mistakes
- Calling
scheduler.step()at the wrong point in the loop (e.g. beforeoptimizer.step(), or once per batch for a schedule designed to run once per epoch) — this silently shifts the schedule's timing relative to what was intended. - Treating "no schedule" as always a safe default — for many modern architectures (especially Transformers), a well-chosen schedule (particularly warmup, covered later) isn't just a minor optimization — training can fail to converge stably without it.
Interview Relevance
Q: "Why not just pick one good, fixed learning rate for the whole training run?" The ideal learning rate genuinely differs across training stages — large early on for fast progress from a random initialization, small later on for precise convergence without overshooting a good minimum. A schedule captures both needs within one run, rather than compromising on a single fixed value that's suboptimal for at least part of training.
Practice Question
You're training a model for 100 epochs. Would you expect a schedule that decreases the learning rate to help more in the first 10 epochs or the last 10 epochs? Why?
