Learning Rate

The learning rate \(\eta\) is widely considered the single most important hyperparameter in deep learning — it controls how large a step every weight takes on every update, and getting it wrong can prevent training from working at all, regardless of architecture or data quality.

Where It Appears

\(\eta\) scales the gradient before it's subtracted — a larger \(\eta\) means bigger, faster (but riskier) steps; a smaller \(\eta\) means smaller, safer (but slower) steps.

Typical Ranges in Practice

These are starting points, not universal answers — the right value depends on the specific architecture, batch size, and optimizer, which is exactly why hyperparameter tuning (covered in its own category later) exists.

Revisiting the Too-Small / Good / Too-Large Picture

The three-way comparison from Gradient Descent (Intro) is worth restating precisely here: too small wastes training time without necessarily reaching a better solution; too large causes the loss to oscillate or diverge outright, since each step can overshoot the minimum by more than the previous step's progress; a well-chosen learning rate balances fast progress against stable convergence.

Numerical Illustration

import torch

def try_lr(lr, steps=20):
    w = torch.tensor(10.0, requires_grad=True)   # start far from the minimum at w=0
    for _ in range(steps):
        loss = w ** 2
        loss.backward()
        with torch.no_grad():
            w -= lr * w.grad
        w.grad.zero_()
    return w.item()

print("lr=0.01: ", try_lr(0.01))    # converges slowly, still far from 0
print("lr=0.3:  ", try_lr(0.3))     # converges nicely toward 0
print("lr=1.1:  ", try_lr(1.1))     # diverges -- overshoots and grows without bound

Learning Rate and Batch Size Interact

A larger mini-batch (see Mini-Batch Gradient Descent) produces a less noisy, more "confident" gradient estimate — a common practical heuristic ("linear scaling rule") is to increase the learning rate roughly proportionally when increasing the batch size, though this isn't a universal law and should be validated empirically for a given setup.

Common Mistakes

• Using the same fixed learning rate across the entire training run by default — as covered later in this category, scheduling the learning rate (decaying it over time, or warming it up at the start) often meaningfully improves final performance.
• Diagnosing a diverging loss as a data or architecture problem before checking the learning rate first — an overly aggressive learning rate is one of the most common root causes of a loss that explodes or oscillates wildly.

Interview Relevance

Q: "Your model's training loss is oscillating wildly and occasionally spiking to very large values. What's the first hyperparameter you'd check?" The learning rate — this exact symptom (oscillation, occasional spikes, sometimes outright divergence) is the classic signature of a learning rate set too high, causing weight updates to overshoot the loss surface's minimum repeatedly rather than converging toward it.

Practice Question

You're fine-tuning a large pretrained model on a small new dataset. Would you expect to use a larger or smaller learning rate than training a similar-sized model from scratch? Why?

Related DL Notes

• Gradient Descent (Intro)
• Learning Rate Scheduling
• Learning Rate Tuning