This note introduces gradient descent as the algorithm that actually uses backpropagation's gradients to improve the network — closing the loop between "measuring error" and "getting better." The full Optimization category covers its many variants (SGD, Adam, etc.) in depth.
The Core Update Rule
\(\eta\) (eta) is the learning rate — a small positive hyperparameter controlling how large a step to take. \(\nabla L(\mathbf{w})\) is the gradient computed by backpropagation, pointing toward steepest loss increase (see Gradient) — subtracting it moves the weights toward decreasing loss.
Numerical Example
A single weight \(w=2.0\), gradient \(\nabla L = 3.0\), learning rate \(\eta=0.1\):
The weight moved from 2.0 to 1.7 — a small step in the direction that reduces the loss, based on the current gradient. Repeating this process many times, across every weight in the network simultaneously, is what "training" means.
Why the Learning Rate Matters So Much
Too small a learning rate wastes training time; too large a learning rate can overshoot the minimum entirely and cause the loss to diverge.
Code — A Complete Minimal Training Step
import torch
import torch.nn as nn
model = nn.Linear(1, 1)
x = torch.tensor([[2.0]])
y_true = torch.tensor([[10.0]])
learning_rate = 0.01
y_pred = model(x) # forward propagation
loss = nn.MSELoss()(y_pred, y_true) # loss function
loss.backward() # backpropagation -- computes gradients
with torch.no_grad(): # gradient descent update -- no gradient tracking needed here
for param in model.parameters():
param -= learning_rate * param.grad
param.grad.zero_() # reset gradient for the next stepIn practice, you'll almost always use an optimizer object (torch.optim.SGD, torch.optim.Adam, ...) to handle this update loop instead of writing it manually — covered in the PyTorch category — but this is exactly what those optimizers do underneath.
The Complete Training Flow, Now Fully Assembled
This closes the loop first previewed in Why Calculus for Neural Networks: Input → Weighted Sum → Activation → Prediction (forward propagation) → Loss (loss function) → Gradient (backpropagation) → Weight Update (gradient descent) — repeated over and over, across many batches and epochs, is the entirety of how a neural network learns.
Common Mistakes
- Setting the learning rate too high "to train faster" — this frequently causes the loss to oscillate or diverge entirely instead, as shown in the diagram above.
- Forgetting to zero out gradients between steps (as flagged already in Gradient Vector) — this silently accumulates stale gradients from previous steps into the current update.
Interview Relevance
Q: "What happens if the learning rate is set far too high?" Each weight update overshoots the loss surface's minimum by a large margin. Instead of smoothly converging, the loss can oscillate wildly or increase without bound (diverge) — the optimizer keeps "jumping over" the minimum in a self-reinforcing way rather than settling into it.
Practice Question
A weight is currently 5.0, with gradient 2.0. Using learning rate 0.5, compute the updated weight. What would happen to the update's size if the learning rate were 5.0 instead — is that likely to help or hurt convergence?
