Stochastic Gradient Descent (SGD) sits at the opposite extreme from batch gradient descent: it updates the weights using the gradient from just one randomly chosen training example at a time.
Formula
Why "Stochastic"
Each individual example's gradient is a noisy, imperfect estimate of the true gradient over the full dataset — but it's an unbiased estimate (its expected value, averaged over many random examples, equals the true full-dataset gradient — see Expected Value). This randomness is the "stochastic" part, and it fundamentally changes the optimization's behavior.
The Noisy Path — A Feature, Not Just a Bug
SGD's noisy path takes a less direct route, but that same noise can help it "jump out of" shallow local minima that trap smoother optimizers.
This noise is a real, useful property: in non-convex loss surfaces (common in deep learning, see Gradient Descent), the randomness can help SGD escape shallow local minima that smoother methods like batch gradient descent might get stuck in.
Advantages and Disadvantages
| Advantage | Disadvantage |
|---|---|
| Extremely fast per-update — no waiting for a full dataset pass | High variance in the update direction; convergence path is noisy |
| Can escape shallow local minima due to noise | Doesn't fully exploit modern hardware's parallelism (processing one example at a time is inefficient on a GPU) |
| Can start updating weights before seeing the whole dataset | Requires a smaller learning rate to avoid excessive oscillation |
Code
import torch
import random
def sgd_step(X, y, w, lr=0.01):
i = random.randint(0, len(X) - 1) # pick ONE random example
x_i, y_i = X[i], y[i]
prediction = x_i @ w
loss = (prediction - y_i) ** 2
loss.backward()
with torch.no_grad():
w -= lr * w.grad
w.grad.zero_()
return wCommon Mistakes
- Confusing "SGD" as used casually in deep learning frameworks (e.g.
torch.optim.SGD) with true one-example-at-a-time SGD — in practice, "SGD" almost always refers to mini-batch gradient descent (next note) under the hood; pure single-example SGD is rarely used directly due to poor hardware utilization. - Using too large a learning rate with pure SGD — the high per-step variance compounds with an aggressive learning rate, often causing divergence.
Interview Relevance
Q: "What's the practical benefit of SGD's noisy gradient estimates, beyond just being faster to compute?" The noise can act as an implicit regularizer and help the optimizer escape shallow local minima or saddle points that a smoother, exact-gradient method (like batch gradient descent) might get stuck at — a genuinely useful side effect of the randomness, not just a necessary evil.
Practice Question
Why does SGD typically require a smaller learning rate than batch gradient descent to train stably?
