The vanishing gradient problem — already previewed in Sigmoid Function and Chain Rule — gets its full, dedicated treatment here: exactly how gradients shrink across many layers, demonstrated numerically, and a complete map of the solutions developed to address it.
The Mechanism, Precisely
From Backward Pass, each layer's error signal involves multiplying by that layer's activation derivative, \(\phi'(\mathbf{z}^{(l)})\). For sigmoid, this derivative is at most 0.25 (see Sigmoid Function); for tanh, at most 1.0 but often much smaller once saturated. Across \(L\) layers, the gradient reaching the earliest layer involves a product of \(L\) such derivative terms:
If every factor is meaningfully less than 1 (routine for saturating activations), this product shrinks exponentially with depth — by layer 10 or 20, the gradient reaching early layers can become vanishingly small, effectively stopping those layers from learning at all.
Numerical Demonstration
Suppose every layer's activation derivative happens to be exactly 0.2 (a plausible value for a saturated sigmoid). After \(L\) layers, the multiplicative factor is \(0.2^L\):
| Depth (\(L\)) | \(0.2^L\) |
|---|---|
| 2 | 0.04 |
| 5 | 0.00032 |
| 10 | 0.0000001024 |
By 10 layers, the gradient reaching the earliest layer has shrunk by a factor of roughly 10 million — for all practical purposes, zero. This exact numerical pattern is why deep sigmoid/tanh networks were notoriously difficult to train before better solutions emerged.
Code — Watching Gradients Shrink Across Layers
import torch
import torch.nn as nn
# A deep network using sigmoid throughout, to expose vanishing gradients
layers = nn.Sequential(*[nn.Sequential(nn.Linear(10, 10), nn.Sigmoid()) for _ in range(10)])
x = torch.randn(1, 10, requires_grad=True)
output = layers(x)
loss = output.sum()
loss.backward()
for i, layer in enumerate(layers):
linear = layer[0]
print(f"Layer {i} weight grad norm:", linear.weight.grad.norm().item())
# Grad norms typically shrink noticeably as you look at earlier layers (lower i)Solutions — A Map of the Rest of This Hub
| Solution | How It Helps | Where It's Covered |
|---|---|---|
| ReLU activation | Derivative is exactly 1 for positive inputs — no shrinkage from this factor | ReLU |
| Residual (skip) connections | Provides an additive gradient path that bypasses the multiplicative chain entirely | ResNet, CNN Architectures category |
| Batch/Layer normalization | Keeps activations in a well-scaled range, avoiding the saturating regions where derivatives vanish | Normalization Techniques category |
| LSTM/GRU gating | Provides an additive cell-state update path, specifically designed to combat vanishing gradients in sequences | LSTM & GRU category |
| Careful weight initialization | Keeps pre-activation values in a range where derivatives aren't already near zero from the start | Training Deep Networks category |
Common Mistakes
- Assuming vanishing gradients only affect very old or unusual architectures — it remains a real concern any time many saturating activations (sigmoid, tanh) are stacked deeply, including inside RNN sequences unrolled over many time steps (see RNN Vanishing Gradient).
- Diagnosing slow or stalled training as a data or learning-rate problem without first checking whether early layers' gradients are near-zero — a quick gradient-norm check per layer, as shown above, can reveal this directly.
Interview Relevance
Q: "Why do vanishing gradients get worse as a network gets deeper?" Because the gradient reaching an early layer is a product of every subsequent layer's local derivative, via the chain rule. If each factor is a fraction less than 1 (routine for saturating activations like sigmoid/tanh), this product shrinks exponentially with the number of layers — a modest per-layer shrinkage compounds into a near-zero gradient after enough layers, effectively halting learning in the earliest layers.
Practice Question
If every layer's activation derivative in a network is approximately 0.5, roughly how many times smaller is the gradient reaching layer 1 compared to the output layer, in a 20-layer network?
