The exploding gradient problem is vanishing gradients' mirror image: instead of shrinking toward zero across layers, gradients grow uncontrollably large — often ending training abruptly with NaN (not-a-number) losses.
The Mechanism
The same multiplicative chain-rule product responsible for vanishing gradients can just as easily explode if the factors involved are consistently greater than 1 instead of less than 1:
If weight matrices have large-magnitude entries (or, equivalently, large eigenvalues — see Eigenvalues), this product can grow exponentially with depth, in exactly the same way the vanishing-gradient product shrinks exponentially.
Numerical Demonstration
Suppose every layer's combined weight-and-derivative factor is 1.5 instead of a fraction less than 1:
| Depth (\(L\)) | \(1.5^L\) |
|---|---|
| 5 | ≈ 7.6 |
| 10 | ≈ 57.7 |
| 20 | ≈ 3,325 |
By 20 layers, the gradient has grown over three thousand-fold — resulting weight updates can be enormous, immediately destabilizing training, often producing NaN values as numbers overflow standard floating-point precision.
Where This Is Especially Common
Recurrent neural networks (see the RNN category, particularly RNN Exploding Gradient) are especially prone to this, since the same weight matrix is applied repeatedly across every time step of a long sequence — effectively multiplying by the same matrix many times, which can compound very quickly if that matrix's eigenvalues exceed 1.
Recognizing It in Practice
| Symptom | Likely Cause |
|---|---|
Loss suddenly becomes NaN or inf | Gradients (or the resulting weight updates) have overflowed floating-point range |
| Loss oscillates wildly, occasionally spiking to huge values, without fully diverging | Gradients are large but not yet fully overflowing — a milder form of the same problem |
| Weight values grow to unreasonably large magnitudes over training | Repeated large updates, consistent with exploding gradients |
Code — Observing Gradient Explosion
import torch
import torch.nn as nn
# Deliberately large initial weights to provoke exploding gradients
layers = nn.Sequential(*[nn.Linear(10, 10) for _ in range(15)])
for layer in layers:
nn.init.normal_(layer.weight, mean=0, std=3.0) # unusually large initialization
x = torch.randn(1, 10, requires_grad=True)
output = layers(x)
loss = output.sum()
loss.backward()
for i, layer in enumerate(layers):
print(f"Layer {i} weight grad norm:", layer.weight.grad.norm().item())
# Grad norms typically grow sharply for earlier layers with this setupCommon Mistakes
- Assuming a sudden
NaNloss always indicates a data problem (e.g. a bad label or corrupted input) — exploding gradients are an equally common, purely optimization-side cause, and checking gradient norms before a NaN occurs can help distinguish the two. - Responding to instability only by lowering the learning rate — while this can help, gradient clipping (next note) directly and more reliably addresses exploding gradients at their source.
Interview Relevance
Q: "Your training loss suddenly becomes NaN partway through training. What's a likely optimization-related cause, and how would you fix it?" Exploding gradients — likely from weight matrices (especially in a recurrent architecture applying the same matrix repeatedly) whose combined effect across layers or time steps causes gradients to grow exponentially, eventually overflowing floating-point precision. Gradient clipping (capping the gradient's norm before the weight update) is the standard, direct fix.
Practice Question
Why are recurrent neural networks especially prone to exploding gradients compared to typical feedforward networks of similar depth?
