This comparison note settles the most common practical activation-function question: when does the small extra complexity of Leaky ReLU actually earn its keep over plain ReLU?
Side-by-Side Comparison
| ReLU | Leaky ReLU | |
|---|---|---|
| Formula for \(z<0\) | 0 | \(\alpha z\) (small \(\alpha\), e.g. 0.01) |
| Gradient for \(z<0\) | Exactly 0 | \(\alpha\) (small but non-zero) |
| Dying neuron risk | Real — a neuron can get permanently stuck outputting 0 | Much lower — negative inputs still produce a (small) gradient |
| Extra hyperparameters | None | One (\(\alpha\)) |
| Computational cost | Slightly cheaper | Slightly more (one extra multiplication for negative inputs) |
| Common default | Yes — still the most common starting choice | Used when dying neurons are observed to be a real problem |
When the Difference Actually Matters
In practice, the gap between ReLU and Leaky ReLU is often small — many well-regularized, well-initialized networks trained with ReLU don't suffer significant dying-neuron issues. The difference matters most when: learning rates are aggressive (increasing dying-neuron risk), the network is very deep (more opportunities for neurons to die somewhere), or you empirically observe a large fraction of dead neurons (near-zero activation for most/all inputs) during training diagnostics.
Code — Comparing Their Behavior
import torch
import torch.nn as nn
relu = nn.ReLU()
leaky = nn.LeakyReLU(negative_slope=0.01)
z = torch.tensor([-10.0, -1.0, 0.0, 1.0, 10.0])
print("ReLU: ", relu(z))
print("LeakyReLU: ", leaky(z))
# ReLU: tensor([ 0., 0., 0., 1., 10.])
# LeakyReLU: tensor([-0.1000, -0.0100, 0.0000, 1.0000, 10.0000])Common Mistakes
- Switching to Leaky ReLU reflexively without evidence of a dying-neuron problem — it's a reasonable default-adjacent choice, but the added hyperparameter and marginal compute cost aren't automatically worth it without a specific reason.
- Assuming Leaky ReLU completely eliminates the possibility of near-dead neurons — a very small \(\alpha\) still means a very small gradient; it reduces but doesn't fully remove the risk of slow learning for consistently-negative neurons.
Interview Relevance
Q: "When would you specifically choose Leaky ReLU over plain ReLU?" When training diagnostics show a significant fraction of dead neurons (activations stuck at exactly zero across most/all training examples), often correlated with a high learning rate or very deep architecture. In the absence of that specific evidence, plain ReLU remains a reasonable, simpler default.
Practice Question
A network trained with ReLU shows that 40% of neurons in one layer output exactly 0 for every example in the validation set. What does this suggest, and what's one activation-function-level fix you could try?
