The tanh (hyperbolic tangent) function is sigmoid's zero-centered cousin — it squashes inputs into (-1, 1) instead of (0, 1), fixing one of sigmoid's two drawbacks while still suffering from the other.
Formula
The second form makes the relationship to sigmoid explicit: tanh is just a rescaled, shifted sigmoid.
Derivative
| Property | Value |
|---|---|
| Range | \((-1, 1)\) |
| \(\tanh(0)\) | 0 |
| Maximum derivative | 1, at \(z=0\) |
Graph
Same S-shape as sigmoid, but centered at zero and stretched to range from -1 to 1.
Why Zero-Centering Matters
Because tanh's output can be negative, gradients flowing back through it aren't systematically biased in one direction the way sigmoid's always-positive output can bias them (see the note on sigmoid's second drawback). This typically makes optimization converge somewhat faster and more smoothly for hidden layers, which is why tanh was historically preferred over sigmoid specifically for hidden layers, back when saturating activations were the norm.
The Shared Flaw: Still Saturates
Tanh has the exact same vanishing gradient problem as sigmoid — for large \(|z|\), the curve flattens and \(\tanh'(z)\to0\). Being zero-centered fixes one issue but not the fundamental saturation problem, which is why tanh, like sigmoid, has also largely been replaced by ReLU-family activations for hidden layers in deep networks.
Numerical Example
Code
import numpy as np
import torch
print(np.tanh([-1, 0, 1])) # [-0.762 0. 0.762]
z = torch.tensor([-1.0, 0.0, 1.0])
print(torch.tanh(z))Where It's Still Used Today
Tanh appears inside LSTM and GRU cells (alongside sigmoid gates) for producing candidate values within a bounded, zero-centered range — covered fully in the LSTM & GRU category. It's rarely the default choice for feedforward or convolutional hidden layers today.
Common Mistakes
- Assuming zero-centering alone solves vanishing gradients — it addresses a separate, secondary issue; the saturation-driven gradient shrinkage is still present and often just as severe as sigmoid's.
Interview Relevance
Q: "Why might tanh converge faster than sigmoid for the same network, even though both can suffer vanishing gradients?" Tanh's zero-centered output (range -1 to 1) avoids systematically biasing gradient directions the way sigmoid's always-positive output (0 to 1) can, which tends to make gradient descent's path to a good solution more direct and efficient — though both still saturate for large-magnitude inputs.
Practice Question
Using the identity \(\tanh(z) = 2\sigma(2z)-1\), verify that \(\tanh(0)=0\) follows directly from \(\sigma(0)=0.5\).
