This comparison note puts sigmoid and tanh side by side directly — useful both for interviews and for the concrete decision of which to use inside an LSTM/GRU gate, where both still appear regularly.
Side-by-Side Comparison
| Sigmoid | Tanh | |
|---|---|---|
| Formula | \(\sigma(z)=\frac{1}{1+e^{-z}}\) | \(\tanh(z)=2\sigma(2z)-1\) |
| Range | (0, 1) | (-1, 1) |
| Zero-centered? | No — always positive | Yes |
| Value at \(z=0\) | 0.5 | 0 |
| Max derivative | 0.25 | 1.0 |
| Saturates for large \(|z|\)? | Yes | Yes |
| Typical use today | Binary classification output; LSTM/GRU gates (values interpreted as "how much to let through," 0 to 1) | LSTM/GRU candidate values (a bounded, signed update) |
Why Both Still Appear Inside LSTM/GRU Cells
Both survive specifically because their bounded ranges are semantically meaningful there, not because they avoid vanishing gradients (they don't, within a single cell's internal computation) — sigmoid's (0,1) range is exactly right for a "gate" controlling how much information to keep or discard, and tanh's (-1,1) range is exactly right for a candidate value that can push a cell's memory in either direction. This is covered in full in the LSTM & GRU category.
Code — Plotting the Difference
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
def tanh(z): return np.tanh(z)
z = np.linspace(-5, 5, 5)
print("sigmoid:", sigmoid(z))
print("tanh: ", tanh(z))
# sigmoid stays entirely positive; tanh is symmetric around 0Common Mistakes
- Assuming tanh solves vanishing gradients because it's zero-centered — it saturates just as severely as sigmoid for large \(|z|\); zero-centering only addresses a separate, secondary optimization issue.
- Using sigmoid for a hidden layer "because it's simpler to reason about" in a deep feedforward network — for hidden layers, ReLU (or a variant) is virtually always the better modern default for both of these reasons.
Interview Relevance
Q: "If both sigmoid and tanh suffer from vanishing gradients, why does an LSTM still use both internally?" Their bounded ranges carry specific semantic meaning inside a gated cell: sigmoid's (0,1) output is used directly as a "how much to let through" gate value, and tanh's (-1,1) output is used as a signed candidate update. The vanishing-gradient concern is mitigated in LSTMs by a separate mechanism — the cell state's largely additive update path — not by avoiding these activations.
Practice Question
For \(z=3\), compute both \(\sigma(3)\) and \(\tanh(3)\). Which one is closer to its respective saturation bound?
