The candidate state \(\tilde{\mathbf{C}}_t\) proposes what new information the current input and previous hidden state suggest should potentially be added to the cell state — the "what" that the input gate then decides "how much" of to actually use.
Formula
Note the activation: tanh, not sigmoid — this is the one place among LSTM's core computations where tanh (not sigmoid) is used, and the reason is specific: tanh's range, \((-1,1)\), lets the candidate propose information that could push the cell state's value in either a positive or negative direction, matching the cell state's own unbounded, signed nature — sigmoid's exclusively-positive \((0,1)\) range would only ever allow the cell state to increase, never decrease, via this pathway.
Why This Is Structurally Different From the Gates
| Forget/Input/Output Gates | Candidate State | |
|---|---|---|
| Activation | Sigmoid — outputs \((0,1)\) | Tanh — outputs \((-1,1)\) |
| Interpreted as | "How much" — a fraction/weight | "What" — actual signed content to potentially add |
| Role in cell state update | Scale (multiply) other quantities | Is itself the quantity being scaled and added |
Numerical Example
Same setup as the previous two gates: \(\mathbf{h}_{t-1}=[0.2,-0.1]\), \(\mathbf{x}_t=[1.0]\), with candidate-specific weights \(\mathbf{W}_C=[0.3, 0.5, -0.4]\), \(b_C=0.2\):
This negative candidate value, if the input gate allows a substantial fraction through, would push the corresponding cell-state dimension in the negative direction.
Code
import torch
h_prev = torch.tensor([0.2, -0.1])
x_t = torch.tensor([1.0])
combined = torch.cat([h_prev, x_t])
W_C = torch.tensor([0.3, 0.5, -0.4])
b_C = torch.tensor(0.2)
z_C = torch.dot(W_C, combined) + b_C
C_candidate = torch.tanh(z_C)
print(C_candidate) # tensor(-0.1880) -- matches the hand-worked exampleAssembling the Full Cell-State Update
Using this note's candidate value alongside the previous two notes' forget and input gate values (\(f_t\approx0.717\), \(i_t\approx0.599\)) with \(C_{t-1}=2.0\) (an illustrative starting value):
Common Mistakes
- Using sigmoid instead of tanh for the candidate state — this would restrict the candidate to always propose non-negative additions, breaking the cell state's ability to have any dimension's value decrease via this specific pathway (it could still decrease via the forget gate scaling it down).
- Referring to the candidate state as "the new cell state" — it's only a proposal; the actual new cell state, \(\mathbf{C}_t\), combines it with the retained old cell state, scaled by both gates.
Interview Relevance
Q: "Why does the LSTM candidate state use tanh instead of sigmoid, unlike the three gates?" The candidate state represents actual signed content that might be added to the cell state, not a "how much" scaling fraction. Tanh's \((-1,1)\) range lets the proposed new information push the cell state's value in either direction, matching the cell state's own unbounded, signed nature — sigmoid's exclusively positive range would incorrectly restrict new information to only ever increase the cell state.
Practice Question
Using the same weights as the worked example, compute the candidate state for a different input: \(\mathbf{h}_{t-1}=[0,0]\), \(x_t=2.0\).
