The cell state \(\mathbf{C}_t\) is LSTM's central innovation — the "conveyor belt" pathway introduced conceptually in Why LSTM. This note gives it the precise mathematical treatment.
The Update Formula
\(\odot\) is element-wise multiplication. Reading this left to right: take the previous cell state, scale it down (element-wise) by the forget gate \(\mathbf{f}_t\) (deciding what to keep), then add new information — the candidate state \(\tilde{\mathbf{C}}_t\), scaled by how much the input gate \(\mathbf{i}_t\) says to actually incorporate.
Why This Specific Formula Protects Gradients
Compare this to a plain RNN's hidden state update, \(\mathbf{h}_t = \tanh(\ldots)\) — every single step forces the carried value through a fresh non-linear squashing. The cell state's update, by contrast, is dominated by element-wise multiplication and addition — no matrix multiplication by a shared weight matrix, and no forced non-linearity applied to the running total itself. Taking the derivative of \(\mathbf{C}_t\) with respect to \(\mathbf{C}_{t-1}\):
If the forget gate \(\mathbf{f}_t\) stays close to 1 (deciding "keep almost everything"), this local gradient stays close to 1 too — meaning the gradient can flow backward through many time steps along this pathway with far less shrinkage than the repeated \(\mathbf{W}_{hh}^\top \cdot \text{diag}(\tanh')\) multiplication from RNN Vanishing Gradient.
Numerical Example
Scalar simplification: \(C_{t-1}=2.0\), \(f_t=0.9\) (forget gate says "keep 90%"), \(i_t=0.3\) (input gate says "add 30% of the new candidate"), \(\tilde C_t = 1.5\) (the proposed new information):
Most of the old memory (\(1.8\) out of \(2.0\)) is retained, with a modest amount of new information (\(0.45\)) blended in — exactly the "gentle update" behavior the cell state is designed for.
Code
import torch
C_prev = torch.tensor(2.0)
f_t = torch.tensor(0.9)
i_t = torch.tensor(0.3)
C_candidate = torch.tensor(1.5)
C_t = f_t * C_prev + i_t * C_candidate
print(C_t) # tensor(2.2500) -- matches the hand-worked exampleCommon Mistakes
- Assuming the cell state passes through unchanged unless a gate intervenes — it always undergoes at least the forget-gate scaling and input-gate addition at every step; "unchanged" would specifically require \(\mathbf{f}_t \approx \mathbf{1}\) and \(\mathbf{i}_t \approx \mathbf{0}\), a learned special case, not the default behavior.
- Forgetting that the cell state itself has no non-linearity applied directly to it in this update equation — the non-linearities (sigmoid for gates, tanh for the candidate) are applied to the inputs that determine how the cell state changes, not to the cell state's running value itself.
Interview Relevance
Q: "Why is the cell state update formula, \(\mathbf{C}_t = \mathbf{f}_t\odot\mathbf{C}_{t-1}+\mathbf{i}_t\odot\tilde{\mathbf{C}}_t\), specifically designed to help with vanishing gradients?" Because it's built from element-wise multiplication and addition rather than a full matrix multiplication followed by a squashing non-linearity, the local gradient \(\frac{\partial \mathbf{C}_t}{\partial \mathbf{C}_{t-1}}\) is simply \(\mathbf{f}_t\) — and if the forget gate learns to stay close to 1 for information that should be remembered long-term, gradients can flow backward through many time steps with far less exponential shrinkage than a plain RNN's hidden-state update allows.
Practice Question
If the forget gate for a specific element of the cell state is consistently learned to be very close to 0 (rather than close to 1), what does that imply about how the network is using that element of memory?
