This note assembles every equation from the previous five notes into one complete, unified reference — the full LSTM cell, all six formulas together, exactly as you'd need to recall them for an exam or an interview.
The Complete Set of Equations
The Full Cell, Diagrammed Together
Every piece from the previous five notes, assembled into the complete LSTM cell.
A Quick-Reference Summary Table
| Symbol | Name | Activation | Purpose |
|---|---|---|---|
| \(\mathbf{f}_t\) | Forget gate | Sigmoid | How much old cell state to keep |
| \(\mathbf{i}_t\) | Input gate | Sigmoid | How much new candidate to add |
| \(\tilde{\mathbf{C}}_t\) | Candidate state | Tanh | What new content to potentially add |
| \(\mathbf{C}_t\) | Cell state | — | The long-term memory pathway itself |
| \(\mathbf{o}_t\) | Output gate | Sigmoid | How much cell state to expose |
| \(\mathbf{h}_t\) | Hidden state | — | This step's working output |
Code — All Six Equations, End to End
import torch
def lstm_cell_manual(x_t, h_prev, C_prev, weights):
combined = torch.cat([h_prev, x_t])
f_t = torch.sigmoid(weights['W_f'] @ combined + weights['b_f'])
i_t = torch.sigmoid(weights['W_i'] @ combined + weights['b_i'])
C_candidate = torch.tanh(weights['W_C'] @ combined + weights['b_C'])
C_t = f_t * C_prev + i_t * C_candidate
o_t = torch.sigmoid(weights['W_o'] @ combined + weights['b_o'])
h_t = o_t * torch.tanh(C_t)
return h_t, C_tCommon Mistakes
- Mixing up which activation belongs to which equation under exam/interview pressure — the reliable rule: every gate (forget, input, output) uses sigmoid; the candidate state and the final hidden-state computation involve tanh.
- Forgetting that all four linear layers (\(f_t, i_t, \tilde C_t, o_t\)) take the same input, \([\mathbf{h}_{t-1}, \mathbf{x}_t]\), just with four independent sets of weights — this is exactly why PyTorch packs all four into one combined weight tensor internally, as noted in LSTM Architecture.
Interview Relevance
Q: "Write out all six LSTM equations from memory." This exact exercise — reproducing the forget gate, input gate, candidate state, cell state update, output gate, and hidden state formulas in order, correctly matching sigmoid to the three gates and tanh to the candidate and final hidden-state computation — is one of the most common whiteboard questions for sequence-modeling-focused ML/DL interviews.
Practice Question
Without looking back at the formulas, write out the cell state update equation and explain, in one sentence each, what \(\mathbf{f}_t\) and \(\mathbf{i}_t\) each control within it.
