The input gate is the second of LSTM's three gates — it decides how much of the newly proposed information (the candidate state, next note) actually gets written into the cell state.
Formula
Structurally identical to the forget gate — a sigmoid-activated linear layer over the concatenated previous hidden state and current input — but with its own independently-learned weights \(\mathbf{W}_i, \mathbf{b}_i\), serving a distinct purpose: controlling how much new information gets added, rather than how much old information gets kept.
How It Works Together With the Candidate State
Recall the cell state update from LSTM Cell State: \(\mathbf{C}_t = \mathbf{f}_t\odot\mathbf{C}_{t-1} + \mathbf{i}_t\odot\tilde{\mathbf{C}}_t\). The candidate state \(\tilde{\mathbf{C}}_t\) (next note) proposes what new information could be added; the input gate \(\mathbf{i}_t\) decides how much of that proposal to actually incorporate. This separation — "what could be added" versus "how much to add" — mirrors the forget gate's separate "how much to keep" role, giving the network fine, independent control over both halves of the memory update.
Reading the Gate's Values
| \(i_t\) Value (per element) | Meaning |
|---|---|
| Close to 1 | "This new information is important — write it into memory fully" |
| Close to 0 | "This new information isn't relevant right now — largely ignore it, leave this part of memory unchanged by new input" |
Numerical Example
Same inputs as the forget gate example, but with the input gate's own weights: \(\mathbf{h}_{t-1}=[0.2,-0.1]\), \(\mathbf{x}_t=[1.0]\), \(\mathbf{W}_i=[-0.2, 0.6, 0.4]\), \(b_i=0.1\):
Code
import torch
h_prev = torch.tensor([0.2, -0.1])
x_t = torch.tensor([1.0])
combined = torch.cat([h_prev, x_t])
W_i = torch.tensor([-0.2, 0.6, 0.4])
b_i = torch.tensor(0.1)
z_i = torch.dot(W_i, combined) + b_i
i_t = torch.sigmoid(z_i)
print(i_t) # tensor(0.5987) -- matches the hand-worked exampleCommon Mistakes
- Confusing the input gate's role with the candidate state's role — the input gate decides how much to write; the candidate state (computed by a separate set of weights, using tanh) decides what the proposed new content actually is. They're multiplied together, not the same computation.
- Assuming the forget gate and input gate are complementary (i.e. \(\mathbf{i}_t = 1-\mathbf{f}_t\)) — they're independently learned with separate weights; nothing forces them to sum to 1, though the GRU architecture (covered later in this category) does deliberately couple an analogous pair of decisions this way.
Interview Relevance
Q: "Why does LSTM use a separate input gate rather than just always fully adding the candidate state to the cell state?" Not all new information at every time step is equally relevant to remember — the input gate lets the network learn, per dimension and per time step, how much of the newly proposed candidate content is actually worth writing into long-term memory, rather than indiscriminately adding everything and letting memory become cluttered with irrelevant information.
Practice Question
If both the forget gate and input gate for a specific cell-state dimension output values close to 0 at the same time step, what happens to that dimension's value in \(\mathbf{C}_t\)?
