This note assembles GRU's complete set of equations — just three, compared to LSTM's six — into one unified reference, then works through a full numerical example by hand.
The Complete Set of Equations
Full Numerical Worked Example
Setup: \(h_{t-1}=0.3\), \(x_t=1.0\), combined \([h_{t-1},x_t]=[0.3,1.0]\).
Update gate:
Reset gate:
Candidate hidden state (using \(r_t \odot h_{t-1} = 0.6248 \times 0.3 \approx 0.1874\)):
Final hidden state:
Code — Verifying Every Value
import torch
h_prev = torch.tensor(0.3)
x_t = torch.tensor(1.0)
W_z, b_z = torch.tensor([0.4, 0.2]), torch.tensor(0.0)
W_r, b_r = torch.tensor([-0.3, 0.5]), torch.tensor(0.1)
W_h, b_h = torch.tensor([0.6, -0.2]), torch.tensor(0.0)
combined = torch.stack([h_prev, x_t])
z_t = torch.sigmoid(torch.dot(W_z, combined) + b_z)
r_t = torch.sigmoid(torch.dot(W_r, combined) + b_r)
reset_combined = torch.stack([r_t * h_prev, x_t])
h_candidate = torch.tanh(torch.dot(W_h, reset_combined) + b_h)
h_t = (1 - z_t) * h_prev + z_t * h_candidate
print("z_t:", z_t.item()) # 0.5793
print("r_t:", r_t.item()) # 0.6248
print("h_candidate:", h_candidate.item()) # -0.0874
print("h_t:", h_t.item()) # 0.0756 -- matches every hand-computed valueComparing Complexity Directly: 3 Equations vs 6
GRU's three equations (plus the final blend) achieve a structurally similar gated-memory effect to LSTM's six, with fewer independent weight sets to learn — this is the concrete numerical face of the parameter-count reduction discussed in GRU Architecture.
Common Mistakes
- Forgetting that the reset gate must be applied before concatenating with \(x_t\) for the candidate computation, not as a separate later step — \(\mathbf{r}_t\odot\mathbf{h}_{t-1}\) is computed first, then concatenated with \(\mathbf{x}_t\), then passed through \(\mathbf{W}_h\).
- Using \(\mathbf{z}_t\) instead of \((1-\mathbf{z}_t)\) for the old-state term in the final blend, or vice versa — mixing these up inverts the intended behavior of the update gate entirely.
Interview Relevance
Q: "Write out all four GRU equations from memory." The update gate, reset gate, candidate hidden state (using the reset-gated previous hidden state), and the final hidden state blend using the update gate and its complement — being able to reproduce these correctly, including exactly where the reset gate's multiplication occurs, is a common practical check of genuine understanding versus memorized keywords.
Practice Question
Using the same weights as the worked example, compute the update gate value for a new input \(x_t = -1.0\) with the same \(h_{t-1}=0.3\).
