The reset gate is GRU's second and final gate — it controls how much of the previous hidden state gets used when computing the new candidate content, letting the network effectively "start fresh" when the old state isn't relevant to what's coming next.
Formula
Same sigmoid-gated pattern as every other gate covered so far — its distinct role becomes clear in how it's actually used, in the candidate computation:
Notice \(\mathbf{r}_t\) multiplies \(\mathbf{h}_{t-1}\) before it's used to compute the candidate — if \(\mathbf{r}_t\) is close to 0, the candidate computation effectively ignores the previous hidden state almost entirely, computing new content based mostly on the current input alone.
What This Enables — "Forgetting" Before Proposing
This is subtly different from the update gate's role. The update gate (previous note) decides how much of the final blended hidden state comes from old vs. new information. The reset gate decides how much the old hidden state influences the computation of the new candidate itself — letting the network propose genuinely fresh content, largely uninfluenced by potentially irrelevant past context, when a reset is warranted (for instance, at a clear sequence boundary, like the start of a new sentence).
Numerical Example
\(r_t = 0.1\) (mostly reset — largely ignore the old hidden state for this candidate), \(h_{t-1}=0.4\): \(r_t \odot h_{t-1} = 0.1 \times 0.4 = 0.04\) — a heavily attenuated version of the old state feeds into the candidate computation, compared to the full \(0.4\) that would be used if \(r_t\) were close to 1.
Code
import torch
h_prev = torch.tensor([0.4, -0.2])
x_t = torch.tensor([1.0])
r_t = torch.tensor([0.1, 0.9]) # a hypothetical per-dimension reset gate
reset_hidden = r_t * h_prev
print(reset_hidden) # tensor([0.0400, -0.1800]) -- dimension 1 heavily attenuated, dimension 2 barely changed
combined_for_candidate = torch.cat([reset_hidden, x_t])
# this combined vector is what feeds into the candidate computation, NOT the raw h_prevFull Assembly Preview
Together, the reset gate (shaping what goes into the candidate) and the update gate (blending old state with the candidate into the final output) give GRU its complete gating behavior with just two gates instead of LSTM's three — the full set of equations, assembled together, is the subject of the next note.
Common Mistakes
- Confusing the reset gate with the update gate — the reset gate affects the candidate computation (what new content gets proposed); the update gate affects the final blend (how much of that proposal actually gets used versus the old state).
- Applying the reset gate to the final hidden state update instead of specifically within the candidate computation — its multiplicative effect on \(\mathbf{h}_{t-1}\) is scoped specifically to computing \(\tilde{\mathbf{h}}_t\), not to the update-gate blending step.
Interview Relevance
Q: "What's the difference between GRU's reset gate and update gate?" The reset gate controls how much of the previous hidden state influences the computation of the new candidate content — a reset value near 0 lets the candidate be computed largely fresh, ignoring past context. The update gate controls how much of that candidate actually replaces the old hidden state in the final output — these are two distinct decisions applied at two different points in the computation.
Practice Question
In what kind of situation might you expect a well-trained GRU's reset gate to output a value close to 0?
