This note lays out the RNN's structure precisely — the specific weight matrices involved, and how they connect the input, hidden state, and output at each time step.
The Three Weight Matrices
| Matrix | Connects | Shape |
|---|---|---|
| \(\mathbf{W}_{xh}\) | Current input \(\mathbf{x}_t\) to the hidden state | \((\text{hidden\_size}, \text{input\_size})\) |
| \(\mathbf{W}_{hh}\) | Previous hidden state \(\mathbf{h}_{t-1}\) to the current hidden state | \((\text{hidden\_size}, \text{hidden\_size})\) |
| \(\mathbf{W}_{hy}\) | Current hidden state \(\mathbf{h}_t\) to the output \(\mathbf{y}_t\) | \((\text{output\_size}, \text{hidden\_size})\) |
All three matrices — along with their bias vectors \(\mathbf{b}_h\) and \(\mathbf{b}_y\) — are shared across every single time step, exactly as established in Why RNN.
RNN vs a Standard Feedforward Layer
| Feedforward Layer | RNN Cell | |
|---|---|---|
| Input | One fixed-size vector | A new input at every time step, plus the previous hidden state |
| Weights | One set, used once per forward pass | One set, reused at every time step |
| Output | One output vector | Optionally, one output vector per time step, plus the evolving hidden state |
Diagram — The Recurrent Cell's Internals
The cell combines the current input and the previous hidden state, applies a non-linearity, and produces the new hidden state — used both for this step's output and passed to the next step.
Code — Using PyTorch's Built-In RNN Layer
import torch
import torch.nn as nn
rnn = nn.RNN(input_size=10, hidden_size=20, batch_first=True)
sequence = torch.randn(4, 7, 10) # batch of 4, 7 time steps, 10 features each
output, h_final = rnn(sequence)
print(output.shape) # torch.Size([4, 7, 20]) -- hidden state at EVERY time step
print(h_final.shape) # torch.Size([1, 4, 20]) -- only the FINAL hidden state
print(rnn.weight_ih_l0.shape) # (20, 10) -- this is W_xh
print(rnn.weight_hh_l0.shape) # (20, 20) -- this is W_hhCommon Mistakes
- Confusing the "output" tensor's shape (hidden states at every time step) with the "final hidden state" tensor (only the last step) —
nn.RNNreturns both, and it's a common source of shape-mismatch bugs to use the wrong one for a downstream task. - Forgetting that \(\mathbf{W}_{hy}\) is a separate matrix from \(\mathbf{W}_{hh}\) — the hidden state and the output are not the same thing, even though the hidden state is what the output is computed from.
Interview Relevance
Q: "How many distinct weight matrices does a basic RNN cell have, and what does each connect?" Three: \(\mathbf{W}_{xh}\) (input to hidden state), \(\mathbf{W}_{hh}\) (previous hidden state to current hidden state), and \(\mathbf{W}_{hy}\) (hidden state to output) — plus their associated bias vectors. All are shared and reused identically across every time step of the sequence.
Practice Question
For an RNN with input size 8 and hidden size 16, what is the shape of \(\mathbf{W}_{xh}\) and \(\mathbf{W}_{hh}\)?
