The hidden state is the single most important concept in an RNN — a fixed-size vector that acts as the network's evolving "memory," compressing everything relevant about the sequence up to the current time step into one representation.
What the Hidden State Actually Represents
At each time step \(t\), \(\mathbf{h}_t\) is meant to capture whatever information from \(\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_t\) is useful for the task — not the raw inputs themselves, but a learned, compressed summary. For a language model predicting the next word, for instance, the hidden state might implicitly encode things like "the sentence so far is a question," or "the subject of this sentence is plural" — whatever turns out to be useful for the network's objective, discovered automatically through training rather than hand-designed.
A Fixed-Size Bottleneck
Critically, \(\mathbf{h}_t\) has a fixed size (the "hidden size" hyperparameter), regardless of how long the sequence processed so far has been. This means the hidden state must be selective about what it retains — after 100 time steps, it cannot literally store all 100 raw inputs; it has to have compressed and often partially discarded earlier information to make room for newer, presumably more immediately relevant information. This fixed-capacity bottleneck is directly responsible for RNNs' difficulty retaining very long-range dependencies, a limitation explored fully later in this category.
Initial Hidden State
Before processing the first element of a sequence, there's no "previous" hidden state to work from — \(\mathbf{h}_0\) is typically initialized to a vector of zeros (or occasionally a learned initial value), representing "no information yet."
Code — Inspecting Hidden State Evolution
import torch
import torch.nn as nn
rnn = nn.RNN(input_size=5, hidden_size=8, batch_first=True)
sequence = torch.randn(1, 4, 5) # one sequence, 4 time steps
h = torch.zeros(1, 1, 8) # initial hidden state -- all zeros
for t in range(4):
x_t = sequence[:, t:t+1, :]
_, h = rnn(x_t, h)
print(f"After step {t}, hidden state (first 3 values): {h[0,0,:3]}")
# notice the hidden state changes at every step, carrying forward a running summaryChoosing the Hidden Size — A Real Hyperparameter Tradeoff
| Hidden Size | Effect |
|---|---|
| Small | Less capacity to retain information — may struggle with longer or more complex sequences, but faster and cheaper |
| Large | More capacity to retain nuanced information across the sequence, at higher compute and memory cost, and higher overfitting risk on small datasets |
Common Mistakes
- Assuming a larger hidden size automatically means "remembers more of the sequence perfectly" — it increases capacity, but doesn't eliminate the fundamental compression bottleneck, nor does it fix the vanishing gradient issues covered later in this category, which limit how far back in the sequence gradient-based learning can actually reach.
- Confusing the hidden state (a vector, updated every time step) with the final output (which may be a separate, differently-shaped projection of the hidden state via \(\mathbf{W}_{hy}\)).
Interview Relevance
Q: "Why can't an RNN's hidden state just perfectly remember everything from earlier in a long sequence?" The hidden state has a fixed size, chosen in advance as a hyperparameter, regardless of sequence length — it's a compressed summary, not a growing record. As the sequence gets longer, the network must increasingly compress or overwrite older information to represent newer information within that same fixed capacity, which is a core reason RNNs struggle with very long-range dependencies.
Practice Question
If an RNN's hidden size is 64, does that number change depending on whether the input sequence has 10 time steps or 1,000 time steps?
