The Recurrent Neural Network (RNN) was designed to satisfy exactly the three requirements laid out in Sequential Data: handle variable-length sequences, share parameters across every position, and maintain a memory of earlier elements while processing later ones.
The Core Idea
Instead of processing an entire sequence at once, an RNN processes it one element at a time, maintaining a running "hidden state" that acts as a compressed summary of everything seen so far. At each step, the network combines the new input with this running summary to produce an updated summary — carrying information forward through the sequence.
How This Solves Each Requirement
| Requirement | How RNNs Satisfy It |
|---|---|
| Variable-length sequences | The same step-by-step process simply repeats for however many elements the sequence has — no fixed input size required |
| Shared parameters across positions | The exact same weight matrices are reused at every single time step — the network learns one set of "how to update the summary" rules, applied uniformly regardless of position |
| Memory of earlier elements | The hidden state, carried from step to step, is precisely this memory |
Diagram — The High-Level Idea
The same cell processes each element of the sequence in turn, its hidden state carrying forward a summary of everything seen so far.
Code — The Conceptual Loop
import torch
import torch.nn as nn
rnn_cell = nn.RNNCell(input_size=10, hidden_size=20)
sequence = torch.randn(5, 1, 10) # 5 time steps, batch size 1, 10 features each
h = torch.zeros(1, 20) # initial hidden state, usually all zeros
for t in range(5):
x_t = sequence[t]
h = rnn_cell(x_t, h) # the SAME cell (same weights) is reused at every time step
print(f"step {t}: hidden state shape {h.shape}")Common Mistakes
- Assuming an RNN has a separate set of weights for each time step — it uses exactly one shared set of weights, reused identically at every step; this is precisely what lets it generalize to sequences of any length.
- Confusing an RNN's "hidden state" with a hidden layer in a standard feedforward network — an RNN's hidden state specifically carries information across time steps, not just across layers within one forward pass, as detailed in the next note.
Interview Relevance
Q: "What's the key architectural idea that lets an RNN handle sequences of any length with a fixed number of parameters?" Weight sharing across time steps — the exact same weight matrices are applied at every position in the sequence, rather than learning distinct weights per position. This means the number of parameters depends only on the hidden state size and input feature size, never on how long the sequence happens to be.
Practice Question
Why does processing a sequence element-by-element with a carried hidden state solve the "variable length" problem that a fixed-size MLP input cannot?
