Recurrent Connections

The "recurrent" in Recurrent Neural Network refers to one specific structural feature: a connection that feeds the hidden state back into the same computation at the next time step. This note isolates that feature precisely and explains exactly why weight sharing across time is what makes it work.

The Recurrent Connection, Formally

The defining feature: \(\mathbf{h}_t\) depends on \(\mathbf{h}_{t-1}\), which itself depended on \(\mathbf{h}_{t-2}\), and so on back to \(\mathbf{h}_0\) — a chain of dependencies threading through the entire sequence. This is a genuine loop in the computation graph, unlike a standard feedforward network where information only ever flows forward through a fixed sequence of distinct layers.

Why the SAME Weights Must Be Used at Every Step

Imagine instead giving each time step its own separate weight matrix, \(\mathbf{W}_{hh}^{(t)}\), rather than reusing one shared \(\mathbf{W}_{hh}\). Two serious problems would immediately follow: (1) the number of parameters would grow with sequence length, so a network trained on 10-step sequences couldn't process a 50-step sequence at all — no weights would exist for steps 11 through 50; (2) the network would need separate training examples specifically covering every possible sequence position to learn what to do there, rather than learning one general "how to update the hidden state" rule that applies uniformly, however long the sequence turns out to be. Sharing \(\mathbf{W}_{hh}\) (and \(\mathbf{W}_{xh}\)) across every step is what solves both problems simultaneously.

Diagram — The Loop, Explicitly

Code — Verifying the Weights Are Genuinely Shared

import torch.nn as nn

rnn = nn.RNN(input_size=5, hidden_size=8)
# There is exactly ONE weight_hh_l0 tensor, used for every time step --
# not one per step, regardless of how long a sequence you feed it
print(rnn.weight_hh_l0.shape)   # torch.Size([8, 8]) -- same shape and same values, every step

Common Mistakes

• Picturing the "loop" as happening simultaneously with all time steps at once — it's a genuinely sequential dependency; \(\mathbf{h}_t\) cannot be computed until \(\mathbf{h}_{t-1}\) exists, which is exactly why RNN training and inference can't be parallelized across time the way convolution or attention operations can (a limitation explored in Limitations of RNN).

Interview Relevance

Q: "Why must an RNN reuse the exact same weight matrices at every time step, rather than learning separate weights per step?" Reusing the same weights lets the network apply one general, learned rule for updating its hidden state, regardless of sequence length or position — this is what allows it to generalize to sequences of arbitrary length and to positions it may not have seen as often during training, without needing a proportionally larger number of parameters for longer sequences.

Practice Question

If an RNN's recurrent weights were NOT shared across time steps, and a network was trained only on sequences of length 20, could it process a sequence of length 30 at inference time? Why or why not?

Related DL Notes

• Hidden State
• Unrolling RNN
• Limitations of RNN