Recurrent Neural Networks (RNNs)

Understand how Recurrent Neural Networks (RNNs) process sequential data by maintaining a hidden state that acts as memory, enabling the model to capture temporal dependencies in time-series or text data.

What it is

An RNN is a neural network architecture designed for sequence modeling. Unlike standard feed-forward networks that treat inputs as independent, an RNN processes elements one at a time, passing information from previous steps to the current step via a hidden state. This creates a form of short-term memory. The core mental model is a loop: at each time step $t$, the network computes a new hidden state $h_t$ based on the current input $x_t$ and the previous hidden state $h_{t-1}$. Related terms include Backpropagation Through Time (BPTT), Vanishing Gradient Problem, and variants like LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit).

Why it matters

• Natural Language Processing: Essential for tasks where word order defines meaning, such as sentiment analysis or machine translation.
• Time-Series Forecasting: Captures trends and seasonality in financial stocks, weather patterns, or sensor data.
• Audio and Speech Recognition: Processes raw audio waveforms or spectrograms sequentially to identify phonemes.
• Video Analysis: Tracks objects or actions across frames by leveraging temporal continuity.

Syntax or steps

The fundamental equation for a vanilla RNN cell is:

h_t = tanh(W_xh * x_t + W_hh * h_{t-1} + b_h)

Where:

• $h_t$ is the hidden state at time $t$.
• $x_t$ is the input vector at time $t$.
• $W_{xh}$ and $W_{hh}$ are weight matrices for input-to-hidden and hidden-to-hidden connections.
• $b_h$ is the bias term.
• $\tanh$ is the activation function ensuring values stay between -1 and 1.

Example

Below is a minimal implementation of a simple RNN layer using PyTorch to predict the next value in a sine wave sequence.

import torch
import torch.nn as nn
import numpy as np

# Define a simple RNN model
class SimpleRNN(nn.Module):
    def __init__(self, input_size=1, hidden_size=64, output_size=1):
        super(SimpleRNN, self).__init__()
        self.hidden_size = hidden_size
        # RNN layer takes input and previous hidden state
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        # Linear layer maps hidden state to output
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x, hidden=None):
        # x shape: (batch, seq_len, input_size)
        out, hidden = self.rnn(x, hidden)
        # Use only the last time step's output for prediction
        out = self.fc(out[:, -1, :])
        return out, hidden

# Generate synthetic data: Sine wave
seq_length = 50
data = np.sin(np.arange(seq_length))
X = torch.tensor(data[:-1], dtype=torch.float32).view(1, -1, 1) # Input: t-1
Y = torch.tensor(data[1:], dtype=torch.float32).view(1, -1, 1)   # Target: t

model = SimpleRNN()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

# Training loop (simplified)
for epoch in range(100):
    optimizer.zero_grad()
    outputs, _ = model(X)
    loss = criterion(outputs, Y)
    loss.backward()
    optimizer.step()
    
    if epoch % 20 == 0:
        print(f'Epoch [{epoch}/100], Loss: {loss.item():.4f}')

Explanation: The nn.RNN module handles the internal looping over the sequence dimension. We pass the entire sequence into the model, but we only use the final hidden state (out[:, -1, :]) to make a single-step prediction. In real applications, you might unroll this to predict multiple future steps.

Common mistakes

• Ignoring Sequence Length: Vanilla RNNs struggle with long sequences due to the vanishing gradient problem. If your sequence exceeds ~20-50 steps, consider LSTM or GRU instead.
• Incorrect Shape Handling: Forgetting to set batch_first=True or mismatching tensor dimensions (e.g., expecting (seq, batch) when getting (batch, seq)) causes runtime errors.
• Not Detaching Hidden States: When training on very long sequences, keeping gradients for all steps can cause memory overflow. Sometimes you need to detach the hidden state periodically.
• Assuming Global Memory: Standard RNNs have limited "memory" span. They do not inherently remember the start of a long sentence unless specifically architected (like with Attention mechanisms).

When to use it

Practice

Guided Exercise: Modify the example above to predict the next two values in the sine wave instead of just one. You will need to adjust the target tensor Y and potentially run the RNN step-by-step during inference.

Challenge: Implement a character-level language model using an RNN. Feed it a string like "hello world", encode characters as integers, and train it to predict the next character. Hint: Use torch.nn.Embedding before the RNN layer.

Quick check

Question: Why does a vanilla RNN often fail to learn dependencies between events separated by many time steps?

Answer: Due to the vanishing gradient problem. During backpropagation through time, gradients are multiplied repeatedly by the same weight matrix. If these weights are small, the gradient shrinks exponentially toward zero, preventing the network from updating weights relevant to earlier inputs.

Summary

RNNs introduce memory into neural networks by feeding the previous hidden state into the current computation, making them suitable for sequential data. While powerful for short-term dependencies, they suffer from vanishing gradients in long sequences, leading to the development of gated variants like LSTMs and GRUs for robust temporal modeling.