Implementation exercises for recurrent networks — implementing an RNN cell manually, then building a real LSTM-based sequence classifier in PyTorch.
🟢 Problem 1: Implement a single RNN cell's forward step manually
Task: Implement one time step of a vanilla RNN's hidden state update: \(h_t = \tanh(W_{xh}x_t + W_{hh}h_{t-1} + b)\).
def rnn_cell_forward(x_t, h_prev, W_xh, W_hh, b):
return np.tanh(x_t @ W_xh + h_prev @ W_hh + b)
input_size, hidden_size = 4, 3
W_xh = np.random.randn(input_size, hidden_size) * 0.1
W_hh = np.random.randn(hidden_size, hidden_size) * 0.1
b = np.zeros(hidden_size)
x_t = np.random.randn(1, input_size)
h_prev = np.zeros((1, hidden_size))
h_t = rnn_cell_forward(x_t, h_prev, W_xh, W_hh, b)
print(f"New hidden state: {h_t}")🟡 Problem 2: Unroll the RNN cell across a full sequence
Task: Given a sequence of 5 time steps, apply your RNN cell repeatedly, carrying the hidden state forward, and collect all hidden states.
def rnn_forward_sequence(X_seq, W_xh, W_hh, b, hidden_size):
batch_size = X_seq.shape[0]
h_t = np.zeros((batch_size, hidden_size))
all_hidden_states = []
for t in range(X_seq.shape[1]): # iterate over time steps
x_t = X_seq[:, t, :]
h_t = rnn_cell_forward(x_t, h_t, W_xh, W_hh, b)
all_hidden_states.append(h_t)
return np.stack(all_hidden_states, axis=1) # (batch, seq_len, hidden_size)
X_seq = np.random.randn(2, 5, input_size) # batch=2, seq_len=5
hidden_states = rnn_forward_sequence(X_seq, W_xh, W_hh, b, hidden_size)
print(hidden_states.shape) # (2, 5, 3)Hint if stuck: The key idea is that h_t from one iteration becomes h_prev for the next — this is exactly what "recurrent" means, and is the only thing carrying information across time steps.
🟡 Problem 3: Build an LSTM-based sentiment classifier in PyTorch
Task: Build a model that embeds input tokens, processes them with an LSTM, and classifies the final hidden state as positive/negative sentiment.
class LSTMClassifier(nn.Module):
def __init__(self, vocab_size, embed_dim=64, hidden_dim=128, num_classes=2):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
self.fc = nn.Linear(hidden_dim, num_classes)
def forward(self, x):
embedded = self.embedding(x) # (batch, seq_len, embed_dim)
_, (h_n, c_n) = self.lstm(embedded) # h_n: (1, batch, hidden_dim)
final_hidden = h_n.squeeze(0) # (batch, hidden_dim)
return self.fc(final_hidden)
model = LSTMClassifier(vocab_size=5000)
sample_input = torch.randint(0, 5000, (4, 20)) # batch=4, seq_len=20
logits = model(sample_input)
print(logits.shape) # (4, 2)Hint if stuck: nn.LSTM returns two things: the output at every time step, and a tuple of the final hidden and cell states — for classification using only the final summary of the sequence, you typically want h_n, the final hidden state.
🔴 Problem 4: Compare a vanilla RNN vs LSTM on a long-dependency task
Task: Create a synthetic task where the correct output depends on information from early in a long sequence (e.g. classify based on the first token, with 50 irrelevant tokens after it). Train both a vanilla RNN and an LSTM on this task and compare their accuracy.
# Synthetic task: label = 1 if first token > 0, else 0 -- but the sequence is 50 tokens long
def generate_long_dependency_data(num_samples, seq_len=50):
X = np.random.randn(num_samples, seq_len, 1)
y = (X[:, 0, 0] > 0).astype(int) # label depends ONLY on the very first time step
return torch.tensor(X, dtype=torch.float32), torch.tensor(y, dtype=torch.long)
# Train both nn.RNN and nn.LSTM based classifiers on this data with identical
# hyperparameters, and compare final test accuracy -- the LSTM should notably
# outperform the vanilla RNN, directly demonstrating why LSTM's gating helps
# preserve information over long sequences that a vanilla RNN tends to loseHint if stuck: If both models perform similarly, try increasing seq_len further — the vanilla RNN's disadvantage becomes more pronounced as the dependency distance grows, which is exactly the point this exercise is designed to demonstrate.
