Sentiment Analysis (LSTM) Project

A complete sentiment analysis project — the classic first real NLP project, building an LSTM-based text classifier from raw text through a trained, evaluated model.

Problem Statement

Build a binary sentiment classifier (positive/negative) for movie reviews, achieving at least 85% test accuracy, with a full pipeline from raw text to trained model.

Dataset

The IMDB movie review dataset (50,000 labeled reviews) is the standard dataset for this task, available directly through torchtext or as a downloadable CSV.

Architecture & Approach

Text is tokenized, converted to integer indices via a vocabulary, embedded, then processed by an LSTM — using the LSTM's final hidden state as a summary of the entire review for classification, directly applying the Practice: RNN & LSTM exercise's pattern to a real dataset.

Step-by-Step Build

import torch
import torch.nn as nn
from collections import Counter

# 1. Build a vocabulary from the training text
def build_vocab(texts, max_vocab_size=10000):
    counter = Counter(word for text in texts for word in text.lower().split())
    vocab = {"<pad>": 0, "<unk>": 1}
    for word, _ in counter.most_common(max_vocab_size - 2):
        vocab[word] = len(vocab)
    return vocab

def text_to_indices(text, vocab, max_len=200):
    indices = [vocab.get(w, vocab["<unk>"]) for w in text.lower().split()][:max_len]
    indices += [vocab["<pad>"]] * (max_len - len(indices))   # pad to fixed length
    return indices

vocab = build_vocab(train_texts)

# 2. The model
class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim=100, hidden_dim=128):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(hidden_dim * 2, 2)   # *2 because bidirectional
        self.dropout = nn.Dropout(0.3)

    def forward(self, x):
        embedded = self.dropout(self.embedding(x))
        _, (h_n, _) = self.lstm(embedded)
        final_hidden = torch.cat([h_n[-2], h_n[-1]], dim=1)   # concat forward + backward final states
        return self.fc(self.dropout(final_hidden))

model = SentimentLSTM(vocab_size=len(vocab))
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.CrossEntropyLoss()

# 3. Train and evaluate (standard loop)
for epoch in range(10):
    model.train()
    for x_batch, y_batch in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(x_batch), y_batch)
        loss.backward()
        optimizer.step()

    model.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for x_batch, y_batch in test_loader:
            preds = model(x_batch).argmax(dim=1)
            correct += (preds == y_batch).sum().item()
            total += y_batch.size(0)
    print(f"Epoch {epoch+1}: test accuracy = {correct/total:.4f}")

Expected Results

A bidirectional LSTM with a reasonable vocabulary size and a handful of training epochs should reach roughly 85-88% test accuracy on IMDB sentiment classification — comparable to, though generally somewhat below, what a fine-tuned Transformer-based model (like BERT) would achieve on the same task.

Key Learnings & Extensions

• Padding index handling matters — using padding_idx=0 in the embedding layer tells PyTorch not to update the padding token's embedding during training, since it carries no real semantic content.
• Extension: Replace the LSTM with a small pretrained Transformer (e.g. via HuggingFace's transformers library) and compare accuracy — a direct, hands-on comparison of the two architecture families on the same task.
• Extension: Add attention over the LSTM's outputs (rather than just using the final hidden state) and see if it improves accuracy or interpretability.

Related DL Notes

• Practice: RNN & LSTM
• Text Generation Project
• Tokenization