Encoder-Decoder Architecture

The encoder-decoder architecture solves a problem none of the previous RNN/LSTM/GRU notes addressed: what happens when the input sequence and output sequence have different lengths, as in translating a 5-word English sentence into a 7-word French one?

The Core Idea

Split the model into two separate RNN-family networks with distinct jobs: an encoder reads the entire input sequence and compresses it into a fixed-size summary; a decoder then generates the output sequence, one element at a time, using that summary as its starting point.

Diagram

Why Two Separate Networks, Not One

A single RNN, as covered in the RNN category, naturally produces one output per input time step — it can't easily handle an input of length 5 producing an output of length 7. Separating "understand the input" (encoder) from "generate the output" (decoder) into two distinct networks removes this constraint entirely — the decoder can run for as many steps as needed, completely decoupled from how many steps the encoder ran for.

Code — A Minimal PyTorch Sketch

import torch
import torch.nn as nn

class Encoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, batch_first=True)

    def forward(self, x):
        embedded = self.embedding(x)
        _, (h, c) = self.rnn(embedded)
        return h, c   # the "summary" -- passed to the decoder

class Decoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.output_layer = nn.Linear(hidden_dim, vocab_size)

    def forward(self, x, h, c):
        embedded = self.embedding(x)
        output, (h, c) = self.rnn(embedded, (h, c))
        return self.output_layer(output), h, c

Common Mistakes

• Assuming the encoder and decoder must use the same architecture or hidden size — while common for simplicity, they're independent networks and can differ, as long as the summary's dimensionality matches what the decoder expects.
• Forgetting that the encoder's own output sequence (its hidden state at every intermediate time step) is typically discarded in the basic version of this architecture — only the final summary is used, which is exactly the bottleneck examined in Context Vector.

Interview Relevance

Q: "Why is an encoder-decoder architecture needed for tasks like machine translation, rather than a single RNN?" Input and output sequences often have different lengths (a sentence in one language rarely has the same word count as its translation), and a single RNN's output length is naturally tied to its input length. Splitting into a separate encoder (compresses input into a summary) and decoder (generates output of whatever length is needed from that summary) removes this length constraint.

Practice Question

Why does the decoder need some representation of the input sequence to generate a correct output, even though it never directly "sees" the input tokens itself?

Related DL Notes

• Context Vector
• Seq2Seq Model
• LSTM Architecture