The attention mechanism, introduced by Bahdanau et al. in 2014, resolves the exact bottleneck identified in Seq2Seq Limitations: instead of compressing the entire input into one fixed-size context vector, let the decoder look back at every encoder hidden state, and learn to weight them by relevance, fresh at each decoding step.
The Core Idea, Before Any Formulas
At each decoding step, instead of using a single, fixed context vector computed once, attention computes a new, dynamically weighted combination of all the encoder's hidden states — weighted by how relevant each input position is to what's being generated right now. Generating the French word for "cat" would give high weight to the encoder's hidden state at the position of the English word "cat"; generating "woke up" near the end would give high weight to whichever encoder positions are actually relevant to that specific word, regardless of how far back in the sentence they occurred.
Diagram — Selective Focus, Not Uniform Compression
Line thickness represents the learned attention weight — the decoder dynamically decides which encoder positions matter most, fresh at every single generation step.
Why This Directly Fixes the Bottleneck
| Basic Seq2Seq | With Attention | |
|---|---|---|
| What the decoder sees | One fixed-size context vector, computed once | A fresh, dynamically weighted combination of ALL encoder hidden states, recomputed at every decoding step |
| Information loss for long inputs | Severe — everything must fit through one vector | Minimal — every encoder position's information remains individually accessible |
| Can focus on distant, relevant input | No — must have survived compression | Yes — direct access, regardless of position |
Code — A First Glimpse (Conceptual)
import torch
# Encoder hidden states for a 4-word input, hidden_dim=8
encoder_states = torch.randn(4, 8) # one row per input position
# Attention weights for a specific decoding step (learned/computed, not fixed)
attention_weights = torch.tensor([0.05, 0.75, 0.10, 0.10]) # heavily favors position 2 (h2)
context_for_this_step = attention_weights @ encoder_states # a WEIGHTED SUM, recomputed every step
print(context_for_this_step.shape) # (8,) -- a fresh context vector, specific to this exact decoding stepNotice this "context" is recomputed fresh for every decoding step — a fundamentally different, far more flexible mechanism than basic Seq2Seq's single, static context vector.
Common Mistakes
- Assuming attention replaces the encoder-decoder structure entirely — it's an addition to it, giving the decoder richer access to encoder information; the underlying encoder-decoder framing from earlier in this category still applies.
- Thinking attention weights are hand-designed rules — they're learned parameters, computed via a small neural network component (formalized starting with the next note, Query, Key, Value) trained end-to-end alongside the rest of the model.
Interview Relevance
Q: "In one sentence, what problem does attention solve that basic Seq2Seq doesn't?" It gives the decoder direct, selective access to every one of the encoder's hidden states at every decoding step — rather than forcing all input information through one fixed-size context vector — letting the model dynamically focus on whichever input positions are actually relevant to what's currently being generated, regardless of how far away they are in the sequence.
Practice Question
Why does attention's benefit tend to grow more pronounced as input sequence length increases, compared to short sequences where basic Seq2Seq might already work reasonably well?
