An RNN (Recurrent Neural Network) processes a sequence one token at a time, carrying a running "memory" (hidden state) forward. Transformers replaced this sequential processing with parallel self-attention — a fundamentally different way of handling sequences.
Side-by-Side
| RNN | Transformer | |
|---|---|---|
| Processing order | Strictly sequential — token 2 needs token 1's result first | Parallel — all tokens processed simultaneously via attention |
| How context is carried | A single running hidden state, updated at each step | Direct attention between any two tokens, regardless of distance |
| Long-range dependencies | Tend to weaken over long distances — information from early tokens can fade by the time later tokens are processed | Directly relates distant tokens with no inherent decay |
| Training speed at scale | Slower — limited parallelization due to sequential dependency | Much faster to train at scale — highly parallelizable |
The Core Limitation RNNs Had
RNN processing "The cat, which had been sitting on the warm
windowsill all afternoon watching birds, finally jumped down":
By the time the RNN reaches "jumped", information about "cat"
(all the way back at the start) has passed through many
sequential updates to the hidden state — and can weaken or
get diluted along the way, especially over long sequences.Transformers sidestep this entirely — "jumped" can attend directly to "cat" via self-attention, regardless of how many words are in between.
Practical Use Case
This is the core technical reason transformer-based LLMs handle long documents and long conversations more effectively than RNN-based approaches did — and why RNNs, while still used in some specialized, resource-constrained, or streaming contexts, are no longer the default choice for large-scale general-purpose language modeling.
Common Mistakes
- Assuming RNNs are simply "obsolete" in every context — they remain relevant in some lightweight, low-latency, or streaming-specific applications where their simpler sequential structure is actually an advantage
- Assuming transformers have no sequence-length limitations at all — they trade "no long-range decay" for a hard, fixed context window limit (see Context Window), a different kind of constraint, not the absence of one
Interview Relevance
Q: "Why did transformers largely replace RNNs for large language models?" — parallelizable training and better handling of long-range dependencies via direct attention are the two pillars of a strong answer.
Practice Question
Explain why an RNN's sequential processing makes it harder to train efficiently on the very large datasets modern LLMs use, compared to a transformer.