This note maps out the complete Transformer architecture at a glance — every major component and how they fit together — before the rest of this category examines each piece individually.
The High-Level Structure
The original Transformer follows the same encoder-decoder pattern from Encoder-Decoder Architecture, but both the encoder and decoder are built entirely from stacked attention and feedforward layers, with no recurrence anywhere.
The Complete Component List
| Component | Role | Covered In |
|---|---|---|
| Input embeddings | Convert input tokens into vectors | NLP with Deep Learning category |
| Positional encoding | Inject order information self-attention lacks natively | Positional Encoding |
| Multi-head self-attention | Relate every position to every other position | Seq2Seq & Attention category |
| Masked self-attention | Prevents the decoder from "seeing" future tokens | Masked Self-Attention |
| Cross-attention | Lets the decoder attend to the encoder's output | Seq2Seq & Attention category |
| Position-wise feed-forward network | Adds per-token non-linear transformation capacity | Feed-Forward Network |
| Residual connections | Enable training very deep stacks of these layers | Residual Connections |
| Layer normalization | Stabilizes activations throughout the deep stack | Layer Normalization |
Diagram — The Overall Shape
An encoder stack and a decoder stack, both built from repeated identical layers, connected by cross-attention — the same overall shape as the "Attention Is All You Need" architecture.
Encoder-Only, Decoder-Only, and Encoder-Decoder Variants
The original Transformer used both an encoder and decoder stack (well suited to translation, where input and output are genuinely different sequences). Many later, highly influential models use only one half: encoder-only models (like BERT, covered in the NLP category) are suited to understanding tasks (classification, extracting information); decoder-only models (like GPT, also covered in the NLP category) are suited to generation tasks, using masked self-attention throughout and no cross-attention at all, since there's no separate encoder output to attend to.
Code — A High-Level Skeleton
import torch.nn as nn
class TransformerSkeleton(nn.Module):
def __init__(self, d_model, num_heads, num_layers, vocab_size):
super().__init__()
self.embedding = nn.Embedding(vocab_size, d_model)
# PyTorch provides both halves directly, built from the components in this category:
self.encoder = nn.TransformerEncoder(
nn.TransformerEncoderLayer(d_model, num_heads), num_layers
)
self.decoder = nn.TransformerDecoder(
nn.TransformerDecoderLayer(d_model, num_heads), num_layers
)
self.output_layer = nn.Linear(d_model, vocab_size)Common Mistakes
- Assuming every Transformer-based model has both an encoder and a decoder — many of the most widely used models (BERT, GPT) use only one half, adapted to their specific task.
- Treating the Transformer as one monolithic new idea rather than a specific, deliberate assembly of components mostly already covered — self-attention, residual connections (borrowed conceptually from ResNet, covered in the CNN Architectures category), and normalization all predate or were adapted from existing ideas; the Transformer's contribution was combining them into a fully recurrence-free architecture.
Interview Relevance
Q: "What's the difference between an encoder-only, decoder-only, and full encoder-decoder Transformer, and when would you use each?" Encoder-only models (like BERT) are suited to understanding/classification tasks, since they can attend bidirectionally over the full input. Decoder-only models (like GPT) are suited to generation, using masked self-attention so each position only sees earlier positions, matching how text is generated left to right. Full encoder-decoder models (the original Transformer design) suit tasks with genuinely distinct input and output sequences, like translation, where cross-attention lets the decoder draw on the encoder's full representation of the input.
Practice Question
Would a Transformer-based model for text summarization (input: a long article; output: a short summary) more naturally use an encoder-decoder design, or a decoder-only design? Justify your answer.
