GPT (Generative Pre-trained Transformer) takes the opposite architectural approach from BERT: a decoder-only Transformer, trained purely on next-token prediction, generating text strictly left to right. This is the architectural foundation underlying essentially every modern large language model.
Decoder-Only — What That Actually Means
GPT uses only the decoder half of the original Transformer, and even then, without the cross-attention sublayer (there's no separate encoder output to attend to) — just repeated blocks of masked self-attention (see Masked Self-Attention) followed by a position-wise feed-forward network, each wrapped in residual connections and layer normalization, stacked many times.
The Pretraining Objective: Next-Token Prediction
Given every token that came before, predict the next one — trained via standard cross-entropy loss (see Categorical Cross-Entropy) over the entire vocabulary at every position. This is an even simpler self-supervised objective than BERT's masked language modeling — the "labels" are just the very next word in the raw training text, requiring no masking scheme or auxiliary task design at all.
GPT vs BERT — Architectural Comparison
| BERT | GPT | |
|---|---|---|
| Transformer half used | Encoder only | Decoder only (no cross-attention) |
| Self-attention | Bidirectional (unmasked) | Masked (causal) — each token sees only earlier tokens |
| Pretraining objective | Masked language modeling + (originally) next sentence prediction | Next-token prediction |
| Natural strength | Understanding tasks (classification, extraction) | Generation tasks (writing, completion, dialogue) |
| Generates text naturally? | Not designed for this | Yes — this is exactly its core training objective |
Why Decoder-Only Became the Dominant Choice for Modern LLMs
Next-token prediction is a remarkably simple, uniform, and scalable training objective — it needs no masking design decisions, no auxiliary tasks, and works directly on raw, unlabeled text at massive scale. This simplicity, combined with the fact that many useful tasks (question answering, summarization, coding, reasoning) can all be reframed as "generate the appropriate continuation of this text," is a large part of why decoder-only architectures became the dominant foundation for essentially every major modern large language model — covered in full depth in the LLM Fundamentals category next.
Code
from transformers import GPT2Tokenizer, GPT2LMHeadModel
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
input_text = "The future of artificial intelligence is"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids
output_ids = model.generate(input_ids, max_length=30, do_sample=True, top_k=50)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
# generates a plausible continuation, one token at a time, autoregressivelyCommon Mistakes
- Assuming GPT-style models can't be used for understanding/classification tasks — with the right prompting (framing the task as generating an appropriate label as text), decoder-only models can and do handle classification-style tasks, though it's not their most architecturally "native" fit compared to BERT.
- Forgetting that GPT's masked (causal) self-attention is architecturally identical to what the Transformer decoder already used in Transformer Decoder — GPT isn't introducing a new attention mechanism, just applying the existing decoder-only pattern without cross-attention, since there's no separate encoder sequence involved.
Interview Relevance
Q: "Why is next-token prediction such an appealing pretraining objective for large-scale language models?" It requires no labeled data or special masking design — every piece of raw text, at any scale, automatically provides training signal simply from what word comes next. This simplicity makes it trivially scalable to enormous, diverse text corpora, and the resulting model directly learns to generate coherent continuations, which turns out to generalize surprisingly well to a huge range of downstream tasks when framed as "generate an appropriate continuation."
Key Takeaways — NLP with Deep Learning
- Subword tokenization (BPE) balances vocabulary size against sequence length while gracefully handling unseen words — the modern standard over word-level or character-level tokenization.
- Word embeddings replaced one-hot encoding's huge, similarity-blind vectors with dense, learned representations capturing semantic relationships.
- Word2Vec (CBOW/Skip-Gram) and GloVe are static embedding methods — one fixed vector per word; contextual embedding models (BERT, GPT) compute a fresh, context-dependent vector for every occurrence.
- BERT (encoder-only, bidirectional) suits understanding tasks; GPT (decoder-only, causal) suits generation; T5 unifies every task into a single text-to-text framing using the full encoder-decoder architecture.
Next: LLM Fundamentals goes deep into exactly how decoder-only models like GPT are trained at scale and actually run — pretraining, fine-tuning, alignment (RLHF/DPO), sampling strategies, context windows, and the KV cache that makes efficient generation possible.
Practice Question
Would a decoder-only (GPT-style) or encoder-only (BERT-style) architecture more naturally suit a chatbot that needs to generate open-ended conversational responses? Justify your answer.
