Under the hood, most modern LLMs are structurally very close to the decoder-only Transformer from GPT Architecture — stacked blocks of masked self-attention and feed-forward layers. This note covers the handful of common modern refinements layered on top of that base design.
The Core, Unchanged Structure
Every modern LLM still fundamentally repeats the same block: masked multi-head self-attention, residual connection, normalization, feed-forward network, residual connection, normalization — stacked \(N\) times, exactly as in Transformer Decoder (minus cross-attention, since there's no separate encoder).
Common Modern Refinements
| Component | Original Transformer | Common in Modern LLMs |
|---|---|---|
| Normalization | LayerNorm | RMSNorm — a simplified, cheaper variant (covered in the Normalization category) |
| Positional information | Sinusoidal, added once at the input | Rotary Positional Embeddings (RoPE), applied directly within attention — covered in Positional Embeddings |
| Attention | Standard multi-head | Grouped-query or multi-query attention — sharing key/value projections across multiple query heads to reduce memory during inference |
| Activation in feed-forward | ReLU | GELU or SwiGLU variants |
None of these are fundamentally new ideas — they're targeted efficiency and stability refinements on top of the same core architecture this hub has already built up piece by piece.
Scale, Not Novelty, Is the Main Differentiator
| Model (illustrative) | Approx. Layers | Approx. Parameters |
|---|---|---|
| GPT-2 (small) | 12 | ~124M |
| GPT-2 (XL) | 48 | ~1.5B |
| GPT-3 | 96 | ~175B |
The architectural pattern barely changes across this table — what changes is the number of stacked blocks, the hidden dimension, and the number of attention heads, scaled up dramatically.
Code
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2")
print(model.config.n_layer) # 12 -- number of stacked decoder blocks
print(model.config.n_head) # 12 -- attention heads per block
print(model.config.n_embd) # 768 -- d_modelCommon Mistakes
- Assuming every efficiency refinement (RMSNorm, RoPE, grouped-query attention) represents a fundamentally different architecture — they're incremental, well-motivated engineering improvements on the same core Transformer decoder design.
- Assuming bigger is always simply "the same model, scaled" — larger models often also adjust hyperparameters (learning rate schedules, batch sizes) non-trivially to train stably at that scale, not just naively stacking more identical blocks.
Interview Relevance
Q: "What actually differs between a small GPT-2-style model and a modern frontier LLM, architecturally?" Surprisingly little at the structural level — both are decoder-only Transformer stacks using masked self-attention and feed-forward blocks. The differences are mostly scale (far more layers, larger hidden dimensions, more parameters) plus a handful of targeted efficiency refinements (RMSNorm, RoPE, grouped-query attention) rather than a fundamentally different design.
Practice Question
Why might reducing the number of distinct key/value projections (grouped-query attention) meaningfully help inference efficiency, even though it slightly reduces attention's representational flexibility?
