Every "Add & Norm" step in the Transformer architecture pairs the residual connection from the previous note with layer normalization — this note focuses specifically on where and why it's placed exactly where it is within the Transformer block (the general formula and mechanics are covered fully in the Normalization category).
Quick Recap of the Formula
\(\mu\) and \(\sigma^2\) are computed across the feature dimension for each individual token, independently — unlike batch normalization, layer normalization doesn't depend on other examples in the batch at all, which matters enormously for sequence models where sequence lengths and batch composition vary (see the full comparison in Batch Normalization and Layer Normalization).
Post-Norm (Original) vs Pre-Norm (Common Modern Variant)
| Variant | Formula | Detail |
|---|---|---|
| Post-norm (original 2017 paper) | \(\text{LayerNorm}(\mathbf{x}+\text{Sublayer}(\mathbf{x}))\) | Normalization applied after the residual addition |
| Pre-norm (common in many later models) | \(\mathbf{x}+\text{Sublayer}(\text{LayerNorm}(\mathbf{x}))\) | Normalization applied before the sublayer, with the residual addition left un-normalized |
Pre-norm has been widely adopted in many later, especially very large, Transformer-based models because it tends to produce more stable training dynamics for very deep stacks — the un-normalized residual path in pre-norm provides an even more direct, unimpeded gradient highway than post-norm's arrangement.
Diagram — The Two Placements, Compared
Pre-norm's un-normalized residual path (bottom right) provides an even more direct gradient highway through the whole stack.
Code
import torch
import torch.nn as nn
class PreNormSublayer(nn.Module):
def __init__(self, sublayer, d_model):
super().__init__()
self.norm = nn.LayerNorm(d_model)
self.sublayer = sublayer
def forward(self, x, **kwargs):
return x + self.sublayer(self.norm(x), **kwargs) # pre-norm: normalize BEFORE the sublayer
class PostNormSublayer(nn.Module):
def __init__(self, sublayer, d_model):
super().__init__()
self.norm = nn.LayerNorm(d_model)
self.sublayer = sublayer
def forward(self, x, **kwargs):
return self.norm(x + self.sublayer(x, **kwargs)) # post-norm: normalize AFTER the residual additionCommon Mistakes
- Assuming pre-norm and post-norm are interchangeable implementation details with no real consequence — they can produce meaningfully different training stability, especially at large scale and depth, and published architectures specify which one they use precisely because it matters.
- Using batch normalization instead of layer normalization in a Transformer — batch norm's dependence on batch statistics is poorly suited to variable-length sequences and small batch sizes common in sequence modeling, which is exactly why layer normalization (not batch normalization) became the standard choice for Transformers.
Interview Relevance
Q: "Why do many modern large Transformer-based models use pre-norm rather than the original paper's post-norm arrangement?" Pre-norm applies layer normalization before each sublayer rather than after the residual addition, leaving the residual path itself completely un-normalized — this provides an even more direct, unimpeded gradient path through the full depth of the stack, which has empirically been found to produce more stable training for very deep, large-scale Transformer models.
Practice Question
Why is layer normalization (not batch normalization) the standard choice for Transformer architectures processing variable-length sequences?
