Group Normalization

Group Normalization (GroupNorm) splits a layer's channels into groups and normalizes within each group, per example — landing structurally between LayerNorm (all channels together) and InstanceNorm (each channel alone), while sharing neither BatchNorm's batch dependency nor its small-batch fragility.

Formula

Channels are divided into \(G\) groups (a chosen hyperparameter); statistics are computed per example \(n\), per group \(g\), pooling across every channel in that group and every spatial location. Setting \(G\) equal to the total number of channels recovers exactly InstanceNorm; setting \(G=1\) (all channels in one group) recovers exactly LayerNorm applied per-channel-group — GroupNorm is a genuine generalization that includes both as special cases.

Why GroupNorm Fixes BatchNorm's Small-Batch Weakness

Because GroupNorm's statistics are computed entirely within a single example (across a subset of its own channels), they never depend on batch size or composition at all — identical behavior whether the batch size is 256 or 2. This makes GroupNorm a strong choice specifically for tasks where large batch sizes aren't practical — high-resolution image tasks (object detection, segmentation) where memory constraints often force small batch sizes, exactly the regime where BatchNorm's statistics become unreliably noisy.

Diagram — Where GroupNorm Sits

Code

import torch
import torch.nn as nn

group_norm = nn.GroupNorm(num_groups=8, num_channels=64)   # 64 channels split into 8 groups of 8
x = torch.randn(4, 64, 32, 32)   # batch of just 4 -- GroupNorm handles this fine, unlike BatchNorm
output = group_norm(x)
print(output.shape)   # torch.Size([4, 64, 32, 32])

Where It's Used Today

Common in object detection and segmentation architectures, where high-resolution inputs force small batch sizes due to GPU memory limits — precisely the regime where BatchNorm's statistics become unreliable and GroupNorm's batch-independence becomes a genuine practical advantage.

Common Mistakes

• Choosing a number of groups that doesn't evenly divide the number of channels — GroupNorm requires num_channels to be divisible by num_groups.
• Assuming GroupNorm's specific group count is a "set and forget" default — it is a genuine hyperparameter, and different group counts can meaningfully affect performance for a given architecture.

Interview Relevance

Q: "Why might you choose GroupNorm over BatchNorm for an object detection model?" Object detection often works with high-resolution images, which forces small batch sizes due to GPU memory limits. BatchNorm's statistics become unreliable with small batches, while GroupNorm computes statistics entirely within each example (across a subset of its channels), making it completely independent of batch size — a significant practical advantage in exactly this memory-constrained regime.

Practice Question

With 32 channels split into 4 groups, how many channels does each group contain? What does GroupNorm reduce to if you instead set the number of groups equal to 32?

Related DL Notes

• Batch Normalization
• Instance Normalization
• Layer Normalization