This closing note of the Regularization category puts every technique covered so far side by side — what each one actually constrains, when each is most appropriate, and how they combine in real training recipes.
Complete Comparison Table
| Technique | What It Constrains | Produces Sparsity? | Typical Use |
|---|---|---|---|
| L1 Regularization | Sum of absolute weight values | Yes — drives some weights to exactly zero | When feature selection or a sparser, more interpretable model is desired |
| L2 Regularization | Sum of squared weight values | No — shrinks smoothly toward zero | General-purpose default; discourages any single weight from growing too large |
| Weight Decay | Direct proportional shrinkage of every weight, per update | No | Equivalent to L2 under SGD; requires AdamW (not plain Adam) to behave correctly under adaptive optimizers |
| Dropout | Reliance on any single neuron's specific activation | N/A — a structural technique, not a weight-magnitude penalty | Fully-connected layers especially; less commonly applied inside convolutional layers directly |
| Data Augmentation | Reliance on exact, unvaried training examples | N/A | Especially effective and standard for image, audio and some text tasks with rich, label-preserving transformations available |
| Early Stopping | How long training is allowed to keep fitting the training set | N/A | Nearly always worth using — cheap, effective, and complements every other technique here |
These Techniques Are Combined, Not Chosen Exclusively
Real training recipes routinely stack several of these simultaneously — for example, a CNN might use L2 regularization (via weight_decay), dropout in its fully-connected layers, data augmentation on its input images, and early stopping based on validation loss, all at once. Each technique constrains a different aspect of the model or training process, so their benefits are largely additive rather than redundant.
A Practical Decision Guide
| Situation | Reasonable Starting Point |
|---|---|
| General overfitting, no specific structure to the problem | L2 regularization / weight decay, as a low-effort default |
| Suspect many irrelevant input features | L1 regularization, for its sparsity/feature-selection effect |
| Fully-connected layers in a moderately deep network | Dropout, at a moderate rate (e.g. 0.3–0.5) |
| Image, audio, or other data with natural label-preserving transformations available | Data augmentation — often the single highest-leverage technique for these data types |
| Any training run, regardless of other choices | Early stopping based on validation performance — essentially always worth including |
Code — A Combined Example
import torch.optim as optim
import torch.nn as nn
model = nn.Sequential(
nn.Linear(784, 256), nn.ReLU(), nn.Dropout(0.5),
nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.3),
nn.Linear(128, 10)
)
# L2 regularization (as weight_decay) combined with dropout layers above,
# plus data augmentation applied to the input pipeline, plus early stopping
# in the training loop -- several regularization techniques stacked together
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)Common Mistakes
- Assuming one regularization technique alone is a complete solution — in practice, the strongest results usually come from combining several complementary techniques, each addressing a slightly different mechanism of overfitting.
- Adding every regularization technique at maximum strength simultaneously without validating each one's individual contribution — over-regularizing can push a model into underfitting just as easily as under-regularizing leaves it overfitting.
Interview Relevance
Q: "You're training a CNN for image classification and observe significant overfitting. Walk through the regularization techniques you'd consider, in order of what you'd try first." A strong answer would typically start with the cheapest, most broadly applicable options — early stopping and data augmentation (especially valuable for image data) — then add dropout in fully-connected layers, and apply weight decay (via AdamW specifically, for correct behavior under an adaptive optimizer) — combining several complementary techniques rather than relying on just one.
Key Takeaways — Regularization
- Regularization trades a small increase in bias for a larger reduction in variance, improving generalization — the direct countermeasure to overfitting.
- L1 produces sparse weights (exact zeros); L2 shrinks weights smoothly; weight decay is L2's direct-shrinkage equivalent under SGD, but requires AdamW to behave correctly under Adam.
- Dropout and data augmentation regularize through entirely different mechanisms — randomly disabling neurons, and expanding effective training data — and both apply only during training, never evaluation.
- These techniques combine rather than compete — real training recipes typically stack several simultaneously.
Next: Normalization Techniques covers BatchNorm, LayerNorm, and their variants — a related but distinct set of techniques that stabilize and speed up training by controlling the scale of activations, rather than directly penalizing weights.
Practice Question
A model shows mild overfitting on an image classification task with limited training data. List two or three regularization techniques from this category you'd try together, and briefly justify each choice.
