Regularization is any technique that deliberately constrains a model, trading a small amount of training performance for meaningfully better generalization — the direct, practical countermeasure to the overfitting problem covered in Overfitting.
The General Idea
Most regularization techniques add a penalty term to the loss that discourages the model from becoming "too complex" in some specific sense — often, too dependent on any single weight being very large. \(\lambda\) is a hyperparameter controlling how strongly this penalty is enforced: \(\lambda=0\) recovers plain, unregularized training; larger \(\lambda\) trades away some training fit in exchange for a simpler, hopefully better-generalizing model.
Why This Trade Is Worth Making
Recall the bias-variance tradeoff from Bias-Variance Tradeoff: an overfitting model has low bias but high variance — it fits its specific training set very closely, including noise that won't repeat in new data. Regularization deliberately increases bias slightly (the model can no longer fit training data quite as tightly) in exchange for a often much larger reduction in variance (the model becomes less sensitive to which specific training examples it happened to see) — a net win for generalization, which is the metric that actually matters.
Not All Regularization Looks Like a Penalty Term
While L1/L2 penalties (next two notes) fit the formula above directly, other important regularization techniques constrain the model differently: Dropout randomly disables neurons during training; data augmentation expands the effective training set with transformed examples; early stopping (already covered in Early Stopping) limits how long the model is allowed to keep fitting the training set at all. All of these share the same underlying goal — improving generalization — even though only some of them literally add a term to the loss function.
A Map of What's Coming
| Technique | Core Idea |
|---|---|
| L1 Regularization | Penalizes the sum of absolute weight values — tends to produce sparse weights (some exactly zero) |
| L2 Regularization | Penalizes the sum of squared weight values — shrinks all weights toward zero smoothly |
| Weight Decay | Directly shrinks weights each update — mathematically related to L2, with an important subtlety under adaptive optimizers |
| Dropout | Randomly disables neurons during training, preventing over-reliance on any single one |
| Data Augmentation | Expands the effective training set with label-preserving transformations of existing data |
Common Mistakes
- Applying strong regularization to a model that's actually underfitting — as flagged in Underfitting, this makes an already-too-simple model even less capable of fitting the true pattern; regularization strength should track the actual overfitting/underfitting diagnosis, not be applied reflexively.
- Treating regularization as a single, interchangeable "overfitting fix" — different techniques have different mechanisms and are often combined, not chosen exclusively from one another.
Interview Relevance
Q: "In one sentence, what is regularization trying to achieve?" It deliberately trades a small amount of training-set fit (slightly increased bias) for a larger reduction in the model's sensitivity to the specific training data it happened to see (reduced variance) — improving generalization to new, unseen data, which is the actual goal of training in the first place.
Practice Question
Why would applying strong regularization to a model that's already underfitting be counterproductive?
