Overfitting is the opposite failure mode from underfitting: the model learns the training data's specific noise and idiosyncrasies too well, achieving excellent training performance that doesn't transfer to new, unseen data.
The Defining Signature
| Metric | Overfitting Signature |
|---|---|
| Training loss | Very low — the model fits the training data extremely well, sometimes almost perfectly |
| Validation loss | Noticeably higher than training loss, and may start increasing even as training loss keeps decreasing |
| Gap between train and validation performance | Large and often growing over the course of training |
Why It Happens — Memorization vs Generalization
A sufficiently high-capacity model (many parameters relative to the amount of training data) can, in principle, memorize the training set almost exactly — including its noise, labeling errors, and coincidental patterns that don't reflect the true underlying relationship being learned. A model that has memorized noise performs excellently on that exact noisy data but has learned nothing generalizable, so it performs poorly on new data drawn from the same true distribution but with different specific noise.
Visualizing Overfitting
The overfit curve hits every training point exactly, but its wild oscillation between points reflects noise, not the true underlying trend.
Common Causes
- Model too complex relative to the amount of available training data — too many parameters for too little data.
- Training for too many epochs — even a well-sized model can eventually start memorizing noise if trained far past the point of genuine improvement (exactly what Early Stopping prevents).
- Insufficient regularization — no dropout, no weight decay, no data augmentation to discourage memorization.
- Small or unrepresentative training dataset — less data means more opportunity for the model to fit noise rather than genuine signal.
Fixes for Overfitting
| Fix | Category Covered In |
|---|---|
| Add or increase dropout, weight decay (L1/L2) | Regularization (next category) |
| Add data augmentation | Regularization (next category) |
| Use early stopping | This category — Early Stopping |
| Collect more training data | — |
| Reduce model capacity (fewer layers/parameters) | — |
| Use transfer learning from a model pretrained on more data | Transfer Learning category |
Code — Diagnosing Overfitting From Training Logs
train_losses = [2.3, 1.5, 0.8, 0.4, 0.2, 0.1, 0.05] # keeps dropping, very low
val_losses = [2.4, 1.6, 1.0, 0.9, 0.95, 1.1, 1.3] # drops initially, then RISES
# Training loss keeps improving; validation loss turns upward around epoch 4 --
# this diverging pattern is the classic signature of overfittingCommon Mistakes
- Judging model quality from training accuracy alone — a model with 99% training accuracy could be badly overfit, performing far worse on genuinely new data; validation/test performance is what actually matters.
- Applying every regularization technique at maximum strength reflexively — over-regularizing can push a properly-fitting model into underfitting instead; regularization strength should be tuned, not maximized blindly.
Interview Relevance
Q: "What's the clearest single signal that a model is overfitting, visible in a training log?" A growing gap between training loss (continuing to decrease, often to a very low value) and validation loss (plateauing or starting to increase) — the model keeps improving on data it has memorized while getting worse on data it hasn't seen, which is exactly the diverging pattern early stopping and regularization address.
Practice Question
A model achieves 98% training accuracy and 75% validation accuracy. Name two specific techniques you'd try first to close this gap, and briefly justify each.
