Early stopping halts training automatically once validation performance stops improving — a simple, remarkably effective form of regularization that directly prevents the model from continuing to overfit past its best generalization point.
The Core Idea
Revisit the training/validation loss diagram from Validation Loop: training loss keeps decreasing, but validation loss eventually turns upward as the model starts memorizing training-specific noise. Early stopping watches for exactly this turning point and stops training there — rather than continuing for a pre-fixed number of epochs regardless of what validation performance is doing.
The Algorithm — Patience-Based Early Stopping
- Track the best validation loss seen so far, and how many epochs it's been since that best value improved (the "patience counter").
- After each epoch, if validation loss improved, reset the patience counter to 0 and save a checkpoint (see Checkpointing).
- If it didn't improve, increment the patience counter.
- If the patience counter exceeds a threshold (e.g. 5 or 10 epochs with no improvement), stop training and restore the best checkpoint.
Code
best_val_loss = float('inf')
patience = 5
epochs_without_improvement = 0
for epoch in range(num_epochs):
# ... training loop for this epoch ...
# ... validation loop for this epoch, producing avg_val_loss ...
if avg_val_loss < best_val_loss:
best_val_loss = avg_val_loss
epochs_without_improvement = 0
torch.save(model.state_dict(), 'best_model.pt')
else:
epochs_without_improvement += 1
if epochs_without_improvement >= patience:
print(f"Early stopping at epoch {epoch} -- no improvement for {patience} epochs")
break
model.load_state_dict(torch.load('best_model.pt')) # restore the best version, not the last oneWhy "Patience," Not Stopping at the First Non-Improvement
Validation loss is itself somewhat noisy from epoch to epoch (it's still computed on a finite sample) — stopping the very first time it fails to improve would trigger prematurely on normal noise, not genuine overfitting. A patience window of several epochs distinguishes a real, sustained upward trend from ordinary epoch-to-epoch fluctuation.
Choosing the Patience Value
| Patience | Tradeoff |
|---|---|
| Too small (e.g. 1) | Stops prematurely on normal validation noise, before the model has fully converged |
| Too large (e.g. 50) | Wastes significant compute time continuing to train well past the point of genuine improvement |
| Moderate (5–15, task-dependent) | Reasonable balance — tolerates normal noise while still stopping promptly once overfitting is genuinely underway |
Common Mistakes
- Restoring the last epoch's weights instead of the best checkpoint after early stopping triggers — the whole point of early stopping is to use the best-performing version, not whatever happened to be current when the patience threshold was hit.
- Using training loss instead of validation loss to decide when to stop — training loss will keep improving even as the model overfits, making it useless as an early-stopping signal.
Interview Relevance
Q: "Why does early stopping need a 'patience' parameter instead of stopping immediately when validation loss fails to improve?" Validation loss fluctuates somewhat from epoch to epoch due to normal noise, even when the model is still genuinely improving overall. Stopping at the very first non-improvement would trigger on this noise prematurely; a patience window requires several consecutive non-improving epochs before concluding that overfitting has genuinely begun, distinguishing signal from noise.
Practice Question
A model's validation loss improves for 8 epochs, then stays flat (neither improving nor worsening significantly) for the next 6 epochs, with patience set to 5. At which epoch does early stopping trigger?
