Practical guidance on choosing dropout rate — building on the conceptual mechanism from Dropout.
Typical Rates in Practice
| Rate | Typical Use |
|---|---|
| 0.1 – 0.2 | Lighter regularization, common in already well-regularized architectures (e.g. inside Transformer blocks alongside other regularizers) |
| 0.3 – 0.5 | Standard range for fully-connected layers in classic feedforward/CNN classification heads |
| 0.5+ | Heavier regularization, typically reserved for cases with severe overfitting or very limited training data |
Where Dropout Is (and Isn't) Typically Applied
Dropout is most commonly applied to fully-connected layers, particularly those with many parameters relative to the amount of training data. It's used more sparingly (or with lower rates) in convolutional layers, since a convolutional layer's weight-sharing already provides a meaningful degree of implicit regularization, and dropping individual spatial activations can disrupt the local feature correlations convolutions are specifically designed to exploit.
Diagnosing From Symptoms
| Symptom | Adjustment |
|---|---|
| Training loss much lower than validation loss (overfitting) | Increase dropout rate |
| Both training and validation performance are poor, or training loss is unexpectedly high (underfitting) | Decrease dropout rate — as flagged in Underfitting, excessive regularization can directly cause underfitting |
Code
import torch.nn as nn
# A dropout sweep, evaluated via validation performance
for dropout_rate in [0.2, 0.3, 0.5, 0.7]:
model = nn.Sequential(
nn.Linear(784, 256), nn.ReLU(), nn.Dropout(dropout_rate),
nn.Linear(256, 10)
)
train(model, train_loader, epochs=20)
val_acc = evaluate(model, val_loader)
print(f"dropout={dropout_rate}: val_accuracy={val_acc:.4f}")Common Mistakes
- Applying the same dropout rate uniformly across every layer without considering each layer's specific overfitting risk — layers with more parameters (typically the largest fully-connected layers) often benefit from relatively higher dropout than smaller layers.
- Increasing dropout in response to poor validation performance without first checking whether the actual problem is underfitting (in which case increasing dropout would make things worse, not better).
Interview Relevance
Q: "Why is dropout typically applied more cautiously (or not at all) in convolutional layers compared to fully-connected layers?" Convolutional layers already have a meaningful degree of implicit regularization from weight sharing (the same small kernel is reused across every spatial position, drastically limiting parameter count relative to what a fully-connected layer over the same input would need). Additionally, dropping individual spatial activations can disrupt the local feature correlations that convolutions are specifically designed to exploit, making dropout's benefit there less clear-cut than in fully-connected layers.
Practice Question
A model shows training accuracy of 98% and validation accuracy of 70%. Would you recommend increasing or decreasing its dropout rate?