Dropout

Dropout takes a completely different approach to regularization from L1/L2/weight decay: instead of penalizing weight magnitudes, it randomly disables a fraction of neurons during every training step — forcing the network to avoid relying too heavily on any single neuron.

The Mechanism

\(p\) is the dropout probability — the chance any given neuron gets "dropped" (zeroed) this step. \(\mathbf{m}\) is a randomly sampled binary mask, one independent Bernoulli draw per neuron, applied fresh on every forward pass during training. Dividing by \((1-p)\) — called inverted dropout — rescales the surviving activations so their expected total magnitude matches what it would be without dropout, keeping the scale consistent between training and evaluation.

Numerical Example

With \(p=0.5\) on a layer with activations \([2, 4, 1, 3]\): suppose the random mask happens to be \([1,0,1,0]\) (neurons 2 and 4 dropped). Masked activations: \([2,0,1,0]\). Scaled by \(\frac{1}{1-0.5}=2\): final output \([4,0,2,0]\) — the surviving neurons are doubled to compensate for the (on average) half of the layer's usual total signal being zeroed out this step.

Why Randomly Disabling Neurons Helps

If a network could rely on one specific neuron always being available, it might build up a fragile dependency on that neuron's exact behavior — a form of "co-adaptation" where neurons become highly specialized to compensate for each other's specific quirks rather than each independently learning something generally useful. Since dropout can zero out any neuron at any step, the network is forced to learn representations that remain useful even when various subsets of neurons are missing — a form of built-in redundancy that tends to improve generalization. Dropout is also sometimes understood as training an implicit ensemble of many different "thinned" sub-networks simultaneously, sharing weights, and approximately averaging them at test time.

Critical: Dropout Only Applies During Training

This is exactly why Validation Loop and Training Loop flagged model.train() vs model.eval() as essential — during evaluation/inference, dropout must be disabled entirely (every neuron active, no masking, no scaling), so the model's output is deterministic and uses its full learned capacity.

Code

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Dropout(p=0.5),   # 50% of neurons randomly zeroed EACH forward pass, during training only
    nn.Linear(64, 10)
)

model.train()
x = torch.randn(1, 128)
print(model(x))   # dropout is ACTIVE -- different random neurons zeroed on each call

model.eval()
print(model(x))   # dropout is DISABLED -- deterministic, full-capacity output

Choosing a Dropout Rate

Common Mistakes

• Forgetting to switch to model.eval() before validation or inference — dropout would remain active, producing noisy, non-deterministic, and needlessly degraded outputs.
• Applying a very high dropout rate to a small model or a model that's already underfitting — this compounds the underfitting problem rather than addressing overfitting.
• Using standard dropout with SELU activations — as flagged in SELU, this breaks SELU's self-normalizing property; "alpha dropout" is the compatible alternative there specifically.

Interview Relevance

Q: "Why does dropout need to rescale surviving activations by \(\frac{1}{1-p}\) during training?" Without rescaling, the total activation magnitude flowing out of a dropout layer would be smaller during training (since some neurons are zeroed) than during evaluation (where all neurons are active) — a mismatch that would require the next layer to somehow adapt to two different activation scales. Dividing surviving activations by \((1-p)\) keeps the expected total magnitude consistent between training and evaluation, so no rescaling is needed at inference time.

Practice Question

With dropout rate \(p=0.3\) applied to activations \([5, 10, 2]\), and a random mask of \([1,1,0]\), compute the final scaled output.

Related DL Notes

• Training Loop
• Validation Loop
• SELU