A probability distribution is the complete rule describing how likely each possible value of a random variable is. Knowing which distribution a network's output represents tells you exactly which loss function to use.
Discrete: Probability Mass Function (PMF)
The Bernoulli distribution (a single yes/no outcome, e.g. binary classification) is the most common discrete distribution in deep learning: \(P(X=1)=p\), \(P(X=0)=1-p\). The categorical distribution generalizes this to more than two outcomes — exactly what a softmax output represents.
Continuous: Probability Density Function (PDF)
For continuous variables, \(f(x)\) is a density, not a probability directly — \(P(a \le X \le b) = \int_a^b f(x)\,dx\) is the probability of landing in a range. The most important continuous distribution in deep learning is the normal (Gaussian) distribution:
\(\mu\) is the mean (center), \(\sigma\) is the standard deviation (spread). Weight initialization schemes (He, Xavier) sample from a normal distribution with a carefully chosen \(\sigma\).
Visualizing the Normal Distribution
A larger standard deviation σ spreads the distribution out; a smaller one concentrates it tightly around the mean μ.
Code
import numpy as np
import torch
# Sampling from a normal distribution -- common for weight initialization
samples = np.random.normal(loc=0.0, scale=0.05, size=5)
print(samples)
# PyTorch's default initialization for many layers uses a similar approach
layer = torch.nn.Linear(10, 5)
print(layer.weight.std().item()) # a small standard deviation, by designWhere This Shows Up in Deep Learning
| Distribution | DL Use Case |
|---|---|
| Bernoulli | Binary classification output, dropout mask |
| Categorical | Multi-class classification output (softmax) |
| Normal (Gaussian) | Weight initialization, noise in VAEs and diffusion models |
| Uniform | Some weight initialization schemes, random data augmentation parameters |
Common Mistakes
- Treating a PDF value \(f(x)\) as a probability directly — for continuous distributions, only the integral over a range gives a probability; \(f(x)\) itself can even exceed 1.
- Assuming every network output is naturally categorical — regression outputs are typically modeled as continuous (often implicitly Gaussian, which is why mean squared error is the natural loss — see Mean Squared Error).
Interview Relevance
Q: "Why is a network's classification output modeled as a categorical distribution, but a regression output usually isn't?" Classification has a finite, discrete set of possible labels, matching the categorical distribution's support exactly (softmax enforces this). Regression targets are continuous, so they're typically modeled as coming from a continuous distribution (often implicitly Gaussian), which is what makes squared-error loss the natural choice.
Practice Question
A weight initialization scheme samples from a normal distribution with \(\mu=0, \sigma=0.01\). What does a small \(\sigma\) tell you about the typical initial weight values?
