KL divergence (Kullback-Leibler divergence) measures how much one probability distribution differs from another — it's the "extra surprise" you incur by using an approximate distribution \(q\) when the true distribution is actually \(p\). It closes out this module by tying entropy and cross-entropy together into one clean relationship.
Formula
This is exactly cross-entropy minus entropy: the extra bits of "surprise" incurred specifically because your model \(q\) isn't a perfect match for the true distribution \(p\). If \(q=p\) exactly, \(D_{KL}(p\parallel q) = 0\) — no extra surprise at all.
Numerical Example
True distribution \(p=[0.7, 0.3]\), model's predicted distribution \(q=[0.5, 0.5]\).
A positive value, confirming \(q\) is not a perfect match for \(p\) — the further \(q\) drifts from \(p\), the larger this value grows.
Why Training Minimizes Cross-Entropy, Not KL Divergence Directly
Rearranged: \(H(p,q) = D_{KL}(p\parallel q) + H(p)\). Since \(p\) is the fixed true label distribution during standard classification training, \(H(p)\) is a constant — it doesn't depend on the model's parameters at all. Minimizing cross-entropy \(H(p,q)\) with respect to the model is therefore exactly equivalent to minimizing KL divergence \(D_{KL}(p\parallel q)\) — they differ only by a constant that doesn't affect where the minimum is. This is why deep learning frameworks implement cross-entropy loss directly rather than KL divergence — the two objectives point to the same optimal weights.
Key Properties
| Property | Detail |
|---|---|
| Non-negativity | \(D_{KL}(p\parallel q) \ge 0\) always, with equality only when \(p=q\) |
| Asymmetry | \(D_{KL}(p\parallel q) \ne D_{KL}(q\parallel p)\) in general — it is not a true distance metric |
Code
import torch
import torch.nn.functional as F
p = torch.tensor([0.7, 0.3])
q = torch.tensor([0.5, 0.5])
kl = F.kl_div(q.log(), p, reduction='sum') # PyTorch expects log-probabilities as the first argument
print(kl.item()) # approximately 0.082, matching the manual calculationWhere This Shows Up in Deep Learning
- VAEs (Variational Autoencoders): KL divergence is used directly as a regularization term, pulling the model's learned latent distribution toward a simple prior (e.g. a standard normal).
- Knowledge distillation: a smaller "student" model is trained to match a larger "teacher" model's full output distribution by minimizing KL divergence between them, not just matching the single correct label.
- RLHF / policy optimization: KL divergence penalizes a fine-tuned language model's output distribution from drifting too far from its original pretrained distribution, keeping updates controlled.
Common Mistakes
- Treating KL divergence as a true "distance" between distributions — it isn't symmetric and doesn't satisfy the triangle inequality, so it can't be used interchangeably with a metric like Euclidean distance.
- Assuming cross-entropy and KL divergence give different training results in standard classification — with a fixed true-label distribution, they differ only by a constant, so their gradients (and therefore training dynamics) are identical.
Interview Relevance
Q: "Why do VAEs use KL divergence as an explicit loss term, while standard classifiers just use cross-entropy?" A standard classifier compares a predicted distribution against a fixed, one-hot true-label distribution — cross-entropy and KL divergence differ only by a constant there, so cross-entropy alone suffices. A VAE compares two learned distributions (the encoder's output vs a chosen prior) where that constant term genuinely matters for shaping the latent space, so KL divergence is used explicitly as its own loss component.
Key Takeaways — Probability & Statistics for DL
- Probability distributions are the formal language behind every classifier's softmax output and every generative model's predictions.
- Expected value, variance and covariance quantify center, spread, and joint behavior — directly motivating feature standardization and PCA.
- Bayes' theorem lets you correctly reason from evidence back to hypothesis probability, essential for interpreting model outputs on imbalanced data.
- Maximum likelihood estimation is the theoretical justification for both cross-entropy loss (categorical assumption) and mean squared error (Gaussian assumption).
- Entropy, cross-entropy and KL divergence form one connected family: KL divergence = cross-entropy − entropy, which is exactly why minimizing cross-entropy during training also minimizes KL divergence to the true label distribution.
Next: Neural Network Fundamentals finally assembles the linear algebra, calculus and probability from these three categories into your first trainable network — from the biological neuron through the complete forward-propagation-to-weight-update flow.
Practice Question
If a model's predicted distribution \(q\) exactly matches the true distribution \(p\), what is \(D_{KL}(p\parallel q)\), and what does that tell you about the relationship between cross-entropy and entropy in that specific case?
