Entropy measures how much uncertainty (or "surprise") a probability distribution contains. A distribution that's very predictable has low entropy; one that's maximally uncertain has high entropy. This single idea, extended to compare two distributions, becomes cross-entropy — the loss function behind almost every classifier.
Formula
Entropy is the expected value of \(-\log P(x)\) — the "surprise" of an outcome. Rare outcomes (\(P(x)\) small) carry high surprise (\(-\log P(x)\) large); common outcomes carry low surprise. Entropy averages this surprise, weighted by how often each outcome actually occurs.
Numerical Example — Two Distributions
A fair coin: \(P(\text{heads})=P(\text{tails})=0.5\).
A biased coin: \(P(\text{heads})=0.9, P(\text{tails})=0.1\).
The biased coin has lower entropy — its outcome is more predictable, so there's less genuine uncertainty (and less "information" gained) from observing a flip.
Visualizing Entropy vs Distribution "Peakedness"
A flat (uniform) distribution has maximum entropy — every outcome equally likely, maximum uncertainty. A peaked distribution has low entropy — one outcome dominates, little uncertainty.
Code
import numpy as np
def entropy(probs):
probs = np.array(probs)
return -np.sum(probs * np.log2(probs))
print(entropy([0.5, 0.5])) # 1.0 bit -- maximum entropy for 2 outcomes
print(entropy([0.9, 0.1])) # ~0.469 bits -- much less uncertainWhere This Shows Up in Deep Learning
Entropy alone isn't a loss function — but it's the direct building block for cross-entropy (comparing a model's predicted distribution against the true label distribution) and KL divergence (measuring how one distribution differs from another), covered in the next two notes. Both are central to essentially every classification and generative model's loss function.
Common Mistakes
- Assuming lower entropy is always "better" — for a classifier's predictions, low entropy (confident, peaked predictions) is desirable when the model is actually correct, but a confidently wrong prediction is worse than an uncertain one; entropy alone doesn't measure correctness.
- Forgetting entropy depends on the log base used — base 2 gives units of "bits" (common in information theory), while natural log (base \(e\)) gives "nats" (the convention deep learning frameworks typically use internally).
Interview Relevance
Q: "What does it mean if a classifier's softmax output has high entropy?" High entropy means the predicted probability is spread relatively evenly across classes — the model is uncertain about which class is correct. Low entropy means the prediction is concentrated on one class — the model is confident (though confidence and correctness are not the same thing).
Practice Question
Compute the entropy of a three-class prediction \([0.33, 0.33, 0.34]\) and compare it to \([0.98, 0.01, 0.01]\). Which has higher entropy, and does that match your intuition about which prediction is more "confident"?
