Bayes' theorem gives you a precise way to flip a conditional probability around — to go from \(P(\text{evidence}\mid\text{hypothesis})\), which is often easy to reason about, to \(P(\text{hypothesis}\mid\text{evidence})\), which is usually what you actually want to know. It resolves the exact confusion flagged at the end of the previous note.
Formula
| Term | Name | Meaning |
|---|---|---|
| \(P(A)\) | Prior | Belief in \(A\) before seeing any evidence |
| \(P(B\mid A)\) | Likelihood | How probable the evidence \(B\) is, if \(A\) is true |
| \(P(A\mid B)\) | Posterior | Updated belief in \(A\), after observing evidence \(B\) |
| \(P(B)\) | Evidence / normalizer | Total probability of observing \(B\), across all hypotheses |
Solving the Disease Test Example
Continuing the practice question from Conditional Probability: \(P(\text{disease})=0.01\), \(P(\text{positive}\mid\text{disease})=0.95\), \(P(\text{positive}\mid\text{no disease})=0.10\) (a 90% accurate negative test means a 10% false-positive rate).
Despite a "95% accurate" test, a positive result only means an 8.76% chance of actually having the disease — because the disease is rare (1% prior) and the 10% false-positive rate applies to the much larger healthy population. This is the classic illustration of why the prior matters enormously, not just the test's accuracy.
Code
p_disease = 0.01
p_pos_given_disease = 0.95
p_pos_given_no_disease = 0.10
p_no_disease = 1 - p_disease
p_positive = p_pos_given_disease * p_disease + p_pos_given_no_disease * p_no_disease
p_disease_given_positive = (p_pos_given_disease * p_disease) / p_positive
print(p_disease_given_positive) # 0.0876...Where This Shows Up in Deep Learning
- Naive Bayes classifiers apply this formula directly, assuming feature independence given the class, to compute \(P(\text{class}\mid\text{features})\).
- Bayesian deep learning treats a network's weights themselves as random variables with a prior distribution, updating to a posterior distribution over weights given the training data — used for uncertainty estimation.
- Class imbalance intuition: the disease example generalizes directly — a rare class (low prior) needs strong evidence to be predicted confidently, which is exactly why naively-trained classifiers underperform on imbalanced datasets unless the class imbalance is explicitly addressed.
Common Mistakes
- Ignoring the prior \(P(A)\) and reasoning only from the test's accuracy — as the worked example shows, a rare condition combined with even a modest false-positive rate can make the posterior probability surprisingly low.
- Forgetting the denominator \(P(B)\) must sum over all possible hypotheses, not just the one you're currently interested in.
Interview Relevance
Q: "A model is 99% accurate on a dataset where only 1% of examples are the positive class. Why might this be a red flag rather than good news?" A model that always predicts the negative class would already achieve 99% accuracy on such an imbalanced dataset, without learning anything useful. This is directly analogous to the disease-test example — accuracy alone, without accounting for class priors (base rates), can be deeply misleading; precision, recall and F1 (covered in Evaluation Metrics) are needed instead.
Practice Question
Using Bayes' theorem, if a spam filter has \(P(\text{spam})=0.4\), \(P(\text{"free"}\mid\text{spam})=0.6\), and \(P(\text{"free"}\mid\text{not spam})=0.05\), compute \(P(\text{spam}\mid\text{"free"})\).
