Machine learning rarely deals in certainties — a classifier doesn't just say "spam," it estimates how likely an email is spam. Probability is the language for reasoning about that uncertainty precisely.
Core Definitions
| Term | Meaning | ML Example |
|---|---|---|
| Sample space | The set of all possible outcomes | {spam, not spam} |
| Event | A subset of outcomes you care about | "email is spam" |
| Random variable | A variable whose value is an outcome of a random process | The predicted class for a given email |
| P(A) | Probability of event A, a number between 0 and 1 | P(spam) = 0.3 |
Formula — Basic Probability
For independent events (one doesn't affect the other's probability):
Numerical Example
Out of 200 historical emails, 60 were spam. \(P(\text{spam}) = \frac{60}{200} = 0.3\). If word choice and sender domain were independent spam signals with \(P(\text{suspicious word})=0.4\) and \(P(\text{spam})=0.3\), then \(P(\text{both}) = 0.4 \times 0.3 = 0.12\) — if they're truly independent, which in real email data they usually aren't (this is exactly the simplifying assumption Naive Bayes makes).
total_emails = 200
spam_emails = 60
p_spam = spam_emails / total_emails
print(p_spam) # 0.3Why Probability Matters Throughout ML
- Classification outputs:
model.predict_proba()in scikit-learn returns a probability per class, not just a hard label - Naive Bayes: built entirely on conditional probability and Bayes' theorem
- Uncertainty and thresholds: deciding "predict positive if probability > 0.5" (or a different threshold) is a probability-based business decision — see Logistic Regression
- Statistical evaluation: deciding whether one model is genuinely better than another, not just luckier on one test set
Common Mistakes
- Treating a model's predicted probability as a guaranteed frequency — a well-calibrated model's "70% confident" predictions should be correct about 70% of the time on average, but any single prediction can still be wrong.
- Assuming independence when events aren't actually independent — this is a simplifying assumption, not a law, and gets it wrong when features are correlated.
Interview Relevance
Q: "What's the difference between a model's predicted class and its predicted probability?" The predicted class is the probability thresholded into a hard decision (e.g. probability > 0.5 → class 1); the probability itself carries more information — two predictions both classified "positive" can have very different confidence (0.51 vs 0.99).
Practice Question
A dataset has 1000 transactions, 40 of which are fraudulent. What is \(P(\text{fraud})\)? If a second, independent signal has a 10% chance of triggering regardless of fraud status, what's the probability both the fraud event and the signal occur together?
