By the end of this lesson, you will be able to calculate and interpret a 95% confidence interval for a population mean using Python, understanding that it represents a range of plausible values rather than a guaranteed capture.
What it is
A confidence interval (CI) is a statistical estimate derived from sample data that provides a range of values likely to contain an unknown population parameter. The most common mental model is "uncertainty quantification." If you take many random samples from a population and calculate a 95% CI for each, approximately 95% of those intervals will contain the true population mean. It is crucial to distinguish this from probability: once a specific interval is calculated from your single dataset, the true mean is either inside it or outside it; there is no probability attached to that specific event. Related terms includemargin of error, standard error, and confidence level.
Why it matters
- Honest Reporting: Point estimates (like a simple average) are almost always wrong. CIs acknowledge sampling variability.
- Decision Making: In A/B testing, if two confidence intervals overlap significantly, the difference between groups may not be statistically meaningful.
- Precision Assessment: A narrow interval indicates high precision (often due to large sample size), while a wide interval suggests low precision.
- Contextualizing Results: It helps stakeholders understand the "worst-case" and "best-case" scenarios for a metric.
Syntax or steps
To calculate a confidence interval for a mean when the population standard deviation is unknown (the typical real-world scenario), we use the t-distribution. The formula is:Mean ± (Critical Value * Standard Error)
Where Standard Error = Sample Std Dev / sqrt(Sample Size). In Python, the scipy.stats library handles the complex math of finding the critical value based on degrees of freedom.
Example
import numpy as np
from scipy import stats
# Simulate sample data: 100 users with avg session time ~30 mins
np.random.seed(42)
sample_data = np.random.normal(loc=30, scale=5, size=100)
# Calculate Mean and Standard Error
mean_val = np.mean(sample_data)
std_err = stats.sem(sample_data) # Standard Error of the Mean
# Calculate 95% Confidence Interval
# alpha = 0.05 means 95% confidence
ci_low, ci_high = stats.t.interval(0.95, len(sample_data)-1, loc=mean_val, scale=std_err)
print(f"Sample Mean: {mean_val:.2f}")
print(f"95% CI: [{ci_low:.2f}, {ci_high:.2f}]")
loc) and the standard error (scale). The function stats.t.interval uses the t-distribution because our sample size is finite and we don't know the true population variance. It returns the lower and upper bounds where we are 95% confident the true population mean lies.
Common mistakes
- Misinterpreting Probability: Saying "There is a 95% chance the true mean is in this interval." Correct phrasing: "We are 95% confident that this interval captures the true mean."
- Ignoring Assumptions: CIs assume independent observations. If your data has clustering (e.g., multiple sessions from one user treated as separate rows), the standard error is underestimated, making the CI too narrow.
- Using Z instead of T: For small samples (
n < 30) or unknown population sigma, using the normal distribution (Z-score) yields inaccurate intervals. Always prefer the t-distribution for sample means unlessnis very large. - Confusing CI with Prediction Interval: A CI estimates the mean. A prediction interval estimates where a single new observation might fall, which is much wider.
When to use it
Compare Confidence Intervals with Hypothesis Testing (P-values). Both assess significance, but they offer different insights.| Feature | Confidence Interval | Hypothesis Test (P-value) |
|---|---|---|
| Output | Range of values | Binary decision (Reject/Fail to Reject) |
| Information | Shows magnitude and direction of effect | Shows only strength of evidence against null |
| Best For | Estimating metrics (e.g., "Conversion rate is between 2-4%") | Strict pass/fail decisions (e.g., "Did the change work?") |
