Hypothesis Testing

By the end of this lesson, you will be able to determine if an observed difference in data is statistically significant or likely due to random chance using hypothesis testing.

What it is

Hypothesis testing is a statistical method used to make decisions about population parameters based on sample data. The core mental model involves two competing claims: the Null Hypothesis ($H_0$), which assumes no effect or difference exists (the status quo), and the Alternative Hypothesis ($H_1$), which suggests there is a real effect. We calculate a test statistic from our data and compare it against a theoretical distribution to find a p-value. If the p-value is lower than a predetermined threshold (usually 0.05), we reject the null hypothesis. Related terms include significance level ($\alpha$), test statistic, and confidence interval.

Why it matters

• Avoiding False Positives: It prevents you from claiming a marketing campaign worked when the increase was just random noise.
• Standardized Decision Making: It provides a rigorous, mathematical framework for making binary decisions (accept/reject) rather than relying on gut feeling.
• Resource Allocation: Helps businesses decide whether to roll out new features by quantifying the certainty of improvement.
• Scientific Rigor: Essential for validating findings in research, ensuring results are reproducible and not artifacts of small sample sizes.

Syntax or steps

The general workflow for a one-sample t-test (comparing a mean to a known value) is: 1. Define $H_0$ and $H_1$. 2. Choose a significance level $\alpha$ (e.g., 0.05). 3. Calculate the test statistic (t-score) using sample mean, population mean, standard deviation, and sample size. 4. Determine the p-value associated with that statistic. 5. Compare p-value to $\alpha$: if $p < \alpha$, reject $H_0$.

Example

Here is a Python example using `scipy` to test if the average delivery time has changed from the historical 30 minutes.

import numpy as np
from scipy import stats

# Historical mean delivery time
historical_mean = 30

# Sample data: 50 recent deliveries
np.random.seed(42) # For reproducibility
sample_data = np.random.normal(loc=28.5, scale=5, size=50)

# Perform one-sample t-test
# H0: Mean == 30
# H1: Mean != 30
t_stat, p_value = stats.ttest_1samp(sample_data, historical_mean)

print(f"T-statistic: {t_stat:.4f}")
print(f"P-value: {p_value:.4f}")

if p_value < 0.05:
    print("Reject Null Hypothesis: The mean delivery time is significantly different.")
else:
    print("Fail to Reject Null Hypothesis: No significant difference detected.")

Part-by-part explanation:

1. We generate synthetic data centered around 28.5 minutes to simulate a slight improvement.
2. `stats.ttest_1samp` calculates how far the sample mean deviates from 30 relative to the variance.
3. The output shows a very low p-value (likely < 0.05), indicating the difference between 28.5 and 30 is unlikely to be random.

Common mistakes

• P-hacking: Running multiple tests until you find a "significant" result. Fix: Pre-register your hypotheses and adjust alpha levels (Bonferroni correction).
• Confusing Statistical Significance with Practical Importance: A tiny difference can be statistically significant with huge samples but irrelevant in business. Fix: Always report effect size alongside p-values.
• Ignoring Assumptions: Using parametric tests (like t-tests) on non-normal data. Fix: Check normality plots or use non-parametric alternatives like Mann-Whitney U.
• Interpreting "Fail to Reject" as "Proof of Null": Not finding evidence of an effect doesn't prove there is no effect; it just means the data didn't show it. Fix: Use confidence intervals to see the range of plausible values.

When to use it

Compare hypothesis testing with simple descriptive analysis.

Method	Best Used When	Limitation
Hypothesis Testing	You need to generalize from a sample to a population or compare groups rigorously.	Requires assumptions about data distribution; sensitive to sample size.
Descriptive Stats	You only care about summarizing the specific data you have (no inference needed).	Cannot tell you if patterns are real or random noise.

Practice

Guided Exercise: Modify the code above to change the `loc` parameter in `np.random.normal` to 30.0. Run the test again. Observe how the p-value changes.
Challenge: Write a script that generates two independent samples of size 100 each, both with a mean of 50 and standard deviation of 10. Perform a two-sample t-test (`stats.ttest_ind`). What do you expect the p-value to be?
Hint: Since both samples come from the same distribution, you should fail to reject the null hypothesis most of the time.

Quick check

Question: If your p-value is 0.06 and your significance level is 0.05, what is your conclusion?
Answer: You fail to reject the null hypothesis. The result is not statistically significant at the 5% level.

Summary

Hypothesis testing transforms raw data into actionable insights by quantifying uncertainty. It allows analysts to distinguish between signal and noise, ensuring that business decisions are backed by statistical evidence rather than anecdotal observation.