Statistical Computing

By the end of this lesson, you will be able to perform basic statistical computations on data using Python's SciPy and NumPy libraries.

What it is

Statistical computing in Python refers to the use of specialized libraries to perform mathematical analysis on datasets. The primary tools are NumPy for numerical operations and array handling, and SciPy.stats for probability distributions and hypothesis testing. Unlike general-purpose programming, statistical computing focuses on descriptive statistics (mean, median, standard deviation) and inferential statistics (t-tests, p-values). Key related terms include "vectorization" (applying functions to entire arrays at once) and "distributions" (mathematical models describing how data is spread).

Why it matters

• Efficiency: Libraries like NumPy are optimized in C, making calculations on large datasets significantly faster than pure Python loops.
• Accuracy: Established libraries implement rigorous mathematical standards, reducing errors from manual implementation.
• Reproducibility: Code-based analysis allows others to verify results exactly by running the same script.
• Integration: Statistical outputs can easily feed into visualization libraries like Matplotlib or machine learning frameworks like Scikit-learn.

Syntax or steps

The smallest useful pattern involves importing the library, creating a data structure (usually an array), and calling a statistical function. For example, to calculate the mean, you import numpy, define an array, and call np.mean(). For hypothesis testing, you import scipy.stats and call functions like ttest_ind(). Always ensure your data is numeric; strings must be converted or filtered out before computation.

Example

import numpy as np
from scipy import stats

# 1. Create sample data
data_a = np.array([10, 12, 14, 16, 18])
data_b = np.array([15, 17, 19, 21, 23])

# 2. Descriptive Statistics
mean_a = np.mean(data_a)
std_dev_a = np.std(data_a, ddof=1) # ddof=1 for sample standard deviation

print(f"Mean A: {mean_a}")
print(f"Std Dev A: {std_dev_a:.2f}")

# 3. Inferential Statistics (Independent T-Test)
# Tests if the means of two independent groups are different
t_statistic, p_value = stats.ttest_ind(data_a, data_b)

print(f"T-statistic: {t_statistic:.4f}")
print(f"P-value: {p_value:.4f}")

Part-by-part explanation:

• import numpy as np: Loads the core numerical library.
• np.array(...): Converts lists into efficient numerical arrays.
• np.mean(): Calculates the arithmetic average.
• np.std(ddof=1): Calculates standard deviation. ddof=1 corrects for sample bias (Bessel's correction).
• stats.ttest_ind(): Performs a Student's t-test assuming equal variance. It returns two values: the test statistic and the p-value.

Common mistakes

• Ignoring Data Types: Passing strings or mixed types to NumPy functions causes errors or unexpected object arrays. Always clean data first.
• Population vs. Sample Std Dev: Using np.std() without ddof=1 calculates population standard deviation. For most statistical inference on samples, you need ddof=1.
• Misinterpreting P-values: A low p-value indicates evidence against the null hypothesis, not that the effect size is large. Always check effect sizes alongside significance.
• Assuming Normality: Many tests (like t-tests) assume normal distribution. If data is heavily skewed, consider non-parametric tests like mannwhitneyu.

When to use it

Compare Python statistical computing with R, another popular language for this domain.

Feature	Python (SciPy/Statsmodels)	R
Best For	Production pipelines, ML integration, general scripting.	Exploratory data analysis, academic research, complex modeling.
Learning Curve	Easier if you already know Python.	Steeper for programmers, intuitive for statisticians.
Performance	Excellent for large-scale data via NumPy/Pandas.	Can struggle with very large datasets without optimization.

Use Python when your workflow includes web scraping, database interaction, or deploying models to production. Use R when deep statistical exploration and custom plotting are the primary goals.

Practice

Guided Exercise: Calculate the median and interquartile range (IQR) for data_a from the example above. Hint: Use np.median() and np.percentile(). Challenge: Modify the code to compare data_a against a third dataset data_c = [10, 11, 12, 13, 14]. Does the p-value change significantly? Why might comparing similar means yield a higher p-value?

Quick check

Question: What does the ddof=1 parameter do in np.std()? Answer: It sets the Delta Degrees of Freedom to 1, which changes the denominator from N to N-1, calculating the sample standard deviation instead of the population standard deviation.

Summary

Statistical computing in Python leverages NumPy for fast numerical operations and SciPy for rigorous hypothesis testing. Mastering these basics allows you to move beyond simple averages to understanding whether differences in data are statistically significant, forming the foundation for reliable data-driven decisions.