Bias-Variance, Overfitting & Underfitting

Understand how model complexity affects error by balancing bias and variance, enabling you to diagnose overfitting and underfitting effectively.

What it is

The Bias-Variance Tradeoff describes the decomposition of prediction error into three parts: bias, variance, and irreducible noise. Bias is the error introduced by approximating a real-world problem with a simplified model (underfitting). Variance is the error introduced by the model's sensitivity to small fluctuations in the training set (overfitting). As model complexity increases, bias decreases but variance increases. The goal is to find the "sweet spot" where total error is minimized.

Why it matters

• Diagnoses whether a model needs more capacity (high bias) or better regularization (high variance).
• Prevents deploying models that perform well on training data but fail in production.
• Guides feature engineering and hyperparameter tuning decisions.
• Provides a theoretical framework for understanding generalization error.

Syntax or steps

To evaluate this tradeoff, compare training error against validation/test error:

1. High Training Error + High Test Error: High Bias (Underfitting). The model is too simple.
2. Low Training Error + High Test Error: High Variance (Overfitting). The model memorizes noise.
3. Low Training Error + Low Test Error: Good Fit. The model generalizes well.

Example

This Python example uses polynomial regression to visualize how increasing degree affects bias and variance.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Generate synthetic data with noise
np.random.seed(42)
X = np.sort(5 * np.random.rand(40, 1), axis=0)
y = np.sin(X).ravel() + 0.1 * np.random.randn(40)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

degrees = [1, 4, 15] # Underfit, Good Fit, Overfit
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

for i, d in enumerate(degrees):
    ax = axes[i]
    poly = PolynomialFeatures(degree=d)
    X_poly_train = poly.fit_transform(X_train)
    X_poly_test = poly.transform(X_test)
    
    model = LinearRegression().fit(X_poly_train, y_train)
    
    # Plot predictions
    X_plot = np.linspace(0, 5, 100).reshape(-1, 1)
    X_plot_poly = poly.transform(X_plot)
    y_pred = model.predict(X_plot_poly)
    
    ax.scatter(X, y, color='black', label='Data')
    ax.plot(X_plot, y_pred, color='red', linewidth=2, label=f'Degree {d}')
    ax.set_title(f'Train Score: {model.score(X_poly_train, y_train):.2f}\nTest Score: {model.score(X_poly_test, y_test):.2f}')
    ax.legend()

plt.tight_layout()
plt.show()

Explanation: Degree 1 shows high bias (straight line misses sine curve). Degree 4 balances fit. Degree 15 shows high variance (wiggles excessively through noise points).

Common mistakes

• Ignoring Validation Sets: Using only training error hides overfitting. Always use a hold-out set or cross-validation.
• Confusing Noise with Signal: Trying to reduce variance by fitting every outlier leads to overfitting.
• Assuming Complexity Equals Accuracy: More parameters do not always mean better performance; they often degrade generalization.
• Neglecting Irreducible Error: Some error is due to inherent randomness in data and cannot be eliminated by any model.

When to use it

Practice

Guided Exercise: Modify the code above to include a degree 2 polynomial. Observe if the test score improves compared to degree 1.

Challenge: Implement L2 Regularization (Ridge Regression) on the degree 15 model. Does the test score improve? Why?

Quick check

Q: If your training error is very low but your validation error is high, what is the primary issue?

A: High Variance (Overfitting). The model has learned the noise in the training data rather than the underlying pattern.

Summary

The bias-variance tradeoff is fundamental to machine learning, illustrating that minimizing training error alone is insufficient for good generalization. By monitoring the gap between training and validation errors, practitioners can adjust model complexity to achieve optimal predictive performance.