By the end of this lesson, you will be able to calculate and interpret Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and R-squared to evaluate regression model performance.
What it is
Error metrics quantify how far a model's predictions deviate from actual values. MAE measures the average absolute difference, providing an intuitive sense of error magnitude in the original units. MSE squares these differences before averaging, which penalizes larger errors more heavily than smaller ones. RMSE is the square root of MSE, returning the error metric to the original scale while retaining the penalty for large outliers. R-squared ($R^2$) represents the proportion of variance in the dependent variable that is predictable from the independent variables, ranging from 0 to 1 (or lower if the model performs worse than a horizontal line). Related terms include residuals (the difference between observed and predicted values) and loss functions (mathematical formulas used during training to minimize error).Why it matters
- Model Selection: Comparing RMSE or MAE across different algorithms helps identify the most accurate predictor.
- Business Impact: MAE provides interpretable numbers (e.g., "average prediction is off by $5") for stakeholders.
- Outlier Sensitivity: Choosing between MAE and MSE allows you to decide whether large errors should be punished severely (MSE) or treated equally (MAE).
- Baseline Comparison: R-squared tells you if your model explains data better than simply guessing the mean value.
Syntax or steps
To compute these metrics, you need two arrays:y_true (actual values) and y_pred (predicted values). The general workflow involves calculating the difference for each pair, applying the specific mathematical transformation (absolute value, squaring, etc.), and then averaging the results. For R-squared, you compare the sum of squared residuals against the total sum of squares relative to the mean of the actual values.
Example
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
# Sample data: Actual vs Predicted house prices (in thousands)
y_true = np.array([300, 450, 500, 600])
y_pred = np.array([310, 440, 520, 580])
# Calculate Metrics
mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_true, y_pred)
print(f"MAE: {mae:.2f}")
print(f"MSE: {mse:.2f}")
print(f"RMSE: {rmse:.2f}")
print(f"R-Squared: {r2:.4f}")y_true and y_pred as NumPy arrays.
2. mean_absolute_error computes $\frac{1}{n}\sum |y_i - \hat{y}_i|$. Here, errors are 10, 10, 20, 20. Average is 15.
3. mean_squared_error computes $\frac{1}{n}\sum (y_i - \hat{y}_i)^2$. Errors squared are 100, 100, 400, 400. Average is 250.
4. np.sqrt(mse) converts MSE back to original units (RMSE ≈ 15.81).
5. r2_score calculates $1 - \frac{\text{SS}_{res}}{\text{SS}_{tot}}$, indicating how well the model fits the trend.
Common mistakes
- Ignoring Scale: Comparing RMSE across datasets with different units is meaningless. Always normalize features or use relative metrics like MAPE if scales differ.
- Misinterpreting R-squared: A high $R^2$ does not imply causation or that the model is unbiased. It only indicates fit quality regarding variance.
- Using MSE for Interpretability: Stakeholders often find squared units confusing. Use RMSE or MAE for reporting business outcomes.
- Negative R-squared: If $R^2$ is negative, your model predicts worse than simply using the mean of the target variable. This usually indicates severe overfitting or incorrect feature engineering.
When to use it
| Metric | Best Used When... | Key Characteristic |
|---|---|---|
| MAE | You need robustness to outliers and simple interpretation. | Linear penalty for errors. |
| MSE/RMSE | Large errors are unacceptable (e.g., safety-critical systems). | Quadratic penalty; sensitive to outliers. |
| R-squared | You want to know how much variance is explained compared to a baseline. | Scale-independent; good for model comparison on same dataset. |
Practice
Guided Exercise: Modify the example above so that one prediction is significantly wrong (e.g., changey_pred[3] to 700). Observe how RMSE increases disproportionately compared to MAE.
Challenge: Write a function to calculate MAPE (Mean Absolute Percentage Error) manually using NumPy. Hint: Formula is $\frac{1}{n} \sum |\frac{y_{true} - y_{pred}}{y_{true}}| \times 100$.