Model Evaluation - Regression Metrics

calendar_today May 9, 2025 schedule 12 min read

1. Welcome and Icebreaker:

Fun Question: If your model is predicting house prices, would you rather it be off by $10,000 on every house, or perfect on most but off by $100,000 on a few? Why?
Share: Think about a time you received a prediction (weather forecast, ETA). What made you consider it a “good” or “bad” prediction?

2. Outline the Session:

Agenda:
1. Welcome & Icebreaker (5 min)
2. Why Evaluate Models? The Importance of Metrics (15 min)
3. Common Regression Metrics:
  - Mean Absolute Error (MAE) (15 min)
  - Mean Squared Error (MSE) (15 min)
  - Root Mean Squared Error (RMSE) (10 min)
  - R-squared (Coefficient of Determination) (20 min)
4. Choosing the Right Metric (10 min)
5. Live Coding: Calculating Regression Metrics in Python (15 min)
6. Task: Interpreting Metrics (15 min)
7. Q&A and Wrap-up (5 min)
Learning Outcomes: By the end of this session, you will be able to:
- Explain why model evaluation is a critical step in the ML pipeline.
- Define and calculate MAE, MSE, and RMSE.
- Interpret R-squared and understand its strengths and limitations.
- Discuss factors to consider when choosing a regression metric.
- Implement these metrics using Python libraries.
Structure: We’ll discuss the necessity of evaluation, then delve into each key regression metric with definitions, formulas, interpretations, and finally, a coding example.
Interactivity: Understanding these metrics is key to knowing if your model is any good! Please ask questions and share your thoughts on how these metrics relate to real-world impact.

3. Concept Explanation:

Why Evaluate Models? The Importance of Metrics

Once we’ve trained a machine learning model, how do we know if it’s actually good? How does it compare to other models? Will it perform well on new, unseen data? This is where model evaluation comes in. We use specific metrics to quantify the performance of our model.

Importance of Evaluation:

Quantify Performance: Metrics provide an objective measure of how well the model is doing. Instead of saying “it seems okay,” we can say “it has an RMSE of X.”
Compare Models: When trying different algorithms, hyperparameters, or feature sets, metrics allow us to compare their performance systematically and choose the best one.
Guide Improvement: Metrics can highlight areas where the model is struggling, guiding further development and tuning.
Prevent Overfitting/Underfitting: By evaluating on a separate test set (or using cross-validation), we can assess if the model generalizes well to unseen data, or if it has simply memorized the training data (overfitting) or failed to capture the underlying patterns (underfitting).
Business Decisions: In real-world applications, model performance metrics are often tied to business outcomes. For example, the accuracy of a fraud detection model directly impacts financial losses.

For regression problems, where the goal is to predict a continuous numerical value (e.g., house price, temperature, stock value), we use specific metrics designed for this type of task.

Common Regression Metrics

Let:

n be the number of data points.
y_i be the actual (true) value for the i-th data point.
ŷ_i (y-hat_i) be the predicted value for the i-th data point.
ȳ (y-bar) be the mean of the actual values: ȳ = (1/n) * Σ y_i.

1. Mean Absolute Error (MAE)

MAE measures the average absolute difference between the predicted values and the actual values.

Formula: MAE = (1/n) * Σ |y_i - ŷ_i|

Interpretation:

MAE gives the average magnitude of the errors in your predictions, in the original units of the target variable.
For example, if you are predicting house prices in dollars, an MAE of 15000 means your predictions are, on average, $15,000 away from the true prices.
It’s less sensitive to outliers than MSE because it doesn’t square the errors.

Pros:

Easy to understand and interpret (same units as the target).
Robust to outliers.

Cons:

The absolute value function is not differentiable at zero, which can be an issue for some optimization algorithms if MAE is used directly as a loss function (though subgradients can be used).

2. Mean Squared Error (MSE)

MSE measures the average of the squared differences between the predicted values and the actual values.

Formula: MSE = (1/n) * Σ (y_i - ŷ_i)^2

Interpretation:

MSE penalizes larger errors more heavily than smaller errors because the errors are squared. This makes it sensitive to outliers.
The units of MSE are the square of the original units of the target variable (e.g., dollars squared for house prices), which can make it harder to interpret directly.
It’s widely used as a cost function in training regression models (like linear regression) because it’s differentiable and has a nice mathematical form.

Pros:

Mathematically convenient (differentiable).
Penalizes large errors significantly.

Cons:

Not in the original units of the target, making interpretation less direct.
Very sensitive to outliers (a single large error can dominate the MSE).

3. Root Mean Squared Error (RMSE)

RMSE is simply the square root of the MSE. It brings the metric back into the original units of the target variable.

Formula: RMSE = sqrt(MSE) = sqrt((1/n) * Σ (y_i - ŷ_i)^2)

Interpretation:

RMSE is the square root of the mean squared prediction error. It is not generally the residual standard deviation; the two are equal only under additional conditions, such as a zero residual mean when using the population denominator.
Like MAE, it’s in the original units of the target variable, making it easier to interpret than MSE.
For example, an RMSE of 20000 for house prices means the typical deviation of your predictions from the actual prices is $20,000.
It still penalizes larger errors more than MAE due to the squaring step before the square root.

Pros:

In the original units of the target, aiding interpretation.
Penalizes large errors (though less extremely than MSE due to the square root).
Commonly reported and understood.

Cons:

More sensitive to outliers than MAE (though less so than MSE).

MAE vs. RMSE:

If you want to treat all errors equally on average, MAE might be preferred.
If you want to penalize large errors more significantly (i.e., outliers are particularly bad for your application), RMSE (or MSE) is more appropriate.
RMSE will always be greater than or equal to MAE. The greater the difference between them, the greater the variance in the individual errors in the sample. If all errors are the same, RMSE = MAE.

4. R-squared (R²) or Coefficient of Determination

R-squared measures the proportion of the variance in the dependent variable (target) that is predictable from the independent variables (features).

Formula: R² = 1 - (SS_res / SS_tot) Where:

Sum of Squares of Residuals (SS_res): SS_res = Σ (y_i - ŷ_i)^2 (This is n * MSE)
Total Sum of Squares (SS_tot): SS_tot = Σ (y_i - ȳ)^2 (This is proportional to the variance of the actual values y)

SS_tot represents the variance you’d have if you just predicted the mean ȳ for every data point (a baseline model). SS_res represents the variance left unexplained by your model.

Interpretation:

R² ranges from -∞ to 1 (though typically 0 to 1 for reasonable models).
R² = 1: The model perfectly predicts the target variable. All variance in y is explained by the features.
R² = 0: The model performs no better than a baseline model that always predicts the mean of y. The features explain none of the variance in y.
R² < 0: The model performs worse than the baseline mean model. This can happen if the model is very poor.
An R² of 0.75 means that 75% of the variance in the target variable can be explained by the model’s inputs.

Pros:

Provides a relative measure of goodness-of-fit (how much variance is explained).
Unitless, making it comparable across different problems (with caution).

Cons:

Increases with more predictors: R² will always increase or stay the same if you add more features to the model, even if those features are irrelevant. This can be misleading. Adjusted R-squared is a modification that penalizes the addition of useless predictors.
Doesn’t indicate if the model is biased or if the coefficient estimates are reliable.
A high R² doesn’t necessarily mean a good model for prediction if the data is overfit or if the underlying assumptions of the model are violated.
Not ideal for comparing models with different dependent variables or transformations of the dependent variable.

Adjusted R-squared: Adjusted R² = 1 - [(1 - R²) * (n - 1) / (n - p - 1)] Where p is the number of predictors (features). Adjusted R² only increases if the new predictor improves the model more than would be expected by chance.

Choosing the Right Metric

The best metric depends on the specific problem, business goals, and the nature of the data (especially outliers).

Interpretability: MAE and RMSE are often preferred for their direct interpretability in the target variable’s units.
Sensitivity to Outliers:
- If large errors are particularly undesirable and should be heavily penalized, MSE or RMSE are good choices.
- If outliers are common and you want a metric that is less affected by them, MAE is better.
Business Context: What kind of error is more costly? A few large errors or many small errors?
- Example: Predicting demand. Overestimating by a lot (large positive error) might lead to wasted inventory (costly). Underestimating by a lot (large negative error) might lead to lost sales (also costly). The relative costs of these errors might guide metric choice.
Relative vs. Absolute Performance: R² gives a sense of how much better your model is than a naive mean model, while MAE/RMSE give absolute error magnitudes.
Common Practice: RMSE and R² are very commonly reported for regression tasks.

It’s often a good idea to look at multiple metrics to get a comprehensive understanding of your model’s performance.

4. Live Coding Demonstration: Calculating Regression Metrics in Python

We’ll use scikit-learn to easily calculate these metrics.

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Sample actual and predicted values
# Imagine these are house prices (e.g., in $1000s)
y_true = np.array([250, 300, 150, 450, 500, 220, 380])
y_pred = np.array([260, 310, 170, 430, 480, 250, 370])

# --- Calculate MAE ---
mae = mean_absolute_error(y_true, y_pred)
print(f"Actual values:    {y_true}")
print(f"Predicted values: {y_pred}")
print(f"Mean Absolute Error (MAE): {mae:.2f}")
# Interpretation: On average, our predictions are off by about ${mae*1000:.2f}.

# Manual MAE calculation for understanding
manual_mae = np.mean(np.abs(y_true - y_pred))
print(f"Manual MAE: {manual_mae:.2f}")

print("-" * 40)

# --- Calculate MSE ---
mse = mean_squared_error(y_true, y_pred)
print(f"Mean Squared Error (MSE): {mse:.2f}")
# Interpretation: The units are (thousands of dollars)^2. Harder to directly interpret.

# Manual MSE calculation
manual_mse = np.mean((y_true - y_pred)**2)
print(f"Manual MSE: {manual_mse:.2f}")

print("-" * 40)

# --- Calculate RMSE ---
rmse = np.sqrt(mse) # Or use mean_squared_error with squared=False
# rmse_sklearn = mean_squared_error(y_true, y_pred, squared=False)
print(f"Root Mean Squared Error (RMSE): {rmse:.2f}")
# Interpretation: The typical deviation of our predictions from the actual values is about ${rmse*1000:.2f}.

# Manual RMSE calculation
manual_rmse = np.sqrt(np.mean((y_true - y_pred)**2))
print(f"Manual RMSE: {manual_rmse:.2f}")

print("-" * 40)

# --- Calculate R-squared (R²) ---
r2 = r2_score(y_true, y_pred)
print(f"R-squared (R²): {r2:.4f}")
# Interpretation: Approximately {r2*100:.2f}% of the variance in the actual house prices
# can be explained by our model's predictions.

# Manual R² calculation
ss_res = np.sum((y_true - y_pred)**2)
ss_tot = np.sum((y_true - np.mean(y_true))**2)
manual_r2 = 1 - (ss_res / ss_tot)
print(f"Manual R²: {manual_r2:.4f}")

print("-" * 40)

# --- Example with an outlier ---
y_true_outlier = np.array([250, 300, 150, 450, 500, 220, 380, 1000]) # Added a high true value
y_pred_outlier = np.array([260, 310, 170, 430, 480, 250, 370, 400])  # Model predicts much lower for it

mae_outlier = mean_absolute_error(y_true_outlier, y_pred_outlier)
mse_outlier = mean_squared_error(y_true_outlier, y_pred_outlier)
rmse_outlier = np.sqrt(mse_outlier)
r2_outlier = r2_score(y_true_outlier, y_pred_outlier)

print("With an outlier:")
print(f"Actual values:    {y_true_outlier}")
print(f"Predicted values: {y_pred_outlier}")
print(f"MAE (outlier):    {mae_outlier:.2f}")
print(f"MSE (outlier):    {mse_outlier:.2f}")
print(f"RMSE (outlier):   {rmse_outlier:.2f}")
print(f"R² (outlier):     {r2_outlier:.4f}")

# Compare impact of outlier
print(f"\nImpact of outlier:")
print(f"MAE changed from {mae:.2f} to {mae_outlier:.2f} (factor: {mae_outlier/mae:.2f}x)")
print(f"MSE changed from {mse:.2f} to {mse_outlier:.2f} (factor: {mse_outlier/mse:.2f}x)") # MSE increases much more
print(f"RMSE changed from {rmse:.2f} to {rmse_outlier:.2f} (factor: {rmse_outlier/rmse:.2f}x)")

Explanation of Code:

We import the necessary functions from sklearn.metrics.
We define sample y_true (actual values) and y_pred (model predictions).
For each metric (MAE, MSE, RMSE, R²):
- We use the corresponding sklearn function.
- We also show a manual calculation using NumPy to reinforce understanding of the formula.
- We provide a brief interpretation.
Outlier Example: We add a data point where the true value is high but the model predicts much lower. We then recalculate the metrics to observe:
- MAE increases, but proportionally less than MSE/RMSE.
- MSE and RMSE increase significantly more due to the squared error term, highlighting their sensitivity to outliers.
- R² typically drops when the model fails to capture such variations.

5. Tasks:

Interpreting Metrics: Suppose you have trained two different regression models (Model A and Model B) to predict the price of a used car (in USD). You evaluate them on a test set and get the following metrics:

Model A:

MAE: $2,500
RMSE: $3,500
R-squared: 0.85

Model B:

MAE: $2,800
RMSE: $4,800
R-squared: 0.82

Answer the following questions:

Average Error: Which model has a lower average error in its predictions, in terms of dollar amount? Which metric tells you this?
Large Errors: Which model seems to be making larger individual errors more often, or is more heavily penalized for them? Which metrics suggest this, and why?
Variance Explained: Which model explains a higher proportion of the variance in car prices? Which metric tells you this?
Consistency of Errors: For Model A, the RMSE ($3,500) is higher than the MAE ($2,500). What does this difference suggest about the distribution of errors for Model A?
Overall Choice: If your primary goal is to have predictions that are on average as close as possible to the true price, and you are less concerned about occasional large errors, which model might you lean towards? Why?
If your business is very sensitive to large prediction errors (e.g., a very wrong price quote could lead to significant loss or customer dissatisfaction), which model might be more concerning, despite its R-squared? Why?

Bonus Question (Conceptual): If you built a model that always predicted the average car price from the training set for every car in the test set, what would you expect its R-squared value to be (approximately)?

6. Q&A and Wrap-up:

Open floor for questions about MAE, MSE, RMSE, R², or choosing between them.
Emphasize that no single metric is universally best; the choice is context-dependent.
It’s good practice to report multiple metrics and understand their implications.
Preview: Next, we’ll move on to evaluating classification models, which have a different set of common metrics.