Overfitting and Underfitting

calendar_today May 13, 2025 schedule 3 min read

1. Welcome and Icebreaker:

Icebreaker: “Can you think of a situation where a model might perform well on training data but poorly on new data?”
Encourage students to share their thoughts or experiences with learning new skills and how practice differs from real-world application.

2. Outline the Session:

Agenda:
- Understand the concepts of overfitting and underfitting.
- Learn how to detect these issues.
- Explore strategies to address them.
Learning Outcomes:
- Define overfitting and underfitting.
- Identify signs of these issues in machine learning models.
- Apply techniques to mitigate them.

3. Concept Explanation:

Overfitting:
- Occurs when a model learns the training data too well, including noise and outliers.
- Results in high accuracy on training data but poor generalization to unseen data.
- Analogy: Memorizing answers to a test without understanding the material.
Underfitting:
- Happens when a model is too simple to capture the underlying patterns in the data.
- Results in poor performance on both training and test data.
- Analogy: Trying to fit a straight line to data that clearly follows a curve.
Detecting Overfitting and Underfitting:
- Compare training and validation errors:
  - High training accuracy but low validation accuracy → Overfitting.
  - Low training and validation accuracy → Underfitting.
- Visualize learning curves.
Importance in Machine Learning:
- Balancing bias and variance is crucial for building robust models.
- Overfitting → High variance.
- Underfitting → High bias.

4. Live Coding Demonstration:

Example: Polynomial regression on a dataset.
- Generate synthetic data with a quadratic relationship.
- Fit models with different degrees of polynomial features (e.g., linear, quadratic, cubic).
- Visualize the results to show underfitting, good fit, and overfitting.

Code Snippet:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Generate data with a quadratic signal and noise
np.random.seed(0)
X = np.random.rand(100, 1) * 10
y = 2 * X**2 - 3 * X + 5 + np.random.randn(100, 1) * 5

# Use a validation split to compare fit with generalization
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.3, random_state=0
)
X_grid = np.linspace(X.min(), X.max(), 300).reshape(-1, 1)
degrees = [1, 2, 10]
plt.figure(figsize=(15, 4))

for i, degree in enumerate(degrees):
    poly = PolynomialFeatures(degree=degree, include_bias=False)
    X_train_poly = poly.fit_transform(X_train)
    X_val_poly = poly.transform(X_val)

    model = LinearRegression()
    model.fit(X_train_poly, y_train)

    train_pred = model.predict(X_train_poly)
    val_pred = model.predict(X_val_poly)
    grid_pred = model.predict(poly.transform(X_grid))

    plt.subplot(1, 3, i + 1)
    plt.scatter(X_train, y_train, color="blue", alpha=0.6, label="Train")
    plt.scatter(X_val, y_val, color="orange", alpha=0.7, label="Validation")
    plt.plot(X_grid, grid_pred, color="red", label=f"Degree {degree}")
    train_mse = mean_squared_error(y_train, train_pred)
    val_mse = mean_squared_error(y_val, val_pred)
    plt.title(f"Degree {degree}\\nTrain MSE: {train_mse:.1f} | Val MSE: {val_mse:.1f}")
    plt.legend()

plt.tight_layout()
plt.show()

Highlight:
- Degree 1 often underfits the quadratic pattern; degree 2 matches the generating relationship more closely.
- A high-degree model may have much lower training error but higher validation error, showing overfitting. The exact result depends on the split.
- Use cross-validation for model selection and reserve a separate test set for final evaluation.

5. Tasks:

Short Exercise:
- Use the provided code to experiment with different polynomial degrees.
- Observe how the model’s performance changes.
- Discuss findings with peers.