Machine Learning chevron_right Regularization

Regularization

calendar_today schedule 2 min read

1. Welcome and Icebreaker:

  • Icebreaker: “Have you ever tried to simplify a complex problem by ignoring less important details?”
  • Encourage students to share their thoughts on balancing complexity and simplicity in problem-solving.

2. Outline the Session:

  • Agenda:
    • Understand the concept of regularization.
    • Learn about L1 (Lasso) and L2 (Ridge) regularization.
    • Explore their effects on model weights and performance.
  • Learning Outcomes:
    • Define regularization and its purpose.
    • Differentiate between L1 and L2 regularization.
    • Apply regularization techniques to reduce overfitting.

3. Concept Explanation:

  • What is Regularization?
    • A technique to prevent overfitting by adding a penalty term to the cost function.
    • Encourages simpler models with smaller weights.
  • L1 Regularization (Lasso):
    • Adds the absolute values of weights to the cost function.
    • Encourages sparsity (some weights become exactly zero).
    • Can act as a feature-selection aid, but when predictors are correlated, L1 may select one somewhat arbitrarily or produce an unstable selected set.
  • L2 Regularization (Ridge):
    • Adds the squared values of weights to the cost function.
    • Penalizes large weights but does not enforce sparsity.
    • Encourages smoothness and stability.
  • Effect on Model Weights:
    • L1: Shrinks some weights to zero, effectively removing features.
    • L2: Shrinks all weights proportionally, reducing their magnitude.
  • Feature Scaling: Scale numeric features before applying L1 or L2 penalties. Otherwise, coefficient sizes depend on measurement units, so the penalty can treat otherwise comparable features differently. Fit the scaler on training data only.
  • Importance in Machine Learning:
    • Helps balance bias and variance.
    • Can improve generalization to unseen data, though it does not guarantee it.

4. Live Coding Demonstration:

  • Example: Linear regression with L1 and L2 regularization.
  • Code Snippet: 
    import numpy as np
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Generate synthetic data
np.random.seed(0)
X = np.random.rand(100, 10)
y = 3 * X[:, 0] - 2 * X[:, 1] + np.random.randn(100) * 0.5

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# Ridge Regression
ridge = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
ridge.fit(X_train, y_train)
y_pred_ridge = ridge.predict(X_test)

# Lasso Regression
lasso = make_pipeline(StandardScaler(), Lasso(alpha=0.1))
lasso.fit(X_train, y_train)
y_pred_lasso = lasso.predict(X_test)

print("Ridge MSE:", mean_squared_error(y_test, y_pred_ridge))
print("Lasso MSE:", mean_squared_error(y_test, y_pred_lasso))
  • Highlight:
    • Compare the performance of Ridge and Lasso.
    • Observe the effect of regularization on model weights.

5. Tasks:

  • Short Exercise:
    • Modify the regularization strength (alpha) and observe its impact on model performance.
    • Discuss the trade-offs between L1 and L2 regularization.