Machine Learning chevron_right Supervised Learning - Classification (Logistic Regression)

Supervised Learning - Classification (Logistic Regression)

calendar_today schedule 11 min read

1. Welcome and Icebreaker:

  • Fun Question: If you had a magic button that could perfectly label emails as “spam” or “not spam,” what’s the first thing you’d do with your newfound free time?
  • Share: Can anyone think of a yes/no question they answer many times a day that they wish a computer could handle for them?

2. Outline the Session:

  • Agenda:
    1. Welcome & Icebreaker (5 min)
    2. Recap: Supervised Learning & Regression (5 min)
    3. Introduction to Classification Problems (10 min)
    4. The Logistic Regression Model (25 min)
      • Why not Linear Regression for Classification?
      • The Sigmoid (Logistic) Function
      • Interpreting the Output as Probability
    5. Cost Function for Logistic Regression (Log Loss / Binary Cross-Entropy) (25 min)
      • Why MSE is not suitable
      • Intuition behind Log Loss
    6. Decision Boundary (15 min)
    7. Live Coding: Visualizing Sigmoid & Basic Logistic Regression Idea (Python) (15 min)
    8. Task: Conceptual Questions (15 min)
    9. Q&A and Wrap-up (5 min)
  • Learning Outcomes: By the end of this session, you will be able to:
    • Define a classification problem and differentiate it from regression.
    • Explain why linear regression is unsuitable for classification tasks.
    • Describe the role and properties of the sigmoid function in logistic regression.
    • Interpret the output of a logistic regression model as a probability.
    • Understand the intuition behind the Log Loss (Binary Cross-Entropy) cost function.
    • Explain the concept of a decision boundary.
  • Structure: We’ll transition from regression to classification, introduce the logistic model and its components, discuss how to measure its error, and visualize its core mathematical function.
  • Interactivity: Classification is a cornerstone of ML! Please ask questions as they arise. We’ll discuss why certain mathematical choices are made.

3. Concept Explanation:

Recap: Supervised Learning & Regression

In supervised learning, we have input features (X) and corresponding output labels (y). Our goal is to learn a mapping function f such that f(X) ≈ y. In regression, the output label y is a continuous numerical value (e.g., predicting house prices, temperature). We saw that Linear Regression models this relationship as a straight line (or hyperplane in higher dimensions) y = wX + b, and we typically use Mean Squared Error (MSE) as a cost function to find the optimal weights w and bias b using Gradient Descent.

Introduction to Classification Problems

Classification is another type of supervised learning problem. However, unlike regression where we predict a continuous value, in classification, the output label y is a discrete category or class.

Examples of Classification Problems:

  • Binary Classification (2 classes):
    • Email spam detection (spam or not spam)
    • Medical diagnosis (disease present or absent)
    • Image classification (cat or dog)
    • Credit card fraud detection (fraudulent or legitimate transaction)
  • Multi-class Classification (>2 classes):
    • Handwritten digit recognition (0, 1, 2, …, 9)
    • Object recognition in images (car, pedestrian, bicycle, tree)
    • Sentiment analysis (positive, negative, neutral review)

Today, we will focus on binary classification and introduce Logistic Regression, a fundamental algorithm for this task.

The Logistic Regression Model

Why not Linear Regression for Classification?

Let’s say we want to classify tumors as malignant (1) or benign (0). If we try to fit a linear regression line to this data, we might encounter several problems:

  1. Output beyond [0, 1]: A linear model h_θ(x) = θ^T X can produce outputs much greater than 1 or much less than 0. This doesn’t make sense for probabilities, which should be between 0 and 1.
  2. Poor Fit: The straight line might not capture the decision point well. Forcing a line through 0/1 data can lead to a suboptimal threshold.
  3. Sensitivity to Outliers: Adding an outlier far to the right might significantly shift the regression line and change the (arbitrary) threshold we might pick (e.g., if h_θ(x) > 0.5, predict 1).

We need a model that outputs a probability between 0 and 1.

The Sigmoid (Logistic) Function

Logistic Regression addresses this by using a different hypothesis function. It still calculates a linear combination of inputs and weights z = θ^T X (similar to linear regression), but then it passes this result z through a special function called the sigmoid function (or logistic function), denoted as σ(z) or g(z).

Sigmoid Function Formula: σ(z) = 1 / (1 + e^(-z))

Properties of the Sigmoid Function:

  • Output Range: It squashes any input z (from -∞ to +∞) into an output range between 0 and 1.
    • As z → ∞, e^(-z) → 0, so σ(z) → 1 / (1 + 0) = 1.
    • As z → -∞, e^(-z) → ∞, so σ(z) → 1 / (1 + ∞) → 0.
    • If z = 0, e^(0) = 1, so σ(z) = 1 / (1 + 1) = 0.5.
  • S-shaped Curve: Its graph is an S-shaped curve.
  • Differentiable: It’s easily differentiable, which is important for gradient descent. σ'(z) = σ(z) * (1 - σ(z))

So, the hypothesis for Logistic Regression is: h_θ(x) = σ(θ^T X) = 1 / (1 + e^(-θ^T X))

Interpreting the Output as Probability

The output h_θ(x) from the logistic regression model can be interpreted as the estimated probability that the output y is 1, given the input x and parameters θ. h_θ(x) = P(y=1 | x; θ)

A threshold of 0.5 is a common default for converting this probability to a class label, not a universal rule. Choose a threshold using validation data and the costs of false positives and false negatives; do not tune it on the final test set.

For example, if h_θ(x) = 0.8 for a patient’s tumor data, it means the model estimates an 80% probability that the tumor is malignant. Since it’s a binary classification: P(y=0 | x; θ) = 1 - P(y=1 | x; θ) = 1 - h_θ(x)

Cost Function for Logistic Regression (Log Loss / Binary Cross-Entropy)

Now that we have a hypothesis function that outputs probabilities, we need a cost function to measure how good our predictions are. We can’t use Mean Squared Error (MSE) effectively here.

Why MSE is not suitable for Logistic Regression:

If we use MSE with the sigmoid function h_θ(x) = σ(θ^T X), the resulting cost function J(θ) = (1/m) * Σ (σ(θ^T X_i) - y_i)^2 becomes non-convex. This means it can have many local minima, making it difficult for Gradient Descent to find the global minimum.

We need a cost function that is convex for logistic regression and that penalizes the model appropriately when it makes confident wrong predictions.

Intuition behind Log Loss (Binary Cross-Entropy)

The cost function used for Logistic Regression is called Log Loss or Binary Cross-Entropy.

For a single training example (x, y): Cost(h_θ(x), y) = -y * log(h_θ(x)) - (1-y) * log(1 - h_θ(x))

Let’s break this down:

  • Case 1: If y = 1 (true label is 1)

    • Cost = -1 * log(h_θ(x)) - (1-1) * log(1 - h_θ(x)) = -log(h_θ(x))
    • If h_θ(x) → 1 (model predicts 1 with high probability, correct): log(h_θ(x)) → log(1) = 0. So, Cost → 0.
    • If h_θ(x) → 0 (model predicts 0 with high probability, incorrect): log(h_θ(x)) → log(0) = -∞. So, Cost → ∞ (huge penalty).

  • Case 2: If y = 0 (true label is 0)

    • Cost = -0 * log(h_θ(x)) - (1-0) * log(1 - h_θ(x)) = -log(1 - h_θ(x))
    • If h_θ(x) → 0 (model predicts 0 with high probability, correct): 1 - h_θ(x) → 1, log(1 - h_θ(x)) → log(1) = 0. So, Cost → 0.
    • If h_θ(x) → 1 (model predicts 1 with high probability, incorrect): 1 - h_θ(x) → 0, log(1 - h_θ(x)) → log(0) = -∞. So, Cost → ∞ (huge penalty).

This cost function has the desired properties:

  • It heavily penalizes confident wrong predictions.
  • It gives low cost for confident correct predictions.
  • It is convex, making it suitable for Gradient Descent.

The overall cost function J(θ) for m training examples is the average of these individual costs: J(θ) = -(1/m) * Σ [ y_i * log(h_θ(x_i)) + (1-y_i) * log(1 - h_θ(x_i)) ]

Our goal is to find θ that minimizes this J(θ). We can do this using Gradient Descent. The derivative of this J(θ) with respect to θ_j surprisingly looks similar to the one for linear regression (after some calculus): ∂J/∂θ_j = (1/m) * Σ (h_θ(x_i) - y_i) * x_{i,j} So the Gradient Descent update rule remains: θ_j := θ_j - α * ∂J/∂θ_j

Decision Boundary

Once we have trained our logistic regression model and found the optimal parameters θ, how do we make a prediction (0 or 1)? We use the predicted probability h_θ(x):

  • If h_θ(x) ≥ 0.5, predict y = 1.
  • If h_θ(x) < 0.5, predict y = 0.

When does h_θ(x) ≥ 0.5? This happens when σ(θ^T X) ≥ 0.5. Looking at the sigmoid function, σ(z) ≥ 0.5 when z ≥ 0. So, we predict y = 1 if θ^T X ≥ 0. And we predict y = 0 if θ^T X < 0.

The equation θ^T X = 0 defines the decision boundary. This is a line (or hyperplane in higher dimensions) that separates the region where we predict y=1 from the region where we predict y=0.

Example: If θ = [θ_0, θ_1, θ_2] and our features are x_1, x_2 (with x_0=1 for the bias term), then θ_0 + θ_1*x_1 + θ_2*x_2 = 0 is the equation of the line that serves as the decision boundary.

  • Points on one side of the line will result in θ^T X > 0 (predict 1).
  • Points on the other side will result in θ^T X < 0 (predict 0).

Unlike linear regression where the line is the prediction, in logistic regression, the line is the boundary between predictions. The decision boundary can be linear or non-linear depending on the features used. If we use polynomial features (e.g., x_1^2, x_1*x_2, x_2^2), we can get non-linear decision boundaries.

4. Live Coding Demonstration: Visualizing Sigmoid & Basic Logistic Regression Idea

import numpy as np
import matplotlib.pyplot as plt

# 1. Sigmoid Function Visualization
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z_values = np.linspace(-10, 10, 200) # Generate 200 points from -10 to 10
sigmoid_values = sigmoid(z_values)

plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.plot(z_values, sigmoid_values, label='σ(z) = 1 / (1 + e^(-z))')
plt.title('Sigmoid (Logistic) Function')
plt.xlabel('z')
plt.ylabel('σ(z)')
plt.grid(True)
plt.axhline(0.5, color='red', linestyle='--', label='σ(z) = 0.5')
plt.axvline(0, color='gray', linestyle='--')
plt.legend()

# 2. Log Loss Visualization (Conceptual for a single prediction)
# Cost when true label y = 1
# Cost = -log(h_theta_x)
h_theta_x_y1 = np.linspace(0.001, 0.999, 100) # Probabilities from close to 0 to close to 1
cost_y1 = -np.log(h_theta_x_y1)

# Cost when true label y = 0
# Cost = -log(1 - h_theta_x)
h_theta_x_y0 = np.linspace(0.001, 0.999, 100)
cost_y0 = -np.log(1 - h_theta_x_y0)

plt.subplot(1, 2, 2)
plt.plot(h_theta_x_y1, cost_y1, label='Cost if y=1 (-log(h))')
plt.plot(h_theta_x_y0, cost_y0, label='Cost if y=0 (-log(1-h))', linestyle=':')
plt.title('Log Loss Cost Function Components')
plt.xlabel('Predicted Probability h_θ(x)')
plt.ylabel('Cost')
plt.grid(True)
plt.legend()

plt.tight_layout()
plt.show()

# 3. Simple Decision Boundary Example (Conceptual)
# Suppose after training, theta = [-3, 1, 1] (theta0, theta1, theta2)
# So, decision boundary is -3 + 1*x1 + 1*x2 = 0  => x2 = -x1 + 3

# Generate some dummy data points for visualization
np.random.seed(0)
X1_class0 = np.random.normal(1, 1, 50)
X2_class0 = np.random.normal(1, 1, 50)
X1_class1 = np.random.normal(4, 1, 50)
X2_class1 = np.random.normal(4, 1, 50)

plt.figure(figsize=(7, 6))
plt.scatter(X1_class0, X2_class0, color='blue', label='Class 0 (y=0)', alpha=0.7)
plt.scatter(X1_class1, X2_class1, color='red', label='Class 1 (y=1)', alpha=0.7)

# Plot the decision boundary: x2 = -x1 + 3
x1_boundary = np.array([min(X1_class0.min(), X1_class1.min()), max(X1_class0.max(), X1_class1.max())])
x2_boundary = -x1_boundary + 3
plt.plot(x1_boundary, x2_boundary, color='green', linestyle='--', label='Decision Boundary (-3 + x1 + x2 = 0)')

plt.title('Conceptual Decision Boundary')
plt.xlabel('Feature x1')
plt.ylabel('Feature x2')
plt.legend()
plt.grid(True)
plt.axhline(0, color='black', linewidth=0.5)
plt.axvline(0, color='black', linewidth=0.5)
plt.show()

Explanation of Code:

  1. Sigmoid Function Plot: We define the sigmoid function and plot it to show its S-shape and how it maps any input z to the range (0, 1). The z=0 point corresponding to σ(z)=0.5 is highlighted.
  2. Log Loss Plot: We plot the two components of the log loss function: -log(h) (when y=1) and -log(1-h) (when y=0). This visually demonstrates:
    • If y=1, cost approaches 0 as predicted probability h approaches 1, and cost approaches ∞ as h approaches 0.
    • If y=0, cost approaches 0 as h approaches 0, and cost approaches ∞ as h approaches 1.
  3. Decision Boundary Example: We assume some theta values after a hypothetical training. The equation θ^T X = 0 gives us a line. We plot this line along with some dummy data points to illustrate how it separates the two classes.

5. Tasks:

Conceptual Questions:

  1. Why is linear regression generally not suitable for binary classification tasks where the output is 0 or 1? List at least two reasons.
  2. What is the range of the sigmoid function σ(z)? What is the value of σ(0)?
  3. If a logistic regression model outputs h_θ(x) = 0.3 for a particular input x, how would you interpret this value in the context of a spam email classification problem (where y=1 means spam)? What would be the predicted class (spam or not spam) if the decision threshold is 0.5?
  4. Consider the Log Loss function component for when the true label y=1, which is Cost = -log(h_θ(x)). Explain why this cost is high if h_θ(x) is close to 0, and low if h_θ(x) is close to 1.
  5. What does the decision boundary represent in a logistic regression model? If you have two features x1 and x2, what shape will a simple logistic regression decision boundary typically have?

6. Q&A and Wrap-up:

  • Open floor for questions about classification, logistic regression, sigmoid, log loss, or decision boundaries.
  • Emphasize that logistic regression is a foundational algorithm for binary classification, and its concepts (like log loss) extend to more complex models.
  • Preview: Next, we’ll discuss how to properly evaluate our models, including splitting data for training and testing.