Machine Learning chevron_right Model Evaluation - Metrics for Classification

Model Evaluation - Metrics for Classification

calendar_today schedule 13 min read

1. Welcome and Icebreaker:

  • Fun Question: If a weather forecast model predicts “rain” 100 times, and it only rains 10 of those times, but it never misses a rainy day, is it a good model? Why or why not?
  • Encourage students to share any experiences they’ve had with classification tasks or metrics they might have heard of.

2. Outline the Session:

  • Agenda:
    1. Welcome & Icebreaker (5 min)
    2. Recap: Why We Need Evaluation Metrics & Data Splitting (5 min)
    3. The Confusion Matrix: The Foundation of Classification Metrics (20 min)
      • True Positives (TP), True Negatives (TN)
      • False Positives (FP) (Type I Error), False Negatives (FN) (Type II Error)
      • Real-world examples for each quadrant.
    4. Common Classification Metrics (30 min):
      • Accuracy: When is it useful? When is it misleading (imbalanced datasets)?
      • Precision: What does it tell us? (Minimizing FP)
      • Recall (Sensitivity, True Positive Rate): What does it tell us? (Minimizing FN)
      • F1-Score: The harmonic mean of Precision and Recall.
      • Specificity (True Negative Rate)
    5. Choosing the Right Metric (15 min):
      • Context matters: Medical diagnosis vs. Spam detection.
      • Understanding business needs.
    6. ROC Curve and AUC (Area Under the ROC Curve) (20 min):
      • Understanding True Positive Rate vs. False Positive Rate.
      • Interpreting the AUC value.
    7. Live Coding: Calculating Metrics with Scikit-learn (15 min)
    8. Task: Metric Selection for Scenarios (15 min)
    9. Q&A and Wrap-up (5 min)
  • Learning Outcomes: By the end of this session, you will be able to:
    • Construct and interpret a confusion matrix.
    • Define and calculate accuracy, precision, recall, F1-score, and specificity.
    • Explain when each metric is most appropriate and the implications of imbalanced datasets.
    • Understand and interpret ROC curves and AUC values.
    • Implement metric calculations using Python’s scikit-learn library.
    • Select appropriate evaluation metrics based on the problem context.
  • Structure: We’ll start with the confusion matrix, then derive various metrics from it, discuss how to choose metrics, and finally look at ROC/AUC. Coding examples will solidify understanding.
  • Interactivity: This session involves many definitions. Please stop me to clarify! We’ll discuss scenarios where metric choices have real consequences.

3. Concept Explanation:

Recap: Why We Need Evaluation Metrics & Data Splitting

In our previous session, we discussed the critical importance of splitting our data into training, validation, and test sets. This ensures we can build a model that generalizes well to new, unseen data, rather than just memorizing the training examples. Once we have a trained model and predictions on our validation or test set, we need a way to quantify its performance. This is where evaluation metrics come in. They provide objective measures of how well our model is doing its job.

For classification tasks (where the goal is to assign a category or class label to an input, e.g., spam/not-spam, cat/dog, fraudulent/legitimate), simply saying a model is “good” or “bad” isn’t enough. We need specific metrics to understand its strengths and weaknesses.

The Confusion Matrix: The Foundation of Classification Metrics

The confusion matrix is a table that visualizes the performance of a classification algorithm. It compares the actual target values with those predicted by the machine learning model. For a binary classification problem (two classes, e.g., Positive and Negative), the matrix looks like this:

Predicted: POSITIVE Predicted: NEGATIVE
Actual: POSITIVE True Positive (TP) False Negative (FN)
Actual: NEGATIVE False Positive (FP) True Negative (TN)

Let’s break down each component:

  • True Positive (TP):

    • Definition: The model correctly predicted the positive class.
    • Example: Actual is “Spam”, Predicted is “Spam”. Actual is “Disease Present”, Predicted is “Disease Present”.

  • True Negative (TN):

    • Definition: The model correctly predicted the negative class.
    • Example: Actual is “Not Spam”, Predicted is “Not Spam”. Actual is “No Disease”, Predicted is “No Disease”.

  • False Positive (FP) - Type I Error:

    • Definition: The model incorrectly predicted the positive class (it was actually negative).
    • Example: Actual is “Not Spam”, Predicted is “Spam” (a legitimate email incorrectly flagged as spam). Actual is “No Disease”, Predicted is “Disease Present” (a healthy patient told they have a disease).
    • Mnemonic: “False Alarm”

  • False Negative (FN) - Type II Error:

    • Definition: The model incorrectly predicted the negative class (it was actually positive).
    • Example: Actual is “Spam”, Predicted is “Not Spam” (a spam email missed and delivered to inbox). Actual is “Disease Present”, Predicted is “No Disease” (a sick patient told they are healthy).
    • Mnemonic: “Missed Detection”

Understanding these four components is crucial because all common classification metrics are derived from them.

Common Classification Metrics

1. Accuracy

  • Definition: The proportion of total predictions that were correct.
  • Formula: (TP + TN) / (TP + TN + FP + FN)
  • When useful: When the classes are balanced (i.e., roughly equal number of samples for each class) and all errors (FP and FN) are equally undesirable.
  • When misleading: Highly misleading for imbalanced datasets. Imagine a dataset with 99% non-fraudulent transactions and 1% fraudulent. A model that predicts “non-fraudulent” for every transaction will have 99% accuracy! However, it completely fails at its primary task of identifying fraud. In this case, accuracy gives a false sense of good performance.

2. Precision (Positive Predictive Value)

  • Definition: Of all the instances the model predicted as POSITIVE, what proportion was actually POSITIVE?
  • Formula: TP / (TP + FP)
  • Focus: Minimizing False Positives.
  • Interpretation: High precision means that when the model predicts the positive class, it is very likely to be correct.
  • Example Scenarios where high precision is crucial:
    • Spam Detection: You want to avoid legitimate emails (Negative) being classified as Spam (Positive). A False Positive here is annoying for the user.
    • Search Engine Results: When a search engine returns a result (Positive), you want it to be highly relevant. Irrelevant results (False Positives) are frustrating.

3. Recall (Sensitivity, True Positive Rate - TPR)

  • Definition: Of all the actual POSITIVE instances, what proportion did the model correctly identify?
  • Formula: TP / (TP + FN)
  • Focus: Minimizing False Negatives.
  • Interpretation: High recall means that the model is good at finding most of the positive instances.
  • Example Scenarios where high recall is crucial:
    • Medical Diagnosis (e.g., cancer detection): You want to identify all patients who actually have the disease (Positive). A False Negative (missing a disease) can have severe consequences.
    • Fraud Detection: You want to catch as many fraudulent transactions (Positive) as possible. Missing a fraudulent transaction (False Negative) can be costly.

4. F1-Score

  • Definition: The harmonic mean of Precision and Recall. It tries to find a balance between the two.
  • Formula: 2 * (Precision * Recall) / (Precision + Recall)
  • Interpretation: A high F1-score indicates that the model has both good precision and good recall. It’s particularly useful when you need a single metric to compare models and the class distribution is imbalanced.
  • Why harmonic mean? The harmonic mean punishes extreme values more than the arithmetic mean. If either precision or recall is very low, the F1-score will also be low. This makes it a better measure than accuracy for imbalanced classes.

5. Specificity (True Negative Rate - TNR)

  • Definition: Of all the actual NEGATIVE instances, what proportion did the model correctly identify?
  • Formula: TN / (TN + FP)
  • Focus: Minimizing False Positives (from the perspective of actual negatives).
  • Interpretation: High specificity means the model is good at correctly identifying negative instances.
  • Relationship to FP Rate: Specificity = 1 - False Positive Rate (FPR), where FPR = FP / (TN + FP).
  • Example Scenarios: Often important in medical testing alongside sensitivity (recall). High specificity means few actual negatives are incorrectly labeled positive. It does not directly tell us how likely a negative result is to be correct; that is the negative predictive value, TN / (TN + FN), which also depends on disease prevalence.

Choosing the Right Metric

There’s no single “best” metric for all classification problems. The choice depends heavily on:

  1. The Problem Context and Business Goal:

    • What are the consequences of a False Positive vs. a False Negative?
    • Spam detection: FP (legit email as spam) is often worse than FN (spam in inbox). Prioritize Precision.
    • Cancer screening: FN (missing a cancer case) is far worse than FP (healthy person gets more tests). Prioritize Recall.
    • Loan default prediction: FN (approving a loan that defaults) can be very costly. High Recall for “default” class is important, but also consider Precision for “not default”.

  2. Class Imbalance:

    • If classes are imbalanced, accuracy is misleading. Focus on Precision, Recall, F1-score, or AUC.

  3. Stakeholder Requirements: What does the end-user or business care most about?

Often, there’s a trade-off between Precision and Recall. Improving one might decrease the other. For example, if you make your spam filter very aggressive (to catch all spam - high recall), you might incorrectly flag more legitimate emails as spam (lowering precision). Many models have a threshold that can be adjusted to favor precision or recall.

ROC Curve and AUC (Area Under the ROC Curve)

Many classification models output a probability or a score indicating the likelihood of an instance belonging to the positive class. To make a discrete class prediction (e.g., Spam/Not Spam), we apply a threshold to this probability. For example, if probability > 0.5, predict Spam.

Changing this threshold affects the number of TPs, FPs, FNs, and TNs, and thus changes Precision and Recall.

  • ROC (Receiver Operating Characteristic) Curve:

    • A plot that visualizes the diagnostic ability of a binary classifier system as its discrimination threshold is varied.
    • It plots the True Positive Rate (TPR) (Recall/Sensitivity) against the False Positive Rate (FPR) at various threshold settings.
      • TPR = TP / (TP + FN)
      • FPR = FP / (FP + TN) = 1 - Specificity
    • Interpretation:
      • A diagonal line from (0,0) to (1,1) represents a random classifier (no better than guessing).
      • The ideal point is (0,1): 0% FPR and 100% TPR (perfect classification).
      • A good classifier’s ROC curve bows towards the top-left corner.

  • AUC (Area Under the ROC Curve):

    • Definition: The area under the ROC curve. It provides a single scalar value summarizing the performance of the classifier across all possible thresholds.
    • Interpretation:
      • AUC = 1: Perfect classifier.
      • AUC = 0.5: Random classifier (no discriminative ability).
      • AUC < 0.5: On the evaluated data, the score ranks the chosen positive class below the negative class more often; check whether the score or positive-label mapping is inverted.
      • There is no universal cutoff for a “good” AUC. Interpret it against the task, baseline, class balance, and costs of errors.
    • Advantages:
      • Threshold-independent: It measures the model’s ability to rank positive instances higher than negative ones, regardless of the chosen threshold.
      • Useful for imbalanced datasets because it considers both TPR and FPR.

4. Live Coding Demonstration: Calculating Metrics with Scikit-learn

import numpy as np
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, roc_curve, classification_report
import matplotlib.pyplot as plt

# Sample actual values and predicted values
# Let's assume a binary classification problem (0 or 1)
y_true = np.array([0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0])
y_pred = np.array([0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0]) # Some correct, some incorrect

# 1. Confusion Matrix
# For binary classification, tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
# For multi-class, it returns the full matrix.
cm = confusion_matrix(y_true, y_pred)
print("Confusion Matrix:")
print(cm)
# [[TN, FP]
#  [FN, TP]]
# From our data: y_true has 10 zeros and 10 ones.
# TN: Actual 0, Predicted 0 -> Count where y_true[i]==0 and y_pred[i]==0
# FP: Actual 0, Predicted 1 -> Count where y_true[i]==0 and y_pred[i]==1
# FN: Actual 1, Predicted 0 -> Count where y_true[i]==1 and y_pred[i]==0
# TP: Actual 1, Predicted 1 -> Count where y_true[i]==1 and y_pred[i]==1

# To get individual TP, TN, FP, FN for binary case (assuming 1 is the positive class)
# This is useful if you want to manually calculate metrics or understand the components
if len(np.unique(y_true)) == 2: # Binary case
    tn, fp, fn, tp = cm.ravel()
    print(f"\nTrue Negatives (TN): {tn}")
    print(f"False Positives (FP): {fp}")
    print(f"False Negatives (FN): {fn}")
    print(f"True Positives (TP): {tp}")
else: # Multiclass case
    print("\nMetrics for multi-class are typically averaged or class-specific.")

# 2. Accuracy
accuracy = accuracy_score(y_true, y_pred)
print(f"\nAccuracy: {accuracy:.4f}")

# 3. Precision
# pos_label=1 means we are calculating precision for class 1 (the positive class)
precision = precision_score(y_true, y_pred, pos_label=1)
print(f"Precision (for class 1): {precision:.4f}")

# 4. Recall (Sensitivity)
recall = recall_score(y_true, y_pred, pos_label=1)
print(f"Recall (Sensitivity, for class 1): {recall:.4f}")

# 5. F1-Score
f1 = f1_score(y_true, y_pred, pos_label=1)
print(f"F1-Score (for class 1): {f1:.4f}")

# Specificity = TN / (TN + FP)
if len(np.unique(y_true)) == 2:
    specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    print(f"Specificity (for class 0): {specificity:.4f}")

# 6. Classification Report (provides a nice summary)
print("\nClassification Report:")
print(classification_report(y_true, y_pred, target_names=['Class 0', 'Class 1']))

# 7. ROC Curve and AUC
# For ROC AUC, we need probability scores, not just hard predictions.
# Let's assume our model gives these probabilities for the positive class (class 1)
# In a real scenario, these come from model.predict_proba(X_test)[:, 1]
y_pred_proba = np.array([0.1, 0.9, 0.6, 0.8, 0.2, 0.3, 0.1, 0.7, 0.95, 0.4, 
                           0.05, 0.85, 0.15, 0.92, 0.75, 0.25, 0.65, 0.35, 0.45, 0.12])

roc_auc = roc_auc_score(y_true, y_pred_proba)
print(f"\nROC AUC Score: {roc_auc:.4f}")

# Calculate ROC curve points
fpr, tpr, thresholds = roc_curve(y_true, y_pred_proba, pos_label=1)

# Plot ROC Curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (area = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', label='Random Guess')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate (FPR)')
plt.ylabel('True Positive Rate (TPR) / Recall')
plt.title('Receiver Operating Characteristic (ROC) Curve')
plt.legend(loc="lower right")
# plt.show() # Uncomment to display plot if running locally
print("ROC Curve plot generated (display depends on environment).")

Explanation of Code:

  1. We import necessary functions from sklearn.metrics.
  2. We define sample y_true (actual labels) and y_pred (model’s predictions).
  3. Confusion Matrix: confusion_matrix(y_true, y_pred) computes it. For binary cases, .ravel() flattens it to tn, fp, fn, tp.
  4. Accuracy, Precision, Recall, F1-Score: Calculated using accuracy_score, precision_score, recall_score, f1_score. pos_label=1 specifies that we are calculating these metrics for class 1 as the positive class.
  5. Specificity: Calculated manually from TN and FP for demonstration.
  6. Classification Report: classification_report provides a text summary of precision, recall, and F1-score for each class, along with support (number of instances in each class) and overall accuracy.
  7. ROC Curve and AUC:
    • roc_auc_score(y_true, y_pred_proba) calculates AUC. Note that it requires probability scores (y_pred_proba) for the positive class, not just the final hard predictions.
    • roc_curve(y_true, y_pred_proba) returns the FPR, TPR, and thresholds used to compute them.
    • We then use matplotlib to plot the ROC curve.

5. Tasks:

Metric Selection for Scenarios: For each scenario from the previous lesson’s task (or new ones below), identify: a. Which class would you consider “Positive”? b. Which metric(s) would be most important to optimize or monitor closely? Why? c. What are the potential consequences of high False Positives in this context? d. What are the potential consequences of high False Negatives in this context?

Scenarios:

  1. Email Spam Detection: Classifying emails as “Spam” or “Not Spam”.
  2. Credit Card Fraud Detection: Identifying transactions as “Fraudulent” or “Legitimate”.
  3. Medical Screening for a Rare but Serious Disease: Testing patients for the disease. “Disease Present” vs. “Disease Absent”.
  4. Product Recommendation System: Recommending products a user might like. “User will like” vs. “User will not like”.
  5. Quality Control in Manufacturing: Identifying defective products. “Defective” vs. “Not Defective”.

6. Q&A and Wrap-up:

  • Open floor for questions about specific metrics, their calculation, or interpretation.
  • Discuss the precision-recall trade-off and how business needs influence which to prioritize.
  • Reiterate that no single metric is universally best; context is key.
  • Preview: Next, we’ll discuss common issues like overfitting and underfitting, and how our understanding of data splitting and evaluation metrics helps us identify and address them.