Machine Learning chevron_right Model Evaluation - Training, Validation, Test Splits

Model Evaluation - Training, Validation, Test Splits

calendar_today schedule 12 min read

1. Welcome and Icebreaker:

  • Fun Question: If you were baking a cake (a model) for a competition (real-world performance), would you taste it yourself while baking (training), ask a friend to taste a small piece before the competition (validation), or only let the judges taste it at the competition (testing)? Why?
  • Encourage students to share their experiences with splitting data or why they think it might be important.

2. Outline the Session:

  • Agenda:
    1. Welcome & Icebreaker (5 min)
    2. The Need for Data Splitting: Why Not Use All Data for Training? (15 min)
    3. Understanding Training, Validation, and Test Sets (20 min)
      • Purpose of each set
      • Typical proportions
      • Analogy: Student learning for an exam
    4. Common Splitting Strategies (20 min)
      • Random Split
      • Stratified Split (especially for classification)
      • Time Series Split (considerations for temporal data)
    5. The Dangers of Data Leakage (15 min)
    6. Live Coding: Splitting Data with Scikit-learn (15 min)
    7. Task: Design a Splitting Strategy (15 min)
    8. Q&A and Wrap-up (5 min)
  • Learning Outcomes: By the end of this session, you will be able to:
    • Explain the rationale behind splitting data into training, validation, and test sets.
    • Describe the role and purpose of each dataset.
    • Implement data splitting using Python’s scikit-learn library.
    • Understand and avoid common pitfalls like data leakage.
    • Choose appropriate splitting strategies for different scenarios.
  • Structure: We’ll start with why we split data, define each set, discuss how to split, and then practice with code.
  • Interactivity: This is a foundational concept! Please ask questions as we go. Share any scenarios where you think a particular split might be tricky.

3. Concept Explanation:

The Need for Data Splitting: Why Not Use All Data for Training?

Imagine you’re a student preparing for a final exam. You have a textbook full of practice problems. If you study by memorizing the answers to only those specific problems, you might ace a test that uses those exact same problems. But what happens when the final exam has new problems you’ve never seen before? You’d likely perform poorly.

Machine learning models are similar. If we train a model on our entire dataset, it might learn the data—including its noise and specific quirks—so well that it performs brilliantly on that same data. This is called overfitting. However, when this overfit model encounters new, unseen data, its performance can be drastically worse. It hasn’t learned the underlying general patterns; it has just memorized the training examples.

The primary goal of splitting data is to simulate how our model will perform on new, unseen data. This allows us to:

  1. Train the Model: Develop the model’s parameters by learning from a portion of the data (training set).
  2. Tune the Model: Adjust hyperparameters (settings of the learning algorithm, not learned from data directly) and select the best model architecture by evaluating on a separate portion of data (validation set). This helps prevent overfitting to the training set.
  3. Evaluate the Model: Get an unbiased estimate of the model’s performance on truly unseen data (test set) before deploying it.

Using all data for training gives us no reliable way to assess this generalization ability.

Understanding Training, Validation, and Test Sets

1. Training Set

  • Purpose: This is the largest portion of the data and is used to actually train the model. The model “learns” the relationships, patterns, and features from this data by adjusting its internal parameters.
  • Analogy: The textbook and practice problems you use to study and learn the material.
  • Typical Proportion: Usually 60-80% of the total dataset.

2. Validation Set (or Development Set / Dev Set)

  • Purpose: This dataset is used to tune the model’s hyperparameters and make decisions about the model’s architecture (e.g., number of layers in a neural network, type of algorithm). The model sees this data during the development phase, but it doesn’t directly learn its parameters from it. Instead, we train multiple model versions (e.g., with different hyperparameter settings) on the training set and then evaluate them on the validation set to see which performs best. This helps in selecting the best model configuration while avoiding overfitting to the training set.
  • Analogy: Mock exams or quizzes you take during your study period. They help you see what you’ve learned well and where you need to adjust your study strategy (tune your “learning hyperparameters”). You might retake mock exams with different approaches.
  • Typical Proportion: Usually 10-20% of the total dataset.
  • Important Note: If you repeatedly use the validation set to tune your model, you might inadvertently start overfitting to the validation set as well. The model’s configuration becomes too specific to the quirks of the validation data.

3. Test Set

  • Purpose: This dataset is held back until the very end of the model development process. Once you have selected your final model (after training on the training set and tuning on the validation set), you evaluate its performance on the test set. This provides an unbiased estimate of how the model is likely to perform on new, unseen real-world data.
  • Analogy: The final, official exam. You only take it once, and its result is your “final grade” or performance measure. You don’t get to change your study methods based on the final exam results for that exam.
  • Typical Proportion: Usually 10-20% of the total dataset.
  • Crucial Rule: The test set should only be used once, at the very end. If you use the test set to make decisions about your model (e.g., “Oh, performance is bad, let me go back and change this hyperparameter and re-test”), it ceases to be a true test set and becomes another validation set. Your performance estimate will be overly optimistic.

Why not just Training and Test? If you only have a training and a test set, you might train your model, evaluate on the test set, and if performance is poor, you’d be tempted to tweak hyperparameters and re-evaluate on the test set. Doing this repeatedly means your model selection process is now influenced by the test set, and your final test score is no longer an unbiased estimate of generalization. The validation set provides a “safe” place to do this tuning.

Common Splitting Strategies

1. Random Split

  • How it works: Data points are randomly assigned to the training, validation, or test sets according to specified proportions.
  • When to use: Suitable for large datasets where the underlying distribution of classes or values is fairly uniform and you don’t expect any specific ordering or grouping to be critical.
  • Scikit-learn: train_test_split function.

2. Stratified Split

  • How it works: This method ensures that each set (train, validation, test) has approximately the same percentage of samples of each target class (for classification problems) or similar distributions of the target variable (for regression, though less common and harder to implement perfectly).
  • When to use: Crucial for classification tasks, especially with imbalanced datasets (where some classes are much rarer than others). A simple random split might, by chance, leave very few (or even zero) instances of a rare class in the validation or test set, making evaluation unreliable.
  • Scikit-learn: train_test_split has a stratify parameter.

3. Time Series Split (or Temporal Split)

  • How it works: For time-ordered data (e.g., stock prices, weather data), random splitting is usually inappropriate because it can lead to data leakage from the future into the past. You don’t want your model to be trained on data from Tuesday and validated on data from Monday if you’re trying to predict Wednesday based on past data.
    • Common approach: Use earlier data for training, the next chunk for validation, and the most recent data for testing. For example, train on 2020-2021 data, validate on 2022 data, and test on 2023 data.
    • Rolling forecast origin / Walk-forward validation: A more sophisticated approach where the model is trained on a window of past data, predicts the next point(s), then the actual value for that point is observed and added to the training set for the next prediction.
  • When to use: Essential for any data where the order of observations matters and future values depend on past values.
  • Scikit-learn: TimeSeriesSplit for cross-validation scenarios. For a simple train-validation-test split, you’d typically slice the data manually based on dates or time indices.

The Dangers of Data Leakage

Data leakage occurs when information from outside the training dataset is used to create the model. This can lead to overly optimistic performance estimates because the model inadvertently “knows” something about the validation or test data that it wouldn’t know in a real-world scenario.

Common Sources of Data Leakage:

  1. Using validation/test data during training: Directly training on these sets.
  2. Using validation/test data for pre-processing: For example, calculating the mean and standard deviation for normalization using the entire dataset (including validation/test) and then applying it to all sets. Pre-processing steps (like scaling, imputation) should be “fit” only on the training data and then “transformed” on the validation and test data.
  3. Feature engineering using future information: Creating features that wouldn’t be available at the time of prediction. For example, in predicting stock prices, using the day’s closing price to predict the closing price.
  4. Non-randomness in data collection that correlates with the target: If data collection for a certain class happened at a different time or with different equipment, and this influences features.
  5. Duplicates: Having identical or near-identical samples in both training and test sets.

Consequences:

  • Model performs exceptionally well during development but fails miserably in production.
  • Misleading conclusions about the model’s capabilities.

Prevention:

  • Strict separation of training, validation, and test sets.
  • Fit all pre-processing steps (scalers, imputers) only on the training data.
  • Carefully consider the temporal aspect of data if applicable.
  • Be vigilant during feature engineering.

4. Live Coding Demonstration: Splitting Data with Scikit-learn

import numpy as np
from sklearn.model_selection import train_test_split

# --- Basic Random Split ---
# Imagine X is our feature matrix (e.g., 100 samples, 5 features)
X = np.random.rand(100, 5)
# Imagine y is our target variable (e.g., 100 labels)
y = np.random.randint(0, 2, 100) # Binary classification target

print("Original data shapes:")
print(f"X: {X.shape}, y: {y.shape}")

# First, split into training and a temporary set (which will be split into validation and test)
# Let's aim for 70% train, 15% validation, 15% test
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.3, random_state=42 # 30% for temp (val + test)
)

# Now, split the temporary set into validation and test sets
# We want validation and test to be equal, so 0.5 of the temp set (0.5 * 0.3 = 0.15 of original)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42 # 50% of temp for test
)

print("
After splitting:")
print(f"X_train: {X_train.shape}, y_train: {y_train.shape} (~{len(X_train)/len(X)*100:.0f}%)")
print(f"X_val:   {X_val.shape}, y_val:   {y_val.shape} (~{len(X_val)/len(X)*100:.0f}%)")
print(f"X_test:  {X_test.shape}, y_test:  {y_test.shape} (~{len(X_test)/len(X)*100:.0f}%)")

# --- Stratified Split (for classification) ---
# Let's create an imbalanced dataset for demonstration
y_imbalanced = np.array([0]*80 + [1]*20) # 80% class 0, 20% class 1
X_imbalanced = np.random.rand(100, 5)

print(f"
Imbalanced y: {np.bincount(y_imbalanced)} (Class 0, Class 1)")

# Stratified split
X_train_strat, X_temp_strat, y_train_strat, y_temp_strat = train_test_split(
    X_imbalanced, y_imbalanced, test_size=0.3, random_state=42, stratify=y_imbalanced
)
X_val_strat, X_test_strat, y_val_strat, y_test_strat = train_test_split(
    X_temp_strat, y_temp_strat, test_size=0.5, random_state=42, stratify=y_temp_strat
)

print("
After STRATIFIED splitting (proportions in y):")
print(f"y_train_strat: {np.bincount(y_train_strat)} (ratio: {np.bincount(y_train_strat)[1]/len(y_train_strat):.2f})")
print(f"y_val_strat:   {np.bincount(y_val_strat)} (ratio: {np.bincount(y_val_strat)[1]/len(y_val_strat):.2f})")
print(f"y_test_strat:  {np.bincount(y_test_strat)} (ratio: {np.bincount(y_test_strat)[1]/len(y_test_strat):.2f})")

# Compare with a non-stratified split on imbalanced data
X_train_nostrat, X_temp_nostrat, y_train_nostrat, y_temp_nostrat = train_test_split(
    X_imbalanced, y_imbalanced, test_size=0.3, random_state=42 # NO stratify
)
X_val_nostrat, X_test_nostrat, y_val_nostrat, y_test_nostrat = train_test_split(
    X_temp_nostrat, y_temp_nostrat, test_size=0.5, random_state=42 # NO stratify
)
print("
After NON-STRATIFIED splitting (proportions in y):")
print(f"y_train_nostrat: {np.bincount(y_train_nostrat)} (ratio: {np.bincount(y_train_nostrat)[1]/len(y_train_nostrat):.2f})")
# Note: y_val_nostrat or y_test_nostrat might have very different ratios, or even miss a class if small enough
# For this specific random_state and size, it might still look okay, but the risk is there.
# For example, if y_val_nostrat is small, it could easily miss the minority class.
if len(np.bincount(y_val_nostrat)) > 1:
    print(f"y_val_nostrat:   {np.bincount(y_val_nostrat)} (ratio: {np.bincount(y_val_nostrat)[1]/len(y_val_nostrat):.2f})")
else:
    print(f"y_val_nostrat:   {np.bincount(y_val_nostrat)} (Minority class might be missing or ratio skewed!)")

if len(np.bincount(y_test_nostrat)) > 1:
    print(f"y_test_nostrat:  {np.bincount(y_test_nostrat)} (ratio: {np.bincount(y_test_nostrat)[1]/len(y_test_nostrat):.2f})")
else:
    print(f"y_test_nostrat:  {np.bincount(y_test_nostrat)} (Minority class might be missing or ratio skewed!)")

# --- Time Series Data (Conceptual - Manual Split) ---
# For time series, you usually sort by time and then slice.
# Example: 100 days of data
time_series_data = np.arange(100)
n_total = len(time_series_data)

train_size = int(n_total * 0.7)
val_size = int(n_total * 0.15)
# test_size is the remainder

ts_train = time_series_data[:train_size]
ts_val = time_series_data[train_size : train_size + val_size]
ts_test = time_series_data[train_size + val_size:]

print("
Time Series Split (manual):")
print(f"ts_train: {ts_train[0]}...{ts_train[-1]} (length {len(ts_train)})")
print(f"ts_val:   {ts_val[0]}...{ts_val[-1]} (length {len(ts_val)})")
print(f"ts_test:  {ts_test[0]}...{ts_test[-1]} (length {len(ts_test)})")

Explanation of Code:

  1. We import train_test_split from sklearn.model_selection.
  2. Basic Random Split:
    • We create dummy feature data X and target y.
    • We perform a two-step split: first into train and temp, then temp into validation and test. This is a common way to get three sets.
    • test_size specifies the proportion of data to allocate to the set being created in that split.
    • random_state ensures reproducibility. If you use the same random_state, you’ll get the same split every time.
  3. Stratified Split:
    • We create an y_imbalanced target to show the utility.
    • The key is the stratify=y_imbalanced (or stratify=y_temp_strat) argument. Scikit-learn uses the provided array to ensure class proportions are maintained.
    • We compare the class counts/ratios in the resulting splits with and without stratification to highlight the difference.
  4. Time Series Split (Manual):
    • We demonstrate the concept by manually slicing a sorted array. In practice, if you have a Pandas DataFrame with a datetime index, you’d filter by dates.
    • sklearn.model_selection.TimeSeriesSplit is more for cross-validation scenarios but follows the same principle of not shuffling and using earlier data for training.

5. Tasks:

Design a Splitting Strategy: For each of the following scenarios, describe: a. How would you split the data into training, validation, and test sets? b. What are the desired proportions (approximate)? c. Which splitting strategy would you use (random, stratified, time-series)? Why? d. What potential pitfalls or special considerations should you keep in mind?

Scenarios:

  1. Predicting Customer Churn: You have a dataset of 100,000 customers with features like demographics, usage patterns, and a binary label indicating if they churned (left the service) or not. About 5% of customers churned.
  2. Forecasting Daily Store Sales: You have 3 years of daily sales data for a retail store, along with features like promotions, holidays, and day of the week. You want to predict sales for the next month.
  3. Image Classification of Cat Breeds: You have a dataset of 50,000 images, labeled with one of 20 different cat breeds. The number of images per breed varies, but none are extremely rare (e.g., minimum 1000 images per breed).
  4. Identifying Fraudulent Transactions: You have a large dataset of financial transactions (1 million records). Only 0.1% of transactions are fraudulent. The data is collected over a year.

6. Q&A and Wrap-up:

  • Open floor for questions about splitting techniques, random_state, or specific scenarios.
  • Reiterate the importance of the test set being “sacred” and used only once for final unbiased evaluation.
  • Emphasize that the choice of splitting strategy is crucial for obtaining reliable model evaluations.
  • Preview: Now that we know how to split our data, we can look at specific metrics for classification tasks.