Understanding Hyperparameters & Common Issues

Learning Objectives: By the end of this lesson, you will be able to:

• Identify key hyperparameters in neural network training
• Understand the impact of hyperparameter choices on performance
• Diagnose common issues like vanishing/exploding gradients and overfitting

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

• What hyperparameter surprised you most when you first tuned a model?

Lesson Overview

• Duration: 45 minutes
• Prerequisites: Basic model training and backpropagation understanding
• Materials Needed: Jupyter or code editor, training scripts

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

1. Learning rate and its effects
2. Batch size and epochs
3. Network depth/width
4. Optimizer choices
5. Common training issues and diagnostics

Success Criteria:

• Ability to tune and diagnose training runs for better performance

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Learning Rate

• Definition: Step size for weight updates
• Real-World Analogy: Like the speed at which you drive to a destination—too fast, you may crash; too slow, you take forever.

Batch Size

• Definition: Number of samples per gradient update
• Why This Matters: Small batches produce noisier gradient estimates and more frequent updates. That noise can help generalization in some settings, but the result depends on the model, data, optimizer, and batch size.

Key Terminology

• Epoch, Batch, Iteration
• Vanishing/Exploding Gradients
• Underfitting vs. Overfitting

4. Detailed Explanation and Examples (25-30 minutes)

Impact of Hyperparameter Choices

Learning Rate Examples:

# Too large
optimizer = Adam(lr=1.0)
# Loss may diverge
# Too small
optimizer = Adam(lr=1e-6)
# Training may be very slow

Batch Size Examples:

for batch in DataLoader(train_dataset, batch_size=1):
    ... # high variance updates
for batch in DataLoader(train_dataset, batch_size=1024):
    ... # lower-noise updates; memory use and throughput depend on hardware

Optimizer Comparison:

Common Issues:

1. Vanishing Gradients: Saturating activations such as sigmoid or tanh can contribute in deep networks. Inspect gradient magnitudes; consider ReLU-family activations, suitable initialization, or normalization where appropriate.
2. Exploding Gradients: Inspect gradient norms; gradient clipping or a smaller learning rate may help.
3. Overfitting: Compare training and validation curves. Consider regularization, simpler models, or early stopping.
4. Unstable training loss: First check the learning rate and data preprocessing, then change one setting at a time.

Common Misconceptions

1. Misconception: Adam always outperforms SGD. Reality: Depends on dataset and problem.
2. Misconception: More layers always means better performance. Reality: Can lead to vanishing gradients or overfitting.

5. Live Coding Demonstration (20-25 minutes)

Example: Learning Rate Scheduler

from torch.optim.lr_scheduler import StepLR

epochs = 30
scheduler = StepLR(optimizer, step_size=10, gamma=0.1)
for epoch in range(epochs):
    train()  # Run one complete training epoch using the model and data loader.
    scheduler.step()

Explanation:

• range(epochs) runs once for each integer epoch count.
• StepLR multiplies the learning rate by gamma after every 10 completed epochs.
• In a real training script, call any validation step separately and use validation results to decide whether the schedule is helping.

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Experiment with three learning rates and plot training curves.

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Implement gradient clipping in PyTorch and observe effect.

Exercise 3: Creative Application (15 minutes)

Objective: Design a simple hyperparameter search (grid search) script.