Machine Learning chevron_right Training a Neural Network - The Full Picture

Training a Neural Network - The Full Picture

calendar_today schedule 9 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Explain the core components of the neural network training process: epochs, batch size, and learning rate.
  • Write and run a basic training loop in Python using NumPy and a deep learning framework (Keras/PyTorch).
  • Compare Stochastic, Batch, and Mini-batch Gradient Descent and choose the right approach for your datasets.
  • Diagnose common training issues and apply best practices to improve convergence.

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Ask learners: “When you train for a marathon, you don’t run the entire distance in one go. How might that relate to training a neural network?”

Lesson Overview

  • Duration: 60–75 minutes
  • Prerequisites: Understanding of feedforward neural networks, activation functions, loss functions, and backpropagation intuition.
  • Materials Needed: Python 3.x environment, Jupyter Notebook or IDE, NumPy, TensorFlow/Keras or PyTorch installed.

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Training Loop Components – understanding epochs, batches, and learning rate.
  2. Gradient Descent Variants – batch, stochastic, mini-batch.
  3. Implementing a Loop with NumPy – low-level perspective.
  4. Using a Deep Learning Framework – Keras/PyTorch example.
  5. Common Pitfalls & Best Practices – diagnosing learning issues.

Success Criteria:

By the end of this session, you should be able to implement a functional training loop and explain how hyperparameters affect convergence.

3. Foundation Knowledge (15–20 minutes)

Core Concepts

Epochs

An epoch refers to one full pass through the entire training dataset. If you have 1,000 images in your training set, one epoch implies that each image has been used once to update model parameters.

Real-World Analogy: Think of practicing a piano piece. One epoch is like playing the entire piece from start to finish once.

Why This Matters: Too few epochs can underfit the model, too many may overfit.

Batch Size

A batch is a subset of the dataset fed through the network before updating weights. Common sizes are powers of two (32, 64, 128).

Real-World Analogy: Tasting a spoonful of soup to decide if it needs more seasoning rather than cooking the entire pot.

Why This Matters: Smaller batches offer more frequent updates (noisier gradients), larger batches give stable estimates (less noise) but use more memory.

Learning Rate

The learning rate determines the step size at each iteration while moving toward a minimum of the loss function.

Real-World Analogy: Driving toward a destination—small steps (low lr) take longer but precise; large steps (high lr) might overshoot.

Why This Matters: A learning rate that’s too high can cause divergence, too low can stall training.

Key Terminology

  • Iteration: One update of model parameters (each batch processed).
  • Gradient Descent: Optimization algorithm that updates weights by following the negative gradient of the loss.
  • Stochastic Gradient Descent (SGD): Uses batch size of 1.
  • Mini-Batch Gradient Descent: Uses batch size between 1 and full dataset.
  • Optimizer: Algorithm (e.g., SGD, Adam) that dictates how weights are updated.

4. Detailed Explanation and Examples (25–30 minutes)

Understanding the Training Loop

Training a neural network comprises repeating the following steps until a stopping criterion is met (epochs or convergence):

  1. Forward Pass: Compute predictions from inputs.
  2. Loss Calculation: Measure difference between predictions and true targets.
  3. Backward Pass (Backpropagation): Compute gradients of the loss w.r.t. each weight.
  4. Parameter Update: Adjust weights using gradients and learning rate.

These steps are often wrapped in loops over epochs and batches.

Practical Applications

  • Image classification tasks (e.g., MNIST digit recognition).
  • Time-series forecasting in finance or IoT sensor data.
  • Natural language processing (sentiment analysis, text generation).

Common Misconceptions

  1. Misconception: More epochs always improve accuracy. Reality: After a point, the model may overfit and performance on unseen data degrades.
  2. Misconception: Larger batch size always speeds up training or always harms generalization. Reality: Batch size trades off gradient noise, memory, and hardware throughput. Its effect on generalization varies by model and data; compare validation performance rather than relying on a universal rule.

5. Live Coding Demonstration (20–25 minutes)

Example 1: Basic Implementation with NumPy

import numpy as np

# Simple dataset: y = 2x + 1 with noise
X = np.linspace(0, 10, 100).reshape(-1, 1)
y = 2 * X + 1 + np.random.randn(*X.shape) * 0.5

# Initialize parameters
w = np.random.randn(1, 1)
b = np.zeros((1,))

# Hyperparameters
learning_rate = 0.01
epochs = 100
batch_size = 20

# Training loop
for epoch in range(epochs):
    # Shuffle at start of epoch
    perm = np.random.permutation(X.shape[0])
    X_shuffled, y_shuffled = X[perm], y[perm]
    for i in range(0, X.shape[0], batch_size):
        X_batch = X_shuffled[i:i+batch_size]
        y_batch = y_shuffled[i:i+batch_size]

        # Forward pass
        preds = X_batch.dot(w) + b
        loss = np.mean((preds - y_batch) ** 2)

        # Backward pass
        grad_w = 2 * np.mean((preds - y_batch) * X_batch)
        grad_b = 2 * np.mean(preds - y_batch)

        # Update
        w -= learning_rate * grad_w
        b -= learning_rate * grad_b

    if epoch % 10 == 0:
        # Recompute loss after the epoch using all training examples and current parameters.
        epoch_preds = X.dot(w) + b
        epoch_loss = np.mean((epoch_preds - y) ** 2)
        print(f"Epoch {epoch}: full-training-set loss={epoch_loss:.4f}")

Explanation:

  • We shuffle data each epoch for randomness.
  • We process in batches to simulate mini-batch GD.
  • After each reported epoch, we recompute loss over the whole training set using the current parameters, so the displayed epoch loss is comparable across epochs.
  • Gradients are averaged over the batch.
  • Weight update uses learning rate.

Example 2: Using Keras (TensorFlow) for MNIST

from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.optimizers import SGD

# Load data
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Build model
def create_model():
    model = Sequential([
        Flatten(input_shape=(28, 28)),
        Dense(128, activation='relu'),
        Dense(10, activation='softmax')
    ])
    model.compile(
        loss='sparse_categorical_crossentropy',
        optimizer=SGD(learning_rate=0.01),
        metrics=['accuracy']
    )
    return model

model = create_model()
# Train: epochs=5, batch_size=64
history = model.fit(x_train, y_train, epochs=5, batch_size=64, validation_split=0.1)

Key Points:

  • Keras encapsulates the training loop.
  • .fit() handles shuffling, batching, forward/backward passes, and updates.
  • Validation split monitors performance on unseen data.

Example 3: PyTorch Training Loop

import torch
from torch import nn, optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Dataset & DataLoader
transform = transforms.ToTensor()
train_ds = datasets.MNIST(root='data', train=True, transform=transform, download=True)
train_loader = DataLoader(train_ds, batch_size=64, shuffle=True)

# Model definition
class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(28*28, 128)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(128, 10)
    def forward(self, x):
        x = self.flatten(x)
        x = self.relu(self.fc1(x))
        return self.fc2(x)

model = SimpleNN()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)

epochs = 5
for epoch in range(epochs):
    running_loss = 0.0
    for images, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    print(f'Epoch {epoch}: Loss={running_loss/len(train_loader):.4f}')

6. Hands-On Practice (30–40 minutes)

Exercise 1: Build Your Own NumPy Trainer (10 minutes)

Objective: Implement a training loop for y = 3x - 2 using NumPy.

Instructions:

  1. Generate 200 data points with Gaussian noise.
  2. Initialize weights randomly.
  3. Train for 50 epochs with batch size 10.
  4. Plot loss curve over epochs.

Starter Code:

import numpy as np

np.random.seed(0)
X = np.linspace(-2, 2, 200).reshape(-1, 1)
y = 3 * X - 1 + np.random.randn(200, 1) * 0.2
w = np.zeros((1, 1))
b = 0.0
learning_rate = 0.05
epochs = 50
batch_size = 10

for epoch in range(epochs):
    order = np.random.permutation(len(X))
    X_epoch, y_epoch = X[order], y[order]
    for start in range(0, len(X), batch_size):
        X_batch = X_epoch[start:start + batch_size]
        y_batch = y_epoch[start:start + batch_size]

        # TODO: compute predictions, gradients, and update w and b

Solution:

np.random.seed(0) X = np.linspace(-2, 2, 200).reshape(-1, 1) y = 3 * X - 1 + np.random.randn(200, 1) * 0.2 w = np.zeros((1, 1)) b = 0.0 learning_rate = 0.05 epochs = 50 batch_size = 10

for epoch in range(epochs): order = np.random.permutation(len(X)) X_epoch, y_epoch = X[order], y[order] epoch_loss_sum = 0.0 examples_seen = 0 for start in range(0, len(X), batch_size): X_batch = X_epoch[start:start + batch_size] y_batch = y_epoch[start:start + batch_size]

    predictions = X_batch @ w + b
    errors = predictions - y_batch
    loss = np.mean(errors ** 2)
    grad_w = (2 / len(X_batch)) * (X_batch.T @ errors)
    grad_b = 2 * errors.mean()

    w -= learning_rate * grad_w
    b -= learning_rate * grad_b
    epoch_loss_sum += loss * len(X_batch)
    examples_seen += len(X_batch)

if (epoch + 1) % 10 == 0:
    print(f"Epoch {epoch + 1}: sample-weighted epoch MSE={epoch_loss_sum / examples_seen:.4f}")

print(f"Learned slope={w.item():.3f}, intercept={float(b):.3f}")

</details>

### Exercise 2: Experiment with Learning Rates (15 minutes)
**Objective:** Observe the effect of different learning rates on convergence.

**Requirements:**
- Use the MNIST Keras example from above.
- Train with lr = [0.1, 0.01, 0.001].
- Plot training and validation loss for each.

**Hints:**
- Pass `learning_rate` into `SGD`.
- Use `history.history['loss']` and `history.history['val_loss']` for plotting.

### Exercise 3: Compare Batch vs. Mini-Batch vs. SGD (15 minutes)
**Objective:** Understand computational trade-offs.

**Challenge:**
- Use the PyTorch example.
- Train with batch_size = [1, 64, 1024].
- Record epoch time and final accuracy.

**Extension Ideas:**
- Implement momentum in SGD.
- Try Adam and compare.

## 7. Common Pitfalls and Troubleshooting (10 minutes)

### Frequent Mistakes
1. **Problem:** Loss not decreasing.
   **Cause:** Learning rate too high or data not normalized.
   **Solution:** Reduce lr, normalize inputs to [0, 1] or zero-mean/unit variance.

2. **Problem:** Overfitting early.
   **Cause:** Too many epochs or no regularization.
   **Solution:** Add dropout, early stopping, or reduce model capacity.

### Debugging Tips
- Monitor gradient norms to catch vanishing/exploding gradients.
- Use TensorBoard or framework visualizers.
- Print shapes of tensors to ensure compatibility.

## 8. Best Practices and Professional Tips (5 minutes)

### Industry Standards
- Use learning rate schedules (step decay, cosine annealing).
- Employ early stopping based on validation loss.
- Save checkpoints and log experiments (e.g., with MLflow).

### Next Steps in Your Learning Journey
- Explore Convolutional Neural Networks (CNNs).
- Learn Recurrent Neural Networks (RNNs) for sequence data.
- Dive into hyperparameter tuning tools like Optuna.

## 9. Summary and Review (5 minutes)

### Key Takeaways
1. The training loop is the engine of deep learning.
2. Hyperparameters (lr, batch_size, epochs) critically affect performance.
3. Frameworks automate loops but understanding low-level details helps diagnose issues.

### Quick Review Questions
1. What is the difference between an epoch and an iteration?
2. How does batch size influence gradient noise?
3. Why might you use a learning rate scheduler?

---

## For Instructors

### Teaching Tips
- Encourage students to verbalize how hyperparameters affect learning.
- Show live TensorBoard visualizations.

### Assessment Ideas
- Have learners write a short script comparing three optimizers.
- Peer-review their loss plots and discuss patterns.

### Adaptation Notes
- For absolute beginners, stick to NumPy only first.
- For advanced students, introduce custom optimizers and gradient clipping.