Optimization - Gradient Descent for Neural Networks

1. Welcome and Icebreaker:

• Welcome to Day 23! Today we focus on optimization and how gradient descent underlies neural network training.
• Icebreaker question: If you could optimize any daily routine automatically, what would it be and why?
• Share your ideas on how you might iteratively improve that daily task.

2. Outline the Session:

• Agenda:
  1. Recap: What is a loss function and why we need to minimize it.
  2. Concept of Gradient Descent: Intuition and algorithm steps.
  3. Learning Rate and Convergence: Choosing step size and avoiding pitfalls.
  4. Variants of GD: Batch, Mini-batch, and Stochastic Gradient Descent.
  5. Live Coding: Implement gradient descent for a simple neural network in PyTorch.
  6. Task: Experiment with learning rates and observe convergence behavior.
• Learning Outcomes:
  • Understand gradient descent as the core optimizer in neural networks.
  • Distinguish batch, mini-batch, and stochastic variants.
  • Recognize the effect of learning rate and common convergence issues.
  • Implement and tweak gradient descent in code.
• Structure: Theory → Pseudocode → Code demo → Exercises.
• Interactivity: Ask questions throughout. We’ll pause for discussion after the code demo.

3. Concept Explanation:

Neural network training is an optimization problem: find parameters (\theta) (weights and biases) that minimize a loss function (L(\theta)). Gradient descent is an iterative algorithm for this purpose. It relies on computing the gradient (partial derivatives) of the loss with respect to each parameter and then updating the parameters in the direction that reduces the loss.

Basic Gradient Descent Algorithm

Given a parameter vector (\theta) and a differentiable loss (L(\theta)):

1. Initialize parameters randomly (e.g., small random values).
2. Compute the gradient (\nabla_\theta L(\theta)), a vector of partial derivatives.
3. Update parameters: [ \theta \leftarrow \theta - \eta , \nabla_\theta L(\theta) ] where (\eta) is the learning rate (step size).
4. Repeat steps 2–3 until convergence (loss stops decreasing or maximum iterations reached).

Learning Rate and Convergence

The learning rate (\eta) controls how big a step we take along the gradient. A small (\eta) leads to slow convergence, while a large (\eta) may overshoot minima and diverge.

Common pitfalls:

• Too Small: Training is very slow, might get stuck in flat regions.
• Too Large: Loss may oscillate or diverge.
• Adaptive Rates: Techniques like learning rate decay, momentum, or optimizers (Adam, RMSProp) adjust (\eta) during training.

Batch vs Stochastic vs Mini-Batch

• Batch Gradient Descent: Compute gradient using the entire training dataset each step. Accurate gradient but expensive for large datasets.
• Stochastic Gradient Descent (SGD): Compute gradient using a single random sample. Fast but noisy updates, can escape shallow local minima.
• Mini-Batch Gradient Descent: Compromise: use small batches (32, 64, 128 samples). Balances stability and speed, leverages vectorized operations.

4. Live Coding Demonstration:

Let’s implement a simple neural network and train it on synthetic data using PyTorch. We’ll compare different learning rates.

import torch
import torch.nn as nn
import torch.optim as optim

# Create synthetic dataset: y = 2x + 3 with noise
x = torch.unsqueeze(torch.linspace(-1, 1, 100), dim=1)
y = 2 * x + 3 + 0.2 * torch.rand(x.size())

dataset = torch.utils.data.TensorDataset(x, y)
loader = torch.utils.data.DataLoader(dataset, batch_size=20, shuffle=True)

# Define a simple two-layer network
class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(1, 10)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(10, 1)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

# Instantiate model
torch.manual_seed(0)
model = SimpleNN()

# Loss function and optimizer
criterion = nn.MSELoss()
learning_rate = 0.1  # try 0.01, 0.1, 0.5
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

# Training loop
num_epochs = 50
for epoch in range(num_epochs):
    epoch_loss = 0.0
    for batch_x, batch_y in loader:
        optimizer.zero_grad()
        outputs = model(batch_x)
        loss = criterion(outputs, batch_y)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
    if (epoch + 1) % 10 == 0:
        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {epoch_loss/len(loader):.4f}')

# Observe final parameters
for name, param in model.named_parameters():
    print(name, param.data)

Explanation:

1. We build a toy linear regression problem with noise.
2. SimpleNN has one hidden layer with ReLU.
3. We use MSELoss and SGD optimizer.
4. The learning rate is configurable. Try different values to see: small rates converge slowly; very large can diverge.
5. We train for 50 epochs, printing loss every 10 epochs.

Observations:

• If lr=0.01, loss gradually decreases.
• If lr=0.1, faster convergence.
• If lr=0.5, might oscillate or diverge.

5. Tasks:

1. Experiment: Change learning_rate to 0.01, 0.1, and 0.5. Record the loss curve for each setting.
2. Compare Variants: Modify the DataLoader to use batch size = 1 (SGD) and batch size = 100 (batch GD). Observe loss stability and convergence speed.
3. Plot Loss: Plot loss vs epoch for different settings on one graph. Discuss tradeoffs.
4. Reflection: Summarize in a few sentences how learning rate and batch size influence convergence behavior.

This exercise will cement your understanding of optimization via gradient descent in neural networks. Good luck!