Machine Learning chevron_right Backpropagation - A Little Deeper

Backpropagation - A Little Deeper

calendar_today schedule 3 min read

1. Welcome and Icebreaker:

  • Welcome to Day 25! Today we delve into the mathematical formulas behind backpropagation.
  • Icebreaker: If calculus had a mascot, who or what would it be and why?
  • Share your mascot idea and how it represents the chain rule.

2. Outline the Session:

  • Agenda:
    1. Recap: Chain rule and forward pass formulas.
    2. Deriving Gradients for a Single Layer.
    3. Putting It Together: Multi-layer formulas.
    4. Live Derivation: Output layer gradient for cross-entropy loss + softmax.
    5. Task: Derive weight update formula for a simple perceptron layer.
  • Learning Outcomes:
    • Derive parameter gradients for a neural network layer.
    • Understand the mathematical relationship between activation derivatives and weight updates.
    • Derive gradient of softmax + cross-entropy in one step.
  • Structure: Theory → Step-by-step derivation → Code snippet (symbolic) → Exercise.
  • Interactivity: Ask questions during derivations.

3. Concept Explanation:

Let’s formalize backprop for a single layer with inputs (x), weights (W), bias (b), activation (f), producing output (a=f(z)), where (z=W x + b). We want (\frac{\partial L}{\partial W}).

  1. Compute (\frac{\partial L}{\partial z}): [ \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot f’(z) ]

  2. Compute (\frac{\partial L}{\partial W}): [ \frac{\partial L}{\partial W} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial W} = \frac{\partial L}{\partial z} \cdot x^T ] Since (z = W x + b), the partial derivative with respect to (W) is simply the input vector (x).

  3. Bias Gradient: [ \frac{\partial L}{\partial b} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial b} = \frac{\partial L}{\partial z} \cdot 1 = \frac{\partial L}{\partial z} ]

  4. Propagating to Previous Layer: [ \frac{\partial L}{\partial x} = W^T \cdot \frac{\partial L}{\partial z} ]

Example: Softmax + Cross-Entropy

For multi-class classification, let logits (z_j), softmax output (\sigma(z)j), and true one-hot (y_j). The loss for a sample is: [ L = - \sum{j} y_j \log(\sigma(z)_j) ]

It can be shown that the gradient wrt logits is: [ \frac{\partial L}{\partial z_j} = \sigma(z)_j - y_j ]

This elegant result emerges by combining the derivative of the softmax and log terms and simplifies implementation in frameworks.

Why This Matters: The gradient simplifies algebraically to p - y, which is efficient. That identity alone does not make the computation numerically stable: avoid taking logs of probabilities rounded to zero. Use a fused, stable logits-based loss such as PyTorch’s CrossEntropyLoss.

4. Live Derivation:

Step 1: Softmax definition: [ \sigma(z)j = \frac{e^{z_j}}{\sum{k} e^{z_k}} ]

Step 2: Compute (\frac{\partial L}{\partial z_j}): [ \frac{\partial L}{\partial z_j} = - \sum_{i} y_i \frac{\partial}{\partial z_j} [\log \sigma(z)i] = - \sum{i} y_i \frac{1}{\sigma(z)_i} \frac{\partial \sigma(z)_i}{\partial z_j} ]

After algebra (apply quotient rule and Kronecker delta), we get: [ \frac{\partial L}{\partial z_j} = \sigma(z)_j - y_j ]

5. Tasks:

  1. Derive: For a single-layer network with softmax output, derive (\frac{\partial L}{\partial W}) using the result above and the general backprop rule.
  2. Compare: Show how the derivative simplifies compared to separate softmax and cross-entropy derivatives.
  3. Implement: Using sympy in Python, symbolically verify (\frac{\partial L}{\partial z_j} = \sigma(z)_j - y_j).

This concludes our deeper dive into backpropagation math. Excellent work reaching this level of detail!