Machine Learning chevron_right Loss Functions for Neural Networks

Loss Functions for Neural Networks

calendar_today schedule 5 min read

1. Welcome and Icebreaker:

  • Welcome to Day 22! Today we explore loss functions and their role in measuring neural network performance.
  • Icebreaker: If a loss function could be any sport, which sport would best represent its objective and why?
  • Share your thought: How do you imagine measuring how badly a network is performing?

2. Outline the Session:

  • Agenda:
    1. Recap: What is a neural network’s output (logits, probabilities)?
    2. Loss Function Concepts: Definition and purpose of loss functions.
    3. Common Loss Functions: MSE, Cross-Entropy (binary and categorical), Negative Log-Likelihood.
    4. Live Coding: Implementing loss functions with NumPy/PyTorch.
    5. Task: Compute loss for simple examples manually.
  • Learning Outcomes:
    • Understand the role of loss functions in training.
    • Compare different loss functions and their properties.
    • Know when to use MSE vs Cross-Entropy.
    • Implement basic loss calculations in code.
  • Structure: Theory → Math formulations → Code examples → Exercises.
  • Interactivity: Ask questions anytime. Plan to discuss real datasets examples.

3. Concept Explanation:

A loss function (also called objective or cost function) quantifies how well your neural network’s predictions match the true labels. During training, the network adjusts its parameters (weights and biases) to minimize this loss. Without a well-defined loss, no learning occurs.

Why Loss Functions Matter

Neural networks make predictions by performing a feedforward pass, producing outputs that may be continuous values (regression) or class probabilities (classification). The loss function computes a single scalar value summarizing the discrepancy between predictions and ground truth labels. This scalar drives backpropagation: its gradient with respect to network parameters indicates how to adjust weights to improve performance.

Key Properties of Good Loss Functions

  1. Differentiability: The function should be continuous and differentiable almost everywhere so gradient-based optimizers can compute parameter updates via backpropagation.
  2. Proper Scale: The output of the loss must reflect meaningful differences in prediction quality. Too large values can cause unstable learning; too small values make training slow.
  3. Convexity (Optional): A convex objective has no nonglobal local minima, but it can have multiple global minima. Strict convexity gives a unique minimum when one exists. Convexity of a loss as a function of predictions does not by itself make the full neural-network objective convex in its parameters.
  4. Interpretability: Ideally, the loss relates to a well-known metric (e.g., MSE relates to Euclidean distance, Cross-Entropy relates to likelihood).

Common Loss Functions

Mean Squared Error (MSE)

  • Used for regression problems where target labels are continuous values.
  • Formula: L_MSE = (1/N) Σ_i (ŷ_i - y_i)^2
  • Squaring penalizes larger errors more heavily; smooth and convex.
  • Sensitive to outliers: large errors contribute disproportionately.

Binary Cross-Entropy (Log Loss)

  • Used for binary classification (two classes).
  • The model outputs a probability p_i = sigmoid(z_i) for each example.
  • Formula: L_BCE = -(1/N) Σ_i [y_i log(p_i) + (1-y_i) log(1-p_i)]
  • Encourages confident classification: if the true label is 1, it penalizes low p_i; if the label is 0, it penalizes high p_i.
  • Handles probabilities directly, avoids vanishing gradient issues near 0/1.

Categorical Cross-Entropy (Softmax + NLL)

  • Used for multi-class classification with C classes.
  • The model outputs logits z_1, ..., z_C. Softmax converts them to probabilities: p_j = exp(z_j) / Σ_k exp(z_k).
  • Loss: L_CCE = -(1/N) Σ_i Σ_j y_ij log(p_ij)
  • Equivalent to Negative Log-Likelihood (NLL) when labels are one-hot encoded.
  • Penalizes incorrect class probabilities heavily; encourages probability mass on correct class.

Huber Loss

  • Combines MSE and MAE (Mean Absolute Error) for regression; less sensitive to outliers.
  • Let r = y - ŷ be the prediction error. The Huber loss is L_delta(r) = 0.5 r^2 when |r| ≤ delta, and delta (|r| - 0.5 delta) otherwise.
  • Quadratic for small errors, linear for large errors.

Choosing the Right Loss

  • Regression: MSE or Huber when outliers present.
  • Binary Classification: Binary Cross-Entropy.
  • Multi-class Classification: Categorical Cross-Entropy.
  • Imbalanced Classes: Weighted cross-entropy or focal loss.

Common Pitfalls:

  • Using MSE for classification leads to slow learning, saturating gradients.
  • Numerical stability: computing log from probabilities rounded to zero can produce infinities or NaNs. Prefer fused logits-based losses such as BCEWithLogitsLoss or CrossEntropyLoss; clipping or adding a small epsilon is mainly a safeguard for simple hand-written demonstrations.
  • Forgetting to average over batch size; total loss grows with batch size.

4. Live Coding Demonstration:

Below is a Python example using NumPy to compute MSE and Cross-Entropy loss, and PyTorch example for built-in loss functions.

# NumPy implementation
import numpy as np

def mse_loss(y_true, y_pred):
    return np.mean((y_pred - y_true) ** 2)

def binary_cross_entropy(y_true, y_pred, eps=1e-12):
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Example data
y_true_reg = np.array([2.5, 0.0, 2.1, 7.8])
y_pred_reg = np.array([3.0, -0.5, 2.0, 7.5])
print("MSE Loss:", mse_loss(y_true_reg, y_pred_reg))

# Binary classification example
y_true_bin = np.array([1, 0, 1, 0])
y_pred_bin = np.array([0.9, 0.2, 0.8, 0.3])
print("Binary Cross-Entropy Loss:", binary_cross_entropy(y_true_bin, y_pred_bin))

# PyTorch implementation
torch_code = '''
import torch
import torch.nn as nn

# Sample tensors
y_true = torch.tensor([1, 0, 1, 0], dtype=torch.float32)
y_pred_logits = torch.tensor([2.0, -1.0, 1.5, -0.5])  # logits

# Binary Cross-Entropy with logits (combines sigmoid + BCE)
bce_loss_fn = nn.BCEWithLogitsLoss()
loss = bce_loss_fn(y_pred_logits, y_true)
print("PyTorch BCEWithLogitsLoss:", loss.item())
'''
print(torch_code)

Explanation:

  1. np.clip ensures numerical stability by avoiding log(0).
  2. PyTorch’s BCEWithLogitsLoss combines sigmoid activation and BCE in a single, stable function.
  3. For categorical cross-entropy in PyTorch, use nn.CrossEntropyLoss() (expects raw logits and integer class labels).

5. Tasks:

Manual Loss Calculation Exercise

  1. Given true labels y = [1, 0, 1] and predicted probabilities p = [0.8, 0.3, 0.6], compute:
    • MSE loss
    • Binary cross-entropy loss
  2. For multi-class with 3 classes, given one-hot true label y = [0, 1, 0] and logits [2.0, 0.5, -1.0], compute:
    • Softmax probabilities
    • Categorical cross-entropy loss
  3. Compare results and discuss which loss better penalizes wrong predictions and why.

Submit your step-by-step calculations, including intermediate values for softmax and log terms. This practice will solidify your understanding of how loss functions measure network performance.