Activation Functions

1. Welcome and Icebreaker:

• Icebreaker Question: “If you were a mathematical function, which one would you be and why?”
  • Invite students to suggest functions like sine (periodic and rhythmic), exponential (growing), step function (decisive), or ReLU (efficient).
  • Discuss the characteristics of each function and how they metaphorically relate to personality traits or behaviors.
• Encourage participation by asking: “In our daily decisions, do you think your brain applies a threshold like a step function? Give an example!”
  • Possible answers: deciding to take an umbrella if it’s raining beyond a certain intensity, or hitting “like” on social media only if a post appeals to you strongly enough.

2. Outline the Session:

• Agenda:
  1. Introduction and motivation for activation functions.
  2. Mathematical definitions of common activation functions: Sigmoid, Tanh, ReLU, Leaky ReLU.
  3. Intuition: why non-linearity matters.
  4. Pros and cons of each function.
  5. Live coding demonstration: plotting and comparing activations.
  6. Common pitfalls and best practices.
  7. Tasks and exercises to deepen understanding.
• Learning Outcomes:
  • Understand why activation functions are crucial for neural networks.
  • Be able to define and implement Sigmoid, Tanh, ReLU, and Leaky ReLU.
  • Recognize the impact of activation choices on training dynamics (e.g., vanishing gradients).
  • Practice visualizing activation functions and exploring their derivatives.
• Expectations:
  • Active participation during live coding.
  • Ask questions if unclear about non-linearity or derivative calculations.
  • Pair up for tasks: discuss results with peers.

3. Concept Explanation:

Neural networks without activation functions reduce to a simple linear regression model regardless of depth. Activation functions introduce non-linearity, enabling networks to learn complex, non-linear decision boundaries. Let’s explore the most common ones:

3.1 Sigmoid Function

• Definition:
  [ \sigma(z) = \frac{1}{1 + e^{-z}} ]
• Range: (0, 1)
• Shape: S-curve that saturates for large positive or negative inputs.
• Derivative:
  [ \sigma’(z) = \sigma(z) (1 - \sigma(z)) ]
• Intuition:
  • Useful when outputs must be probabilities (binary classification).
  • Near-zero inputs gives output ~0.5, extreme inputs push output close to 0 or 1.
• Pros:
  • Smooth gradient everywhere.
  • Probabilistic interpretation.
• Cons:
  • Vanishing Gradient: derivatives become very small (close to 0) when |z| is large, slowing learning in deep networks.
  • Outputs not zero-centered: can lead to zig-zagging during gradient descent.

3.2 Hyperbolic Tangent (Tanh)

• Definition:
  [ \tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}} ]
• Range: (–1, 1)
• Relationship: ( \tanh(z) = 2\sigma(2z) - 1 )
• Derivative:
  [ \tanh’(z) = 1 - \tanh^2(z) ]
• Intuition:
  • Zero-centered output (negative values possible) helps gradient descent converge more smoothly.
• Pros:
  • Zero-centered.
  • Steeper gradient than sigmoid near the center.
• Cons:
  • Still suffers from vanishing gradients for large |z|.

3.3 Rectified Linear Unit (ReLU)

• Definition:
  [ \text{ReLU}(z) = \max(0, z) ]
• Range: [0, ∞)
• Derivative:
  [ \text{ReLU}’(z) = \begin{cases} 0 & \text{if } z < 0, \ 1 & \text{if } z > 0 \end{cases} ]
• Intuition:
  • Keeps positive inputs linearly, discards negative ones.
  • Mimics biological neuron firing: inactive below threshold (0), active above.
• Pros:
  • Computationally efficient (simple max operation).
  • Alleviates vanishing gradient for positive inputs (gradient = 1).
  • Enables sparse activations (many zeros), which can reduce computation in some implementations; it does not reduce the model’s number of learned parameters.
• Cons:
  • Dying ReLU: neurons can get stuck if inputs become negative, gradient zero, weights never update.

3.4 Leaky ReLU

• Definition:
  [ \text{LeakyReLU}(z) = \max(\alpha z, z) ] where (0 < \alpha \ll 1) (commonly 0.01)
• Range: (–∞, ∞)
• Derivative:
  [ \text{LeakyReLU}’(z) = \begin{cases} \alpha & \text{if } z < 0, \ 1 & \text{if } z > 0 \end{cases} ]
• Intuition:
  • Allows a small, non-zero gradient when z < 0 to prevent neurons from dying.
• Pros:
  • Mitigates the dying ReLU problem.
  • Maintains most benefits of ReLU.
• Cons:
  • Introduces a hyperparameter (\alpha) to tune.

3.5 Other Variants

• ELU (Exponential Linear Unit): smooth negative values, faster learning in deep networks.
• SELU (Scaled ELU): can help keep activations near stable mean and variance in specific self-normalizing setups. This behavior relies on conditions such as standardized inputs, LeCun-normal initialization, a compatible feedforward architecture, and AlphaDropout when dropout is used; it is not automatic for every network.

3.6 Why Non-Linearity Matters

Without non-linear activations, the composition of linear transformations remains linear no matter how many layers you stack. Non-linearity allows networks to approximate complex functions. This capability is critical for tasks like image recognition, language processing, and reinforcement learning.

4. Live Coding Demonstration:

Let’s plot and compare these activation functions and their derivatives using Python and Matplotlib.

import numpy as np
import matplotlib.pyplot as plt

# Define range of input values
z = np.linspace(-5, 5, 500)

# Activation functions and derivatives
sigmoid = 1 / (1 + np.exp(-z))
sigmoid_deriv = sigmoid * (1 - sigmoid)

tanh = np.tanh(z)
tanh_deriv = 1 - tanh**2

relu = np.maximum(0, z)
relu_deriv = np.where(z > 0, 1, 0)

alpha = 0.01
leaky_relu = np.where(z > 0, z, alpha * z)
leaky_relu_deriv = np.where(z > 0, 1, alpha)

# Plot functions
plt.figure(figsize=(12, 8))
plt.subplot(2, 2, 1)
plt.plot(z, sigmoid, label='Sigmoid')
plt.plot(z, sigmoid_deriv, '--', label='Sigmoid Derivative')
plt.title('Sigmoid')
plt.legend()

plt.subplot(2, 2, 2)
plt.plot(z, tanh, label='Tanh')
plt.plot(z, tanh_deriv, '--', label='Tanh Derivative')
plt.title('Tanh')
plt.legend()

plt.subplot(2, 2, 3)
plt.plot(z, relu, label='ReLU')
plt.plot(z, relu_deriv, '--', label='ReLU Derivative')
plt.title('ReLU')
plt.legend()

plt.subplot(2, 2, 4)
plt.plot(z, leaky_relu, label=f'Leaky ReLU (alpha={alpha})')
plt.plot(z, leaky_relu_deriv, '--', label='Leaky ReLU Derivative')
plt.title('Leaky ReLU')
plt.legend()

plt.tight_layout()
plt.show()

Step-by-Step Explanation:

1. Setup Input Range: Create a smooth range of z values from –5 to 5.
2. Compute Activations: Use NumPy operations to calculate each function and its derivative.
3. Plotting: Use Matplotlib’s subplots to visualize functions side-by-side for comparison.
4. Interpretation: Students should observe saturation regions (Sigmoid, Tanh), linear regions (ReLU), and small gradient for negative inputs (Leaky ReLU).

Discuss Observations:

• Why does Sigmoid’s derivative shrink near the extremes?
• How does Tanh’s zero-centering benefit optimization?
• Under what conditions might ReLU neurons die, and how can we detect it?

5. Common Pitfalls & Best Practices:

• Choosing Activation Too Early: Don’t pick an arbitrary function without understanding data and network depth.
• Learning Rate Interaction: ReLU often pairs well with higher learning rates; Sigmoid may require smaller rates to avoid oscillations.
• Batch Normalization: Can mitigate issues like internal covariate shift and make activation choice less critical.
• Monitoring Gradients: Use tools (TensorBoard, custom hooks) to track gradient flow and detect vanishing or exploding gradients.
• Consistency: Keep activation choices consistent across hidden layers (with exceptions), and only change output layer activation based on task (e.g., softmax for multi-class).

6. Tasks:

Task 1: Manual Computation

• Compute the output and derivative of each activation function for input values z = -2, -0.5, 0, 1.5, 4. Show calculations step-by-step.

Task 2: Implement Custom Activation

• Write a Python function for ELU (Exponential Linear Unit):

  def elu(z, alpha=1.0):
      return np.where(z >= 0, z, alpha * (np.exp(z) - 1))
  

• Plot ELU alongside ReLU to observe differences.

Task 3: Activation Choice Experiment

• Build a small neural network (2 hidden layers, 50 neurons each) on the Iris dataset using Keras or PyTorch.
• Train three versions, changing only the hidden activation: Sigmoid, Tanh, ReLU.
• Record training time, final accuracy, and observe any plateauing or oscillations in loss curves. Summarize results.

Reflection:

• Based on your experiments, which activation function would you choose for a deep network with hundreds of layers? Why?
• How might you combine activations or create hybrid architectures using different functions in different parts of the network?