Calculus - Derivatives & Gradients

calendar_today May 5, 2025 schedule 9 min read

1. Welcome and Icebreaker:

Fun Question: If a function were a rollercoaster, what would its derivative represent at the steepest drop?
Share: Can anyone recall a time they intuitively used the idea of a “rate of change” in a real-life situation?

2. Outline the Session:

Agenda:
1. Welcome & Icebreaker (5 min)
2. Understanding Derivatives (20 min)
3. Differentiation Rules (25 min)
4. Partial Derivatives & Gradients (25 min)
5. Live Coding: Calculating Derivatives in Python (15 min)
6. Task: Practice Problems (15 min)
7. Q&A and Wrap-up (5 min)
Learning Outcomes: By the end of this session, you will be able to:
- Explain the concept of a derivative as a rate of change and slope.
- Apply basic differentiation rules (power, sum, product, chain).
- Understand and calculate partial derivatives.
- Define a gradient and understand its significance.
- Implement basic derivative calculations in Python.
Structure: We’ll mix explanations with examples, a live coding session, and a hands-on exercise.
Interactivity: Please ask questions at any point! We’ll have discussions and a practical coding segment. Your participation is key.

3. Concept Explanation:

Understanding Derivatives

The derivative of a function at a certain point is one of the most fundamental concepts in calculus. It essentially measures the rate of change of the function at that specific point. Imagine you’re driving a car; your speed is the rate of change of distance over time. If your distance from the start is represented by a function f(t) where t is time, then the derivative f'(t) (also written as df/dt) gives you your instantaneous speed.

Graphically, the derivative of a function f(x) at a point x = a represents the slope of the tangent line to the graph of f(x) at that point. A steep slope means a rapid change, while a flat slope (derivative is zero) means no change (like the peak of a hill or the bottom of a valley).

Why are derivatives important in Machine Learning? Derivatives are the backbone of optimization algorithms like Gradient Descent. In ML, we often define a “cost” or “loss” function that measures how bad our model’s predictions are. To improve the model, we need to minimize this cost function. Derivatives tell us the direction in which the function is increasing or decreasing most steeply. By moving in the opposite direction of the gradient (which is made of derivatives), we can iteratively find the minimum of the cost function and thus, a better model.

Basic Differentiation Rules

Calculating derivatives from first principles (using limits) can be tedious. Fortunately, there are several rules that simplify this process for many common functions:

Power Rule: If f(x) = x^n, then f'(x) = n * x^(n-1).
- Example: If f(x) = x^3, then f'(x) = 3x^2.
Constant Rule: If f(x) = c (a constant), then f'(x) = 0.
- Example: If f(x) = 5, then f'(x) = 0. (The slope of a horizontal line is zero).
Constant Multiple Rule: If f(x) = c * g(x), then f'(x) = c * g'(x).
- Example: If f(x) = 2x^3, then f'(x) = 2 * (3x^2) = 6x^2.
Sum Rule: If h(x) = f(x) + g(x), then h'(x) = f'(x) + g'(x).
- Example: If h(x) = x^2 + 3x, then h'(x) = 2x + 3.
Difference Rule: If h(x) = f(x) - g(x), then h'(x) = f'(x) - g'(x).
- Example: If h(x) = x^4 - 2x, then h'(x) = 4x^3 - 2.
Product Rule: If h(x) = f(x) * g(x), then h'(x) = f'(x)g(x) + f(x)g'(x).
- Example: If h(x) = x^2 * sin(x), then h'(x) = 2x*sin(x) + x^2*cos(x).
Quotient Rule: If h(x) = f(x) / g(x), then h'(x) = (f'(x)g(x) - f(x)g'(x)) / (g(x))^2.
- This one is a bit more complex but follows a pattern.
Chain Rule: This is crucial for neural networks. If h(x) = f(g(x)) (a function of a function), then h'(x) = f'(g(x)) * g'(x).
- Think of it as “derivative of the outer function (with the inner function unchanged) times the derivative of the inner function.”
- Example: If h(x) = (2x + 1)^3. Let f(u) = u^3 and g(x) = 2x + 1. Then f'(u) = 3u^2 and g'(x) = 2. So, h'(x) = 3(2x+1)^2 * 2 = 6(2x+1)^2.

Partial Derivatives

Often, functions in machine learning depend on multiple variables (e.g., a cost function might depend on several model weights w1, w2, ..., wn). When we have a function of multiple variables, like f(x, y), we can ask how the function changes if we vary one variable while keeping the others constant. This is what a partial derivative tells us.

The partial derivative of f with respect to x is denoted as ∂f/∂x. To calculate it, you treat y (and any other variables) as a constant and differentiate with respect to x using the standard rules.
Similarly, the partial derivative of f with respect to y is ∂f/∂y. Here, you treat x as a constant and differentiate with respect to y.

Example: Let f(x, y) = x^2 * y^3 + 2x + y.

∂f/∂x: Treat y as a constant.
- Derivative of x^2*y^3 w.r.t x is 2x*y^3.
- Derivative of 2x w.r.t x is 2.
- Derivative of y w.r.t x is 0 (since y is treated as a constant).
- So, ∂f/∂x = 2xy^3 + 2.
∂f/∂y: Treat x as a constant.
- Derivative of x^2*y^3 w.r.t y is x^2 * 3y^2 = 3x^2y^2.
- Derivative of 2x w.r.t y is 0.
- Derivative of y w.r.t y is 1.
- So, ∂f/∂y = 3x^2y^2 + 1.

Gradients

The gradient of a multivariable function f(x, y, ...) is a vector that contains all of its partial derivatives. It’s denoted by ∇f (nabla f). If f(x, y), then ∇f = [∂f/∂x, ∂f/∂y]. If f(x1, x2, ..., xn), then ∇f = [∂f/∂x1, ∂f/∂x2, ..., ∂f/∂xn].

Significance of the Gradient:

Direction of Steepest Ascent: The gradient vector ∇f at a point (a, b) points in the direction in which the function f increases most rapidly.
Magnitude of Steepest Ascent: The magnitude (length) of the gradient vector ||∇f|| is the rate of that increase.
Orthogonal to Level Curves: At points where ∇f is nonzero, the gradient is perpendicular to the level curve (or level surface) through that point. If ∇f = 0, it has no defined direction there.

In machine learning, we are usually interested in minimizing a cost function. Since the gradient points in the direction of the steepest increase, the negative gradient -∇f points in the direction of the steepest decrease. This is the core idea behind the Gradient Descent algorithm.

Resources for this section:

Khan Academy (Calculus): Excellent for foundational concepts and practice.
3Blue1Brown (Essence of Calculus playlist): Provides fantastic visual intuitions.

4. Live Coding Demonstration:

We’ll use Python with a library like SymPy for symbolic differentiation to illustrate these concepts. If SymPy is too advanced for a first look, we can do manual calculations and then show how to approximate derivatives numerically.

import numpy as np

# Example 1: Derivative of a simple polynomial
# f(x) = x^3 - 2x^2 + 5
# f'(x) = 3x^2 - 4x

def f(x):
    return x**3 - 2*x**2 + 5

def df_analytical(x):
    return 3*x**2 - 4*x

# Let's check at x = 2
x_val = 2
print(f"f({x_val}) = {f(x_val)}")
print(f"f'({x_val}) analytical = {df_analytical(x_val)}")

# Numerical approximation of derivative (finite difference)
h = 0.0001 # A small step
df_numerical = (f(x_val + h) - f(x_val)) / h
print(f"f'({x_val}) numerical approx = {df_numerical}")

print("-" * 30)

# Example 2: Partial Derivatives
# g(x, y) = x^2*y + 3*y^3
# ∂g/∂x = 2xy
# ∂g/∂y = x^2 + 9y^2

def g(x, y):
    return x**2 * y + 3*y**3

def dg_dx_analytical(x, y):
    return 2*x*y

def dg_dy_analytical(x, y):
    return x**2 + 9*y**2

x_val, y_val = 1, 2
print(f"g({x_val}, {y_val}) = {g(x_val, y_val)}")
print(f"∂g/∂x ({x_val}, {y_val}) analytical = {dg_dx_analytical(x_val, y_val)}")
print(f"∂g/∂y ({x_val}, {y_val}) analytical = {dg_dy_analytical(x_val, y_val)}")

# Numerical approximation of partial derivative ∂g/∂x
h = 0.0001
dg_dx_numerical = (g(x_val + h, y_val) - g(x_val, y_val)) / h
print(f"∂g/∂x ({x_val}, {y_val}) numerical approx = {dg_dx_numerical}")

# Numerical approximation of partial derivative ∂g/∂y
dg_dy_numerical = (g(x_val, y_val + h) - g(x_val, y_val)) / h
print(f"∂g/∂y ({x_val}, {y_val}) numerical approx = {dg_dy_numerical}")

print("-" * 30)
# Gradient example
# For g(x,y) at (1,2), ∇g = [dg_dx(1,2), dg_dy(1,2)]
grad_g_analytical = [dg_dx_analytical(x_val, y_val), dg_dy_analytical(x_val, y_val)]
print(f"Gradient ∇g({x_val}, {y_val}) analytical = {grad_g_analytical}")

# Using SymPy for symbolic differentiation (more robust)
from sympy import symbols, diff

x_sym, y_sym = symbols('x y')

# Define f(x) symbolically
f_sym = x_sym**3 - 2*x_sym**2 + 5
df_sym = diff(f_sym, x_sym)
print(f"Symbolic derivative of f(x): {df_sym}")
print(f"Symbolic f'({x_val}): {df_sym.subs(x_sym, x_val)}")


# Define g(x, y) symbolically
g_sym = x_sym**2 * y_sym + 3*y_sym**3
dg_dx_sym = diff(g_sym, x_sym)
dg_dy_sym = diff(g_sym, y_sym)

print(f"Symbolic ∂g/∂x: {dg_dx_sym}")
print(f"Symbolic ∂g/∂y: {dg_dy_sym}")
print(f"Symbolic ∂g/∂x at ({x_val},{y_val}): {dg_dx_sym.subs([(x_sym, x_val), (y_sym, y_val)])}")
print(f"Symbolic ∂g/∂y at ({x_val},{y_val}): {dg_dy_sym.subs([(x_sym, x_val), (y_sym, y_val)])}")

Explanation of Code:

We first define functions and their analytical derivatives manually.
We show how to approximate a derivative numerically using the finite difference method: f'(x) ≈ (f(x+h) - f(x)) / h for a small h. This is the definition of the derivative before taking the limit h -> 0.
We repeat this for partial derivatives.
We then introduce SymPy, a powerful Python library for symbolic mathematics.
- symbols('x y') declares x and y as symbolic variables.
- diff(expression, variable) computes the derivative of the expression with respect to the variable.
- .subs([(variable, value), ...]) substitutes values into the symbolic expression to get a numerical result. Common Pitfalls:

Numerical instability with very small h in finite differences.
Forgetting the chain rule in manual calculations.
Confusing partial derivatives with total derivatives.

5. Tasks:

Short Exercise:

Calculate Derivatives Manually:
- Find the derivative f'(x) for f(x) = 5x^4 - 7x^2 + 2x - 10.
- Find the derivative g'(t) for g(t) = (t^2 + 1)^5 (Hint: Use the chain rule).
- Find the derivative h'(x) for h(x) = x * e^x (Hint: Use the product rule. The derivative of e^x is e^x).
Compute Partial Derivatives and Gradient:
- Given the function k(a, b) = 3a^2b - ab^3 + 7a - 2b.
- Calculate ∂k/∂a.
- Calculate ∂k/∂b.
- Write down the gradient ∇k(a, b).
- Evaluate the gradient ∇k at the point (a=1, b=2).
(Optional Python Task):
- Using SymPy, verify your manual calculations for question 1 and 2.
- Define the function m(x, y) = sin(x) * y^2. Use SymPy to find ∂m/∂x and ∂m/∂y.

Practice from Resources:

Solve derivative problems on Khan Academy (Calculus section).
Compute gradients for functions of 2-3 variables as suggested in the lesson.md.

6. Q&A and Wrap-up:

Open floor for any questions about derivatives, gradients, or their application.
Briefly reiterate why derivatives are crucial for optimization in machine learning.
Preview: Next, we’ll see how these concepts are used in optimization algorithms like Gradient Descent.