Machine Learning chevron_right Calculus - Integrals & Applications

Calculus - Integrals & Applications

calendar_today schedule 10 min read

1. Welcome and Icebreaker:

  • Fun Question: If a derivative is like finding your speed at any moment, what real-world quantity would an integral help you find if you know your speed over time?
  • Share: Has anyone encountered a situation where they needed to find a total accumulation or an average of something that was changing?

2. Outline the Session:

  • Agenda:
    1. Welcome & Icebreaker (5 min)
    2. Understanding Integrals: The Antiderivative (20 min)
    3. The Definite Integral: Area Under a Curve (25 min)
    4. The Fundamental Theorem of Calculus (15 min)
    5. Applications in Probability and Machine Learning (15 min)
    6. Live Coding: Calculating Integrals in Python (15 min)
    7. Task: Practice Problems (15 min)
    8. Q&A and Wrap-up (5 min)
  • Learning Outcomes: By the end of this session, you will be able to:
    • Explain the concept of an integral as an antiderivative.
    • Understand the definite integral as the area under a curve.
    • State and appreciate the Fundamental Theorem of Calculus.
    • Recognize basic applications of integrals in probability (e.g., CDFs) and ML.
    • Implement basic integral calculations in Python.
  • Structure: We will cover the theory of integrals, connect it to derivatives, explore its geometric meaning, and then look at practical examples and coding.
  • Interactivity: Your questions are welcome throughout! We’ll have discussions and a coding segment. Let’s explore integrals together.

3. Concept Explanation:

Understanding Integrals: The Antiderivative

Integration is often introduced as the reverse process of differentiation. If differentiation is about finding the rate of change (slope), then integration (specifically, indefinite integration) is about finding a function whose derivative is the given function. This original function is called the antiderivative or primitive.

If F'(x) = f(x), then F(x) is an antiderivative of f(x). The indefinite integral of f(x) with respect to x is written as: ∫f(x) dx = F(x) + C

Where:

  • ∫ is the integral symbol.
  • f(x) is the integrand (the function to be integrated).
  • dx indicates that the integration is with respect to the variable x.
  • F(x) is an antiderivative of f(x).
  • C is the constant of integration. Since the derivative of a constant is zero, any constant C can be added to an antiderivative, and it will still have the same derivative f(x). For example, if f(x) = 2x, then F(x) = x^2 is an antiderivative because d/dx (x^2) = 2x. But x^2 + 5, x^2 - 100, and x^2 + C in general are also antiderivatives of 2x.

Basic Integration Rules (reverse of differentiation rules):

  1. Power Rule (for n ≠ -1): ∫x^n dx = (x^(n+1))/(n+1) + C
    • Example: ∫x^3 dx = (x^4)/4 + C.
  2. Constant Rule: ∫k dx = kx + C (where k is a constant)
    • Example: ∫5 dx = 5x + C.
  3. Constant Multiple Rule: ∫k*f(x) dx = k * ∫f(x) dx
    • Example: ∫3x^2 dx = 3 * ∫x^2 dx = 3 * (x^3/3) + C = x^3 + C.
  4. Sum/Difference Rule: ∫(f(x) ± g(x)) dx = ∫f(x) dx ± ∫g(x) dx
    • Example: ∫(x + cos(x)) dx = ∫x dx + ∫cos(x) dx = (x^2/2) + sin(x) + C.
  5. Special Cases:
    • ∫(1/x) dx = ln|x| + C (This is the n = -1 case for the power rule)
    • ∫e^x dx = e^x + C
    • ∫cos(x) dx = sin(x) + C
    • ∫sin(x) dx = -cos(x) + C

The Definite Integral: Area Under a Curve

While the indefinite integral gives us a family of functions, the definite integral gives us a specific number. It represents the net signed area between the graph of f(x), the x-axis, and the vertical lines x = a and x = b.

The definite integral is written as: ∫[a, b] f(x) dx

Where:

  • a is the lower limit of integration.
  • b is the upper limit of integration.

Geometrically, this can be visualized as summing up the areas of infinitely many infinitesimally thin rectangles under the curve f(x) from x = a to x = b. If f(x) is above the x-axis, the area is positive. If f(x) is below the x-axis, the area is considered negative.

Example: ∫[0, 2] x dx This represents the area under the line y = x from x = 0 to x = 2. This forms a triangle with base 2 and height 2. The area is (1/2) * base * height = (1/2) * 2 * 2 = 2.

The Fundamental Theorem of Calculus

This is a cornerstone theorem that links differentiation and integration. It has two parts:

Part 1 (Derivative of an Integral): If f is continuous on [a, b], and G(x) = ∫[a, x] f(t) dt, then G'(x) = f(x). This part essentially says that the process of integrating f and then differentiating the result gets you back to f.

Part 2 (Evaluation of Definite Integrals): If f is continuous on [a, b] and F is any antiderivative of f (i.e., F'(x) = f(x)), then: ∫[a, b] f(x) dx = F(b) - F(a)

This part is incredibly powerful because it gives us a way to calculate definite integrals without having to sum up areas of rectangles (which is often very difficult). We just need to:

  1. Find an antiderivative F(x) of f(x).
  2. Evaluate F(x) at the upper limit b (i.e., F(b)).
  3. Evaluate F(x) at the lower limit a (i.e., F(a)).
  4. Subtract the two: F(b) - F(a).

Example using Part 2: ∫[0, 2] x dx

  1. f(x) = x. An antiderivative is F(x) = x^2/2.
  2. F(b) = F(2) = 2^2/2 = 4/2 = 2.
  3. F(a) = F(0) = 0^2/2 = 0.
  4. ∫[0, 2] x dx = F(2) - F(0) = 2 - 0 = 2. This matches our geometric calculation!

Applications in Probability and Machine Learning

Integrals are vital in many areas relevant to machine learning:

  1. Probability Theory:

    • Continuous Random Variables: For a continuous random variable X with a probability density function (PDF) p(x), the probability that X falls within an interval [a, b] is given by the integral: P(a ≤ X ≤ b) = ∫[a, b] p(x) dx.
    • Cumulative Distribution Function (CDF): The CDF F(x) for a continuous random variable is defined as F(x) = P(X ≤ x) = ∫[-∞, x] p(t) dt. The CDF gives the probability that the variable takes a value less than or equal to x.
    • Expectation (Mean): The expected value of a continuous random variable X is E[X] = ∫[-∞, ∞] x * p(x) dx.
    • Normalization: For any PDF p(x), it must be true that ∫[-∞, ∞] p(x) dx = 1 (the total probability is 1).

  2. Machine Learning:

    • Calculating Marginals: In probabilistic models, to get the marginal probability of one variable, you might need to integrate (or sum, for discrete variables) over other variables. This is called marginalization.
    • Area Under the ROC Curve (AUC): AUC is a common metric for evaluating binary classification models. The ROC curve plots True Positive Rate vs. False Positive Rate, and the AUC is literally the area under this curve, calculated using integration.
    • Signal Processing & Feature Engineering: Integrals can be used to smooth signals, calculate accumulated effects, or define features from raw data (e.g., energy of a signal).
    • Variational Inference: A technique in Bayesian machine learning that often involves optimizing functionals, which can involve integrals.
    • Reinforcement Learning: Calculating expected future rewards can involve integrals if states or actions are continuous.

Common Misconceptions:

  • Forgetting the + C for indefinite integrals. This C can be important if you need to find a specific antiderivative that satisfies certain conditions.
  • Confusing definite and indefinite integrals. One gives a number (area), the other a family of functions.
  • Errors in applying the Fundamental Theorem, especially when substituting limits.

4. Live Coding Demonstration:

We’ll use Python with SciPy for numerical integration and SymPy for symbolic integration.

import numpy as np
from scipy import integrate
import sympy

# --- Indefinite Integrals (Symbolic with SymPy) ---
x_sym = sympy.symbols('x')

# Example 1: ∫(x^3 - 2x + 5) dx
integrand1_sym = x_sym**3 - 2*x_sym + 5
antiderivative1_sym = sympy.integrate(integrand1_sym, x_sym)
print(f"Indefinite integral of {integrand1_sym} is: {antiderivative1_sym} + C")

# Example 2: ∫(1/x) dx
integrand2_sym = 1/x_sym
antiderivative2_sym = sympy.integrate(integrand2_sym, x_sym)
print(f"Indefinite integral of {integrand2_sym} is: {antiderivative2_sym} + C")

# Example 3: ∫(exp(x)) dx
integrand3_sym = sympy.exp(x_sym)
antiderivative3_sym = sympy.integrate(integrand3_sym, x_sym)
print(f"Indefinite integral of {integrand3_sym} is: {antiderivative3_sym} + C")

print("-" * 40)

# --- Definite Integrals (Numerical with SciPy & Symbolic with SymPy) ---

# Example 4: ∫[0, 2] x dx
def f1(x):
    return x

# Numerical integration with SciPy quad
result_f1_scipy, error_f1_scipy = integrate.quad(f1, 0, 2)
print(f"Numerical definite integral of x from 0 to 2 (SciPy): {result_f1_scipy:.4f} (error: {error_f1_scipy:.2e})")

# Symbolic definite integration with SymPy
integrand_f1_sym = x_sym
result_f1_sympy = sympy.integrate(integrand_f1_sym, (x_sym, 0, 2)) # (variable, lower_limit, upper_limit)
print(f"Symbolic definite integral of x from 0 to 2 (SymPy): {result_f1_sympy}")

print("-" * 40)

# Example 5: ∫[0, π] sin(x) dx
# Analytical: -cos(x) | from 0 to π = -cos(π) - (-cos(0)) = -(-1) - (-1) = 1 + 1 = 2
def f2(x):
    return np.sin(x)

# Numerical integration with SciPy quad
result_f2_scipy, error_f2_scipy = integrate.quad(f2, 0, np.pi)
print(f"Numerical definite integral of sin(x) from 0 to pi (SciPy): {result_f2_scipy:.4f} (error: {error_f2_scipy:.2e})")

# Symbolic definite integration with SymPy
integrand_f2_sym = sympy.sin(x_sym)
result_f2_sympy = sympy.integrate(integrand_f2_sym, (x_sym, 0, sympy.pi))
print(f"Symbolic definite integral of sin(x) from 0 to pi (SymPy): {result_f2_sympy}")

print("-" * 40)

# Example 6: Area under a Gaussian PDF (related to probability)
# Normal distribution PDF: (1 / (sigma * sqrt(2*pi))) * exp(- (x-mu)^2 / (2*sigma^2))
# Let mu=0, sigma=1 (standard normal distribution)
def normal_pdf(x, mu=0, sigma=1):
    return (1 / (sigma * np.sqrt(2 * np.pi))) * np.exp(- (x - mu)**2 / (2 * sigma**2))

# Probability P(-1 <= X <= 1) for standard normal
prob_minus1_to_1, _ = integrate.quad(normal_pdf, -1, 1, args=(0, 1))
print(f"P(-1 <= X <= 1) for standard normal: {prob_minus1_to_1:.4f} (approx 68%)")

# Total area under standard normal PDF (should be 1)
total_prob, _ = integrate.quad(normal_pdf, -np.inf, np.inf, args=(0, 1))
print(f"Total area under standard normal PDF: {total_prob:.4f}")

Explanation of Code:

  1. SymPy for Indefinite and Definite Integrals:
    • sympy.symbols('x') creates a symbolic variable.
    • sympy.integrate(integrand, variable) computes the indefinite integral.
    • sympy.integrate(integrand, (variable, lower_limit, upper_limit)) computes the definite integral.
    • SymPy attempts to find exact analytical solutions.
  2. SciPy for Numerical Definite Integrals:
    • scipy.integrate.quad(func, a, b, args=()) is the primary tool for 1D numerical integration (quadrature).
    • func: The Python function to integrate.
    • a, b: Lower and upper limits of integration.
    • args: A tuple of extra arguments to pass to func.
    • It returns a tuple: (estimated_value, estimated_absolute_error).
    • np.inf can be used for infinite limits.
  3. We demonstrate calculating areas and probabilities using these tools.

Common Pitfalls:

  • Syntax errors when defining functions for scipy.integrate.quad.
  • SciPy provides numerical approximations; for exact answers, SymPy is preferred if an analytical solution exists.
  • Forgetting to pass extra arguments to the integrand function via the args parameter in quad.
  • Numerical integration can be slow or inaccurate for highly oscillatory or singular functions.

5. Tasks:

Short Exercise:

  1. Find Indefinite Integrals Manually:
    • ∫(4x^3 + 6x - 2) dx
    • ∫(2e^x + 3/x) dx
    • ∫(2cos(t) - sin(t)) dt
  2. Evaluate Definite Integrals Manually (using the Fundamental Theorem):
    • ∫[1, 3] 2x dx
    • ∫[0, 1] (x^2 + 1) dx
    • ∫[0, π/2] cos(x) dx
  3. Conceptual Question:
    • If p(x) = 1/2 for 0 ≤ x ≤ 2 and p(x) = 0 otherwise, is p(x) a valid probability density function? Why or why not? (Hint: Check the total area).
    • What would ∫[0, 1] p(x) dx represent in this case?
  4. (Optional Python Task):
    • Use SymPy to verify your indefinite integrals from question 1.
    • Use SciPy (and SymPy if you like) to verify your definite integrals from question 2.
    • Consider the function f(x) = x * e^(-x^2). Calculate ∫[0, ∞] f(x) dx using scipy.integrate.quad.

6. Q&A and Wrap-up:

  • Open discussion for any questions about integrals, the Fundamental Theorem, or their uses.
  • Briefly summarize the two main types of integrals and their significance.
  • Preview: We are now equipped with the core calculus tools (derivatives and integrals) needed for many machine learning concepts. Next, we’ll dive deeper into how these are applied in optimization and probability.