Probability - Distributions & Expectation

calendar_today May 4, 2025 schedule 16 min read

1. Welcome and Icebreaker:

Welcome to Day 4! Yesterday, we laid the groundwork for probability with basic concepts, conditional probability, Bayes’ theorem, and random variables. Today, we build on that by exploring some common probability distributions and the important concepts of expectation and variance.

Icebreaker for today: If a Normal distribution, a Binomial distribution, and a Bernoulli distribution were at a party, what kind of conversations do you think they’d be having? What would be their personalities? (e.g., Is Normal a bit… average? Is Bernoulli very decisive?)

Share your creative thoughts! Any lingering questions from PMFs, PDFs, or Bayes’ Theorem before we dive into distributions?

2. Outline the Session:

Today, we’ll get familiar with some of the most frequently encountered probability distributions in data science and machine learning. We’ll also learn how to summarize random variables using expectation and variance.

Agenda:
- Recap: Random Variables (Discrete/Continuous), PMF/PDF.
- Common Discrete Probability Distributions:
  - Bernoulli Distribution: The Single Coin Flip
  - Binomial Distribution: Multiple Coin Flips
- Common Continuous Probability Distributions:
  - Normal (Gaussian) Distribution: The Bell Curve
    - Properties and Importance
    - Standard Normal Distribution
- Key Concepts for Random Variables:
  - Expected Value (Mean): E[X]
    - Definition and Calculation (Discrete & Continuous)
    - Intuition: The “Average” Outcome
  - Variance: Var(X)
    - Definition and Calculation (Discrete & Continuous)
    - Intuition: Measure of Spread or Dispersion
  - Standard Deviation: σ(X)
- Relevance in Machine Learning (e.g., modeling data, initializing weights, understanding errors)
- Live Coding Demonstration (Python):
  - Generating samples from these distributions (NumPy/SciPy).
  - Plotting their PMFs/PDFs.
  - Calculating empirical mean and variance and comparing to theoretical values.
- Hands-on Task: Problems on distributions, expectation, and variance.
- Q&A and Wrap-up
Learning Outcomes: By the end of this session, you should be able to:
- Describe the characteristics of Bernoulli, Binomial, and Normal distributions.
- Understand when each of these distributions might be an appropriate model for a real-world scenario.
- Define and calculate the expected value (mean) of a random variable.
- Define and calculate the variance and standard deviation of a random variable.
- Understand the intuition behind expectation (center) and variance (spread).
- Use Python (NumPy/SciPy) to work with these distributions and calculate their summary statistics.
Structure of the Session: We’ll define each distribution, discuss its properties and uses, then cover expectation and variance. The live coding will focus on practical application with Python libraries.
Expectations for Interactivity: These concepts are foundational. Please ask for clarifications, share examples, and participate actively. The more you engage, the clearer these ideas will become.

3. Concept Explanation:

Recap: Random Variables and PMF/PDF

Remember, a random variable assigns a numerical value to each outcome of a random experiment.

Discrete random variables have countable outcomes, described by a Probability Mass Function (PMF), P(X=x), which gives the probability of each specific outcome. ΣP(X=x) = 1.
Continuous random variables have outcomes over a continuous range, described by a Probability Density Function (PDF), f(x). The area under the PDF curve gives probability, P(a ≤ X ≤ b) = ∫f(x)dx. The total area under the PDF is 1. f(x) itself is not a probability.

Common Discrete Probability Distributions

Bernoulli Distribution (The Single Coin Flip):
- Scenario: Describes an experiment with only two possible outcomes, typically labeled “success” and “failure”.
- Random Variable (X): Takes a value of 1 for success and 0 for failure.
- Parameter (p): The probability of success, P(X=1) = p.
  - Then, the probability of failure is P(X=0) = 1 - p.
- PMF: P(X=x) = p^x × (1-p)^(1-x) for x ∈ {0, 1}.
  - If x=1 (success): p^1 × (1-p)^0 = p.
  - If x=0 (failure): p^0 × (1-p)^1 = 1-p.
- Examples:
  - A single coin flip (Heads=success, Tails=failure).
  - A single component being defective or not.
  - A patient responding to a treatment or not.
  - An email being spam or not spam.
- Expected Value (Mean): E[X] = p
- Variance: Var(X) = p(1-p)
- The Bernoulli distribution is the fundamental building block for the Binomial distribution.
Binomial Distribution (Multiple Coin Flips):
- Scenario: Describes the number of successes in a fixed number of independent Bernoulli trials (n trials), where each trial has the same probability of success (p).
- Random Variable (X): The total number of successes in n trials. X can take values from 0, 1, …, n.
- Parameters:
  - n: The number of trials.
  - p: The probability of success on any single trial.
- PMF: Gives the probability of observing exactly k successes in n trials. P(X=k) = C(n, k) × p^k × (1-p)^(n-k) for k ∈ {0, 1, ..., n}. Where C(n, k) is the binomial coefficient, “n choose k”, calculated as: C(n, k) = n! / [k! × (n-k)!]. This coefficient counts the number of ways to arrange k successes among n trials.
- Conditions for Binomial Distribution:
  1. Fixed number of trials (n).
  2. Each trial is independent.
  3. Each trial has only two outcomes (success/failure).
  4. The probability of success (p) is constant for all trials.
- Examples:
  - The number of Heads in 10 coin flips.
  - The number of defective items in a sample of 20 items from a production line (if defect probability is constant and items are independent).
  - The number of patients who recover out of 50 patients given a treatment with a known success rate.
- Expected Value (Mean): E[X] = np
- Variance: Var(X) = np(1-p)
- Shape: The shape of the Binomial PMF depends on n and p. If p=0.5, it’s symmetric. If p is small, it’s skewed right. If p is large, it’s skewed left. As n gets large, the Binomial distribution can be approximated by the Normal distribution (if np and n(1-p) are not too small).

Common Continuous Probability Distributions

Normal (Gaussian) Distribution (The Bell Curve):
- Scenario: Perhaps the most important and widely used distribution in statistics and machine learning. It describes many natural phenomena and is a cornerstone of statistical inference.
- Random Variable (X): Can take any real value (-∞ to ∞).
- Parameters:
  - μ (mu): The mean of the distribution. It determines the center (peak) of the bell curve.
  - σ² (sigma-squared): The variance of the distribution. (σ is the standard deviation).
    - σ determines the spread or width of the bell curve. A smaller σ means a taller, narrower curve; a larger σ means a shorter, wider curve.
- PDF: f(x | μ, σ) = [1 / (σ√(2π))] × exp(-(x-μ)^2 / (2σ^2)). This formula looks complex, but its shape is the familiar symmetric bell curve.
- Properties and Importance:
  - Symmetric: The curve is perfectly symmetric around its mean μ.
  - Mean, Median, Mode: For a Normal distribution, the mean, median, and mode are all equal (μ).
  - Empirical Rule (68-95-99.7 Rule): For a Normal distribution:
    - Approximately 68% of the data falls within 1 standard deviation of the mean (μ ± σ).
    - Approximately 95% of the data falls within 2 standard deviations of the mean (μ ± 2σ).
    - Approximately 99.7% of the data falls within 3 standard deviations of the mean (μ ± 3σ).
  - Central Limit Theorem (CLT): One of the main reasons for its importance. The CLT states that the sum (or average) of a large number of independent and identically distributed random variables will tend to be Normally distributed, regardless of the original distribution of the variables (under certain conditions). This is why many real-world phenomena that result from multiple small additive effects (e.g., measurement errors, heights, IQ scores) often follow a Normal distribution.
  - Many statistical tests and machine learning models assume Normally distributed data or errors (e.g., Linear Regression errors, Gaussian Naive Bayes).
- Examples:
  - Heights or weights of a population.
  - Measurement errors in experiments.
  - IQ scores.
  - The distribution of sample means (due to CLT).
  - Noise in signals.
- Standard Normal Distribution: A special case of the Normal distribution where μ = 0 and σ = 1.
  - Its PDF is often denoted by φ(z).
  - Any Normal random variable X ~ N(μ, σ²) can be transformed into a standard Normal variable Z ~ N(0, 1) using the z-score transformation: Z = (X - μ) / σ
  - Z-scores tell you how many standard deviations an original value X is away from its mean μ. This is extremely useful for comparing values from different Normal distributions and for using standard Normal tables or functions to find probabilities.

Key Concepts for Random Variables

These are measures that summarize the properties of a probability distribution.

Expected Value (Mean): E[X] or μX
- Definition: The weighted average of all possible values of a random variable X, where the weights are the probabilities (for discrete RVs) or density (for continuous RVs).
- Intuition: It’s the long-run average value of X if you were to repeat the experiment many times. It represents the “center of mass” or the “balancing point” of the distribution.
- Calculation:
  - For a Discrete Random Variable X: E[X] = Σ [x * P(X=x)] (sum over all possible values of x) Example: Fair die roll. X = {1, 2, 3, 4, 5, 6}, P(X=x) = 1/6 for each. E[X] = 1(1/6) + 2(1/6) + 3(1/6) + 4(1/6) + 5(1/6) + 6(1/6) = (1+2+3+4+5+6)/6 = 21/6 = 3.5. Note: The expected value doesn’t have to be a value that X can actually take.
  - For a Continuous Random Variable X: E[X] = ∫-∞∞ [x * f(x)] dx (integral over all possible values of x)
- Properties of Expectation:
  - E[c] = c (where c is a constant)
  - E[cX] = cE[X]
  - E[X + Y] = E[X] + E[Y] (Linearity of Expectation - holds even if X and Y are not independent! This is a very powerful property.)
  - E[X - Y] = E[X] - E[Y]
  - If X and Y are independent, E[XY] = E[X]E[Y]. (This is NOT true if they are dependent).
Variance: Var(X) or σ²X
- Definition: The expected value of the squared difference between the random variable and its mean. It measures how much the values of X typically deviate from the mean E[X].
- Intuition: A measure of the spread, dispersion, or variability of the distribution. A small variance means values tend to be close to the mean; a large variance means values are more spread out.
- Calculation: Var(X) = E[(X - E[X])²] = E[(X - μ)²]
  - For a Discrete Random Variable X: Var(X) = Σ [(x - μ)² * P(X=x)]
  - For a Continuous Random Variable X: Var(X) = ∫-∞∞ [(x - μ)² * f(x)] dx
- Alternative (Computational) Formula: A more convenient way to calculate variance is often: Var(X) = E[X²] - (E[X])² = E[X²] - μ² Where E[X²] = Σ [x² * P(X=x)] (for discrete) or ∫ [x² * f(x)] dx (for continuous).
- Properties of Variance:
  - Var(c) = 0 (where c is a constant, as it doesn’t vary)
  - Var(cX) = c²Var(X)
  - Var(X + c) = Var(X) (adding a constant shifts the distribution but doesn’t change its spread)
  - If X and Y are independent: Var(X + Y) = Var(X) + Var(Y) and Var(X - Y) = Var(X) + Var(Y). (Note: It’s + Var(Y) for subtraction too, because (-1)² = 1 in Var(-Y) = (-1)²Var(Y)). If X and Y are dependent, Var(X + Y) = Var(X) + Var(Y) + 2Cov(X,Y), where Cov is covariance.
Standard Deviation: SD(X) or σX
- Definition: The square root of the variance. SD(X) = σ = √Var(X)
- Intuition: It’s also a measure of spread, but it’s in the same units as the random variable X (and the mean E[X]). This makes it more directly interpretable than variance.
- For a Normal distribution, the standard deviation σ is one of its defining parameters and directly relates to the 68-95-99.7 rule.

Relevance in Machine Learning:

Modeling Data: Choosing appropriate distributions to model features or target variables (e.g., Bernoulli for binary classification, Normal for continuous features that look bell-shaped).
Understanding Errors: Residuals (errors) in regression models are often assumed to be Normally distributed.
Initialization of Weights: In neural networks, initial weights are often drawn from specific distributions (e.g., Normal or Uniform) with carefully chosen means and variances to aid training.
Probabilistic Models: Many models are inherently probabilistic (e.g., Gaussian Mixture Models, Naive Bayes variants based on different feature distributions).
Regularization: Some regularization techniques have Bayesian interpretations involving prior distributions on model parameters.
Anomaly Detection: Outliers might be identified as points that are many standard deviations away from the mean under an assumed distribution.
Hypothesis Testing & Confidence Intervals: These statistical tools, often used to evaluate models or features, rely heavily on understanding distributions and their properties like mean and variance.

Common Misconceptions and Pitfalls:

Assuming Normality: Not all data is Normally distributed. Always visualize your data (e.g., histograms) before assuming a specific distribution.
Interpreting PDF values as probabilities: Remember, for continuous distributions, f(x) is a density. Area under f(x) is probability.
E[X²] vs. (E[X])²: These are different. E[X²] is the mean of the squares, (E[X])² is the square of the mean. Var(X) = E[X²] - (E[X])².
Independence for Variance Sum: Var(X+Y) = Var(X) + Var(Y) only if X and Y are independent.
Standard Deviation vs. Standard Error: Standard deviation measures the spread of data points in a single dataset/population. Standard error (of the mean, for example) measures the variability of a sample statistic (like the sample mean) if you were to take many samples. They are related but distinct.

Understanding these distributions and summary statistics is crucial for making sense of data, building effective models, and interpreting their results.

4. Live Coding Demonstration (Python):

Let’s use Python with NumPy and SciPy to explore these distributions and concepts.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import bernoulli, binom, norm

print("--- Bernoulli Distribution ---")
p_bernoulli = 0.7 # Probability of success

# Generate random samples
bernoulli_samples = bernoulli.rvs(p_bernoulli, size=1000)
# print(f"First 10 Bernoulli samples (p={p_bernoulli}): {bernoulli_samples[:10]}")

# Calculate empirical mean (should be close to p)
empirical_mean_bernoulli = np.mean(bernoulli_samples)
print(f"Theoretical Mean (p): {p_bernoulli}")
print(f"Empirical Mean from {len(bernoulli_samples)} samples: {empirical_mean_bernoulli:.4f}")

# Calculate empirical variance (should be close to p*(1-p))
empirical_var_bernoulli = np.var(bernoulli_samples)
theoretical_var_bernoulli = p_bernoulli * (1 - p_bernoulli)
print(f"Theoretical Variance (p*(1-p)): {theoretical_var_bernoulli:.4f}")
print(f"Empirical Variance from samples: {empirical_var_bernoulli:.4f}")

# PMF
x_bernoulli = [0, 1]
pmf_values_bernoulli = bernoulli.pmf(x_bernoulli, p_bernoulli)
plt.figure(figsize=(6,4))
plt.bar(x_bernoulli, pmf_values_bernoulli, width=0.1, color='cyan')
plt.title(f"Bernoulli PMF (p={p_bernoulli})")
plt.xlabel("Outcome (0=Failure, 1=Success)")
plt.ylabel("Probability")
plt.xticks(x_bernoulli)
# plt.show() # In a real script
print("Bernoulli PMF plot would show two bars at 0 and 1.")


print("\n--- Binomial Distribution ---")
n_binomial = 10   # Number of trials
p_binomial = 0.3  # Probability of success in each trial

# Generate random samples (number of successes in n trials)
binomial_samples = binom.rvs(n_binomial, p_binomial, size=1000)
# print(f"First 10 Binomial samples (n={n_binomial}, p={p_binomial}): {binomial_samples[:10]}")

# Calculate empirical mean (should be close to n*p)
empirical_mean_binomial = np.mean(binomial_samples)
theoretical_mean_binomial = n_binomial * p_binomial
print(f"Theoretical Mean (n*p): {theoretical_mean_binomial}")
print(f"Empirical Mean from {len(binomial_samples)} samples: {empirical_mean_binomial:.4f}")

# Calculate empirical variance (should be close to n*p*(1-p))
empirical_var_binomial = np.var(binomial_samples)
theoretical_var_binomial = n_binomial * p_binomial * (1 - p_binomial)
print(f"Theoretical Variance (n*p*(1-p)): {theoretical_var_binomial:.4f}")
print(f"Empirical Variance from samples: {empirical_var_binomial:.4f}")

# PMF
x_binomial = np.arange(0, n_binomial + 1)
pmf_values_binomial = binom.pmf(x_binomial, n_binomial, p_binomial)
plt.figure(figsize=(8,5))
plt.bar(x_binomial, pmf_values_binomial, color='lightgreen')
plt.title(f"Binomial PMF (n={n_binomial}, p={p_binomial})")
plt.xlabel("Number of Successes (k)")
plt.ylabel("P(X=k)")
plt.xticks(x_binomial)
# plt.show()
print("Binomial PMF plot would show bars for k=0 to n.")


print("\n--- Normal (Gaussian) Distribution ---")
mu_normal = 50    # Mean
sigma_normal = 10 # Standard deviation

# Generate random samples
normal_samples = norm.rvs(loc=mu_normal, scale=sigma_normal, size=10000)
# print(f"First 10 Normal samples (mu={mu_normal}, sigma={sigma_normal}): {normal_samples[:10]}")

# Calculate empirical mean (should be close to mu)
empirical_mean_normal = np.mean(normal_samples)
print(f"Theoretical Mean (mu): {mu_normal}")
print(f"Empirical Mean from {len(normal_samples)} samples: {empirical_mean_normal:.4f}")

# Calculate empirical variance (should be close to sigma^2)
empirical_var_normal = np.var(normal_samples)
theoretical_var_normal = sigma_normal**2
print(f"Theoretical Variance (sigma^2): {theoretical_var_normal}")
print(f"Empirical Variance from samples: {empirical_var_normal:.4f}")

# PDF and Histogram
plt.figure(figsize=(10,6))
plt.hist(normal_samples, bins=50, density=True, alpha=0.7, color='skyblue', label='Histogram of Samples')

# Overlay the theoretical PDF
x_normal_pdf = np.linspace(mu_normal - 4*sigma_normal, mu_normal + 4*sigma_normal, 200)
pdf_values_normal = norm.pdf(x_normal_pdf, loc=mu_normal, scale=sigma_normal)
plt.plot(x_normal_pdf, pdf_values_normal, 'r-', lw=2, label='Theoretical PDF')

plt.title(f"Normal Distribution (μ={mu_normal}, σ={sigma_normal})")
plt.xlabel("Value")
plt.ylabel("Density")
plt.legend()
# plt.show()
print("Normal distribution plot shows a bell curve (PDF) and a histogram of samples.")

# Standard Normal Distribution (Z-scores)
# Convert some samples to Z-scores
sample_values = normal_samples[:5]
z_scores = (sample_values - mu_normal) / sigma_normal
print(f"\nOriginal samples: {sample_values}")
print(f"Corresponding Z-scores: {z_scores}")

# Probability using CDF (Cumulative Distribution Function)
# P(X <= x_val) for a normal distribution
x_val = 60 # e.g., what is P(X <= 60) for our N(50, 10^2)?
prob_less_than_xval = norm.cdf(x_val, loc=mu_normal, scale=sigma_normal)
print(f"P(X <= {x_val}) for N({mu_normal}, {sigma_normal}^2): {prob_less_than_xval:.4f}")
# This corresponds to a Z-score of (60-50)/10 = 1. P(Z <= 1) is approx 0.8413

print("\nNote: Matplotlib plots would typically be displayed in a GUI window.")

Explanation of the Code:

We use scipy.stats which has modules for many distributions (bernoulli, binom, norm).
For each distribution (Bernoulli, Binomial, Normal):
- .rvs(): Generates random variates (samples) from the distribution.
- We calculate the empirical mean (np.mean()) and variance (np.var()) of these samples and compare them to the theoretical formulas.
- .pmf() (for discrete) or .pdf() (for continuous): Calculates the Probability Mass Function or Probability Density Function values.
- We use matplotlib.pyplot to visualize the PMFs (bar charts) and the PDF (line plot, often overlaid on a histogram of samples).
Normal Distribution Specifics:
- loc parameter in scipy.stats.norm corresponds to the mean (μ).
- scale parameter corresponds to the standard deviation (σ).
- We demonstrate calculating Z-scores.
- .cdf(): Calculates the Cumulative Distribution Function, P(X ≤ x), which gives the probability that the random variable is less than or equal to a certain value. This is the area under the PDF to the left of x.

This code allows us to see how theoretical distributions behave in practice with generated samples and how to use Python tools to work with them.

5. Tasks:

Let’s apply these concepts.

Exercise 1: Bernoulli and Binomial A factory produces light bulbs. The probability that a randomly selected bulb is defective is 0.05 (p=0.05).

Bernoulli: If you pick one bulb, let X be a random variable that is 1 if the bulb is defective and 0 otherwise.
- What distribution does X follow? Specify its parameter(s).
- What is E[X]?
- What is Var(X)?
Binomial: You take a sample of 20 bulbs (n=20).
- Let Y be the number of defective bulbs in this sample. What distribution does Y follow? Specify its parameter(s).
- What is the probability that exactly 2 bulbs are defective, P(Y=2)? You can use the PMF formula: P(Y=k) = C(n, k) × p^k × (1-p)^(n-k). (C(20,2) = (20 × 19) / (2 × 1) = 190).
- What is the expected number of defective bulbs, E[Y]?
- What is the variance of the number of defective bulbs, Var(Y)?

Exercise 2: Normal Distribution and Z-scores The scores on a standardized test are Normally distributed with a mean (μ) of 100 and a standard deviation (σ) of 15.

If a student scores 130, what is their Z-score?
What percentage of students score between 85 and 115 (i.e., within ±1 standard deviation of the mean)? (Hint: Empirical Rule)
What percentage of students score below 70 (i.e., P(X < 70))? (Hint: 70 is how many standard deviations below the mean? Use the Empirical Rule or symmetry, or if you have access to a Z-table/calculator, find P(Z < -2)).
Using Python (scipy.stats.norm.cdf), calculate the probability that a randomly selected student scores less than 110.

Exercise 3: Expectation and Variance (Discrete Random Variable) A game involves rolling a biased four-sided die. The possible outcomes are {1, 2, 3, 4}. The probabilities are: P(X=1) = 0.4 P(X=2) = 0.3 P(X=3) = 0.2 P(X=4) = 0.1

Calculate the expected value E[X] of the outcome.
Calculate E[X²].
Using E[X] and E[X²], calculate the variance Var(X) = E[X²] - (E[X])².
What is the standard deviation of X?

Exercise 4: Conceptual Questions

If you increase the number of trials n in a Binomial distribution (keeping p constant), what generally happens to the mean and variance of the number of successes?
If a dataset has a very small standard deviation, what does this tell you about the data points relative to the mean?
Why is the Standard Normal Distribution (μ=0, σ=1) particularly useful?

Take about 20-25 minutes for these. Focus on applying the formulas and understanding the interpretations. We’ll review them together. Good luck!