Machine Learning chevron_right Probability - Basic Concepts & Random Variables

Probability - Basic Concepts & Random Variables

calendar_today schedule 19 min read

1. Welcome and Icebreaker:

Welcome to Day 3! We’ve navigated the structured world of linear algebra, and now we’re venturing into the realm of uncertainty with Probability. It’s a fundamental shift in thinking, but equally crucial for machine learning.

Today’s icebreaker: If you had to explain “probability” to a curious alien who just landed, using only gestures and one common Earth object, what would you do and what object would you use? (A coin? A dice? A weather forecast?)

Let your imagination run wild! Also, any final thoughts or questions on determinants, inverses, or eigenvalues before we switch gears? How did the practice problems go?

2. Outline the Session:

Today, we lay the foundational concepts of probability theory. This will be essential for understanding probabilistic models in machine learning, how models handle uncertainty, and how we evaluate their performance.

  • Agenda:

    • Quick Recap: Why Probability for ML?
    • Core Probability Concepts:
      • Experiment, Sample Space, Events
      • Probability Axioms and Basic Rules
      • Classical, Empirical, and Subjective Probability
    • Conditional Probability and Independence:
      • Understanding P(A|B)
      • The Multiplication Rule
      • Independent vs. Dependent Events
    • Bayes’ Theorem (Introduction):
      • The formula and its components
      • Intuitive understanding: updating beliefs
    • Random Variables:
      • Definition: Mapping outcomes to numbers
      • Discrete Random Variables & Probability Mass Function (PMF)
      • Continuous Random Variables & Probability Density Function (PDF)
    • Relevance in Machine Learning (e.g., Naive Bayes, understanding model outputs as probabilities)
    • Live Coding Demonstration (Conceptual with Python): Simulating events, visualizing PMFs/PDFs (conceptually)
    • Hands-on Task: Basic probability problems
    • Q&A and Wrap-up

  • Learning Outcomes: By the end of this session, you should be able to:

    • Define and provide examples of sample spaces, events, and basic probability.
    • Understand and apply the axioms of probability.
    • Calculate conditional probabilities and determine if events are independent.
    • State Bayes’ Theorem and understand its components and intuitive meaning.
    • Define random variables and differentiate between discrete and continuous types.
    • Understand the concepts of Probability Mass Functions (PMFs) for discrete variables and Probability Density Functions (PDFs) for continuous variables.
    • Appreciate how these probability concepts underpin many machine learning algorithms and interpretations.

  • Structure of the Session: We’ll cover definitions, work through examples, and I’ll use Python to illustrate some ideas (though less computation-heavy than linear algebra, more conceptual simulation).

  • Expectations for Interactivity: Probability can be counter-intuitive at times. Please ask questions, challenge assumptions (politely!), and share your thought processes. This is key to grasping these concepts.

3. Concept Explanation:

Why Probability for Machine Learning?

Machine learning is often about dealing with uncertainty and making predictions or decisions based on incomplete or noisy data. Probability theory provides the mathematical framework to quantify this uncertainty.

  • Modeling Uncertainty: Real-world data is rarely perfect. There’s noise, missing values, and inherent randomness. Probabilistic models can explicitly account for this.
  • Building Generative Models: Some generative models explicitly model a probability distribution or likelihood. GANs instead learn an implicit sampler through adversarial training; standard GANs generally do not provide a tractable explicit PDF or PMF.
  • Quantifying Confidence: Many ML models output probabilities (e.g., “there’s an 80% probability this email is spam”). Understanding probability helps interpret these outputs and set decision thresholds.
  • Loss Functions: Some loss functions in ML are derived from probabilistic principles (e.g., cross-entropy loss for classification is related to maximizing likelihood).
  • Bayesian Methods: A whole branch of machine learning is based on Bayesian statistics, which uses probability to update beliefs as more data becomes available.
  • Algorithm Design & Analysis: Probability is used to analyze the behavior and performance of algorithms (e.g., expected error rates).

In essence, probability helps us move from deterministic statements to nuanced, data-driven inferences.

Core Probability Concepts

  1. Experiment, Sample Space, and Events:

    • Experiment: A process or action whose outcome is uncertain. Examples: flipping a coin, rolling a die, measuring a patient’s temperature, observing tomorrow’s weather.
    • Sample Space (S): The set of all possible distinct outcomes of an experiment. Each outcome is a “sample point.”
      • Coin flip: S = {Heads, Tails}
      • Rolling a six-sided die: S = {1, 2, 3, 4, 5, 6}
      • Measuring temperature (simplified): S = {Hot, Warm, Cold}
    • Event (E): A subset of the sample space. It’s a collection of one or more outcomes that we are interested in.
      • Coin flip, event “getting Heads”: E = {Heads}
      • Rolling a die, event “getting an even number”: E = {2, 4, 6}
      • Rolling a die, event “getting a number greater than 4”: E = {5, 6}

  2. Probability Axioms and Basic Rules: The probability of an event E, denoted P(E), is a number that quantifies the likelihood of E occurring. It adheres to a few fundamental axioms (Kolmogorov Axioms):

    • Axiom 1: Non-negativity: For any event E, P(E) ≥ 0. (Probability cannot be negative).
    • Axiom 2: Normalization: The probability of the entire sample space S is 1. P(S) = 1. (Something from the sample space must happen).
    • Axiom 3: Additivity for Mutually Exclusive Events: If E₁, E₂, E₃, … are a sequence of mutually exclusive events (meaning they cannot happen at the same time, i.e., their intersection is empty), then the probability that at least one of them occurs is the sum of their individual probabilities: P(E₁ ∪ E₂ ∪ E₃ ∪ …) = P(E₁) + P(E₂) + P(E₃) + … (Here, ∪ denotes the union, or “OR” operation).

    Derived Basic Rules:

    • Probability Bounds: For any event E, 0 ≤ P(E) ≤ 1.
    • Probability of an Impossible Event: If an event is impossible (empty set, Ø), P(Ø) = 0.
    • Complement Rule: The probability that an event E does not occur (denoted Ec or E’) is: P(Ec) = 1 - P(E) Example: If P(rain) = 0.3, then P(no rain) = 1 - 0.3 = 0.7.
    • Addition Rule (for any two events, not necessarily mutually exclusive): P(A ∪ B) = P(A) + P(B) - P(A ∩ B) Where P(A ∩ B) is the probability that both A AND B occur (their intersection). If A and B are mutually exclusive, P(A ∩ B) = 0, and this simplifies to Axiom 3 for two events.

  3. Types of Probability (Interpretations):

    • Classical (or Theoretical) Probability: Assumes all outcomes in the sample space are equally likely. P(E) = (Number of favorable outcomes in E) / (Total number of possible outcomes in S) Example: Rolling a fair die, P(getting a 4) = 1/6. This is useful for idealized scenarios like games of chance.
    • Empirical (or Frequentist/Statistical) Probability: Based on observed frequencies from experiments or historical data. P(E) ≈ (Number of times event E occurred) / (Total number of trials) Example: If a coin is flipped 100 times and lands Heads 55 times, the empirical P(Heads) ≈ 55/100 = 0.55. This is widely used in data analysis and machine learning.
    • Subjective Probability: Based on personal belief, intuition, or expert opinion. It’s a degree of confidence. Example: “I think there’s an 80% chance this startup will succeed.” Common in Bayesian statistics and decision-making under uncertainty.

Conditional Probability and Independence

  1. Conditional Probability (P(A|B)):

    • Definition: The probability of event A occurring given that event B has already occurred. It’s read as “the probability of A given B.”
    • It represents how the knowledge that B happened changes our assessment of the likelihood of A.
    • Formula: P(A|B) = P(A ∩ B) / P(B) (provided P(B) > 0) Where P(A ∩ B) is the probability of both A and B occurring.
    • Intuition: We restrict our sample space to only those outcomes where B has occurred. Then, within this new, smaller sample space, we find the proportion of outcomes that also correspond to A.
    • Example: Consider rolling a fair six-sided die. S = {1, 2, 3, 4, 5, 6} Let A = “getting a number less than 4” = {1, 2, 3}. So, P(A) = 3/6 = 1/2. Let B = “getting an even number” = {2, 4, 6}. So, P(B) = 3/6 = 1/2. A ∩ B = “getting an even number less than 4” = {2}. So, P(A ∩ B) = 1/6. Now, what is P(A|B)? (Probability of getting <4 GIVEN it’s even). P(A|B) = P(A ∩ B) / P(B) = (1/6) / (3/6) = (1/6) * (6/3) = 1/3. Intuitive check: If we know the outcome is even (B occurred), our new sample space is {2, 4, 6}. Within this, only {2} satisfies A. So, 1 out of 3 outcomes, hence 1/3.

  2. The Multiplication Rule: Derived directly from the conditional probability formula: P(A ∩ B) = P(A|B) * P(B) Also, P(A ∩ B) = P(B|A) * P(A) This rule is useful for calculating the probability of a sequence of events.

  3. Independent vs. Dependent Events:

    • Independent Events: Two events A and B are independent if the occurrence of one does not affect the probability of the other occurring.
      • Formally: A and B are independent if P(A|B) = P(A).
      • If P(A|B) = P(A), then it must also be true that P(B|A) = P(B).
      • Key Consequence (Multiplication Rule for Independent Events): If A and B are independent, then: P(A ∩ B) = P(A) * P(B)
      • Example: Flipping a fair coin twice. Let A = “first flip is Heads”, B = “second flip is Heads”. P(A) = 0.5, P(B) = 0.5. The outcome of the first flip doesn’t affect the second. So, P(A ∩ B) = P(A) * P(B) = 0.5 * 0.5 = 0.25.
    • Dependent Events: If the occurrence of one event does affect the probability of the other, they are dependent.
      • P(A|B) ≠ P(A)
      • Example: Drawing two cards from a deck without replacement. Let A = “first card is a King”, B = “second card is a King”. P(A) = 4/52. If the first card was a King, then P(B|A) = 3/51 (since there are only 3 Kings left and 51 cards total). If the first card was not a King, then P(B|not A) = 4/51. Clearly, P(B) depends on whether A happened.

Bayes’ Theorem (Introduction)

Bayes’ Theorem is a fundamental result in probability that describes how to update the probability of a hypothesis based on new evidence.

  • The Formula: P(H|E) = [P(E|H) * P(H)] / P(E) Where:

    • P(H|E): Posterior probability – The probability of hypothesis H being true, given the evidence E.
    • P(E|H): Likelihood – The probability of observing evidence E, given that hypothesis H is true.
    • P(H): Prior probability – The initial probability of hypothesis H being true, before observing evidence E.
    • P(E): Probability of evidence – The overall probability of observing evidence E. This can be calculated using the law of total probability: P(E) = P(E|H) * P(H) + P(E|not H) * P(not H).

  • Intuitive Understanding: Updating Beliefs Bayes’ Theorem provides a formal way to combine your prior beliefs (P(H)) with new evidence (E) to arrive at an updated, more informed belief (P(H|E)).

    • If the evidence is more likely under your hypothesis (P(E|H) is high), your belief in the hypothesis increases.
    • If the evidence is very common regardless of the hypothesis (P(E) is high), the evidence doesn’t change your belief as much.

  • Example (Simplified Medical Diagnosis):

    • H = Patient has a specific disease.
    • E = Patient tests positive on a diagnostic test.
    • P(H): Prior probability of having the disease (e.g., prevalence in the population).
    • P(E|H): Likelihood of testing positive if they have the disease (sensitivity of the test).
    • P(E|not H): Likelihood of testing positive if they don’t have the disease (false positive rate).
    • P(H|E): Posterior probability – what we want to find: probability of having the disease given a positive test.

    Bayes’ Theorem is the backbone of Bayesian machine learning models (e.g., Naive Bayes classifier, Bayesian networks) and is used in spam filtering, medical diagnosis, and many other fields.

Random Variables

In many experiments, we are interested in a numerical value associated with the outcome, rather than the outcome itself.

  1. Definition: A random variable (often denoted by an uppercase letter like X, Y, Z) is a variable whose value is a numerical outcome of a random phenomenon.

    • More formally: It’s a function that maps each outcome in the sample space S to a real number.
    • Examples:
      • Experiment: Flip a coin. X = 1 if Heads, X = 0 if Tails.
      • Experiment: Roll two dice. Y = sum of the numbers on the two dice (Y can take values 2, 3, …, 12).
      • Experiment: Measure the height of a randomly selected student. H = height in cm (H can take any positive real value within a range).

  2. Types of Random Variables:

    • Discrete Random Variable:

      • Can take on a finite number of distinct values or a countably infinite number of values (e.g., 0, 1, 2, 3, …).
      • Often arises from counting processes.
      • Examples: Number of heads in 3 coin flips (0, 1, 2, 3), number of defective items in a batch, number of cars passing an intersection in an hour.
      • Probability Mass Function (PMF): For a discrete random variable X, its PMF, denoted p(x) or P(X=x), gives the probability that X is exactly equal to some value x.
        • p(x) = P(X=x)
        • Properties of a PMF:
          1. 0 ≤ p(x) ≤ 1 for all x.
          2. The sum of p(x) over all possible values of x is 1: Σ p(x) = 1.

    • Continuous Random Variable:

      • Can take on any value within a given range or interval (an uncountably infinite number of values).
      • Often arises from measurement processes.
      • Examples: Height of a person, temperature of a room, time until an event occurs.
      • Probability Density Function (PDF): For a continuous random variable X, its PDF, denoted f(x), describes the relative likelihood for the random variable to take on a given value.
        • Important Note: For a continuous variable, P(X=x) = 0 for any specific value x. This is because there are infinitely many possible values in any interval. Instead, we talk about the probability that X falls within an interval [a, b].
        • P(a ≤ X ≤ b) = ∫ab f(x) dx (the area under the PDF curve between a and b).
        • Properties of a PDF:
          1. f(x) ≥ 0 for all x (the density itself is non-negative, but it’s NOT a probability).
          2. The total area under the PDF curve over all possible values of x is 1: ∫-∞∞ f(x) dx = 1.
        • The value f(x) itself is not a probability. It’s a density. Higher f(x) means values around x are more likely.

Relevance in Machine Learning:

  • Naive Bayes Classifier: Uses Bayes’ theorem and assumes features are conditionally independent given the class.
  • Logistic Regression: Outputs probabilities for classification tasks.
  • Probabilistic Graphical Models (e.g., Bayesian Networks, Markov Random Fields): Represent complex probabilistic relationships between many variables.
  • Generative Models: Some models, such as autoregressive models and normalizing flows, provide explicit likelihoods; some variational autoencoders provide likelihood bounds. GANs learn an implicit sampling distribution and generally do not provide a tractable explicit PDF or PMF.
  • Evaluating Models: Concepts like likelihood are used to assess how well a model fits the data.
  • Handling Uncertainty: Many models explicitly output or work with probabilities to represent uncertainty in predictions.

Common Misconceptions and Pitfalls:

  • P(A|B) vs. P(B|A): These are generally not the same (e.g., P(positive test | disease) is not P(disease | positive test) – this is a common error in interpreting medical test results, known as the base rate fallacy).
  • Independence Assumption: Assuming independence when events are actually dependent can lead to incorrect probability calculations. This is a simplification often made in models like Naive Bayes, which can still perform well but is important to be aware of.
  • PDF values are not probabilities: For continuous variables, f(x) can be greater than 1. Only the area under the PDF corresponds to probability.
  • Confusing P(A ∩ B) with P(A ∪ B): Intersection (AND) vs. Union (OR).

Understanding these basic probability concepts is crucial before moving on to specific distributions (like Bernoulli, Binomial, Normal) and statistical concepts like expectation and variance, which we’ll cover next.

4. Live Coding Demonstration (Conceptual with Python):

Since today is more about concepts, the Python part will be less about complex calculations and more about simulating and illustrating ideas.

import random
import numpy as np
import matplotlib.pyplot as plt

# --- Simulating Basic Probability --- 
print("--- Simulating Basic Probability ---")

# Experiment: Rolling a fair six-sided die
sample_space_die = [1, 2, 3, 4, 5, 6]

# Simulate N rolls
N_rolls = 10000
rolls = [random.choice(sample_space_die) for _ in range(N_rolls)]

# Empirical probability of getting a 4
event_get_4 = rolls.count(4)
P_get_4_empirical = event_get_4 / N_rolls
print(f"Simulated rolls: {N_rolls}")
print(f"Times a 4 was rolled: {event_get_4}")
print(f"Empirical P(getting a 4): {P_get_4_empirical:.4f} (Theoretical: {1/6:.4f})")

# Empirical probability of getting an even number
event_get_even = sum(1 for roll in rolls if roll % 2 == 0)
P_get_even_empirical = event_get_even / N_rolls
print(f"Times an even number was rolled: {event_get_even}")
print(f"Empirical P(getting an even number): {P_get_even_empirical:.4f} (Theoretical: {3/6:.4f})")

# --- Conditional Probability Example (Simulated) ---
print("\n--- Conditional Probability (Simulated) ---")
# Let A = "roll is < 4" (i.e., 1, 2, 3)
# Let B = "roll is even" (i.e., 2, 4, 6)
# We want to find P(A|B) = P(A and B) / P(B)

# Outcomes where B occurred (roll is even)
rolls_where_B_occurred = [roll for roll in rolls if roll % 2 == 0]
count_B = len(rolls_where_B_occurred)

# Outcomes where A AND B occurred (roll is < 4 AND even, i.e., roll is 2)
rolls_where_A_and_B_occurred = [roll for roll in rolls_where_B_occurred if roll < 4]
count_A_and_B = len(rolls_where_A_and_B_occurred)

if count_B > 0:
    P_A_given_B_empirical = count_A_and_B / count_B
    print(f"Number of even rolls (B occurred): {count_B}")
    print(f"Number of rolls that were even AND < 4 (A and B occurred): {count_A_and_B}")
    print(f"Empirical P(roll < 4 | roll is even): {P_A_given_B_empirical:.4f} (Theoretical: {1/3:.4f})")
else:
    print("No even rolls observed in simulation to calculate conditional probability.")

# --- Random Variables & PMF (Conceptual for Discrete) ---
print("\n--- Discrete Random Variable & PMF (Conceptual) ---")
# Experiment: Flip 3 fair coins. X = number of Heads.
# Possible values for X: 0, 1, 2, 3
# Theoretical probabilities (can be found using binomial formula or by listing outcomes):
# S = {HHH, HHT, HTH, THH, HTT, THT, TTH, TTT} (8 outcomes)
# P(X=0) (TTT) = 1/8
# P(X=1) (HTT, THT, TTH) = 3/8
# P(X=2) (HHT, HTH, THH) = 3/8
# P(X=3) (HHH) = 1/8

values_X = [0, 1, 2, 3]
pmf_X = [1/8, 3/8, 3/8, 1/8]

plt.figure(figsize=(6, 4))
plt.bar(values_X, pmf_X, width=0.1, color='skyblue')
plt.xlabel("X (Number of Heads in 3 Flips)")
plt.ylabel("P(X=x) (Probability Mass)")
plt.title("PMF of Number of Heads in 3 Fair Coin Flips")
plt.xticks(values_X)
plt.grid(axis='y', linestyle='--')
# plt.show() # In a real script, you'd show this. For here, we'll just describe.
print("PMF plot for 'Number of Heads in 3 Flips' would show bars at 0, 1, 2, 3 with heights 1/8, 3/8, 3/8, 1/8.")

# --- Continuous Random Variable & PDF (Conceptual) ---
print("\n--- Continuous Random Variable & PDF (Conceptual) ---")
# Example: Heights of adult males (often modeled by a Normal/Gaussian distribution)
# We can't plot a true PDF from finite samples easily without estimation techniques (like KDE)
# But we can show a histogram of samples, which approximates the shape of the PDF.

# Simulate some data that might look like heights (e.g., from a normal distribution)
mu, sigma = 175, 10 # mean and standard deviation (example values in cm)
height_samples = np.random.normal(mu, sigma, 1000)

plt.figure(figsize=(8, 5))
plt.hist(height_samples, bins=30, density=True, alpha=0.7, color='lightcoral', label='Histogram of Samples')
# `density=True` normalizes the histogram so the area sums to 1, approximating a PDF.

# For illustration, overlay the true PDF if we knew the parameters
from scipy.stats import norm
x_axis = np.linspace(mu - 4*sigma, mu + 4*sigma, 200)
plt.plot(x_axis, norm.pdf(x_axis, mu, sigma), 'r-', lw=2, label='True PDF (if known)')

plt.xlabel("Height (cm)")
plt.ylabel("Density")
plt.title("Conceptual PDF of Adult Male Heights (Approximated by Histogram)")
plt.legend()
plt.grid(axis='y', linestyle='--')
# plt.show()
print("PDF plot for 'Adult Male Heights' would show a bell-shaped curve. A histogram of samples would approximate this shape.")

print("\nNote: Matplotlib plots would typically be displayed in a GUI window.")

Explanation of the Code:

  • Basic Probability Simulation: We simulate rolling a die many times and calculate empirical probabilities, comparing them to theoretical ones.
  • Conditional Probability Simulation: We filter the simulated die rolls to demonstrate P(A|B) empirically.
  • PMF Plot (Conceptual): We define the theoretical PMF for the number of heads in 3 coin flips and use Matplotlib to create a bar chart representing it. This visually shows the probability for each discrete outcome.
  • PDF Plot (Conceptual): We simulate data from a normal distribution (often used for continuous variables like height). We then plot a histogram of this data with density=True. This normalized histogram gives an approximation of the underlying PDF. We also overlay the true PDF curve for comparison.
  • The actual plt.show() commands are commented out as they would pop up GUI windows. The print statements describe what would be seen.

This code helps visualize how theoretical probability concepts can be observed through simulation and how PMFs/PDFs represent the distributions of random variables.

5. Tasks:

Let’s test your understanding with some problems. Focus on setting up the problem correctly using the definitions and rules we discussed.

Exercise 1: Sample Spaces and Events An experiment consists of drawing one card from a standard 52-card deck.

  1. Describe the sample space S.
  2. Let A be the event “the card drawn is a King.” List the outcomes in A. What is P(A)?
  3. Let B be the event “the card drawn is a Heart.” List the outcomes in B. What is P(B)?
  4. What is the event A ∩ B (A and B)? List its outcomes. What is P(A ∩ B)?
  5. Using the addition rule, what is P(A ∪ B) (the probability of drawing a King OR a Heart)?

Exercise 2: Conditional Probability and Independence Using the same 52-card deck experiment:

  1. What is P(A|B)? (The probability of drawing a King GIVEN that the card is a Heart).
  2. Are events A (drawing a King) and B (drawing a Heart) independent? Justify your answer by comparing P(A|B) with P(A) or by checking if P(A ∩ B) = P(A)P(B).

Exercise 3: Bayes’ Theorem (Simple Scenario) Suppose there are two bags of marbles.

  • Bag 1 contains 3 red marbles and 7 blue marbles (10 total).
  • Bag 2 contains 6 red marbles and 4 blue marbles (10 total).

You randomly pick a bag (P(Bag 1) = 0.5, P(Bag 2) = 0.5). Then, you randomly draw one marble from the chosen bag.

  1. Let R be the event that you draw a red marble.
    • What is P(R | Bag 1) (probability of red GIVEN you chose Bag 1)?
    • What is P(R | Bag 2) (probability of red GIVEN you chose Bag 2)?
  2. Suppose you drew a red marble. Now, you want to find P(Bag 1 | R) – the probability that you picked Bag 1 GIVEN that you drew a red marble. Use Bayes’ Theorem: P(Bag 1 | R) = [P(R | Bag 1) * P(Bag 1)] / P(R) To do this, you first need to calculate P(R) (the overall probability of drawing a red marble). You can use the law of total probability: P(R) = P(R | Bag 1) * P(Bag 1) + P(R | Bag 2) * P(Bag 2) Calculate P(R), and then calculate P(Bag 1 | R).

Exercise 4: Random Variables (Discrete vs. Continuous) For each of the following, state whether it is a discrete or continuous random variable:

  1. The number of emails you receive in a day.
  2. The exact time (e.g., 3.4567 seconds) it takes for a web page to load.
  3. The number of students in a randomly selected classroom who have blue eyes.
  4. The weight of a randomly selected apple.
  5. The outcome of rolling a 20-sided die.

Exercise 5: PMF/PDF Conceptual Question

  1. If X is a discrete random variable representing the score on a fair 6-sided die, what would P(X=3.5) be? Why?
  2. If Y is a continuous random variable representing the exact height of a randomly chosen adult, what would P(Y = 170.000000 cm) be? Why? How would you express the likelihood of someone being around 170cm?

These problems cover the main topics from today. Take about 20-25 minutes. Think carefully about the definitions. We’ll go over them together. Good luck!