Unsupervised Learning - Dimensionality Reduction (PCA)

calendar_today May 17, 2025 schedule 18 min read

1. Welcome and Icebreaker:

Welcome to Day 17 of our machine learning adventure! Yesterday, we explored K-Means clustering, a way to find groups in unlabeled data. Today, we continue our journey into unsupervised learning by tackling another incredibly important concept: Dimensionality Reduction, with a special focus on Principal Component Analysis (PCA).

For our icebreaker today: Think about a time you were faced with a very complex problem or a story with too many details. How did you try to simplify it to grasp the main points or the bigger picture? Did you focus on certain aspects and ignore others? Share your experiences!

Your approaches to simplification are very relevant to what we’ll discuss. Often, datasets come with a vast number of features (dimensions), making them hard to work with, visualize, or even understand. Dimensionality reduction techniques, like PCA, help us simplify these datasets while trying to retain the most crucial information.

2. Outline the Session:

Our goal today is to understand why and how we reduce the dimensionality of data, with a deep dive into PCA.

Our Agenda:

Recap of Unsupervised Learning.
Introduction to Dimensionality Reduction:
- What is it?
- Why is it necessary? The “Curse of Dimensionality.”
- Benefits of reducing dimensions.
- Brief overview of approaches (Feature Selection vs. Feature Extraction).
Principal Component Analysis (PCA) - The Core Idea:
- Intuition: Finding new axes that capture the most variance in the data.
- Key Terminology: Variance, Covariance, Eigenvectors, Eigenvalues, Principal Components.
- Geometric interpretation: Rotation and projection.
The PCA Algorithm - Step-by-Step (Conceptual):
- Centering, with optional feature standardization when appropriate.
- Covariance Matrix Computation.
- Eigendecomposition (Eigenvectors & Eigenvalues).
- Selecting Principal Components.
- Data Transformation.
Choosing the Number of Principal Components.
- Explained Variance Ratio & Cumulative Explained Variance.
- Scree Plots.
Strengths and Weaknesses of PCA.
Applications of PCA.

Learning Outcomes: By the end of this session, you should be able to:

Explain what dimensionality reduction is and why it’s often crucial.
Describe the challenges posed by the “curse of dimensionality.”
Articulate the core intuition behind Principal Component Analysis (PCA).
Understand the roles of variance, covariance, eigenvectors, and eigenvalues in PCA.
Outline the main steps involved in performing PCA.
Discuss common methods for deciding how many principal components to retain.
List the key advantages and limitations of using PCA.
Identify common applications where PCA is beneficial.

Structure of the Session: We’ll start with the general need for dimensionality reduction, then zoom into PCA. We’ll focus on the conceptual understanding and intuition behind PCA, supported by visual ideas, rather than getting bogged down in complex linear algebra derivations. We’ll also discuss how to apply it practically using libraries like scikit-learn. Resources like Andrew Ng’s ML course (Week 8) and StatQuest (PCA) offer excellent supplementary material.

Interactivity: Your questions and insights are vital! The practice task for today, as noted in lesson.md, is to “Visualize a 2D dataset and try to identify the principal components intuitively.” We will discuss this to help solidify the core concepts.

3. Concept Explanation:

Recap of Unsupervised Learning: As we learned with K-Means, unsupervised learning deals with data that doesn’t have predefined labels. The goal is to discover hidden patterns, structures, or relationships within the data itself. Clustering helps us find groups. Dimensionality reduction, another key unsupervised task, helps us simplify the data’s representation.

Introduction to Dimensionality Reduction:

What is Dimensionality Reduction? Dimensionality reduction is the process of transforming data from a high-dimensional space into a lower-dimensional space while retaining some meaningful properties of the original data. In simpler terms, it’s about reducing the number of features (variables or dimensions) in a dataset.

Imagine you have a spreadsheet with hundreds or even thousands of columns (features) for each data point (row). Dimensionality reduction aims to create a new spreadsheet with fewer columns, but these new columns are carefully constructed to capture the essence of the original data.

Why is Dimensionality Reduction Necessary? The “Curse of Dimensionality”: The term “curse of dimensionality” refers to various problems that arise when working with high-dimensional data. As the number of features increases:

Data Sparsity: The data points become increasingly sparse in the high-dimensional space. To maintain the same density of data points as in lower dimensions, an exponential increase in the number of data points is required. This means that in high dimensions, most data points are far away from each other, making it difficult to find local patterns or neighbors.
Distance Metrics Lose Meaning: Many algorithms (like K-Means, KNN) rely on distance metrics (e.g., Euclidean distance). In high dimensions, the distances between pairs of points can become very similar, making it hard to distinguish between close and far points. This is sometimes called distance concentration.
Increased Computational Cost: More features mean more computations. Algorithms take longer to train, and predictions can be slower.
Overfitting Risk in Supervised Models: With many features, supervised models have more parameters to learn. This increases the risk of the model fitting the noise in the training data (overfitting) rather than the true underlying signal, leading to poor generalization on unseen data.
Difficulty in Visualization and Interpretation: Humans are good at visualizing data in 2D or 3D. Beyond that, it becomes impossible to directly visualize and intuitively understand the data’s structure.
Multicollinearity: In high-dimensional data, features are often correlated with each other (multicollinearity). This redundancy can make models unstable and harder to interpret.

Benefits of Dimensionality Reduction: Addressing the curse of dimensionality provides several benefits:

Reduced Storage Space: Fewer features mean less data to store.
Faster Computation: Algorithms run faster on data with fewer dimensions.
Improved Model Performance: By removing irrelevant noise and redundancy, dimensionality reduction can sometimes lead to more robust and accurate supervised models.
Enhanced Data Visualization: Reducing data to 2D or 3D allows for visual exploration and identification of patterns.
Better Interpretability (sometimes): While new features might be combinations of old ones, understanding the main sources of variation can be insightful.
Noise Reduction: Can help filter out noise by focusing on components with higher variance.

Approaches to Dimensionality Reduction (Brief Overview): There are two main approaches:

Feature Selection: We select a subset of the original features and discard the rest. Methods include filter methods (e.g., based on correlation or mutual information), wrapper methods (using a model to evaluate feature subsets), and embedded methods (feature selection as part of model training, like L1 regularization).
Feature Extraction (or Feature Projection): We create a new, smaller set of features by combining or transforming the original features. These new features (components) are intended to capture the most important information from the original set. Principal Component Analysis (PCA) is a prime example of feature extraction. Other examples include Linear Discriminant Analysis (LDA, which is supervised), t-SNE, and UMAP.

Principal Component Analysis (PCA) - The Core Idea:

PCA is arguably the most popular and widely used linear dimensionality reduction technique. Its main goal is to transform a dataset with many features into a smaller set of new features, called principal components (PCs), suchthat these PCs are uncorrelated and ordered by the amount of original variance they capture.

Intuition: Imagine your data points scattered in a 2D plane, forming an elongated cloud. PCA tries to find a new set of axes for this data.

The first principal component (PC1) is the direction (axis) in the data along which the data varies the most (i.e., has the maximum variance). If you were to project all data points onto this line, the spread of these projected points would be maximized.
The second principal component (PC2) is the direction orthogonal (perpendicular) to PC1 that captures the maximum remaining variance. In our 2D example, there’s only one direction perpendicular to PC1.
If we were in 3D, PC3 would be orthogonal to both PC1 and PC2 and capture the next largest variance, and so on.

Essentially, PCA rotates the original axes to a new set of axes (the principal components) that are better aligned with the data’s spread. It then allows us to discard the components that capture the least variance, assuming that variance corresponds to information.

Analogy: Think of a 3D object. If you want to represent it with a 2D shadow, you’d shine a light on it. PCA is like finding the best angle for the light source so that the resulting 2D shadow preserves as much of the object’s shape and structure (variance) as possible.

Key Terminology (Conceptual Understanding):

Variance: A measure of the spread or dispersion of data points along a particular dimension. PCA seeks to find dimensions (principal components) with high variance.
Covariance: A measure of how two variables change together. If features are highly correlated (high positive or negative covariance), they might be redundant. The covariance matrix of the dataset is fundamental to PCA, as it describes the variance of each feature and the covariance between pairs of features.
Eigenvectors and Eigenvalues: These are mathematical concepts from linear algebra that are crucial for PCA. They are derived from the covariance matrix of the data.
- Eigenvectors: These represent the directions of the principal components. They are unit vectors (length of 1) and are orthogonal to each other.
- Eigenvalues: Each eigenvector has a corresponding eigenvalue. The eigenvalue indicates the amount of variance in the data that is captured by its corresponding eigenvector (principal component). Larger eigenvalues mean that the corresponding principal component explains more variance.
- The principal components are, in fact, the eigenvectors of the covariance matrix, sorted in descending order of their corresponding eigenvalues.

Geometric Interpretation: PCA can be seen as a process that:

Shifts the data so that its mean is at the origin.
Rotates the coordinate axes so that the new axes (the principal components) are aligned with the directions of maximum variance in the data. These new axes are orthogonal.
Projects the data onto a subset of these new axes (the ones corresponding to the largest eigenvalues/variances) to achieve dimensionality reduction.

The PCA Algorithm - Step-by-Step (Conceptual):

Center the Data: Subtract each feature’s mean. Centering is part of PCA; scikit-learn’s PCA does this automatically.
- For feature j: X_centered_j = X_j - mean(X_j)
Optionally Standardize Feature Scales: Standardize when features have different units or when equal variance weighting is appropriate. This is a separate choice that changes the relative weight of features; it is not required by PCA.
- For feature j: X_standardized_j = (X_j - mean(X_j)) / std_dev(X_j)
- Compute the covariance matrix from the centered data, or from the standardized data if you chose that preprocessing. Call the selected version X_prepared in the projection step below.
Calculate Eigenvectors and Eigenvalues (Eigendecomposition): Perform eigendecomposition on the covariance matrix. This process finds the eigenvectors (v) and eigenvalues (λ) that satisfy the equation: Σv = λv.
- The eigenvectors represent the principal component directions.
- The eigenvalues represent the magnitude of variance along these directions.
Sort Eigenvectors by Eigenvalues: Sort the eigenvalues in descending order. Rearrange the corresponding eigenvectors accordingly. The eigenvector with the largest eigenvalue is the first principal component (PC1), the one with the second-largest eigenvalue is PC2, and so on.
Choose the Number of Principal Components (k): Decide how many of the top principal components to keep. This k will be the new dimensionality of your data (k < d). We’ll discuss how to choose k in the next section.
Construct the Projection Matrix (W): Create a matrix W whose columns are the k selected eigenvectors (the ones corresponding to the k largest eigenvalues).
Transform the Data: Project the centered data, or the standardized data if you chose that preprocessing, onto the new k-dimensional subspace by multiplying it with the projection matrix W:
- X_pca = X_prepared @ W, where X_prepared is the centered or standardized data used to compute the components. The resulting X_pca is your new dataset with k features (the principal components).

Choosing the Number of Principal Components (k): This is a crucial decision in PCA. We want to reduce dimensions but also retain as much significant information (variance) as possible.

Explained Variance Ratio:
- For each principal component, its explained variance ratio is its eigenvalue divided by the sum of all eigenvalues. This tells you the proportion of the total dataset variance that PC accounts for.
- Explained Variance Ratio (PC_i) = Eigenvalue_i / Σ(All Eigenvalues)
- Cumulative Explained Variance: Plot the cumulative sum of explained variance ratios as you add more principal components (from PC1, PC1+PC2, PC1+PC2+PC3, etc.).
- You can then choose k such that a desired percentage of the total variance is retained (e.g., 90%, 95%, or 99%). For example, find the smallest k that explains at least 95% of the variance.
Scree Plot:
- Plot the eigenvalues themselves in descending order.
- Typically, the eigenvalues will drop off sharply and then start to level out. The point where the drop slows down, forming an “elbow,” is often considered an indicator of a good k. Components beyond this elbow might be capturing mostly noise.
- This method can be somewhat subjective.
Domain Knowledge or Specific Application Needs:
- If you are reducing dimensions for visualization, you’ll choose k=2 or k=3.
- Sometimes, the number of components is chosen based on constraints of a subsequent algorithm or specific insights required.

Strengths of PCA:

Reduces Dimensionality: Effectively decreases the number of features, leading to faster model training, reduced storage needs, and mitigation of the curse of dimensionality.
Noise Reduction: By discarding components with low variance (often associated with noise), PCA can help clean the data.
Multicollinearity Removal: The resulting principal components are orthogonal (uncorrelated), which can be beneficial for some algorithms that are sensitive to multicollinearity (e.g., Linear Regression).
Data Compression: Captures the most important information in fewer dimensions.
No Need to Pre-select K (like in K-Means): You decide on the number of components based on explained variance or other criteria after seeing the eigenvalue spectrum.
Standardized Method: Widely available and understood.

Weaknesses of PCA:

Interpretability of Components: Principal components are linear combinations of the original features. This can make them harder to interpret than the original, domain-specific features. For example, if PC1 is 0.6*featureA - 0.4*featureB + 0.7*featureC, its direct meaning might be obscure.
Information Loss: Dimensionality reduction inherently involves some information loss. The key is to lose as little important information as possible.
Sensitivity to Feature Scales: PCA centers features but does not automatically scale them to unit variance. Features with larger variances dominate the principal components. Standardize only when equal feature weighting is appropriate for the task.
Assumes Linear Relationships: PCA is a linear projection method. It may not be effective if the underlying structure of the data is highly non-linear. For non-linear dimensionality reduction, techniques like t-SNE, UMAP, or Kernel PCA are used.
Variance Might Not Always Equal Importance: PCA assumes that directions with higher variance are more important. This is often true, but for some specific supervised tasks, a low-variance component might still hold crucial information for class separation.
Sensitive to Outliers: Outliers can heavily influence the covariance matrix and thus the principal components.

Applications of PCA: PCA is used in a wide array of fields:

Image Compression: Reducing the data needed to store images by representing them with fewer principal components.
Facial Recognition (Eigenfaces): One of the classic applications where faces are represented by a set of principal components derived from a large dataset of face images.
Bioinformatics: Analyzing gene expression data (microarrays), where datasets often have thousands of genes (features) and relatively few samples.
Finance: Identifying key factors driving stock returns or portfolio risk.
Data Visualization: Reducing high-dimensional data to 2D or 3D for plotting and visual exploration.
Preprocessing for Supervised Learning: Reducing the number of input features to a supervised model to improve training speed, reduce overfitting, and sometimes improve accuracy.
Noise Reduction in Signals and Images.

Common Misconceptions:

PCA is Feature Selection: PCA is not feature selection. Feature selection picks a subset of original features. PCA transforms original features into a new set of (fewer) features.
High Variance Always Means High Importance for a Specific Task: PCA is unsupervised. It finds directions of maximum variance in the data itself. While this often correlates with importance for a supervised task, it’s not guaranteed. A low-variance PC could, in some cases, be crucial for separating classes in a classification problem (though LDA would be more suited for that specific goal).

PCA is a powerful tool for understanding and simplifying complex datasets. Its ability to capture the essence of data in fewer dimensions makes it invaluable in the machine learning pipeline.

4. Live Coding Demonstration:

We’ll use Python with scikit-learn to demonstrate PCA. We can use the Iris dataset, which has 4 features, and reduce it to 2 principal components for visualization.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris

# --- 1. Load Iris Dataset ---
iris = load_iris()
X = iris.data
y = iris.target # We'll use this for coloring the plot, not for PCA itself
feature_names = iris.feature_names

# Convert to DataFrame for easier inspection (optional)
# X_df = pd.DataFrame(X, columns=feature_names)
# print("Original Data Head:")
# print(X_df.head())
# print(f"\nOriginal shape: {X.shape}")

# --- 2. Optional: standardize the Data ---
# StandardScaler is used here to give each feature unit variance. PCA itself centers input but does not scale each feature.
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Scaled data DataFrame (optional for inspection)
# X_scaled_df = pd.DataFrame(X_scaled, columns=feature_names)
# print("\nScaled Data Head:")
# print(X_scaled_df.head())

# --- 3. Apply PCA ---
# Let's reduce to 2 principal components for visualization.
N_COMPONENTS = 2
pca = PCA(n_components=N_COMPONENTS)
X_pca = pca.fit_transform(X_scaled)

# Create a DataFrame for the PCA results (optional)
pca_df = pd.DataFrame(data = X_pca, columns = ['principal component 1', 'principal component 2'])
final_df = pd.concat([pca_df, pd.Series(y, name='target')], axis = 1)

# print(f"\nShape after PCA: {X_pca.shape}")
# print("\nPCA Data Head:")
# print(pca_df.head())

# --- 4. Explained Variance ---
print(f"\nExplained variance ratio by each component: {pca.explained_variance_ratio_}")
print(f"Total explained variance by {N_COMPONENTS} components: {np.sum(pca.explained_variance_ratio_):.4f}")

# To see how many components are needed for, say, 95% variance:
# pca_full = PCA().fit(X_scaled) # Fit PCA without specifying n_components initially
# plt.figure(figsize=(8,4))
# plt.plot(np.cumsum(pca_full.explained_variance_ratio_))
# plt.xlabel('Number of Components')
# plt.ylabel('Cumulative Explained Variance')
# plt.title('Cumulative Explained Variance by PCA Components')
# plt.grid(True)
# plt.axhline(y=0.95, color='r', linestyle='--') # 95% threshold
# plt.axhline(y=0.99, color='g', linestyle='--') # 99% threshold
# plt.show()

# --- 5. Visualize 2D PCA Results ---
plt.figure(figsize=(10, 7))
scatter = plt.scatter(final_df['principal component 1'], final_df['principal component 2'], c=final_df['target'], cmap='viridis', edgecolor='k', alpha=0.8)
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.title(f'{N_COMPONENTS}-Component PCA of Iris Dataset')

# Add legend
legend_labels = [iris.target_names[i] for i in np.unique(y)]
plt.legend(handles=scatter.legend_elements()[0], labels=legend_labels, title="Classes")
plt.grid(True)
# plt.show() # Uncomment to display if running locally

# --- Understanding the Components (Optional) ---
# pca.components_ shows the contribution of each original feature to each principal component.
# print("\nPCA Components (Eigenvectors):")
# print(pca.components_)
# This means PC1 = pca.components_[0,0]*feature1 + pca.components_[0,1]*feature2 + ...

# --- Example: PCA on a synthetic correlated dataset ---
# Generate 2D correlated data
np.random.seed(42)
mean = [0, 0]
cov = [[5, 4], [4, 5]]  # Correlated data
X_corr, _ = np.random.multivariate_normal(mean, cov, 300, check_valid='warn').T
X_corr = np.vstack((X_corr[0], X_corr[1])).T # Make it (300, 2)

# Standardize
X_corr_scaled = StandardScaler().fit_transform(X_corr)

# Apply PCA
pca_corr = PCA(n_components=2)
X_corr_pca = pca_corr.fit_transform(X_corr_scaled)

# Plot original and PCA components
plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)
plt.scatter(X_corr_scaled[:, 0], X_corr_scaled[:, 1], alpha=0.7)
plt.title('Original Correlated Data (Scaled)')
plt.xlabel('Feature 1 (scaled)')
plt.ylabel('Feature 2 (scaled)')
plt.axis('equal')

# Plot PCA components as vectors
for i, (comp, var) in enumerate(zip(pca_corr.components_, pca_corr.explained_variance_)):
    # Scale components by their explained variance for visualization
    comp = comp * np.sqrt(var) * 2 # Arbitrary scaling for visibility
    plt.arrow(0, 0, comp[0], comp[1], head_width=0.1, head_length=0.1, fc='red', ec='red', label=f'PC{i+1}' if i==0 else None)
if pca_corr.n_components_ == 2: plt.legend()

plt.subplot(1, 2, 2)
plt.scatter(X_corr_pca[:, 0], X_corr_pca[:, 1], alpha=0.7)
plt.title('Data Transformed by PCA')
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.axis('equal')
plt.tight_layout()
# plt.show() # Uncomment to display if running locally

print(f"\nExplained variance for correlated data: {pca_corr.explained_variance_ratio_}")

Explanation of Code:

Load Data: We load the Iris dataset (4 features, 3 classes).
Optional Standardization: This example uses StandardScaler to give features zero mean and unit variance before PCA. It is a deliberate weighting choice, not a required PCA step; PCA centers input data automatically.
Apply PCA: We initialize PCA from sklearn.decomposition with n_components=2 to reduce the data to 2 dimensions. Then, fit_transform is called on the scaled data.
Explained Variance: pca.explained_variance_ratio_ shows the percentage of variance explained by each of the selected components. We print this to see how much information is retained.
- The commented-out section shows how to plot cumulative explained variance to help choose n_components if it wasn’t predetermined (e.g., to capture 95% of variance).
Visualize Results: The 2D X_pca data is plotted. Points are colored by their original Iris class to see if PCA has helped in separating them or revealing structure.
Understanding Components (Optional): pca.components_ gives the actual eigenvectors. Each row corresponds to a principal component, and the values in that row are the weights for each original feature in that component.
Synthetic Correlated Data Example: This part generates 2D data with a clear correlation. PCA is applied, and the principal component vectors are overlaid on the original data plot. This visually demonstrates how PC1 aligns with the direction of maximum variance and PC2 is orthogonal to it. The second plot shows the data after being transformed (rotated) to align with these principal components.

This demonstration highlights the importance of scaling, how to apply PCA, how to check explained variance, and how to visualize the results.

5. Tasks:

Short Exercise: Intuitive PCA on 2D Data

As mentioned in lesson.md, the practice is to: “Visualize a 2D dataset and try to identify the principal components intuitively.”

Sketch or Imagine: Consider a scatter plot of 2D data points that form an elongated, tilted oval shape (indicating correlation between the two original features).
- You can quickly sketch this on a piece of paper.
Identify PC1: Draw a line (an axis) through the center of this data cloud that best represents the direction of its longest spread. This line is your intuitive PC1. Why did you choose this direction? (It should be the direction where the data points are most stretched out).
Identify PC2: Now, draw another line (axis) through the center of the data cloud that is perpendicular (orthogonal) to your PC1. This is your intuitive PC2. It should represent the direction of the next largest spread, perpendicular to PC1.
Reflection: If you were to reduce this 2D data to 1D using PCA, you would project all data points onto PC1. The positions of these projected points along PC1 would be their new single feature values. How much information about the original spread do you think is captured by PC1 alone in your sketch? How much is captured by PC2?

This exercise helps build a gut feeling for what PCA is trying to achieve: finding the most informative axes in your data.

This concludes our exploration of Principal Component Analysis. PCA is a fundamental technique for dimensionality reduction, offering benefits in data compression, visualization, and preprocessing for other machine learning algorithms. Next, we’ll look at the overall ML project workflow and get a more formal introduction to the scikit-learn library.