Other Supervised Learning Algorithms (Brief Overview)

calendar_today May 15, 2025 schedule 18 min read

1. Welcome and Icebreaker:

Welcome to Day 15 of our Machine Learning journey! Today, we’re going to take a tour of some other important supervised learning algorithms. Before we dive in, let’s have a quick icebreaker: If your favorite algorithm was a superhero, what would its superpower be and why? Share your thoughts in the chat!

It’s great to see your creative answers! This kind of thinking – associating concepts with relatable ideas – can actually help in understanding and remembering them. Many of you might have encountered some of these algorithms before, perhaps in a project or a course. Feel free to share any prior experiences or initial thoughts you have about K-Nearest Neighbors, Decision Trees, or Support Vector Machines.

2. Outline the Session:

Today’s session is designed to give you a conceptual understanding of three popular and powerful supervised learning algorithms: K-Nearest Neighbors (KNN), Decision Trees, and Support Vector Machines (SVMs).

Our Agenda:

K-Nearest Neighbors (KNN):
- Intuition and How it Works
- Pros and Cons
- Use Cases
Decision Trees:
- Intuition and How it Works (including concepts like root node, branches, leaves, splitting criteria like Gini impurity/entropy)
- Pros and Cons
- Use Cases
Support Vector Machines (SVMs):
- Intuition and How it Works (including concepts like hyperplanes, margins, support vectors, kernels for non-linear data)
- Pros and Cons
- Use Cases

Learning Outcomes: By the end of this session, you should be able to:

Explain the fundamental principles behind KNN, Decision Trees, and SVMs.
Discuss the advantages and disadvantages of each algorithm.
Identify scenarios where each algorithm might be a suitable choice.
Understand the key terminology associated with each model.

Structure of the Session: For each algorithm, we will first explore its core concept and how it makes predictions. Then, we’ll discuss its strengths and weaknesses. We won’t be diving deep into the mathematical derivations today, but rather focusing on building a strong intuition. We’ll touch upon resources like StatQuest and the scikit-learn documentation for those who wish to explore further.

Interactivity: Your participation is key! Please ask questions as they arise. We’ll have pauses for questions after discussing each algorithm. I also encourage you to think about the “Practice” task mentioned in lesson.md: “For each, describe a scenario where it might be useful.” We can discuss these scenarios together.

3. Concept Explanation:

Supervised learning, as we’ve learned, involves training a model on labeled data (data with known outcomes). We’ve already explored Linear Regression and Logistic Regression in detail. However, the world of supervised learning is vast, and different problems often require different algorithmic approaches. Today, we’ll broaden our toolkit by understanding KNN, Decision Trees, and SVMs. These algorithms offer diverse ways to tackle both classification and regression tasks, each with its unique characteristics.

K-Nearest Neighbors (KNN)

Intuition and How it Works: K-Nearest Neighbors is one of the simplest yet surprisingly effective machine learning algorithms. The core idea is “Tell me who your neighbors are, and I’ll tell you who you are.” It’s a non-parametric, instance-based learning algorithm (sometimes called lazy learning because it doesn’t learn a discriminative function from the training data but memorizes the training dataset instead).

For Classification: To classify a new data point, KNN looks at the ‘K’ closest data points (neighbors) in the training set. The new data point is then assigned to the class that is most common among its K neighbors. For example, if K=5, and 3 of the 5 nearest neighbors belong to Class A and 2 belong to Class B, the new point is classified as Class A.
For Regression: Instead of a majority vote, the prediction for a new data point is typically the average (or median) of the values of its K nearest neighbors.

Key Aspects:

Choosing K: The value of ‘K’ is crucial. A small K can make the model sensitive to noise (high variance), while a large K can oversmooth the decision boundary (high bias). K is usually an odd number for binary classification to avoid ties. It’s often chosen through cross-validation.
Distance Metric: The “closeness” of neighbors is determined by a distance metric. Common choices include:
- Euclidean Distance: The most common, representing the straight-line distance between two points.
- Manhattan Distance: Sum of the absolute differences of their Cartesian coordinates.
- Minkowski Distance: A generalization of both Euclidean and Manhattan distances.
- Hamming Distance: Used for categorical variables, counting positions at which corresponding symbols are different. It’s important to scale your data (e.g., using StandardScaler or MinMaxScaler from scikit-learn) before applying KNN, as features with larger ranges can disproportionately influence the distance calculation.

Pros:

Simple to understand and implement: The logic is very intuitive.
No training phase (lazy learner): The algorithm simply stores the dataset. Predictions can be made immediately once you have the data.
Naturally handles multi-class classification.
Flexible decision boundaries: Can learn complex, non-linear decision boundaries.

Cons:

Computationally expensive during prediction: For each new point, it needs to calculate distances to all training points. This can be slow for large datasets.
Sensitive to irrelevant features and the curse of dimensionality: As the number of dimensions (features) increases, the distance between points becomes less meaningful, and the performance of KNN can degrade. Feature selection or dimensionality reduction is often necessary.
Requires feature scaling: As mentioned, features on different scales can lead to biased distance calculations.
Needs a good choice of K and distance metric: Performance is highly dependent on these choices.
Can suffer from imbalanced datasets: If one class is much more frequent than others, it might dominate the predictions.

Relevance and Importance: KNN is a great baseline model. Its simplicity makes it a good starting point for many classification problems. It’s particularly useful when the decision boundary is irregular and not easily approximated by a simple linear model. Understanding KNN also helps in grasping concepts related to instance-based learning and the importance of distance metrics.

Common Misconceptions/Pitfalls:

“No training” means no work: While there’s no explicit training model, choosing K, the distance metric, and preprocessing data (like scaling) are crucial steps.
Ignoring feature scaling: This is a very common mistake that can severely hamper KNN’s performance.
Using it blindly on high-dimensional data: Without feature selection or dimensionality reduction, KNN might perform poorly.

Decision Trees

Intuition and How it Works: Decision Trees are versatile supervised learning algorithms that can be used for both classification and regression tasks. In scikit-learn, categorical-feature support depends on the estimator and version; encode categorical inputs for these examples unless the specific API documents native support. They work by creating a tree-like model of decisions. Imagine a flowchart where each internal node represents a “test” on an attribute (e.g., “Is outlook sunny?”), each branch represents the outcome of the test, and each leaf node represents a class label (in classification) or a continuous value (in regression).

Building a Tree: The tree is built by recursively splitting the data based on the feature that provides the “best” split. The “best” split is typically determined by metrics that measure impurity or information gain:

Gini Impurity (Classification): Measures the probability of misclassifying a randomly chosen element if it were randomly labeled according to the distribution of labels in the subset. A Gini impurity of 0 means all elements in the node belong to the same class.
Entropy / Information Gain (Classification): Entropy measures the disorder or uncertainty in a set. Information Gain is the reduction in entropy achieved by splitting the data on a particular attribute. The algorithm chooses the attribute that provides the highest information gain.
Mean Squared Error (MSE) / Variance Reduction (Regression): For regression trees, the goal is to reduce the variance in the target variable within each leaf. Splits are chosen to minimize the MSE of the predictions in the resulting leaves.

The process starts at the root node (containing all data) and continues until a stopping criterion is met, such as:

All samples in a node belong to the same class.
There are no remaining features to split on.
The tree reaches a predefined maximum depth.
The number of samples in a node is below a certain threshold.

Pruning: A fully grown tree might overfit the training data, capturing noise rather than the underlying signal. Pruning is a technique to reduce the size of the tree by removing sections (subtrees) that provide little power in classifying instances. This helps improve the model’s generalization to unseen data.

Pros:

Easy to understand and interpret: The tree structure is very visual and mimics human decision-making.
Handles both numerical and categorical data: Can be used without much data preprocessing (though encoding categorical variables is often done).
Non-parametric: Makes no assumptions about the underlying data distribution.
Implicitly performs feature selection: Features that are more important for decision-making will appear higher up in the tree.
Relatively fast for prediction.

Cons:

Prone to overfitting: Especially if the tree is deep and not pruned. Small changes in data can lead to very different tree structures.
Can be unstable: Small variations in the data can result in a completely different tree being generated. Ensemble methods like Random Forests and Gradient Boosted Trees are often used to mitigate this.
Can create biased trees if some classes dominate.
Decision boundaries are typically axis-parallel: They might struggle with relationships that are not easily captured by horizontal or vertical splits (e.g., diagonal relationships).
Greedy algorithm: The splitting process is greedy, meaning it makes the best local decision at each step. This doesn’t guarantee a globally optimal tree.

Relevance and Importance: Decision Trees are fundamental building blocks for more complex and powerful ensemble methods like Random Forests and Gradient Boosting. Their interpretability makes them valuable for understanding the relationships in data and for explaining model predictions.

Common Misconceptions/Pitfalls:

Not pruning: Leading to overfitting is a common issue.
Ignoring instability: Relying on a single decision tree without considering ensemble methods for more robust predictions.
Misinterpreting feature importance: While they show which features are used for splitting, the interactions can be complex.

Support Vector Machines (SVMs)

Intuition and How it Works: Support Vector Machines are powerful and versatile supervised learning models used for classification, regression, and outlier detection. The core idea of SVM for classification is to find a hyperplane in an N-dimensional space (where N is the number of features) that distinctly classifies the data points.

The Optimal Hyperplane: For a given dataset, there might be multiple hyperplanes that can separate the classes. SVM aims to find the hyperplane that has the maximum margin, i.e., the maximum distance between data points of both classes. This maximum margin hyperplane provides better generalization to unseen data.

Key Concepts:

Hyperplane: In a 2D space, it’s a line. In 3D, it’s a plane. In higher dimensions, it’s a hyperplane. It’s the decision boundary.
Margin: The distance between the hyperplane and the closest data points from either class. SVM tries to maximize this margin.
Support Vectors: These are the data points that lie closest to the hyperplane (on the margin boundaries). They are critical because if these points were removed, the position of the optimal hyperplane would change. They “support” the hyperplane.
Soft Margin: In real-world datasets, data is often not perfectly separable. Soft margin SVM allows for some misclassifications or points within the margin by introducing a penalty term (C). A smaller C allows for a wider margin but more violations, while a larger C aims for fewer violations but a narrower margin (potentially overfitting).

The Kernel Trick (Handling Non-linear Data): What if the data is not linearly separable in its original feature space? SVMs can still work wonders using the “kernel trick.”

The kernel trick allows SVMs to operate in a higher-dimensional feature space without explicitly computing the coordinates of the data in that space. Instead, it uses a kernel function that computes the dot product of the data points in that higher-dimensional space.
Common Kernels:
- Linear Kernel: For linearly separable data (K(x_i, x_j) = x_i^T x_j).
- Polynomial Kernel: K(x_i, x_j) = (γ x_i^T x_j + r)^d.
- Radial Basis Function (RBF) Kernel: K(x_i, x_j) = exp(-γ ||x_i - x_j||^2). This is a very popular choice as it can map data to an infinitely dimensional space and handle complex non-linear relationships.
- Sigmoid Kernel: K(x_i, x_j) = tanh(γ x_i^T x_j + r).

Pros:

Effective in high-dimensional spaces: Works well even when the number of dimensions exceeds the number of samples.
Memory efficient: Uses a subset of training points (support vectors) in the decision function.
Versatile: Different Kernel functions can be specified for the decision function. Common kernels are provided, but it’s also possible to specify custom kernels.
Robust to overfitting in high-dimensional space, especially with proper choice of C and kernel.
Can model non-linear decision boundaries effectively using kernels.

Cons:

Training time can be long for large datasets: The complexity can be O(n^2) or O(n^3) depending on the implementation and kernel.
Choosing the right kernel and its parameters (like gamma and C) can be challenging: Often requires careful tuning and cross-validation.
Less interpretable than Decision Trees: The resulting model, especially with non-linear kernels, can be difficult to understand directly.
Performance can be poor if the number of features is much larger than the number of samples (without careful regularization).
Sensitive to feature scaling.

Relevance and Importance: SVMs were once considered state-of-the-art for many classification tasks before deep learning gained widespread popularity. They are still very powerful, especially for medium-sized datasets where clear margins of separation can be found or engineered via kernels. They are widely used in text classification, image classification, and bioinformatics.

Common Misconceptions/Pitfalls:

Not scaling features: SVMs are highly sensitive to feature scaling, similar to KNN.
Choosing the wrong kernel or parameters: Using a linear kernel for highly non-linear data or vice-versa, or not tuning C and gamma for RBF kernels.
Expecting high interpretability: While powerful, SVMs are more of a “black box” compared to decision trees.

Summary of Algorithm Choices:

KNN: Simple, good for irregular boundaries, but can be slow and needs careful preprocessing.
Decision Trees: Interpretable, handles mixed data types, but prone to overfitting if not managed. Forms the basis of powerful ensembles.
SVMs: Powerful for complex but separable data (even non-linearly via kernels), robust in high dimensions, but can be slow to train and requires careful tuning.

Understanding these algorithms, their strengths, and their weaknesses allows you to make more informed choices when approaching a new machine learning problem. The “No Free Lunch” theorem in machine learning states that no single algorithm works best for all problems. Therefore, having a diverse set of tools is essential.

4. Live Coding Demonstration:

For the live coding part, we’ll use Python with the scikit-learn library. scikit-learn provides easy-to-use implementations of these algorithms. We’ll focus on the conceptual application rather than deep optimization.

Let’s imagine we have a simple dataset and want to see how these classifiers might work. We’ll use the Iris dataset, which is a classic dataset for classification. It has 3 classes of irises (Setosa, Versicolour, Virginica) and 4 features (sepal length, sepal width, petal length, petal width).

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt
import numpy as np

# Load Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# For simplicity in visualization, let's use only two features
X_simple = X[:, :2] # Sepal length and sepal width

# Split data
X_train, X_test, y_train, y_test = train_test_split(X_simple, y, test_size=0.3, random_state=42, stratify=y)

# Scale features (important for KNN and SVM)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# --- K-Nearest Neighbors (KNN) ---
print("\n--- K-Nearest Neighbors (KNN) ---")
knn = KNeighborsClassifier(n_neighbors=5) # Using K=5
knn.fit(X_train_scaled, y_train)
y_pred_knn = knn.predict(X_test_scaled)
accuracy_knn = accuracy_score(y_test, y_pred_knn)
print(f"KNN (K=5) Accuracy: {accuracy_knn:.4f}")
# Common pitfall: Not scaling data. If we used X_train, X_test directly, results might differ.
# Best practice: Choose K using cross-validation.

# --- Decision Tree ---
print("\n--- Decision Tree ---")
# Using default parameters for simplicity. In practice, tune max_depth, min_samples_split etc.
dt = DecisionTreeClassifier(random_state=42, max_depth=3) # Limiting depth to prevent overfitting and for easier visualization
dt.fit(X_train, y_train) # Decision Trees don't strictly require feature scaling, but it doesn't hurt
y_pred_dt = dt.predict(X_test)
accuracy_dt = accuracy_score(y_test, y_pred_dt)
print(f"Decision Tree (max_depth=3) Accuracy: {accuracy_dt:.4f}")

# Visualize the Decision Tree
plt.figure(figsize=(12,8))
plot_tree(dt, filled=True, feature_names=iris.feature_names[:2], class_names=iris.target_names)
plt.title("Decision Tree for Iris Dataset (Sepal Features, max_depth=3)")
# plt.show() # Uncomment to display if running locally

# Common pitfall: Letting the tree grow too deep, leading to overfitting.
# Best practice: Prune the tree or use ensemble methods like Random Forest.

# --- Support Vector Machine (SVM) ---
print("\n--- Support Vector Machine (SVM) ---")
# Using RBF kernel by default. Parameters C and gamma are important.
svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale', random_state=42) # 'scale' is 1 / (n_features * X.var())
svm_rbf.fit(X_train_scaled, y_train)
y_pred_svm_rbf = svm_rbf.predict(X_test_scaled)
accuracy_svm_rbf = accuracy_score(y_test, y_pred_svm_rbf)
print(f"SVM (RBF Kernel) Accuracy: {accuracy_svm_rbf:.4f}")

# Example with Linear Kernel
svm_linear = SVC(kernel='linear', C=1.0, random_state=42)
svm_linear.fit(X_train_scaled, y_train)
y_pred_svm_linear = svm_linear.predict(X_test_scaled)
accuracy_svm_linear = accuracy_score(y_test, y_pred_svm_linear)
print(f"SVM (Linear Kernel) Accuracy: {accuracy_svm_linear:.4f}")

# Common pitfall: Not scaling data for SVM. Not tuning C and gamma for RBF kernel.
# Best practice: Use GridSearchCV or RandomizedSearchCV to find optimal hyperparameters.

# --- Visualizing Decision Boundaries (Optional, more complex code) ---
# This part is a bit more involved and is for illustrative purposes.
# We'll create a meshgrid to plot decision boundaries for each classifier.

def plot_decision_boundaries(X, y, classifier, classifier_name, ax, scaler=None):
    h = .02  # step size in the mesh
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))

    # If scaler is provided, transform the meshgrid points
    if scaler:
        Z = classifier.predict(scaler.transform(np.c_[xx.ravel(), yy.ravel()]))
    else:
        Z = classifier.predict(np.c_[xx.ravel(), yy.ravel()])
    
    Z = Z.reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.4, cmap=plt.cm.coolwarm)
    ax.scatter(X[:, 0], X[:, 1], c=y, s=20, edgecolor='k', cmap=plt.cm.coolwarm)
    ax.set_title(classifier_name)

fig, axes = plt.subplots(1, 3, figsize=(18, 5))

# Re-fit models on the scaled data for KNN and SVM for boundary plotting
# (DT can use unscaled for this visualization if preferred, but consistency is fine)
knn_plot = KNeighborsClassifier(n_neighbors=5)
knn_plot.fit(X_train_scaled, y_train)
plot_decision_boundaries(X_test_scaled, y_test, knn_plot, "KNN (K=5)", axes[0], scaler=None) # Data already scaled

dt_plot = DecisionTreeClassifier(random_state=42, max_depth=3)
dt_plot.fit(X_train_scaled, y_train) # Using scaled data for consistency in plot scales
plot_decision_boundaries(X_test_scaled, y_test, dt_plot, "Decision Tree (max_depth=3)", axes[1], scaler=None)

svm_rbf_plot = SVC(kernel='rbf', C=1.0, gamma='scale', random_state=42)
svm_rbf_plot.fit(X_train_scaled, y_train)
plot_decision_boundaries(X_test_scaled, y_test, svm_rbf_plot, "SVM (RBF Kernel)", axes[2], scaler=None)

plt.suptitle("Decision Boundaries on Iris Dataset (Sepal Features - Scaled)")
# plt.show() # Uncomment to display if running locally

Explanation of Code:

Load Data & Preprocessing: We load the Iris dataset, select two features for easier visualization, and split it into training and testing sets. Crucially, we scale the features using StandardScaler. This is vital for KNN and SVM.
KNN:
- We initialize KNeighborsClassifier with n_neighbors=5.
- Fit it on the scaled training data.
- Predict on the scaled test data and calculate accuracy.
- Common Mistake Highlighted: Using unscaled data would likely yield different (often worse) results for KNN.
- Best Practice: K should ideally be chosen using techniques like cross-validation.
Decision Tree:
- We initialize DecisionTreeClassifier with max_depth=3 to keep the tree simple and avoid overfitting for this demonstration. random_state ensures reproducibility.
- Fit it on the training data (scaling is less critical for DTs but doesn’t harm).
- Predict and calculate accuracy.
- We use plot_tree to visualize the tree structure. This is a great advantage of decision trees – their interpretability.
- Common Mistake Highlighted: Allowing trees to grow indefinitely can lead to overfitting.
- Best Practice: Prune trees or use ensemble methods.
SVM:
- We initialize SVC (Support Vector Classifier). By default, it uses the ‘rbf’ (Radial Basis Function) kernel, which is good for non-linear data. C is the regularization parameter. gamma='scale' is a common default.
- Fit on scaled training data.
- Predict on scaled test data and calculate accuracy.
- We also show an example with a kernel='linear'.
- Common Mistake Highlighted: Not scaling data or not tuning hyperparameters like C and gamma (for RBF).
- Best Practice: Hyperparameter tuning (e.g., with GridSearchCV) is crucial for SVMs.
Decision Boundary Visualization (Optional):
- The plot_decision_boundaries function creates a mesh of points and uses the trained classifier to predict the class for each point in the mesh. This allows us to visualize the regions each class occupies according to the model.
- This helps to intuitively understand how each algorithm carves up the feature space. You’d notice KNN’s boundary can be irregular, the Decision Tree’s boundary is axis-parallel, and SVM’s (with RBF) can be smooth and non-linear.

This live coding segment demonstrates the basic application of these three algorithms. Remember, real-world application involves more rigorous hyperparameter tuning, cross-validation, and feature engineering.

5. Tasks:

Now it’s your turn to think a bit more deeply about these algorithms!

Short Exercise: For each of the three algorithms discussed today (KNN, Decision Trees, SVMs):

Describe a specific, real-world scenario or problem (different from the Iris dataset) where that algorithm might be a particularly good choice. Explain why its characteristics make it suitable for that scenario.
Mention one key hyperparameter for that algorithm and briefly explain its impact on the model’s behavior.

Example for KNN:

Scenario: Recommending movies to a user on a streaming platform. If a user has liked certain movies, KNN could find other users (neighbors) who have liked a similar set of movies and then recommend movies liked by those neighbors but not yet seen by the target user. It’s suitable because user preferences can be complex and non-linear, and KNN doesn’t assume a specific model for preferences.
Hyperparameter: n_neighbors (K). A small K might lead to recommendations based on very niche similar users (potentially overfitting to noise or very specific tastes), while a large K might lead to very generic recommendations (oversmoothing, high bias).

Take about 10-15 minutes to jot down your thoughts. We can then discuss some of your ideas. This exercise will help solidify your understanding of when and why to choose a particular algorithm.

This concludes our overview of KNN, Decision Trees, and SVMs. These are valuable additions to your supervised learning toolkit! Next, we’ll be diving into Unsupervised Learning.