Linear Algebra - Vectors & Matrices

calendar_today May 1, 2025 schedule 17 min read

1. Welcome and Icebreaker:

Hello everyone, and welcome to our first session on the mathematical foundations for Machine Learning! To kick things off, let’s ponder a fun question: If a tiny vector and a giant matrix got into an argument at a math party, what do you think they’d be bickering about? Maybe their dimensions? Or who’s more fundamental?

Feel free to share your thoughts in the chat! Also, I’d love to hear a bit about your prior experience or any initial thoughts on linear algebra. Have you encountered vectors and matrices before? Any concepts you found particularly intriguing or perhaps a bit challenging? Sharing this helps me tailor our discussions.

2. Outline the Session:

Today, we’re diving into the essential building blocks of linear algebra: vectors and matrices. Here’s what our journey today looks like:

Agenda:
- Quick Welcome & Icebreaker (We just did this!)
- Why Linear Algebra is Crucial for Machine Learning
- Deep Dive into Vectors:
  - What is a Vector? (Definition, geometric and algebraic views)
  - Vector Operations: Addition, Scalar Multiplication, Dot Product, Norm/Magnitude
- Deep Dive into Matrices:
  - What is a Matrix? (Definition, dimensions)
  - Matrix Operations: Addition, Scalar Multiplication, Matrix-Vector Multiplication, Matrix-Matrix Multiplication
  - The Transpose of a Matrix
- Real-world Applications and Importance in ML
- Live Coding Demonstration: Basic Vector and Matrix Operations with Python (using the NumPy library)
- Hands-on Task: Practice problems to solidify understanding
- Q&A and Wrap-up
Learning Outcomes: By the end of this session, you should be able to:
- Clearly define and understand the fundamental concepts of vectors and matrices.
- Confidently perform basic operations on vectors (addition, scalar multiplication, dot product, norm calculation).
- Confidently perform basic operations on matrices (addition, scalar multiplication, matrix-vector, matrix-matrix multiplication, transpose).
- Appreciate why linear algebra is a cornerstone for understanding and working with machine learning algorithms.
- Implement these fundamental operations using Python and its powerful NumPy library.
Structure of the Session: This will be an interactive lecture. I’ll explain the concepts, we’ll look at examples, I’ll do some live coding, and then you’ll get a chance to try things out yourself.
Expectations for Interactivity: Please feel free to ask questions at any point – there are no silly questions! The more you engage, ask, and participate, the more you’ll learn. Use the chat, raise your hand, or unmute. Let’s make this a collaborative learning experience.

3. Concept Explanation:

Why is Linear Algebra So Important for Machine Learning?

Before we jump into definitions, let’s address the “why.” Why do we start a machine learning journey with what might seem like abstract math? The truth is, linear algebra is the bedrock upon which much of machine learning is built.

Data Representation: In ML, data is king. Whether it’s images, text, tabular data from a spreadsheet, or sensor readings, we need a way to represent this data numerically so computers can process it. Vectors and matrices provide a powerful and efficient way to do this. For instance, a single data point (like a customer) with multiple features (age, income, purchase history) can be represented as a vector. A whole dataset of many customers can be represented as a matrix, where each row is a customer (vector) and each column is a feature.
Algorithm Mechanics: Many ML algorithms are described and implemented using the language and operations of linear algebra.
- Linear Regression: The classic y = mx + b (or its higher-dimensional form y = WX + b) is fundamentally a matrix-vector operation.
- Neural Networks: These are essentially complex series of linear algebra operations (matrix multiplications) interspersed with non-linear activation functions. The “weights” in a neural network are often stored in matrices.
- Dimensionality Reduction: Techniques like Principal Component Analysis (PCA) heavily rely on matrix operations (eigen-decomposition) to find more compact representations of data.
- Optimization: The process of “training” an ML model often involves optimization algorithms like gradient descent, where gradients (which are vectors) guide updates to model parameters (often vectors or matrices).
Efficiency: Performing operations on vectors and matrices (especially with libraries like NumPy) is computationally very efficient compared to looping through individual numbers. This allows ML models to process vast amounts of data.

So, a solid grasp of linear algebra isn’t just helpful; it’s essential for understanding what’s happening “under the hood” of ML models, for implementing them, and for reading cutting-edge research.

Vectors: The Building Blocks

What is a Vector?

At its core, a vector is an object that has both magnitude (or length) and direction. Think of it as an arrow in space.

Geometric View: In a 2D plane, you can draw a vector as an arrow starting from the origin (0,0) and pointing to a coordinate (x,y). Its length is the distance from the origin to that point, and its direction is the way it’s pointing.
Algebraic View: More generally, a vector is an ordered list of numbers, called components or elements. For example, v = [v1, v2, v3] is a 3-dimensional vector. These numbers can represent coordinates in space, but they can also represent any set of related numerical values (like the features of a house: [area_sq_ft, num_bedrooms, age_years]).
Notation: Vectors are typically denoted by lowercase bold letters (e.g., v) or lowercase letters with an arrow above (e.g., vec(v)). In programming and text, we often just use a variable name like v.
Column vs. Row Vectors: In mathematical notation, a vector’s orientation matters in matrix products. A column vector has shape n × 1; a row vector has shape 1 × n. A one-dimensional NumPy array has shape (n,), so it is neither an explicit row nor column vector.
- Column vector: [[x], [y], [z]] (shape (3, 1))
- Row vector: [[x, y, z]] (shape (1, 3))

Vector Operations

Vector Addition:
- How it works: If you have two vectors of the same dimension, say v = [v₁, v₂] and w = [w₁, w₂], their sum u = v + w is found by adding corresponding components: u = [v₁ + w₁, v₂ + w₂].
- Geometric Interpretation: Imagine placing the tail of vector w at the head (tip) of vector v. The vector from the tail of v to the head of w is their sum. This is often called the “tip-to-tail” rule. Alternatively, if they share the same origin, their sum is the diagonal of the parallelogram formed by the two vectors.
- Properties: Commutative (v + w = w + v), Associative ((v + w) + z = v + (w + z)).
- Constraint: Vectors must have the same number of components (same dimension) to be added.
Scalar Multiplication:
- How it works: Multiplying a vector v by a scalar (a single number) c means multiplying each component of v by c. If v = [v₁, v₂], then cv = [cv₁, cv₂].
- Geometric Interpretation: Scalar multiplication scales the length of the vector. If c > 1, the vector gets longer. If 0 < c < 1, it gets shorter. If c is negative, the vector reverses its direction in addition to being scaled.
- Properties: Distributive over vector addition (c(v + w) = cv + cw), distributive over scalar addition ((c + d)v = cv + dv).
Dot Product (Inner Product):
- Definition: The dot product of two vectors v and w (of the same dimension) is a scalar value. If v = [v₁, v₂, …, vₙ] and w = [w₁, w₂, …, wₙ], their dot product (denoted v ⋅ w or sometimes vTw) is: v ⋅ w = v₁w₁ + v₂w₂ + … + vₙwₙ
- Geometric Meaning: The dot product is also related to the angle (θ) between the two vectors: v ⋅ w = ||v|| ||w|| cos(θ) where ||v|| is the magnitude (length) of vector v.
- Applications & Interpretations:
  - Similarity/Alignment: If cos(θ) is 1 (θ=0°, vectors point in the same direction), the dot product is maximized for their lengths. If cos(θ) is -1 (θ=180°, opposite directions), it’s minimized (large negative).
  - Orthogonality (Perpendicularity): If two non-zero vectors are orthogonal (perpendicular, θ=90°), their dot product is 0 (since cos(90°)=0). This is a crucial concept.
  - Projection: The dot product can be used to find the projection of one vector onto another (how much of one vector lies in the direction of another).
- Properties: Commutative (v ⋅ w = w ⋅ v), Distributive (v ⋅ (w + z) = v ⋅ w + v ⋅ z).
Norm (Magnitude/Length):
- Definition: The norm of a vector v, denoted ||v||, is its length. The most common type is the Euclidean norm or L2 norm.
- Formula (L2 norm): For v = [v₁, v₂, …, vₙ]: ||v|| = √(v₁² + v₂² + … + vₙ²) Notice that ||v||² = v ⋅ v, so ||v|| = √(v ⋅ v).
- Other Norms: While L2 is common, other norms exist, like the L1 norm (Manhattan distance): ||v||₁ = |v₁| + |v₂| + … + |vₙ|.
- Unit Vector: A vector whose norm is 1 is called a unit vector. To get a unit vector u in the same direction as a non-zero vector v, you divide v by its norm: u = v / ||v||. Unit vectors are important for representing direction without magnitude.

Matrices: Organizing Data and Transformations

What is a Matrix?

A matrix is a rectangular array (or grid) of numbers, symbols, or expressions, arranged in rows and columns.

Dimensions: A matrix with m rows and n columns is called an m x n matrix (read “m by n matrix”).
Notation: Matrices are typically denoted by uppercase bold letters (e.g., A) or just uppercase letters. An element in the i-th row and j-th column of matrix A is denoted as Aij or aij.

Examples:

A = [[1, 2, 3],  // A is a 2x3 matrix
     [4, 5, 6]]

B = [[a, b],     // B is a 2x2 matrix (square matrix)
     [c, d]]

Real-world Analogies:
- Datasets: As mentioned, rows can be samples (e.g., patients) and columns can be features (e.g., blood pressure, heart rate).
- Images: A grayscale image can be a matrix where each element is a pixel intensity. A color image might be three matrices (Red, Green, Blue channels).
- Systems of Linear Equations: Matrices are used to represent and solve systems like: 2x + 3y = 7 x - y = 1 can be written as A x = b, where A is a matrix of coefficients, x is a vector of variables, and b is a vector of constants.
- Transformations: Matrices can represent geometric transformations like rotations, scaling, and shearing of vectors or objects in space.

Matrix Operations

Matrix Addition:
- How it works: Similar to vector addition, two matrices A and B can be added if they have the same dimensions. The sum C = A + B is found by adding corresponding elements: Cij = Aij + Bij.
- Properties: Commutative (A + B = B + A), Associative ((A + B) + D = A + (B + D)).
Scalar Multiplication:
- How it works: Multiplying a matrix A by a scalar c means multiplying every element of A by c. If C = cA, then Cij = cAij.
Matrix-Vector Multiplication:
- How it works: If A is an m x n matrix and v is an n x 1 column vector (meaning n components), their product b = A v is an m x 1 column vector.
- The i-th element of the resulting vector b (i.e., bi) is the dot product of the i-th row of matrix A with the vector v.
- Condition: The number of columns in matrix A must equal the number of components (rows) in vector v. (m x n) * (n x 1) -> (m x 1)
- Interpretation: This is often interpreted as transforming the vector v using the matrix A. For example, A could be a rotation matrix, and A v would be the rotated vector.
Matrix-Matrix Multiplication:
- How it works: If A is an m x k matrix and B is a k x n matrix, their product C = A B is an m x n matrix.
- The element Cij (in the i-th row, j-th column of the result C) is the dot product of the i-th row of A with the j-th column of B.
- Condition: The number of columns in the first matrix (A) must equal the number of rows in the second matrix (B). (m x k) * (k x n) -> (m x n)
- Crucial Property: NOT Commutative! In general, A B ≠ B A. The order matters significantly. Sometimes B A might not even be defined if the dimensions don’t align.
- Other Properties: Associative ((A B)D = A(B D)), Distributive (A(B + D) = A B + A D).
Transpose of a Matrix:
- Definition: The transpose of an m x n matrix A, denoted AT, is an n x m matrix obtained by swapping its rows and columns. The element (AT)ij is Aji.
- Example: If A = [[1, 2, 3], [4, 5, 6]] (a 2x3 matrix) Then A^T = [[1, 4], [2, 5], [3, 6]] (a 3x2 matrix)
- Properties:
  - (AT)T = A
  - (A + B)T = AT + BT
  - (cA)T = cAT
  - (A B)T = BTAT (Note the reversal of order!)

Importance and Relevance Revisited

Now that we’ve seen the definitions, the importance in ML should be even clearer:

Data Representation: A dataset with 1000 samples and 10 features is a 1000x10 matrix. Each sample is a row vector.
Model Parameters: In linear regression y = Xw + b, X is the input data matrix, w is a vector of weights. In neural networks, layers have weight matrices.
Transformations & Computations:
- Calculating predictions in many models involves matrix-vector or matrix-matrix multiplications.
- Applying transformations to data (e.g., PCA) involves matrix operations.
- Updating model weights during training (e.g., via gradient descent) involves vector and matrix arithmetic.

Common Misconceptions and Pitfalls:

Matrix Multiplication Order: Always remember A B is generally not B A. This is a very common source of errors.
Dimensionality Mismatches: Trying to add or multiply matrices/vectors whose dimensions are not compatible is a frequent bug. Always check shapes!
Dot Product vs. Element-wise Multiplication: The dot product results in a scalar (for vectors) or is a component of matrix multiplication. Element-wise multiplication (Hadamard product) is a different operation where you multiply corresponding elements (e.g., [1,2] * [3,4] = [3,8]). Be clear which one you need.
Row vs. Column Vectors: The orientation of your vectors matters, especially when multiplying with matrices. Consistency is key.

We encourage you to pause and think: can you visualize a 2D vector being scaled? Or two 2D vectors being added tip-to-tail? Can you see how a dataset of student grades (Math, Science, History for several students) could form a matrix? This visualization helps immensely.

4. Live Coding Demonstration:

Alright, let’s make these concepts concrete by using Python with its powerful NumPy library. NumPy is the de facto standard for numerical computation in Python, and it’s incredibly efficient for vector and matrix operations.

If you have Python and NumPy installed, you can follow along!

# First, we need to import the NumPy library
import numpy as np

# --- VECTORS ---
print("--- VECTORS ---")

# Creating vectors (NumPy arrays)
v = np.array([1, 2, 3]) # 1D array with shape (3,), not explicitly a row or column vector
v_row = np.array([[1, 2, 3]]) # Explicit row vector with shape (1, 3)
print(f"Row vector v_row: {v_row}, shape: {v_row.shape}")
w = np.array([4, 5, 6])
print(f"Vector v: {v}, shape: {v.shape}")
print(f"Vector w: {w}, shape: {w.shape}")

# To be explicit about a column vector (2D array with one column):
v_col = np.array([[1], [2], [3]])
print(f"Column vector v_col: \n{v_col}, shape: {v_col.shape}")


# Vector Addition
sum_vw = v + w
print(f"v + w = {sum_vw}")

# Scalar Multiplication
scalar = 3
scaled_v = scalar * v
print(f"{scalar} * v = {scaled_v}")

# Dot Product
# Method 1: np.dot()
dot_product_vw = np.dot(v, w)
print(f"v . w (using np.dot) = {dot_product_vw}")

# Method 2: Using the @ operator (Python 3.5+)
# This is preferred for matrix multiplication and dot products involving 1D arrays
dot_product_vw_operator = v @ w
print(f"v @ w (using @ operator) = {dot_product_vw_operator}")

# Norm (Magnitude/Length) - L2 norm by default
norm_v = np.linalg.norm(v)
print(f"Norm of v (||v||) = {norm_v}")
# Verify: sqrt(1^2 + 2^2 + 3^2) = sqrt(1 + 4 + 9) = sqrt(14) approx 3.74

# --- MATRICES ---
print("\n--- MATRICES ---")

# Creating matrices (2D NumPy arrays)
A = np.array([[1, 2], 
              [3, 4]])
B = np.array([[5, 6], 
              [7, 8]])
print(f"Matrix A:\n{A}, shape: {A.shape}")
print(f"Matrix B:\n{B}, shape: {B.shape}")

# Matrix Addition (must have same dimensions)
sum_AB = A + B
print(f"A + B:\n{sum_AB}")

# Scalar Multiplication
scalar_mat = 0.5
scaled_A = scalar_mat * A
print(f"{scalar_mat} * A:\n{scaled_A}")

# Matrix-Vector Multiplication
# A is 2x2, v_col is 2x1 (if we define v_col = np.array([[1],[2]]))
# Let's use a compatible vector
vec_for_A = np.array([10, 20]) # This is 1D, NumPy handles it well with @
# Or explicitly as a column vector:
vec_for_A_col = np.array([[10], [20]])

# Using @ operator
product_A_vec = A @ vec_for_A 
print(f"A @ vec_for_A (1D vector):\n{product_A_vec}, shape: {product_A_vec.shape}") 
# Result: [1*10 + 2*20, 3*10 + 4*20] = [50, 110]

product_A_veccol = A @ vec_for_A_col
print(f"A @ vec_for_A_col (column vector):\n{product_A_veccol}, shape: {product_A_veccol.shape}")

# Matrix-Matrix Multiplication
# A (2x2) @ B (2x2) -> results in a 2x2 matrix
product_AB = A @ B
print(f"A @ B (Matrix Product):\n{product_AB}")
# Let's verify one element: C_11 = row1(A) . col1(B) = [1,2] . [5,7] = 1*5 + 2*7 = 5 + 14 = 19

# What if dimensions don't match for multiplication?
C = np.array([[1, 2, 3], [4, 5, 6]]) # C is 2x3
# A (2x2) @ C (2x3) -> This should work, result 2x3
product_AC = A @ C
print(f"A (2x2) @ C (2x3):\n{product_AC}, shape: {product_AC.shape}")

# C (2x3) @ A (2x2) -> This should FAIL (cols of C != rows of A)
try:
    product_CA = C @ A
    print(f"C @ A:\n{product_CA}")
except ValueError as e:
    print(f"Error trying C @ A: {e}")
    print("This error is expected due to incompatible shapes (3 != 2).")


# Transpose of a Matrix
A_transpose = A.T
print(f"Transpose of A (A.T):\n{A_transpose}")
# Also np.transpose(A)
A_transpose_func = np.transpose(A)
print(f"Transpose of A (np.transpose(A)):\n{A_transpose_func}")

# Best Practice: Always check shapes!
print(f"Shape of A: {A.shape}, Shape of C: {C.shape}")
# If A.shape[1] == C.shape[0], then A @ C is valid.

# Common Pitfall: Element-wise multiplication vs. Matrix multiplication
# Using * with two matrices performs element-wise multiplication
element_wise_AB = A * B 
print(f"A * B (Element-wise product):\n{element_wise_AB}")
print("Note: This is NOT matrix multiplication. For matrix multiplication, use @ or np.dot().")

Explanation of the Code:

We import numpy as np.
Vectors:
- np.array([...]) creates a vector.
- Basic arithmetic operators (+, *) work element-wise for addition and scalar multiplication.
- np.dot(v, w) or v @ w calculates the dot product.
- np.linalg.norm(v) calculates the L2 norm.
Matrices:
- np.array([[...], [...]]) creates a matrix.
- + and * (with a scalar) work as expected.
- A @ vec or np.dot(A, vec) for matrix-vector multiplication.
- A @ B or np.dot(A, B) for matrix-matrix multiplication. Crucially, * between two matrices performs element-wise multiplication, not standard matrix multiplication. This is a common pitfall!
- .T attribute (e.g., A.T) gives the transpose.
We also demonstrated checking .shape and handling potential ValueError for incompatible dimensions in matrix multiplication.

Feel free to ask any questions about this code or NumPy!

5. Tasks:

Now it’s your turn to practice! These exercises are designed to help you solidify your understanding of the concepts we covered. Try to solve them by hand first (for the smaller ones) to build intuition, and then verify your answers using Python and NumPy.

Exercise 1: Vector Operations Given the vectors: a = [2, -1, 5] b = [-3, 0, 4]

Calculate a + b by hand. Then, verify your result using NumPy.
Calculate 3 * a by hand. Verify with NumPy.
Calculate the dot product a ⋅ b by hand. Verify with NumPy.
Calculate the L2 norm (magnitude) of vector a (||a||) by hand. Verify with NumPy.
What is the unit vector in the direction of a? Calculate its components (you can use a calculator for the final division if doing by hand). Verify with NumPy.

Exercise 2: Matrix Operations Given the matrices: M = [[1, -2], [0, 3]] N = [[4, 1], [-1, 2]] P = [[1, 0, 2], [-1, 3, 1]] (Note: P has different dimensions)

And vector: v = [5, -2] (treat as a column vector [[5], [-2]] for matrix-vector products if needed by hand)

Calculate M + N by hand. Verify with NumPy.
Calculate M @ N (matrix product of M and N) by hand. Verify with NumPy.
Calculate N @ M by hand. Is it the same as M @ N? Verify with NumPy.
Find the transpose of matrix P (PT) by hand. Verify with NumPy.
Calculate M @ v (matrix M multiplied by vector v) by hand. Verify with NumPy.
Can you calculate P @ M? If yes, what are the dimensions of the result? If no, why not? Check with NumPy (it will raise an error if not possible).
Can you calculate M @ P? If yes, what are the dimensions of the result? If no, why not? Check with NumPy.

Exercise 3: Thinking Question Imagine you have a dataset of 100 students, and for each student, you have 5 exam scores.

How would you represent this entire dataset using a matrix? What would be its dimensions?
How would you represent the scores of a single student using a vector? Would it be a row or column vector in the context of your matrix?

Take about 15-20 minutes for these. Don’t worry if you get stuck; the goal is to try and apply what we’ve learned. We can discuss the solutions afterwards. Good luck!