Feedforward Neural Networks

calendar_today May 21, 2025 schedule 11 min read

1. Welcome and Icebreaker:

Welcome to Day 21! Today, we’re diving into the mechanics of how information flows through a neural network.
Fun question: If a neural network could learn to do any one household chore perfectly, which chore would you pick and why?
Share any thoughts or perhaps analogies you can think of for how information might pass through a system to produce an output.

2. Outline the Session:

Agenda:
1. Recap: What are artificial neurons and layers?
2. Concept: How information flows in a Feedforward Neural Network.
3. Mathematical Representation: Expressing the flow using matrix operations and activation functions.
4. Live Coding: Building a simple feedforward pass in Python.
5. Task: Manually calculate the output of a small 2-layer NN.
Learning Outcomes: By the end of this session, you should be able to:
- Explain the concept of a feedforward pass in a neural network.
- Describe how matrix multiplications and activation functions are applied layer by layer.
- Represent a simple neural network mathematically.
- Manually trace the computation through a small network.
Structure: We’ll start with the core concepts, then look at the math, followed by a practical coding example and a hands-on exercise.
Interactivity: Please ask questions at any point! Your understanding is key. We’ll have moments for discussion and Q&A.

3. Concept Explanation:

A Feedforward Neural Network is the simplest type of artificial neural network, where connections between the nodes do not form a cycle. Information moves in only one direction—forward—from the input nodes, through the hidden layers (if any), and finally to the output nodes. There are no loops or feedback connections in the network architecture, meaning the output of one layer does not affect that same layer. This is in contrast to recurrent neural networks, which we might touch upon much later, where connections can form cycles, allowing information to persist.

The Journey of Information:

Input Layer: The process begins at the input layer, which receives the initial data or features for the network. Each node in the input layer represents a feature of the input data. For example, if you’re trying to predict house prices, input features might include the size of the house, the number of bedrooms, and the age of the house.
Weights and Biases: As information moves from one layer to the next, it’s transformed. Each connection between neurons in adjacent layers has an associated weight. These weights are the primary parameters that the network learns during the training process. A higher weight means the signal from the preceding neuron has a stronger influence on the succeeding neuron. In addition to weights, each neuron (except in the input layer, typically) has a bias term. The bias allows the neuron to be activated even when all its inputs are zero, or it can shift the activation function to the left or right, providing more flexibility to the model. Think of the bias as the b in the linear equation y = mx + b.
Weighted Sum: For each neuron in a hidden layer or the output layer, the first step is to calculate a weighted sum of the outputs from the neurons in the previous layer that are connected to it. If x_i are the outputs from the previous layer’s neurons and w_i are the corresponding weights of the connections, the weighted sum z for a neuron is calculated as: z = (w_1 * x_1) + (w_2 * x_2) + ... + (w_n * x_n) + bias This can be expressed more compactly using vector notation if X is the vector of inputs and W is the vector of weights: z = W · X + b
Activation Function: After the weighted sum z is calculated, it’s passed through an activation function, f(z). We discussed activation functions yesterday (Day 20). Their purpose is crucial: they introduce non-linearity into the model. Without non-linear activation functions, a deep neural network, no matter how many layers it has, would behave like a single-layer linear model. This non-linearity allows the network to learn complex patterns and relationships in the data that a linear model cannot capture. Common activation functions include Sigmoid, Tanh, ReLU, and Leaky ReLU. The output of the activation function becomes the output of that neuron, which then serves as an input to the neurons in the next layer.
Hidden Layers: A feedforward network can have zero or more hidden layers. These layers are “hidden” because their values are not directly observed in the input data or the final output. Each hidden layer takes the outputs from the previous layer, performs the weighted sum and activation function computation for each of its neurons, and passes its outputs to the next layer. The depth (number of hidden layers) and width (number of neurons per hidden layer) of the network are important architectural choices that affect its capacity to learn. More layers and neurons can model more complex functions but also increase the risk of overfitting and computational cost.
Output Layer: The final layer is the output layer. It produces the network’s prediction. The number of neurons in the output layer and the choice of activation function for this layer depend on the type of problem you’re solving:
- Regression: For predicting a continuous value (e.g., house price), the output layer typically has a single neuron, and often a linear activation function (or no activation function) is used.
- Binary Classification: For classifying input into one of two classes (e.g., spam or not spam), the output layer usually has one neuron with a sigmoid activation function, which squashes the output to a probability between 0 and 1.
- Multi-class Classification: For classifying input into one of several classes (e.g., digit recognition from 0 to 9), the output layer typically has N neurons, where N is the number of classes. A softmax activation function is commonly used here, as it converts the raw outputs (logits) into a probability distribution across the classes, ensuring that the sum of probabilities is 1.

Mathematical Representation:

Let’s consider a simple 2-layer neural network (one hidden layer and one output layer).

X: Input vector (e.g., [x_1, x_2, ..., x_m])
W_1: Weight matrix connecting the input layer to the hidden layer. If the input layer has m neurons and the hidden layer has h neurons, W_1 will be an h x m matrix.
b_1: Bias vector for the hidden layer ( h x 1 vector).
a_1: Activation function for the hidden layer (e.g., ReLU).
H: Output vector of the hidden layer.
W_2: Weight matrix connecting the hidden layer to the output layer. If the output layer has o neurons, W_2 will be an o x h matrix.
b_2: Bias vector for the output layer ( o x 1 vector).
a_2: Activation function for the output layer (e.g., Sigmoid or Softmax).
Y_hat: Output vector of the network (the prediction).

The computation proceeds as follows:

Hidden Layer Calculation:
- Weighted sum for the hidden layer: Z_1 = W_1 · X + b_1 (Here, · denotes matrix multiplication. If X is a column vector, W_1 multiplies X.)
- Output of the hidden layer: H = a_1(Z_1) (activation function applied element-wise)
Output Layer Calculation:
- Weighted sum for the output layer: Z_2 = W_2 · H + b_2
- Output of the network: Y_hat = a_2(Z_2)

This process, from input X to output Y_hat, is called a forward pass or feedforward propagation. It’s how the network makes a prediction for a given input. The actual learning happens when we compare Y_hat with the true label Y using a loss function and then adjust the weights (W_1, W_2) and biases (b_1, b_2) using an optimization algorithm like gradient descent (which involves backpropagation, a topic for future sessions).

Importance and Relevance: Understanding feedforward propagation is fundamental because it’s the basis for how all neural networks, including more complex architectures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), make predictions. Even during the training phase, the forward pass is the first step before calculating the error and updating the weights.

Common Misconceptions/Pitfalls:

Forgetting Non-linearity: Thinking that just stacking layers is enough. Without non-linear activation functions, a deep network is just a linear model.
Matrix Dimension Mismatches: When implementing, ensuring that the dimensions of weight matrices, input vectors, and bias vectors are compatible for matrix multiplication is crucial. A common source of errors.
Bias Term: Sometimes the bias term is forgotten or its role is underestimated. It provides essential flexibility.

We expect you to actively think about how data transforms at each step. Don’t hesitate to sketch out small networks and trace the data flow.

4. Live Coding Demonstration:

Let’s implement a simple feedforward pass for a 2-layer neural network using Python and NumPy.

import numpy as np

# Define activation functions
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def relu(x):
    return np.maximum(0, x)

# --- Network Architecture ---
# Input layer: 2 features
# Hidden layer: 3 neurons, ReLU activation
# Output layer: 1 neuron, Sigmoid activation (for binary classification)

# --- Initialize Weights and Biases (randomly for demonstration) ---
# Weights from input to hidden layer (W_1: 3x2 matrix)
W1 = np.array([[0.1, 0.4],
               [-0.2, 0.3],
               [0.5, -0.1]])
# Biases for hidden layer (b_1: 3x1 vector)
b1 = np.array([[0.1], [0.2], [-0.1]])

# Weights from hidden to output layer (W_2: 1x3 matrix)
W2 = np.array([[0.3, -0.2, 0.6]])
# Bias for output layer (b_2: 1x1 vector)
b2 = np.array([[0.05]])

# --- Input Data (example: one sample with 2 features) ---
X = np.array([[0.5], [1.0]]) # A column vector

# --- Feedforward Pass ---

# 1. Hidden Layer Calculation
# Weighted sum for hidden layer: Z_1 = W_1 · X + b_1
print(f"Input X (shape {X.shape}):
{X}")
print(f"Weights W1 (shape {W1.shape}):
{W1}")
print(f"Biases b1 (shape {b1.shape}):
{b1}")

Z1 = np.dot(W1, X) + b1
print(f"
Weighted Sum Z1 (shape {Z1.shape}):
{Z1}")

# Activation for hidden layer: H = relu(Z_1)
H = relu(Z1)
print(f"Activated Hidden Layer H (shape {H.shape}):
{H}")

# 2. Output Layer Calculation
# Weighted sum for output layer: Z_2 = W_2 · H + b_2
print(f"
Weights W2 (shape {W2.shape}):
{W2}")
print(f"Biases b2 (shape {b2.shape}):
{b2}")

Z2 = np.dot(W2, H) + b2
print(f"
Weighted Sum Z2 (shape {Z2.shape}):
{Z2}")

# Activation for output layer (prediction): Y_hat = sigmoid(Z_2)
Y_hat = sigmoid(Z2)
print(f"Output Y_hat (shape {Y_hat.shape}):
{Y_hat}")

print(f"
Final Prediction: {Y_hat[0,0]:.4f}")

Explanation of Code:

We import numpy for numerical operations, especially matrix multiplication.
We define sigmoid and relu activation functions.
We define the network architecture in comments and then initialize weight matrices (W1, W2) and bias vectors (b1, b2) with some arbitrary values. In a real scenario, these would be learned during training.
- W1 connects 2 input features to 3 hidden neurons.
- b1 are the biases for the 3 hidden neurons.
- W2 connects the 3 hidden neurons to 1 output neuron.
- b2 is the bias for the output neuron.
We create a sample input X with 2 features. It’s crucial to represent X as a column vector (2x1) for correct matrix multiplication with W1 (3x2).
Hidden Layer:
- Z1 = np.dot(W1, X) + b1: We perform the matrix multiplication of W1 and X, then add the bias b1. np.dot handles matrix multiplication.
- H = relu(Z1): We apply the ReLU activation function element-wise to Z1.
Output Layer:
- Z2 = np.dot(W2, H) + b2: We multiply the activated hidden layer output H by W2 and add the bias b2.
- Y_hat = sigmoid(Z2): We apply the sigmoid activation function to get the final prediction, which will be a value between 0 and 1.

Common Mistakes/Best Practices:

Shape Mismatches: The most common error is getting matrix dimensions wrong. np.dot(A, B) requires the number of columns in A to match the number of rows in B. Always print shapes (.shape) to debug.
Bias Broadcasting: NumPy’s broadcasting usually handles adding biases correctly (e.g., adding a 3x1 bias vector to each column of a 3xN matrix if you process N samples at once). Here, X is a single sample, so it’s straightforward.
Element-wise Activation: Activation functions are applied element-wise to the result of the weighted sum.

Feel free to ask any questions about this code or the concepts.

5. Tasks:

Manually Calculate the Output of a Small 2-Layer NN:

Consider the following neural network:

Input Layer: 2 input neurons. Input X = [x1, x2] = [0.8, 0.2]
Hidden Layer: 2 neurons, using ReLU activation.
- Weights from input to hidden neuron 1 (h1): w_h1_x1 = 0.5, w_h1_x2 = -0.3
- Bias for hidden neuron 1: b_h1 = 0.1
- Weights from input to hidden neuron 2 (h2): w_h2_x1 = 0.1, w_h2_x2 = 0.7
- Bias for hidden neuron 2: b_h2 = -0.2
Output Layer: 1 neuron, using Sigmoid activation.
- Weights from hidden neuron 1 to output neuron (o1): w_o1_h1 = 0.6
- Weights from hidden neuron 2 to output neuron (o1): w_o1_h2 = -0.4
- Bias for output neuron: b_o1 = 0.3

Your Task:

Calculate the weighted sum for hidden neuron h1 (z_h1).
Calculate the output of hidden neuron h1 after ReLU activation (out_h1).
Calculate the weighted sum for hidden neuron h2 (z_h2).
Calculate the output of hidden neuron h2 after ReLU activation (out_h2).
Calculate the weighted sum for the output neuron o1 (z_o1), using out_h1 and out_h2 as inputs.
Calculate the final output of the network (Y_hat) after Sigmoid activation.

Show your calculations step-by-step. You can use a calculator for the sigmoid function (1 / (1 + e^-x)). This exercise will help solidify your understanding of the feedforward process. ‘‘‘DRAFT_FILE_CONTENT