Introduction to Neural Networks (NNs)

1. Welcome and Icebreaker:

• Icebreaker question: “If your brain were a network of neurons, what everyday task do you think it does best?”
• Invite students to share creative analogies: memory recall (search), pattern recognition (face ID), decision making (traffic light recognition).
• Artificial neural networks were inspired partly by brains, but they are mathematical models and do not closely reproduce biological neurons or brain function.

2. Outline the Session:

• Agenda:
  1. Biological inspiration & history of NNs.
  2. Artificial neuron (perceptron) structure.
  3. Network architecture: layers, nodes, connectivity.
  4. Role of weights and biases.
  5. Live coding: implement a simple perceptron.
  6. Practice tasks.
• Learning outcomes:
  • Understand how simple neurons process information.
  • Identify components of a neural network: inputs, weights, biases, activation.
  • Build and calculate outputs of a perceptron by hand and with code.
• Participation: ask questions during the code walkthrough, share parts that feel confusing, and suggest instances where linear decision boundaries suffice.

3. Concept Explanation:

Biological Inspiration Is Only an Analogy: Artificial neurons borrow simplified ideas such as weighted inputs and activation, but they do not closely model biological neurons or the brain’s complexity. Neural networks are inspired by the human brain, which contains billions of interconnected neurons. Each biological neuron receives signals via dendrites, processes them in the cell body, and sends output through the axon. In ML, we simplify this process into a mathematical model called an artificial neuron or perceptron.

1. Perceptron Model:

   • Inputs: features from data (e.g., pixel intensities in an image).
   • Weights (w): strength of each input connection. A higher weight means the input has more influence.
   • Bias (b): an additional term that shifts the activation function, similar to a threshold in biology.
   • Summation: compute a weighted sum: z = w1*x1 + w2*x2 + ... + b.
   • Activation: pass z through an activation function (initially a step function). If z > 0, output 1; otherwise 0.

   Real-world analogy: imagine a decision gate that opens only if the combined effect of keys inserted (inputs) exceeds a threshold (bias).

2. Network Structure:

   • Layers: group of neurons. A simple NN has an input layer, one hidden layer, and an output layer.
   • Input Layer: nodes correspond to features; no computation here—just a pass-through.
   • Hidden Layers: each neuron applies weights, bias, and activation. The more hidden layers, the deeper the network.
   • Output Layer: produces final prediction (binary, multiclass, regression value).

3. Weights and Biases:

   • Weights determine the slope or sensitivity to each input dimension.
   • Bias allows shifting of the activation threshold.
   • During learning (training), the network updates weights and biases to minimize a loss function, adjusting the decision surface.

4. Why Multiple Layers?

   • Single perceptron can only classify linearly separable data (straight line boundary).
   • Multiple layers with non-linear activations enable the network to learn complex, non-linear decision boundaries.

5. Common Misconceptions and Pitfalls:

   • Neurons in ML are simple mathematical functions—not biological replicas.
   • More layers/neurons don’t always mean better performance—risk of overfitting and vanishing gradients.
   • Explanation: too many parameters may memorize noise, reducing generalization.

Importance in ML: Neural networks power today’s state-of-the-art systems in computer vision, natural language processing, and speech recognition. Understanding the simplest unit—the perceptron—lays the foundation for deep learning.

4. Live Coding Demonstration:

# Implementing a simple perceptron from scratch
import numpy as np

class Perceptron:
    def __init__(self, input_size, learning_rate=0.1):
        # Initialize weights and bias randomly
        self.weights = np.random.randn(input_size)
        self.bias = np.random.randn()
        self.lr = learning_rate

    def activation(self, z):
        # Step function
        return 1 if z >= 0 else 0

    def predict(self, X):
        # Compute weighted sum and apply activation
        z = np.dot(self.weights, X) + self.bias
        return self.activation(z)

    def train(self, X_train, y_train, epochs=10):
        for epoch in range(epochs):
            total_error = 0
            for x, y in zip(X_train, y_train):
                y_pred = self.predict(x)
                error = y - y_pred
                total_error += abs(error)
                # Update weights and bias
                self.weights += self.lr * error * x
                self.bias += self.lr * error
            print(f"Epoch {epoch+1}: Total Error = {total_error}")

# Sample dataset: OR logic gate
data = np.array([[0,0], [0,1], [1,0], [1,1]])
labels = np.array([0,1,1,1])

# Create and train perceptron
p = Perceptron(input_size=2, learning_rate=0.2)
p.train(data, labels, epochs=10)

# Test predictions
for x in data:
    print(f"Input: {x}, Predicted: {p.predict(x)}")

• Step-by-step:
  1. Initialize weights and bias.
  2. Define a simple threshold activation.
  3. Training loop: update rules using error and learning rate.
  4. Demonstrate on OR gate—linearly separable problem.
• Common mistakes:
  • Forgetting to update bias.
  • Learning rate too large: oscillation; too small: slow convergence.

5. Tasks:

• Short Exercise:

  1. Manually compute the perceptron output for inputs [1, 1] given weights [0.5, -0.6] and bias = 0.1. Use a step activation.
  2. Modify the code to implement an AND gate and train it. Observe differences in weights and bias.

• Reflection:

  • Why can’t a single perceptron solve XOR? Sketch the points and decision boundary.
  • How does adding a hidden layer enable solving XOR?