Multi-Head Attention and Positional Encoding in Transformers

calendar_today May 30, 2025 schedule 6 min read

Learning Objectives: By the end of this lesson, you will be able to:

Explain the purpose and mechanics of multi-head attention

Describe and implement positional encoding

Analyze how these components enhance the Transformer architecture

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Have you ever wondered how a model can focus on different parts of a sentence at the same time? Or how it knows the order of words when there are no loops or recurrence?

Lesson Overview

Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
Prerequisites: Python, understanding of self-attention, basic neural network concepts
Materials Needed: Python (Jupyter Notebook or IDE), NumPy or PyTorch, sample text data

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

The motivation for multi-head attention
How multi-head attention works and why it’s effective
The need for positional encoding in Transformers
Implementing and visualizing positional encoding
Hands-on practice and exercises
Common pitfalls and how to avoid them

Success Criteria:

By the end of this session, you should be able to implement and explain both multi-head attention and positional encoding in the context of Transformers.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Multi-Head Attention

Multi-head attention allows the model to attend to information from different representation subspaces at different positions. Instead of performing a single attention function, the model runs multiple attention operations (“heads”) in parallel, each with its own set of learned projections.

Real-World Analogy: Imagine a team of experts, each focusing on a different aspect of a problem. By combining their insights, you get a more comprehensive understanding.

Why This Matters: Multi-head attention enables the model to capture various types of relationships and patterns in the data, improving its ability to understand complex sequences.

Positional Encoding

Transformers lack recurrence and thus have no inherent sense of word order. Positional encoding injects information about the position of each token into the model, allowing it to distinguish between different sequence orders.

Key Terminology

Head: An independent attention mechanism in multi-head attention
Projection: Linear transformation applied to Q, K, V for each head
Positional Encoding: A vector added to input embeddings to encode position information

4. Detailed Explanation and Examples (25-30 minutes)

Understanding Multi-Head Attention

Each head in multi-head attention projects the input into different subspaces, computes attention, and the results are concatenated and linearly transformed. This allows the model to jointly attend to information from different representation subspaces.

Practical Applications

Machine translation: capturing different syntactic and semantic relationships
Text summarization: focusing on key information from various parts of the text

Common Misconceptions

Misconception: More heads always mean better performance. Reality: Too many heads can lead to overfitting or redundancy; the number of heads should be tuned.
Misconception: Positional encoding is only needed for long sequences. Reality: Even short sequences require positional information for correct interpretation.

5. Live Coding Demonstration (20-25 minutes)

Example 1: Multi-Head Attention Skeleton (PyTorch)

import torch
import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, embed_dim, num_heads):
        super().__init__()
        self.mha = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
    def forward(self, x):
        attn_output, attn_weights = self.mha(
            x, x, x, average_attn_weights=False
        )
        return attn_output, attn_weights

# Example usage:
mha = MultiHeadAttention(embed_dim=32, num_heads=4)
x = torch.randn(8, 10, 32)  # (batch, seq_len, embed_dim)
output, attn_weights = mha(x)
print(output.shape, attn_weights.shape)  # (8, 10, 32), (8, 4, 10, 10)

Explanation:

Each head computes attention in parallel
Outputs are concatenated and projected

Example 2: Implementing Positional Encoding (NumPy)

import numpy as np

def positional_encoding(seq_len, d_model):
    PE = np.zeros((seq_len, d_model))
    for pos in range(seq_len):
        for i in range(0, d_model, 2):
            angle = pos / (10000 ** (i / d_model))
            PE[pos, i] = np.sin(angle)
            if i+1 < d_model:
                PE[pos, i+1] = np.cos(angle)
    return PE

print(positional_encoding(5, 8))

Key Points:

Sine and cosine functions encode position information
Adjacent sine/cosine dimensions form a pair and share one frequency
Added to input embeddings before feeding into the model

Example 3: Visualizing Positional Encodings

import matplotlib.pyplot as plt
PE = positional_encoding(50, 16)
plt.imshow(PE, aspect='auto', cmap='viridis')
plt.colorbar()
plt.title('Positional Encoding Heatmap')
plt.xlabel('Embedding Dimension')
plt.ylabel('Position')
plt.show()

Debugging Demonstration

Show common errors and how to fix them:

# Common mistake: not matching embedding dimension in positional encoding
def bad_positional_encoding(seq_len, d_model):
    return np.zeros((seq_len, d_model-1))  # Wrong shape

# Corrected version
def good_positional_encoding(seq_len, d_model):
    return np.zeros((seq_len, d_model))

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Implement and test multi-head attention in PyTorch.

Instructions:

Define a MultiHeadAttention class as above
Pass a batch of random sequences through it
Print the output and attention weights shapes

Starter Code:

import torch
# Your code here

Solution:

mha = MultiHeadAttention(embed_dim=32, num_heads=4)
x = torch.randn(8, 10, 32)
out, attn_weights = mha(x)
print(out.shape, attn_weights.shape)

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Implement and visualize positional encoding.

Requirements:

Write a function for positional encoding
Plot the encoding as a heatmap

Hints:

Use sine and cosine functions
Use matplotlib for visualization

Exercise 3: Creative Application (15 minutes)

Objective: Analyze the effect of positional encoding on sequence modeling.

Challenge:

Compare model outputs with and without positional encoding
Visualize and interpret the differences

Extension Ideas:

Try different encoding schemes (learned vs. fixed)
Experiment with different sequence lengths

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

Problem: Mismatched dimensions in multi-head attention or positional encoding Cause: Incorrect input shapes Solution: Always check and match dimensions
Problem: Forgetting to add positional encoding to input embeddings Cause: Model loses sense of order Solution: Always add positional encoding before feeding to the model

Debugging Tips

Print tensor shapes at each step
Visualize positional encodings
Use small test cases to verify implementation

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

Use multi-head attention for all Transformer models
Always include positional encoding
Document model architecture and hyperparameters

Next Steps in Your Learning Journey

Explore full Transformer encoder and decoder stacks
Learn about training objectives (e.g., translation)
Try implementing a mini-Transformer

9. Summary and Review (5 minutes)

Key Takeaways

Multi-head attention enables the model to capture diverse relationships
Positional encoding is essential for sequence understanding in Transformers
Careful implementation and visualization help avoid common mistakes

Quick Review Questions

What is the main advantage of multi-head attention?
Why is positional encoding necessary in Transformers?
How can you visualize positional encodings?

For Instructors

Teaching Tips

Use diagrams to illustrate multi-head attention and positional encoding
Encourage students to experiment with different numbers of heads and encoding schemes
Discuss real-world applications (translation, summarization)

Assessment Ideas

Quiz on multi-head attention and positional encoding concepts
Coding assignment: implement and visualize positional encoding
Peer review: students explain their visualizations and code

Adaptation Notes

For beginners: focus on basic implementation and visualization
For advanced students: experiment with learned positional encodings or custom attention heads
For online delivery: use Colab or Jupyter for interactive coding and visualization