Transformer Blocks: Encoder and Decoder Stacks Explained

calendar_today May 28, 2025 schedule 6 min read

Learning Objectives: By the end of this lesson, you will be able to:

Describe the structure and function of Transformer encoder and decoder blocks

Diagram and explain the data flow through encoder and decoder stacks

Identify the role of each component (Self-Attention, Feed-Forward, Add & Norm, Masked Attention, Encoder-Decoder Attention)

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Imagine you are translating a sentence from English to French. How do you think a model keeps track of both the meaning of the input and what it has already generated in the output? Why might this be challenging for traditional models?

Lesson Overview

Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
Prerequisites: Understanding of self-attention, multi-head attention, and positional encoding
Materials Needed: Python (Jupyter Notebook or IDE), PyTorch or TensorFlow, pen and paper for diagramming

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

The overall architecture of Transformer encoder and decoder stacks
The flow of data through each block
The function of each sub-layer (Self-Attention, Feed-Forward, Add & Norm, Masked Attention, Encoder-Decoder Attention)
Hands-on practice and exercises
Common pitfalls and how to avoid them

Success Criteria:

By the end of this session, you should be able to diagram and explain the full Transformer block and its data flow.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Transformer Encoder Block

Each encoder block consists of:

Multi-Head Self-Attention
Add & Layer Normalization
Feed-Forward Network
Add & Layer Normalization

Transformer Decoder Block

Each decoder block consists of:

Masked Multi-Head Self-Attention (prevents attending to future tokens)
Add & Layer Normalization
Encoder-Decoder Attention (attends to encoder outputs)
Add & Layer Normalization
Feed-Forward Network
Add & Layer Normalization

Real-World Analogy: Think of the encoder as a team of experts analyzing a document, and the decoder as a translator who consults the experts while writing the translation, making sure not to “peek” at future words.

Why This Matters: Understanding the block structure is essential for customizing, debugging, or extending Transformer models.

Key Terminology

Add & Norm: Residual connection followed by layer normalization
Masked Attention: Prevents the decoder from seeing future tokens
Encoder-Decoder Attention: Allows the decoder to focus on relevant encoder outputs

4. Detailed Explanation and Examples (25-30 minutes)

Understanding the Transformer Block

The encoder stack processes the input sequence in parallel, passing information up through multiple layers. The decoder stack generates the output sequence one token at a time, using both its own previous outputs and the encoder’s representations.

Practical Applications

Machine translation
Text summarization
Sequence-to-sequence tasks (e.g., question answering)

Common Misconceptions

Misconception: The decoder can see the entire output sequence at once. Reality: Masked attention ensures the decoder only sees previous tokens.
Misconception: Encoder and decoder blocks are identical. Reality: Decoder blocks have additional masked attention and encoder-decoder attention layers.

5. Live Coding Demonstration (20-25 minutes)

Example 1: Encoder Block Skeleton (PyTorch)

import torch
import torch.nn as nn

class EncoderBlock(nn.Module):
    def __init__(self, embed_dim, num_heads, ff_dim):
        super().__init__()
        self.attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
        self.norm1 = nn.LayerNorm(embed_dim)
        self.ff = nn.Sequential(
            nn.Linear(embed_dim, ff_dim),
            nn.ReLU(),
            nn.Linear(ff_dim, embed_dim)
        )
        self.norm2 = nn.LayerNorm(embed_dim)
    def forward(self, x):
        attn_out, _ = self.attn(x, x, x)
        x = self.norm1(x + attn_out)
        ff_out = self.ff(x)
        x = self.norm2(x + ff_out)
        return x

Explanation:

Multi-head self-attention, followed by Add & Norm
Feed-forward network, followed by Add & Norm

Example 2: Decoder Block Skeleton (PyTorch)

class DecoderBlock(nn.Module):
    def __init__(self, embed_dim, num_heads, ff_dim):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
        self.norm1 = nn.LayerNorm(embed_dim)
        self.enc_dec_attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
        self.norm2 = nn.LayerNorm(embed_dim)
        self.ff = nn.Sequential(
            nn.Linear(embed_dim, ff_dim),
            nn.ReLU(),
            nn.Linear(ff_dim, embed_dim)
        )
        self.norm3 = nn.LayerNorm(embed_dim)
    def forward(self, x, enc_out, tgt_mask=None):
        if tgt_mask is None:
            length = x.size(1)
            tgt_mask = torch.triu(
                torch.ones(length, length, dtype=torch.bool, device=x.device),
                diagonal=1,
            )
        self_attn_out, _ = self.self_attn(x, x, x, attn_mask=tgt_mask)
        x = self.norm1(x + self_attn_out)
        enc_dec_attn_out, _ = self.enc_dec_attn(x, enc_out, enc_out)
        x = self.norm2(x + enc_dec_attn_out)
        ff_out = self.ff(x)
        x = self.norm3(x + ff_out)
        return x

Key Points:

Masked self-attention prevents peeking at future tokens
Encoder-decoder attention allows the decoder to use encoder outputs
Each sub-layer is followed by Add & Norm

Example 3: Visualizing the Data Flow

# Pseudocode for data flow
input_seq -> [Encoder Stack] -> encoder_outputs
start_token -> [Decoder Stack (uses encoder_outputs)] -> output_seq

Debugging Demonstration

Show common errors and how to fix them:

# Common mistake: not using masks in decoder self-attention
try:
    self_attn_out, _ = self.self_attn(x, x, x)  # No mask
except Exception as e:
    print(e)

# Corrected version
self_attn_out, _ = self.self_attn(x, x, x, attn_mask=tgt_mask)

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Implement an encoder block and pass random data through it.

Instructions:

Define an EncoderBlock class as above
Pass a batch of random sequences through it
Print the output shape

Starter Code:

import torch
# Your code here

Solution:

block = EncoderBlock(embed_dim=32, num_heads=4, ff_dim=64)
x = torch.randn(8, 10, 32)
out = block(x)
print(out.shape)  # (8, 10, 32)

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Implement a decoder block and use it with encoder outputs.

Requirements:

Use masked self-attention in the decoder
Pass encoder outputs to encoder-decoder attention
Print the output shape

Hints:

Use nn.MultiheadAttention with attn_mask for masking
Input shapes should match between encoder and decoder

Exercise 3: Creative Application (15 minutes)

Objective: Diagram the full data flow through encoder and decoder stacks.

Challenge:

Draw or code a diagram showing how data moves from input to output
Label each sub-layer and connection

Extension Ideas:

Add comments or annotations explaining each step
Try implementing a mini-Transformer with multiple encoder/decoder blocks

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

Problem: Forgetting to use masks in decoder self-attention Cause: Decoder can see future tokens, leading to data leakage Solution: Always use masks in decoder self-attention
Problem: Mismatched input/output shapes between encoder and decoder Cause: Incorrect tensor dimensions Solution: Check and match shapes at each step

Debugging Tips

Print tensor shapes at each layer
Visualize attention masks
Use small test cases to verify implementation

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

Use residual connections and layer normalization after each sub-layer
Always mask future tokens in the decoder
Document model architecture and data flow

Next Steps in Your Learning Journey

Explore full Transformer models for translation
Learn about training objectives and loss functions
Try implementing a full encoder-decoder model

Causal-Mask Requirement

During teacher-forced decoder training, construct an upper-triangular causal mask and pass it on every decoder call; a default of None permits attention to future target tokens. For example, mask = torch.triu(torch.ones(T, T, dtype=torch.bool, device=x.device), diagonal=1). Padding masks are separate and should also be supplied when batches contain padding.

9. Summary and Review (5 minutes)

Key Takeaways

Transformer blocks are built from attention, feed-forward, and normalization layers
Decoder blocks use masking and encoder-decoder attention
Understanding the data flow is crucial for building and debugging models

Quick Review Questions

What is the purpose of Add & Norm in Transformer blocks?
Why is masking necessary in the decoder?
How does encoder-decoder attention work?

For Instructors

Teaching Tips

Use diagrams to illustrate encoder and decoder stacks
Encourage students to step through the data flow
Discuss real-world applications (translation, summarization)

Assessment Ideas

Quiz on block structure and data flow
Coding assignment: implement encoder and decoder blocks
Peer review: students explain their diagrams

Adaptation Notes

For beginners: focus on single encoder/decoder block
For advanced students: stack multiple blocks and implement masking
For online delivery: use Colab or Jupyter for interactive coding and diagramming