LLM chevron_right Sequence Models Before Transformers: RNNs, LSTMs, and GRUs

Sequence Models Before Transformers: RNNs, LSTMs, and GRUs

calendar_today schedule 7 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Explain the motivation and mechanics of RNNs, LSTMs, and GRUs
  • Implement a simple LSTM for sentiment analysis in PyTorch or TensorFlow
  • Identify the limitations of sequence models and why Transformers were needed

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Have you ever wondered how your phone predicts the next word as you type, or how chatbots can generate sentences that make sense? What do you think makes understanding sequences (like sentences) harder than just looking at single words?

Lesson Overview

  • Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
  • Prerequisites: Python, basic understanding of vectors and neural networks, previous lessons on tokenization and embeddings
  • Materials Needed: Python (Jupyter Notebook or IDE), PyTorch or TensorFlow, sample text data

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. The challenge of modeling sequences in NLP
  2. Recurrent Neural Networks (RNNs): how they work and their limitations
  3. LSTMs and GRUs: overcoming RNN challenges
  4. Encoder-Decoder (seq2seq) architecture
  5. Hands-on practice and exercises
  6. Common pitfalls and how to avoid them

Success Criteria:

By the end of this session, you should be able to build and train a simple LSTM for a basic NLP task and explain why newer architectures were developed.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Why Sequence Modeling?

Many NLP tasks require understanding the order and context of words (e.g., translation, sentiment analysis). Sequence models process data step-by-step, maintaining a memory of previous steps.

Real-World Analogy: Think of reading a sentence: you need to remember the words you’ve already read to understand the next one.

Why This Matters: Without sequence modeling, models can’t capture dependencies across time or word order.

Recurrent Neural Networks (RNNs)

RNNs process sequences by maintaining a hidden state that is updated at each step. However, they struggle with long-term dependencies due to vanishing/exploding gradients.

LSTMs and GRUs

LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent Units) are advanced RNN variants designed to remember information over longer sequences using gating mechanisms.

Encoder-Decoder (seq2seq) Architecture

This architecture is used for tasks like translation, where an encoder processes the input sequence and a decoder generates the output sequence.

Key Terminology

  • Hidden State: The memory of the network at each time step
  • Vanishing/Exploding Gradients: Problems that make training deep RNNs difficult
  • Gates (LSTM/GRU): Mechanisms that control the flow of information
  • Encoder-Decoder: Architecture for mapping input sequences to output sequences

4. Detailed Explanation and Examples (25-30 minutes)

Understanding Sequence Models

RNNs process input one step at a time, updating their hidden state. However, as sequences get longer, it becomes harder for the model to remember earlier information.

LSTMs and GRUs introduce gates to control what information is kept or forgotten, making them better at handling long-term dependencies.

Practical Applications

  • Sentiment analysis (e.g., “I didn’t like the movie” vs. “I liked the movie”)
  • Machine translation
  • Text generation

Common Misconceptions

  1. Misconception: RNNs can remember everything in a sequence. Reality: RNNs struggle with long-term dependencies due to vanishing gradients.
  2. Misconception: LSTMs/GRUs are always better than RNNs. Reality: They are better for long sequences, but may be overkill for short/simple tasks.

5. Live Coding Demonstration (20-25 minutes)

Example 1: Simple RNN in PyTorch

import torch
import torch.nn as nn

rnn = nn.RNN(input_size=10, hidden_size=20, num_layers=1, batch_first=True)
inputs = torch.randn(5, 3, 10)  # (batch, seq_len, input_size)
h0 = torch.zeros(1, 5, 20)
output, hn = rnn(inputs, h0)
print(output.shape)  # (5, 3, 20)

Explanation:

  • Defines a simple RNN layer
  • Processes a batch of sequences
  • Returns outputs for each time step

Example 2: LSTM for Sentiment Analysis (PyTorch)

import torch
import torch.nn as nn

class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, output_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)
    def forward(self, x):
        x = self.embedding(x)
        _, (h_n, _) = self.lstm(x)
        out = self.fc(h_n[-1])
        return out

# Example usage:
model = SentimentLSTM(vocab_size=1000, embed_dim=50, hidden_dim=64, output_dim=2)
inputs = torch.randint(0, 1000, (32, 20))  # (batch, seq_len)
outputs = model(inputs)
print(outputs.shape)  # (32, 2)

Key Points:

  • Embedding layer converts word indices to vectors
  • LSTM processes the sequence
  • Final hidden state is used for classification

Example 3: Encoder-Decoder (seq2seq) Skeleton

class Encoder(nn.Module):
    def __init__(self, ...):
        super().__init__()
        # ...
    def forward(self, x):
        # ...
        return hidden

class Decoder(nn.Module):
    def __init__(self, ...):
        super().__init__()
        # ...
    def forward(self, x, hidden):
        # ...
        return output

Debugging Demonstration

Show common errors and how to fix them:

# Common mistake: mismatched input dimensions
try:
    wrong_inputs = torch.randn(5, 10)  # Missing sequence dimension
    output, hn = rnn(wrong_inputs, h0)
except RuntimeError as e:
    print(e)

# Corrected version
inputs = torch.randn(5, 3, 10)
output, hn = rnn(inputs, h0)

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Build a simple RNN or LSTM and run it on random data.

Instructions:

  1. Define an RNN or LSTM layer in PyTorch or TensorFlow
  2. Pass a batch of random sequences through it
  3. Print the output shape

Starter Code:

import torch
import torch.nn as nn

# Your code here

Solution:

import torch
import torch.nn as nn

lstm = nn.LSTM(input_size=8, hidden_size=16, batch_first=True)
inputs = torch.randn(4, 5, 8)
h0 = torch.zeros(1, 4, 16)
c0 = torch.zeros(1, 4, 16)
output, (hn, cn) = lstm(inputs, (h0, c0))
print(output.shape)  # (4, 5, 16)

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Implement a sentiment classifier using an LSTM.

Requirements:

  • Use an embedding layer
  • Use the final hidden state for classification
  • Train on a small dataset (e.g., IMDB, or synthetic data)

Hints:

  • Use nn.Embedding, nn.LSTM, nn.Linear
  • Use CrossEntropyLoss for training

Exercise 3: Creative Application (15 minutes)

Objective: Build a sequence-to-sequence model skeleton for translation.

Challenge:

  • Define Encoder and Decoder classes
  • Pass a sequence through the encoder, then use the hidden state to initialize the decoder

Extension Ideas:

  • Add attention mechanism (advanced)
  • Try with real translation data

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Passing hidden/cell states with the wrong dimensions Cause: Confusing layer, direction, batch, and hidden dimensions Solution: Omit states to use PyTorch zero defaults, or validate shapes when passing them explicitly

  2. Problem: Mismatched input dimensions Cause: Sequence and batch dimensions are swapped Solution: Use batch_first=True or adjust input shape

Debugging Tips

  • Print shapes at each step
  • Use small random data to test before using real data
  • Read error messages carefully for dimension mismatches

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

  • Use LSTMs/GRUs for sequence tasks where context matters
  • Prefer LSTMs/GRUs over vanilla RNNs for long sequences
  • Document model architecture and input shapes

Next Steps in Your Learning Journey

  • Explore attention mechanisms
  • Learn about the limitations of sequence models
  • Prepare for the Transformer architecture

Accuracy Note: Gradient Flow and PyTorch State Defaults

LSTM and GRU gates provide paths that mitigate, but do not eliminate, vanishing or exploding gradients, especially on very long sequences. Gradient clipping, initialization, normalization, and sequence length still matter. In PyTorch, nn.LSTM and nn.GRU do not require an explicit initial state: omitting it creates zero-valued hidden (and, for LSTM, cell) states with the documented shape. Pass a state only when continuing a sequence or intentionally choosing a different initialization.

9. Summary and Review (5 minutes)

Key Takeaways

  1. Sequence models process data step-by-step, maintaining memory
  2. LSTMs and GRUs mitigate, but do not eliminate, vanishing gradients
  3. Transformers were developed to overcome sequence model limitations

Quick Review Questions

  1. What is the main limitation of vanilla RNNs?
  2. How do LSTMs and GRUs improve on RNNs?
  3. What is the purpose of the encoder-decoder architecture?

For Instructors

Teaching Tips

  • Use visualizations to show how information flows through RNNs/LSTMs
  • Encourage students to experiment with sequence lengths
  • Discuss real-world applications (translation, chatbots)

Assessment Ideas

  • Quiz on sequence model concepts
  • Coding assignment: build a sentiment classifier
  • Peer review: students explain their model architectures

Adaptation Notes

  • For beginners: focus on running provided code and understanding outputs
  • For advanced students: implement attention or try real datasets
  • For online delivery: use Colab or Jupyter for interactive coding