LLM chevron_right RLHF: The Core Loop – SFT, Reward Modeling, and PPO

RLHF: The Core Loop – SFT, Reward Modeling, and PPO

calendar_today schedule 6 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Describe the three main stages of Reinforcement Learning from Human Feedback (RLHF)
  • Explain the purpose and process of Supervised Fine-Tuning (SFT), Reward Modeling (RM), and Proximal Policy Optimization (PPO)
  • Identify common challenges and best practices in RLHF pipelines

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Have you ever interacted with a chatbot that felt more helpful or polite than others? What do you think made the difference?

Lesson Overview

  • Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
  • Prerequisites: Understanding of LLMs, supervised fine-tuning, and basic reinforcement learning concepts
  • Materials Needed: Python environment, Hugging Face Transformers, TRL (Transformers Reinforcement Learning) library, access to RLHF-related papers (e.g., InstructGPT, Llama 2)

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. The motivation for RLHF and its role in LLM alignment
  2. Stage 1: Supervised Fine-Tuning (SFT)
  3. Stage 2: Reward Modeling (RM)
  4. Stage 3: Reinforcement Learning (PPO)
  5. Hands-on: Simulating RLHF steps
  6. Common pitfalls and troubleshooting

Success Criteria:

By the end of this session, you should be able to outline and implement a basic RLHF pipeline and explain the function of each stage.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Why RLHF?

Large language models can generate fluent text, but may not always align with human preferences (e.g., helpfulness, harmlessness, honesty). RLHF is a process to align LLMs with human values by incorporating human feedback into training.

Real-World Analogy: Imagine training a customer service agent. First, you teach them with example conversations (SFT). Next, you rate their responses (RM). Finally, you let them practice, rewarding good behavior and correcting mistakes (PPO).

Why This Matters:

  • Produces models that are safer, more helpful, and more aligned with user intent
  • Reduces toxic, biased, or unhelpful outputs
  • Underpins state-of-the-art models like ChatGPT, Llama 2-Chat

The Three Stages of RLHF

  1. Supervised Fine-Tuning (SFT): Train the model on high-quality, instruction-following data
  2. Reward Modeling (RM): Train a model to predict human preferences by ranking model outputs
  3. Reinforcement Learning (PPO): Fine-tune the SFT model using the reward model as a reward signal

Key Terminology

  • RLHF: Reinforcement Learning from Human Feedback
  • SFT: Supervised Fine-Tuning
  • Reward Model (RM): Model trained to score outputs based on human preferences
  • PPO: Proximal Policy Optimization, a reinforcement learning algorithm

4. Detailed Explanation and Examples (25-30 minutes)

Understanding the RLHF Core Loop

Stage 1: Supervised Fine-Tuning (SFT)

  • Collect high-quality instruction-following data (e.g., human-written responses)
  • Fine-tune a pre-trained LLM on this data
  • Result: SFT model that follows instructions better than the base model

Stage 2: Reward Modeling (RM)

  • Collect pairs of model outputs for the same prompt
  • Humans rank which output is better
  • Train a reward model to predict these preferences (higher score = preferred)

Stage 3: Reinforcement Learning (PPO)

  • Use the reward model to score new outputs from the SFT model
  • Fine-tune the SFT model with PPO, maximizing the reward signal
  • Result: Model that generates outputs more aligned with human preferences

Practical Applications

  • Chatbots, virtual assistants, content moderation, and more

Common Misconceptions

  1. Misconception: RLHF is just supervised learning Reality: RLHF combines supervised and reinforcement learning with human feedback
  2. Misconception: The reward model is perfect Reality: Reward models can be biased or gamed by the LLM

5. Live Coding Demonstration (20-25 minutes)

Example 1: Supervised Fine-Tuning (SFT)

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer

dataset = load_dataset('tatsu-lab/alpaca')
model_name = 'tiiuae/falcon-7b-instruct'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Format data as (prompt, response) pairs
# ...see previous lesson for formatting...

# TrainingArguments and Trainer setup as before

Example 2: Reward Modeling (RM)

# Assume you have pairs of outputs and human preference labels
# Each example: {'prompt': ..., 'chosen': ..., 'rejected': ...}
from transformers import AutoModelForSequenceClassification

rm_model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=1)
# Train RM to score 'chosen' higher than 'rejected' (pairwise ranking loss)

Example 3: PPO Architecture (Version-Sensitive Pseudocode)

# API details change between TRL releases; pin and follow that release's docs.
policy = load_trainable_policy("sft-checkpoint")
reference = load_frozen_copy("sft-checkpoint")  # a distinct, frozen model
reward_model = load_frozen_reward_model("reward-checkpoint")

for prompts in preference_dataloader:
    responses = policy.generate(prompts)
    rewards = reward_model.score(prompts, responses)
    ppo_update(policy, reference, prompts, responses, rewards)

SFT → reward modeling → PPO is one historically important RLHF recipe, not the only alignment method. Direct preference optimization and other approaches are also used. A high preference-model score does not guarantee safety, truth, or alignment with diverse human values.

Debugging Demonstration

Show a common error: reward hacking

# Model learns to exploit reward model quirks (e.g., always outputting certain phrases)
# Solution: Regularize, use diverse data, monitor for reward gaming

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Simulate SFT and RM steps with toy data

Instructions:

  1. Create a small dataset of prompts and responses
  2. Simulate human preferences by ranking responses
  3. Train a simple reward model to predict preferences

Starter Code:

# Toy data
examples = [
    {'prompt': 'Say hello.', 'chosen': 'Hello!', 'rejected': 'Hi.'},
    {'prompt': 'Say goodbye.', 'chosen': 'Goodbye!', 'rejected': 'Bye.'}
]
# Your reward model training code here

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Implement a PPO training loop with a reward model

Requirements:

  • Use a small model (e.g., distilgpt2)
  • Generate responses, score with reward model, update with PPO

Hints:

  • Use TRL library for PPO
  • Use dummy reward model if needed

Exercise 3: Creative Application (15 minutes)

Objective: Explore reward hacking

Challenge:

  • Modify the reward model to have a bias (e.g., always prefers longer responses)
  • Observe how the LLM adapts to exploit this bias

Extension Ideas:

  • Try different reward model architectures
  • Experiment with human-in-the-loop feedback

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Reward model overfits or is gamed Cause: Poor data or lack of regularization Solution: Use diverse, high-quality data; monitor for reward hacking
  2. Problem: PPO training is unstable Cause: Bad hyperparameters or reward signal Solution: Tune learning rate, batch size, and reward scaling

Debugging Tips

  • Visualize reward distributions
  • Regularly sample and inspect model outputs
  • Use a reference model to prevent catastrophic forgetting

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

  • Use large, diverse preference datasets for RM
  • Regularize reward models and monitor for gaming
  • Combine automated and human evaluation

Next Steps in Your Learning Journey

  • Explore advanced RL algorithms (e.g., DPO, RLAIF)
  • Read InstructGPT and Llama 2 RLHF sections
  • Try RLHF on your own data

9. Summary and Review (5 minutes)

Key Takeaways

  1. RLHF aligns LLMs with human preferences via SFT, RM, and PPO
  2. Each stage has unique challenges and best practices
  3. Reward hacking is a real risk—monitor and mitigate

Quick Review Questions

  1. What are the three stages of RLHF?
  2. How is a reward model trained?
  3. What is reward hacking and how can it be prevented?

For Instructors

Teaching Tips

  • Emphasize the iterative nature of RLHF
  • Use visual diagrams to explain the core loop
  • Address confusion about reward modeling and PPO

Assessment Ideas

  • Quiz: Identify RLHF stages in a code snippet
  • Project: Simulate RLHF with toy data
  • Peer review: Analyze reward hacking examples

Adaptation Notes

  • For beginners: Focus on SFT and RM basics
  • For advanced: Dive into PPO and reward hacking
  • Online: Provide Colab notebooks; In-person: Live coding demos