LLM chevron_right Evaluating RAG Systems and Iterative Improvement

Evaluating RAG Systems and Iterative Improvement

calendar_today schedule 6 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Apply key metrics to evaluate retrieval and generation in RAG systems
  • Identify and address common failure modes in RAG pipelines
  • Use frameworks and best practices for iterative improvement

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Have you ever used a search or Q&A system that gave you a wrong or irrelevant answer? How did you know it was wrong, and what would you do to improve it?

Lesson Overview

  • Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
  • Prerequisites: Familiarity with RAG pipelines, retrieval, and LLM generation
  • Materials Needed: Python environment, RAG system from previous lesson, evaluation datasets, RAGAS/TruLens (optional)

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Metrics for evaluating retrieval and generation
  2. End-to-end evaluation frameworks
  3. Common failure modes in RAG
  4. Hands-on: Critically evaluating and improving a RAG system
  5. Best practices for iterative improvement

Success Criteria:

By the end of this session, you should be able to measure, analyze, and improve the quality of a RAG system.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Why Evaluate RAG Systems?

Evaluation ensures that your RAG system retrieves relevant information and generates accurate, grounded answers. Without proper evaluation, you risk deploying systems that hallucinate, ignore context, or provide irrelevant results.

Real-World Analogy: Think of a librarian and an expert answering questions. You want to check if the librarian finds the right books and if the expert gives correct, well-sourced answers.

Why This Matters:

  • Ensures reliability and trustworthiness
  • Identifies weaknesses for targeted improvement
  • Supports continuous, data-driven iteration

Key Terminology

  • Precision@k: Fraction of top-k retrieved documents that are relevant
  • Recall@k: Fraction of all relevant documents retrieved in top-k
  • Mean Reciprocal Rank (MRR): Average of reciprocal ranks of the first relevant result
  • Faithfulness/Attribution: Does the answer stick to the retrieved context?
  • Relevance: Is the answer on-topic and useful?
  • Fluency: Is the answer well-formed and readable?

4. Detailed Explanation and Examples (25-30 minutes)

Retrieval Metrics

  • Precision@k: How many of the top-k retrieved chunks are actually relevant?
  • Recall@k: How many relevant chunks are found in the top-k?
  • MRR: How soon does the system retrieve a relevant chunk?

Generation Metrics

  • Faithfulness/Attribution: Does the answer use only the retrieved context?
  • Relevance: Is the answer useful for the query?
  • Fluency: Is the answer grammatically correct and clear?
  • Human Evaluation: Manual review is often necessary for nuanced tasks

End-to-End Evaluation Frameworks

  • RAGAS: Open-source toolkit for RAG evaluation (retrieval, generation, and end-to-end)
  • TruLens: Framework for monitoring and evaluating LLM applications

Common Failure Modes

  1. Poor Retrieval: Irrelevant or missing context
  2. LLM Ignores Context: Hallucinated or off-topic answers
  3. Hallucination Despite Context: LLM invents facts not in retrieved data

5. Live Coding Demonstration (20-25 minutes)

Example 1: Calculating Precision@k and Recall@k

# Assume you have a list of retrieved chunks and a set of ground-truth relevant chunks
retrieved = ["chunk1", "chunk2", "chunk3"]
relevant = set(["chunk2", "chunk4"])

precision_at_k = len([c for c in retrieved if c in relevant]) / len(retrieved)
recall_at_k = len([c for c in retrieved if c in relevant]) / len(relevant)
print(f"Precision@k: {precision_at_k:.2f}, Recall@k: {recall_at_k:.2f}")

Example 2: Faithfulness and Attribution

# A tiny deterministic check for this controlled example. Real evaluation
# should split an answer into claims and use entailment plus human review.
supported_facts = {"eiffel_tower_location": "paris"}
answer_claims = {"eiffel_tower_location": "paris"}
faithful = all(supported_facts.get(k) == v for k, v in answer_claims.items())
print("All answer claims supported:", faithful)

Example 3: Using RAGAS (Optional)

# Pseudocode only: pin a Ragas version and implement its documented API.
def evaluate_rag_system(held_out_examples):
    return run_version_pinned_ragas_evaluation(held_out_examples)

Debugging Demonstration

Show a common error: LLM hallucination

# Problem: LLM answers "The Eiffel Tower is in London." (not in context)
# Solution: Improve prompt, retrieval, or add faithfulness checks

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Calculate retrieval metrics for your RAG system

Instructions:

  1. For a set of queries, record retrieved chunks and ground-truth relevant chunks
  2. Compute Precision@k and Recall@k

Starter Code:

retrieved = ["chunk1", "chunk2", "chunk3"]
relevant = set(["chunk2", "chunk4"])
# Your metric calculation code here

Solution:

precision_at_k = len([c for c in retrieved if c in relevant]) / len(retrieved)
recall_at_k = len([c for c in retrieved if c in relevant]) / len(relevant)
print(f"Precision@k: {precision_at_k:.2f}, Recall@k: {recall_at_k:.2f}")

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Evaluate generation faithfulness and relevance

Requirements:

  • Compare generated answers to retrieved context
  • Rate faithfulness and relevance (e.g., 1-5 scale)

Hints:

  • Use manual review or simple string matching
  • Try RAGAS or TruLens if available

Exercise 3: Creative Application (15 minutes)

Objective: Iteratively improve your RAG system

Challenge:

  • Identify a failure mode (e.g., poor retrieval, hallucination)
  • Brainstorm and implement an improvement (e.g., better chunking, prompt tuning)
  • Measure before/after metrics

Extension Ideas:

  • Add automated faithfulness checks
  • Experiment with re-ranking or prompt optimization

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Overfitting to evaluation set Cause: Tuning system on test data Solution: Use separate validation and test sets
  2. Problem: Ignoring human evaluation Cause: Relying only on automated metrics Solution: Include manual review for nuanced tasks

Debugging Tips

  • Visualize retrieval and answer distributions
  • Sample and review failures regularly
  • Track metrics over time for continuous improvement

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

  • Use a mix of automated and human evaluation
  • Track multiple metrics (retrieval, generation, end-to-end)
  • Iterate based on failure analysis and user feedback

Next Steps in Your Learning Journey

  • Explore advanced evaluation frameworks (RAGAS, TruLens)
  • Add re-ranking, prompt optimization, or feedback loops
  • Read case studies on RAG system improvement

Evaluation Protocol

A substring such as "paris" in context cannot establish faithfulness—it returns true regardless of what the answer claims. Split the answer into claims, retrieve the cited passages, and evaluate whether each claim is entailed, contradicted, or unsupported; aggregate those judgments and retain examples for human audit. Treat library snippets as version-pinned examples because Ragas APIs evolve. Build evaluation sets from held-out questions, include difficult and adversarial cases, and never tune on the final test set.

9. Summary and Review (5 minutes)

Key Takeaways

  1. Evaluation is critical for reliable RAG systems
  2. Use retrieval and generation metrics to guide improvement
  3. Iterative analysis and testing drive better performance

Quick Review Questions

  1. What are Precision@k and Recall@k?
  2. How do you check if an answer is faithful to context?
  3. What is a common failure mode in RAG systems?

For Instructors

Teaching Tips

  • Emphasize hands-on metric calculation and failure analysis
  • Encourage students to iterate and measure improvements
  • Address confusion about faithfulness and attribution

Assessment Ideas

  • Quiz: Calculate metrics from sample data
  • Project: Evaluate and improve a RAG system
  • Peer review: Analyze and discuss failure cases

Adaptation Notes

  • For beginners: Focus on retrieval metrics and manual review
  • For advanced: Use RAGAS/TruLens and automate evaluation
  • Online: Provide Colab notebooks; In-person: Live metric calculation demos