LLM chevron_right Introduction to RAG: Why and How

Introduction to RAG: Why and How

calendar_today schedule 6 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Explain the motivation for Retrieval Augmented Generation (RAG) in LLMs
  • Describe the high-level RAG pipeline and its components
  • Identify use cases where RAG is beneficial and understand its limitations

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Have you ever asked a language model a question and received an outdated or incorrect answer? What do you wish it could do better?

Lesson Overview

  • Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
  • Prerequisites: Familiarity with LLMs and basic NLP concepts
  • Materials Needed: Python environment, access to Hugging Face Transformers, sample documents for retrieval

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. The limitations of parametric knowledge in LLMs
  2. The core concept of RAG: combining retrieval and generation
  3. The high-level RAG pipeline
  4. Hands-on: Diagramming and discussing RAG use cases
  5. Common pitfalls and best practices

Success Criteria:

By the end of this session, you should be able to explain why RAG is needed, describe its pipeline, and identify practical applications.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Why Do LLMs Need Retrieval?

LLMs are trained on large but static datasets. Their knowledge is fixed at the time of training, leading to issues like knowledge cut-offs, hallucinations, and difficulty updating information.

Real-World Analogy: Imagine a student who memorized all their textbooks up to 2022. If you ask about a 2024 event, they can’t answer unless they can look it up!

Why This Matters:

  • LLMs can’t access new or private information after training
  • They may hallucinate facts when unsure
  • Updating model weights is expensive and slow

What is Retrieval Augmented Generation (RAG)?

RAG is a framework that augments LLMs with an external retriever. When given a query, the system first retrieves relevant documents, then feeds them (as context) to the LLM for generation.

Key Terminology

  • Parametric Knowledge: Information stored in model weights
  • Retriever: Component that finds relevant documents or passages
  • Generator: The LLM that produces the final answer using retrieved context
  • Context Window: The maximum input length the LLM can process

4. Detailed Explanation and Examples (25-30 minutes)

Understanding the RAG Pipeline

High-Level Steps

  1. Query: User asks a question
  2. Retriever: Finds relevant documents from a database or corpus
  3. Context: Retrieved documents are added to the prompt
  4. LLM (Generator): Generates an answer using both the query and context

Practical Applications

  • Q&A over private or proprietary documents
  • Keeping LLMs up-to-date with new information
  • Reducing hallucinations by grounding answers in retrieved evidence

Common Misconceptions

  1. Misconception: RAG makes the LLM “smarter” by itself Reality: RAG augments the LLM with external knowledge, but the LLM’s reasoning is still limited by its training
  2. Misconception: RAG is only for search engines Reality: RAG is useful for chatbots, assistants, research tools, and more

5. Live Coding Demonstration (20-25 minutes)

Example 1: Diagramming the RAG Pipeline

graph TD;
    A[User Query] --> B[Retriever];
    B --> C[Retrieved Context];
    C --> D[LLM Generator];
    D --> E[Augmented Output];

Example 2: Manual RAG with Python

# Sample documents
corpus = [
    "The Eiffel Tower is in Paris.",
    "Mount Everest is the tallest mountain.",
    "The capital of Japan is Tokyo."
]
query = "Where is the Eiffel Tower?"

# Simple retriever: keyword search
retrieved = [doc for doc in corpus if "Eiffel Tower" in doc]
context = "\n".join(retrieved)

# Simulate LLM generation
prompt = f"Context: {context}\nQuestion: {query}\nAnswer:"
print(prompt)
# (In practice, feed this to an LLM)

Explanation:

  • The retriever finds relevant facts
  • The LLM uses these facts to answer the question

Example 3: Prompt Engineering for RAG

prompt = f"Answer the question based on the following context:\n{context}\n\nQuestion: {query}\nAnswer:"

Debugging Demonstration

Show a common error: context window overflow

# If too many documents are retrieved, the prompt may exceed the LLM's context window
# Solution: Limit number of retrieved documents or summarize context

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Simulate a RAG pipeline with a small corpus

Instructions:

  1. Create a list of 5-10 short documents
  2. Write a function to retrieve documents containing a keyword from a query
  3. Construct a prompt for the LLM using the retrieved context

Starter Code:

corpus = [
    "Python is a popular programming language.",
    "The sun rises in the east.",
    # ...more documents...
]
query = "What is Python?"
# Your retrieval and prompt code here

Solution:

def retrieve(corpus, query):
    return [doc for doc in corpus if any(word in doc for word in query.split())]
context = "\n".join(retrieve(corpus, query))
prompt = f"Context: {context}\nQuestion: {query}\nAnswer:"
print(prompt)

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Discuss RAG use cases and limitations

Requirements:

  • List 3 real-world scenarios where RAG is beneficial
  • Identify 2 limitations or challenges of RAG

Hints:

  • Think about up-to-date information, private data, hallucination reduction
  • Consider context window size, retrieval quality

Exercise 3: Creative Application (15 minutes)

Objective: Design a RAG pipeline for a specific use case

Challenge:

  • Choose a domain (e.g., legal, medical, academic)
  • Sketch a RAG pipeline: What is the retriever? What is the corpus? How is context integrated?

Extension Ideas:

  • Consider how to update the corpus dynamically
  • Think about privacy and security concerns

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Retrieving irrelevant or too many documents Cause: Poor retrieval strategy Solution: Use better retrieval methods, limit number of documents
  2. Problem: LLM ignores context Cause: Poor prompt engineering Solution: Explicitly instruct the LLM to use the context

Debugging Tips

  • Always check the retrieved context for relevance
  • Monitor prompt length and context window usage
  • Experiment with prompt phrasing

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

  • Use semantic search (embeddings) for better retrieval
  • Limit and prioritize context for the LLM
  • Log queries, retrieved docs, and outputs for analysis

Next Steps in Your Learning Journey

  • Explore vector databases and semantic search
  • Try RAG with real LLMs (e.g., via Hugging Face pipelines)
  • Read RAG research papers and case studies

Grounding Is Not a Guarantee

RAG can supply timely or private evidence and often reduces unsupported answers, but it does not prevent hallucinations: retrieval can miss, rank poorly, or return misleading text, and the generator can ignore evidence. Model weights do not automatically acquire new or private information; it must be supplied through retrieval, tools, or retraining. The keyword-matching implementation in this lesson is a toy retriever for learning, not a production relevance system.

9. Summary and Review (5 minutes)

Key Takeaways

  1. RAG augments LLMs with external knowledge for better, up-to-date answers
  2. The RAG pipeline involves retrieval, context integration, and generation
  3. Prompt engineering and retrieval quality are critical for success

Quick Review Questions

  1. Why do LLMs need retrieval augmentation?
  2. What are the main steps in a RAG pipeline?
  3. What is a common challenge when integrating retrieval with LLMs?

For Instructors

Teaching Tips

  • Use diagrams to illustrate the RAG pipeline
  • Encourage students to experiment with different retrieval strategies
  • Address confusion about context integration and prompt design

Assessment Ideas

  • Quiz: Identify RAG components in a scenario
  • Project: Build a simple RAG demo with a local corpus
  • Peer review: Analyze prompt effectiveness and retrieval quality

Adaptation Notes

  • For beginners: Focus on manual retrieval and prompt construction
  • For advanced: Introduce semantic search and vector databases
  • Online: Provide Colab notebooks; In-person: Live coding demos