LLM chevron_right The Retriever Component in RAG: Vector Databases and Semantic Search

The Retriever Component in RAG: Vector Databases and Semantic Search

calendar_today schedule 6 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Explain the role of the retriever in Retrieval Augmented Generation (RAG)
  • Describe and implement document chunking, embedding, and vector database indexing
  • Perform semantic search and evaluate retrieval quality

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Have you ever tried to search for information in a large document or website? What made it easy or difficult to find what you needed?

Lesson Overview

  • Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
  • Prerequisites: Familiarity with LLMs, RAG basics, and Python programming
  • Materials Needed: Python environment, Hugging Face Transformers, FAISS or ChromaDB, sample text documents

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Document preprocessing and chunking strategies
  2. Embedding models for semantic search
  3. Vector databases: FAISS, ChromaDB, Pinecone, etc.
  4. Performing similarity search and evaluating retrieval
  5. Hands-on: Building a simple retriever pipeline

Success Criteria:

By the end of this session, you should be able to preprocess documents, generate embeddings, index them in a vector database, and perform semantic search.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Why Chunk and Embed Documents?

LLMs have limited context windows. To retrieve relevant information, documents are split into manageable chunks, each converted into a vector (embedding) that captures its meaning. This enables fast, semantic search over large corpora.

Real-World Analogy: Imagine a library where every paragraph is summarized on a card. To answer a question, you search the cards for the most relevant summaries, not the whole books.

Why This Matters:

  • Enables efficient search over large datasets
  • Improves retrieval relevance and LLM grounding
  • Supports up-to-date and domain-specific knowledge

Embedding Models

  • Convert text chunks into high-dimensional vectors
  • Examples: Sentence-BERT, OpenAI Ada, MiniLM
  • Similarity between vectors reflects semantic similarity

Vector Databases

  • Store and index embeddings for fast retrieval
  • Examples: FAISS (local), Pinecone, ChromaDB, Weaviate, Milvus
  • Support similarity search (cosine, dot product, ANN)

Key Terminology

  • Chunking: Splitting documents into smaller pieces for embedding
  • Embedding: Numeric vector representation of text
  • Vector Database: Specialized database for storing and searching embeddings
  • Semantic Search: Retrieval based on meaning, not just keywords

4. Detailed Explanation and Examples (25-30 minutes)

Document Preprocessing and Chunking

  • Fixed-size chunks (e.g., 512 tokens)
  • Semantic or recursive chunking (split at sentence/paragraph boundaries)
  • Trade-off: Larger chunks = more context, but risk of context window overflow

Embedding Generation

  • Use pre-trained models to convert text to vectors
  • Normalize vectors for cosine similarity
  • Index embeddings in a vector database
  • Query: Embed the question, search for nearest neighbors
  • Retrieve top-k most similar chunks

Practical Applications

  • Q&A over large document sets
  • Knowledge management and search
  • Chatbots with access to private or up-to-date data

Common Misconceptions

  1. Misconception: Keyword search is enough Reality: Semantic search finds relevant info even if keywords differ
  2. Misconception: All vector databases are the same Reality: They differ in scalability, speed, and features

5. Live Coding Demonstration (20-25 minutes)

Example 1: Chunking and Embedding Documents

from sentence_transformers import SentenceTransformer

# Sample documents
docs = [
    "The Eiffel Tower is in Paris.",
    "Mount Everest is the tallest mountain.",
    "The capital of Japan is Tokyo."
]
# Chunking (trivial here, but use NLTK or spaCy for real docs)
chunks = docs

# Embedding
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(chunks, normalize_embeddings=True)

Example 2: Indexing with FAISS

import faiss
import numpy as np

index = faiss.IndexFlatIP(embeddings.shape[1])  # Inner product (cosine if normalized)
index.add(np.array(embeddings))

# Query
query = "Where is the Eiffel Tower?"
q_emb = model.encode([query], normalize_embeddings=True)
D, I = index.search(np.array(q_emb), k=2)
print("Top results:", [chunks[i] for i in I[0]])

Example 3: Using ChromaDB (Optional)

import chromadb
from chromadb.utils import embedding_functions

chroma_client = chromadb.Client()
collection = chroma_client.create_collection("docs")
ids = [f"chunk-{i}" for i in range(len(chunks))]
collection.add(ids=ids, documents=chunks, embeddings=embeddings.tolist())
# Because vectors were supplied explicitly at indexing time, use the same
# SentenceTransformer to embed queries rather than Chroma's default function.
query_embedding = model.encode([query], normalize_embeddings=True)
results = collection.query(query_embeddings=query_embedding.tolist(), n_results=2)
print(results)

Chunk size and overlap are retrieval hyperparameters, not universal constants; tune them on held-out queries. Embedding similarity is a useful approximation of relevance, so production systems often add metadata filters or reranking.

Debugging Demonstration

Show a common error: inconsistent chunk sizes

# Problem: Some chunks are too large for the LLM context window
# Solution: Use consistent, manageable chunk sizes (e.g., 256-512 tokens)

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Chunk and embed a set of documents

Instructions:

  1. Create a list of 5-10 short text passages
  2. Use SentenceTransformer to generate embeddings
  3. Print the shape of the embeddings array

Starter Code:

from sentence_transformers import SentenceTransformer

docs = [
    "Python is a popular programming language.",
    "The sun rises in the east.",
    # ...more documents...
]
model = SentenceTransformer('all-MiniLM-L6-v2')
# Your embedding code here

Solution:

embeddings = model.encode(docs, normalize_embeddings=True)
print(embeddings.shape)

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Build a simple FAISS index and perform semantic search

Requirements:

  • Index the embeddings
  • Query with a new question
  • Print the top-2 most relevant passages

Hints:

  • Use FAISS IndexFlatIP for cosine similarity
  • Normalize embeddings

Exercise 3: Creative Application (15 minutes)

Objective: Experiment with chunking strategies

Challenge:

  • Try splitting a long document into sentences, paragraphs, or fixed token chunks
  • Compare retrieval results for different chunking methods

Extension Ideas:

  • Use a different embedding model (e.g., OpenAI Ada)
  • Try ChromaDB or Pinecone for indexing

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Chunks too large for LLM context Cause: Not splitting documents properly Solution: Use sentence/paragraph boundaries or fixed token sizes
  2. Problem: Poor retrieval quality Cause: Low-quality embeddings or bad chunking Solution: Use strong embedding models and experiment with chunk sizes

Debugging Tips

  • Visualize chunk sizes and embedding distributions
  • Test retrieval with known queries
  • Monitor retrieval speed and accuracy

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

  • Use semantic chunking for better context
  • Normalize embeddings for cosine similarity
  • Choose vector database based on scale and requirements

Next Steps in Your Learning Journey

  • Integrate retriever with LLM generator (full RAG pipeline)
  • Explore advanced retrieval (re-ranking, hybrid search)
  • Read about vector database scaling and deployment

9. Summary and Review (5 minutes)

Key Takeaways

  1. The retriever is essential for grounding LLMs in external knowledge
  2. Chunking, embedding, and indexing enable fast, semantic search
  3. Retrieval quality depends on chunking, embeddings, and database choice

Quick Review Questions

  1. Why do we chunk documents for RAG?
  2. What is the purpose of a vector database?
  3. How does semantic search differ from keyword search?

For Instructors

Teaching Tips

  • Demonstrate chunking and embedding visually
  • Encourage students to experiment with chunk sizes and models
  • Address confusion about vector database setup

Assessment Ideas

  • Quiz: Identify chunking and embedding steps in code
  • Project: Build a retriever for a custom document set
  • Peer review: Compare retrieval quality for different strategies

Adaptation Notes

  • For beginners: Focus on chunking and basic FAISS usage
  • For advanced: Try ChromaDB, Pinecone, or hybrid retrieval
  • Online: Provide Colab notebooks; In-person: Live coding demos