RAG — Retrieval Augmented Generation
RAG with LangChain
Retrieval Augmented Generation (RAG) is a framework that connects large language models to your own private or dynamic data sources. It matters because it grounds model outputs in factual evidence, significantly reducing hallucinations compared to relying solely on pre-trained parametric memory. You reach for RAG whenever you need an AI to answer questions about specific documents, proprietary databases, or information that changes faster than a model can be retrained.
The Core Concept of Retrieval
At its simplest, RAG is a search problem paired with a generation problem. LLMs suffer from a limited context window and a cutoff date for their knowledge. Retrieval solves this by fetching relevant snippets of information from an external database before passing that information to the model as context. The reasoning behind this is 'grounding': by providing the model with a reference text, you shift its role from a creative storyteller to a summarizer or analyzer of provided facts. This significantly reduces hallucinations because the model is incentivized to use the provided context. If the model finds the answer in the retrieved text, it will almost always favor that over its internal, potentially outdated, weights. Understanding this flow is essential because it allows you to optimize either the quality of the retrieval (searching better) or the quality of the generation (prompting better).
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
# Prepare data: convert raw text into numerical vector embeddings
texts = ["The store opens at 9 AM", "The manager is Sarah"]
embeddings = OpenAIEmbeddings()
# Create a searchable vector store index
vector_store = FAISS.from_texts(texts, embeddings)
# Retrieve relevant documents based on user query
retriever = vector_store.as_retriever()
docs = retriever.invoke("When does the store open?")
print(docs[0].page_content) # Output: The store opens at 9 AMDocument Loading and Splitting
Raw documents are rarely in a state ready for vectorization. You must first load the data—whether it is a PDF, a website, or a text file—and then break it down into manageable chunks. This is crucial because LLMs have context limits, and embeddings are most effective when they map to semantically dense, smaller units of text. If you pass an entire book as one chunk, the embedding will be diluted and lack focus. By splitting the text into paragraphs or overlapping sentences, you ensure that the retrieval mechanism can find exactly the paragraph that addresses the user's query. The key is to select a 'chunk size' that is large enough to contain meaning but small enough to be highly relevant to specific questions. Overlapping these chunks ensures that context at the boundary of a cut is not lost, maintaining flow between sections.
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Load a large text block
text = "A very long document content..."
# Define the splitter to break down text into smaller fragments
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
# Generate chunks from the document
docs = text_splitter.create_documents([text])
print(f"Created {len(docs)} chunks.")Chaining Retrieval to Generation
Once you have your retrieved context, you need to combine it with the user's original prompt to create the final instruction for the LLM. In LangChain, this is handled via 'chains'. A chain acts as a pipeline where the output of the retriever becomes the context input for a prompt template. The template instructs the model: 'Use the following pieces of retrieved context to answer the user question. If you do not know the answer, say you do not know.' This specific prompt engineering forces the model to treat the retrieved text as the source of truth. Without this pipeline, the model would treat the retrieved text as just another part of the conversation rather than a governing constraint. The chain handles the logic of injecting the retrieved documents into the context variable, ensuring a clean and consistent flow of information during every inference cycle.
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
# Construct a prompt that demands context usage
prompt = ChatPromptTemplate.from_template(
"Answer based only on this context: {context}. Question: {question}"
)
# Chain the retriever and the model together
llm = ChatOpenAI()
# The 'context' will be dynamically filled by the retriever
chain = prompt | llm
result = chain.invoke({"context": "The store opens at 9 AM", "question": "What time?"})
print(result.content)Vector Stores as Search Engines
Vector stores are specialized databases that perform 'semantic search' rather than keyword search. While keyword search looks for exact string matches, semantic search looks for mathematical closeness in a high-dimensional vector space. When you 'embed' text, you turn it into a list of numbers representing its meaning. A vector store measures the distance (usually cosine similarity) between your query vector and your document vectors. This is transformative for RAG because it allows the retriever to find documents that are conceptually related, even if they share no words with the user's query. For example, a search for 'financial report' will correctly identify documents about 'annual earnings' or 'quarterly revenue'. Understanding this vector space is key to tuning retrieval performance; if your results are irrelevant, it is often because your embeddings are not capturing the nuance of your specific domain language.
from langchain_community.vectorstores import Chroma
# Store and search embeddings using a persistent vector store
# Chroma handles the mathematical index internally
vector_db = Chroma.from_texts(
["Cloud computing is reliable", "Local servers have high latency"],
embedding=OpenAIEmbeddings(),
persist_directory="./chroma_db"
)
# Query for a semantic match
results = vector_db.similarity_search("Is the internet fast?", k=1)
print(results[0].page_content)Advanced Optimization with Re-ranking
The final stage of a production-grade RAG system is re-ranking. Even the best vector search can return documents that are conceptually similar but ultimately wrong for the user's specific request. Re-ranking involves using a secondary, more computationally expensive model to score the relevance of the retrieved chunks after they have been pulled from the vector store. This model takes the user query and the retrieved documents and calculates a probability score of relevance. By only passing the top-ranked results to the generator, you minimize noise in the LLM's context window. This is highly effective when you are dealing with large datasets where many chunks might look similar. It effectively creates a 'two-pass' system: a fast, broad retrieval followed by a precise, narrow evaluation. This architecture ensures that your model's generated answer is grounded in only the most relevant, highly-vetted information available.
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor
# Setup a base retriever
base_retriever = vector_store.as_retriever()
# Wrap with a compressor to re-rank/filter results
compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
base_retriever=base_retriever, base_compressor=compressor
)
# Get refined results
refined_docs = compression_retriever.invoke("What is the store status?")
print(refined_docs[0].page_content)Key points
- RAG bridges the gap between private data and static model knowledge.
- The retrieval process transforms a search problem into a grounding task.
- Document splitting enables models to process information without exceeding context limits.
- Semantic search via vector embeddings captures meaning better than traditional keyword matching.
- Prompt templates are essential to enforce the use of retrieved context over internal weights.
- Chains in LangChain automate the flow of data from databases to the LLM.
- Overlapping text chunks prevent loss of context at the edges of document segments.
- Re-ranking retrieved results improves the precision of final generated answers.
Common mistakes
- Mistake: Not configuring a proper chunking strategy. Why it's wrong: Naive splitting breaks semantic context across boundaries. Fix: Use recursive character text splitting with appropriate overlap settings.
- Mistake: Over-relying on default similarity search. Why it's wrong: Default distance metrics often fail to capture nuanced semantic relationships. Fix: Implement hybrid search combining dense vector embeddings with sparse keyword search.
- Mistake: Sending the entire retrieved document set to the LLM context. Why it's wrong: This causes 'lost in the middle' phenomena and increases costs. Fix: Apply a reranker or re-ranker stage to select the most relevant segments before generation.
- Mistake: Neglecting metadata filtering during retrieval. Why it's wrong: The system retrieves irrelevant documents simply because they share vector proximity. Fix: Use metadata filters to constrain the vector search space.
- Mistake: Assuming a fixed prompt works for all queries. Why it's wrong: Static prompts don't account for user intent or data sparsity. Fix: Use query transformation techniques like Multi-Query or Hypothetical Document Embeddings (HyDE).
Interview questions
What is the fundamental purpose of RAG in a Generative AI application?
Retrieval-Augmented Generation, or RAG, serves to solve the inherent limitations of large language models, specifically their tendency to hallucinate and their lack of knowledge regarding private or real-time data. By connecting an AI model to an external knowledge base, RAG allows the system to retrieve relevant documents first and then use that context to generate a factual, grounded response. This approach is essential for enterprise applications where accuracy is prioritized over generic creative generation.
How does LangChain facilitate the document loading and splitting process in a RAG pipeline?
LangChain simplifies RAG by providing standardized interfaces for data ingestion and processing. You typically use a 'DocumentLoader' to extract text from files like PDFs or websites, followed by a 'TextSplitter' to break large bodies of text into smaller, manageable chunks. This is vital because LLMs have context window limits; by using recursive character text splitters, we ensure the retrieval process focuses only on highly relevant, semantically dense segments rather than overwhelming the model with irrelevant noise.
What is the role of embeddings and vector databases within the LangChain framework?
Embeddings are numerical representations of text that capture semantic meaning. LangChain integrates these by converting text chunks into high-dimensional vectors. These vectors are then stored in a vector database, such as Chroma or Pinecone, which supports similarity searches. When a query comes in, the system converts the user's intent into an embedding and performs a vector search to find the most relevant context, which is then injected into the LLM's prompt.
Explain the difference between a simple RetrievalQA chain and a ConversationalRetrievalChain in LangChain.
The simple RetrievalQA chain is designed for a single-turn interaction: you ask a question, the system retrieves context, and the model provides an answer. In contrast, the ConversationalRetrievalChain maintains a history of the interaction. This is superior because it uses a 'condense question' mechanism to rephrase the user's current query based on prior conversation turns, ensuring that the retrieval step is context-aware and maintains continuity throughout the user's session.
Compare the 'Stuffing' technique versus 'Map-Reduce' for handling retrieved documents in LangChain.
Stuffing is the simplest approach, where you push all retrieved documents into the LLM prompt at once; it is efficient but risks hitting the context window limit. Map-Reduce, however, processes each document independently to generate partial answers before summarizing them into a final response. While Map-Reduce handles larger datasets and avoids token limit errors, it is slower and potentially loses nuances that stuffing preserves by keeping the entire context available simultaneously.
How do you implement advanced retrieval techniques like 'Self-Querying' or 'Parent Document Retrieval' to improve RAG performance?
Basic semantic search often fails with complex queries. Self-Querying uses an LLM to parse a user prompt and extract specific metadata filters, which are then applied to the vector database search to narrow down results. Parent Document Retrieval involves splitting documents into small chunks for high-precision searching while retrieving larger 'parent' chunks to provide the LLM with sufficient surrounding context. Implementing code such as 'from langchain.retrievers import ParentDocumentRetriever' ensures that the LLM receives the most informative context possible for its generation step.
Check yourself
1. Why is 'chunk overlap' essential when preparing documents for a vector store?
- A.To increase the total number of vectors for better statistical significance
- B.To preserve the semantic continuity of information that happens to fall at the split point
- C.To ensure the vector store indexes the same text multiple times for redundancy
- D.To allow the LLM to process more tokens per API call
Show answer
B. To preserve the semantic continuity of information that happens to fall at the split point
Overlap preserves context that would otherwise be severed by a hard cut. Option 0 is false as it increases storage but not utility; option 2 is incorrect as it creates data noise; option 3 is false because overlap does not influence model token limits.
2. When implementing a RAG pipeline, why is a Reranker step often superior to a raw vector search?
- A.It generates new text to replace the user query
- B.It provides a secondary assessment of relevance between the query and retrieved context
- C.It eliminates the need for vector embeddings entirely
- D.It compresses the index size of the vector database
Show answer
B. It provides a secondary assessment of relevance between the query and retrieved context
Rerankers act as a second-pass cross-encoder model that scores query-document pairs more accurately than simple cosine distance. Option 0 describes a generator, not a reranker; option 2 is false as embeddings are required for the first pass; option 3 is false as reranking is a post-retrieval process.
3. What is the primary benefit of using a 'Hybrid Search' approach?
- A.It reduces the latency of vector generation
- B.It makes the system immune to hallucinations
- C.It combines semantic understanding with exact keyword matching for better precision
- D.It allows the model to ignore the system prompt
Show answer
C. It combines semantic understanding with exact keyword matching for better precision
Hybrid search balances vector embeddings (broad intent) with sparse retrieval (keywords), capturing both synonyms and specific technical terms. Option 0 is false; option 1 is false because RAG still relies on the LLM's grounding; option 3 is unrelated.
4. Which of the following describes the purpose of 'Query Transformation' in RAG?
- A.To convert all user input into low-case strings to simplify matching
- B.To rewrite or expand the user's query into a format more likely to retrieve relevant documents
- C.To automatically generate a list of citations for the user
- D.To verify if the user's question violates safety guidelines
Show answer
B. To rewrite or expand the user's query into a format more likely to retrieve relevant documents
Query transformation handles underspecified or poorly phrased questions by rewriting them to match the document retrieval space. Option 0 is trivial normalization; option 2 describes a citing agent; option 3 is a moderation task.
5. How does 'Metadata Filtering' improve a retrieval system?
- A.By increasing the temperature of the LLM output
- B.By allowing the system to restrict search results to specific categories or timeframes
- C.By converting text into binary format for faster processing
- D.By automatically summarizing documents based on their file size
Show answer
B. By allowing the system to restrict search results to specific categories or timeframes
Metadata filtering allows the user to pre-constrain the search space, ensuring results are contextually appropriate (e.g., only searching documents from 2024). Options 0, 2, and 3 describe processes unrelated to retrieval scope restriction.