Embeddings and Semantic Search
Chunking Strategies for Documents
Chunking is the process of breaking large documents into smaller, manageable text segments suitable for embedding models and vector search. It is critical because embedding models have strict token limits and semantic integrity suffers when context is either too fragmented or too diluted by irrelevant information. You apply these strategies whenever you need to implement a Retrieval-Augmented Generation (RAG) system to ensure accurate context retrieval for LLM prompts.
Fixed-Size Character Chunking
Fixed-size character chunking is the most fundamental approach, where text is split into segments of a predetermined number of characters or tokens regardless of the semantic content. This method is highly computationally efficient because it requires minimal processing overhead and predictable indexing time. However, the primary drawback is the risk of cutting sentences or paragraphs in half, which often destroys the semantic meaning of the chunk. When you choose this strategy, you must balance chunk size with the model's context window. Smaller chunks might miss necessary context, while larger chunks might blend unrelated topics, leading to noisy retrieval results. It is best used for simple text formats where structural boundaries are not clearly defined or when speed is the absolute priority over high-quality semantic precision.
# Simple fixed-size chunking logic
text = "The quick brown fox jumps over the lazy dog. The cat sat on the mat."
chunk_size = 20
# Split text into segments of exactly chunk_size
chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]
print(chunks) # Output: ['The quick brown fox ', 'jumps over the lazy ', 'dog. The cat sat on', ' the mat.']Recursive Character Text Splitting
Recursive character splitting improves upon basic fixed-size methods by attempting to honor the natural structure of the document through a hierarchy of separators. Instead of blindly cutting at a fixed index, this strategy tries to split on paragraphs, then sentences, and finally words or characters if the chunk is still too large. By prioritizing logical break points, you maintain better semantic integrity within each chunk. The reasoning behind this is to minimize the destruction of meaningful entities. When the splitter encounters a paragraph, it checks if the result is under the limit; if not, it recurses into smaller units. This is the recommended default for most general-purpose documentation because it creates a balance between maintaining local context and meeting the constraints of the vector embedding model's input architecture.
def recursive_split(text, separators=['\n\n', '\n', '. ', ' '], max_size=50):
# Simplified recursive splitting demonstration
if len(text) <= max_size:
return [text]
# Find the best separator that maintains logical flow
for sep in separators:
if sep in text:
parts = text.split(sep)
return [p + sep for p in parts if len(p) > 0] # Return segments
return [text[:max_size]]
print(recursive_split("First paragraph. Second paragraph.", max_size=20))Overlap-Based Chunking
Overlap-based chunking is a technique designed to mitigate the loss of context that occurs at the boundaries of any split. When you cut a document into distinct chunks, the information residing exactly at the junction point might be lost or become unintelligible, potentially confusing the embedding model. By introducing an overlap—where a portion of the end of chunk A is repeated at the beginning of chunk B—you ensure that cross-boundary context is preserved. This strategy is vital for maintaining narrative flow or technical relationships that span across the arbitrary split point. The trade-off is increased storage requirements and redundant data processing. You should always reach for this strategy when dealing with long-form prose or technical documentation where sentences often reference preceding information across structure breaks.
# Overlapping chunking to preserve context
text = "Artificial Intelligence is transforming industries. Modern systems rely on data."
chunk_size, overlap = 30, 10
# Iterate with a stride smaller than the size
chunks = [text[i : i + chunk_size] for i in range(0, len(text), chunk_size - overlap)]
print(chunks) # Note the overlap in indicesSemantic Chunking via Embeddings
Semantic chunking moves away from character counts and instead focuses on the underlying meaning of the text. This method calculates the cosine similarity between sentences or paragraphs using an embedding model and groups them only when the semantic content remains consistent. If a sharp drop in similarity is detected—indicating a shift in topic—the splitter triggers a new chunk. This is computationally expensive but provides the highest quality results, as chunks represent cohesive thoughts or sections rather than arbitrary segments. You should use this when your documents contain diverse topics or sections that vary significantly in length. By aligning the chunk boundaries with actual shifts in meaning, you ensure that the vector search returns results that are highly relevant to the user query rather than generic text fragments.
import math
# Simplified semantic distance check
def is_topic_shift(vec1, vec2, threshold=0.8):
# Higher similarity means same topic, lower means shift
similarity = sum(a*b for a,b in zip(vec1, vec2)) # Mock dot product
return similarity < threshold
# If shift, split; otherwise, merge into current chunkAgentic or Structural Chunking
Agentic or structural chunking utilizes the internal metadata of documents, such as Markdown headers, JSON fields, or specific HTML tags, to determine boundaries. This method assumes that the document author already provided the logical structure that is most meaningful to the end user. By treating a 'Header' or 'Section' as the primary unit of a chunk, you ensure that all information related to a specific topic stays together regardless of its actual length. This is particularly effective for technical manuals or legal contracts where document hierarchies are strictly enforced. When you encounter well-structured documents, this is the most reliable strategy, as it avoids the guesswork of character splitting entirely and respects the intentional organization of the original source material during retrieval tasks.
# Structural chunking based on markdown-style headers
doc = "# Intro\nWelcome.\n# Methods\nData collection."
# Split by headers while keeping the header context
chunks = doc.split("# ")
# Process chunks while retaining header metadata for context
for chunk in chunks:
print(f"Processing section: {chunk.split('\n')[0]}")Key points
- Chunking is essential for fitting large documents into the fixed token limitations of embedding models.
- Fixed-size chunking is fast but risks breaking sentences and losing the semantic unity of the content.
- Recursive character splitting is the industry default because it prioritizes structural boundaries like paragraphs and sentences.
- Overlap between chunks prevents the loss of context at the specific point where a document is partitioned.
- Semantic chunking uses embedding similarity to identify natural topic shifts, resulting in more coherent retrieval units.
- Structural chunking leverages document metadata like headers to maintain the original author's logical organization.
- Choosing the right strategy requires balancing the computational cost of the method against the need for retrieval precision.
- Poor chunking strategies result in fragmented data, which causes the model to lose the relevant context necessary for accurate answers.
Common mistakes
- Mistake: Using a fixed character count for all document types. Why it's wrong: It ignores semantic boundaries like paragraphs or sentences. Fix: Use semantic-aware splitters that respect document structure.
- Mistake: Overlapping chunks are seen as unnecessary overhead. Why it's wrong: Without overlap, contextual information at the edge of a chunk is often lost. Fix: Implement 10-20% overlap to preserve context continuity.
- Mistake: Treating all document types with the same chunking strategy. Why it's wrong: Code files require different structural awareness than prose. Fix: Tailor the chunking strategy to the data format, such as using AST-based splitters for code.
- Mistake: Ignoring the retrieval window limits of the model. Why it's wrong: Chunks that exceed the token limit will be truncated, leading to data loss. Fix: Ensure chunk sizes are well within the context window limits of the embedding model.
- Mistake: Not evaluating chunking effects on retrieval quality. Why it's wrong: Poor chunking can dilute information, causing the model to miss relevant details. Fix: Perform retrieval benchmarking to compare different chunk sizes against query relevance.
Interview questions
What is the fundamental purpose of chunking in Generative AI document processing?
Chunking is the essential process of breaking down large documents into smaller, manageable segments before they are embedded and stored in a vector database. The primary purpose is to respect the context window limits of Large Language Models while ensuring that the retrieved information is highly relevant to the user's prompt. By converting text into smaller vectors, we enable the system to perform high-precision semantic search. Without chunking, large documents would exceed input token limits, and the signal-to-noise ratio in semantic retrieval would be too low, leading to inaccurate model responses.
How does a simple fixed-size chunking strategy work, and what are its potential drawbacks?
Fixed-size chunking divides a document into segments of a predefined number of characters or tokens, such as 500 tokens per chunk. While this is computationally efficient and simple to implement, it often ignores the semantic structure of the content. For example, a hard limit might cut a sentence or a paragraph in half, stripping it of its context. In code, this might look like: chunks = [text[i:i+500] for i in range(0, len(text), 500)]. The drawback is that these arbitrary cuts often result in fragmented information that confuses the model, leading to hallucinated interpretations because the semantic boundary does not align with the chunk boundary.
Can you explain recursive character text splitting and why it is generally preferred over basic fixed-size splitting?
Recursive character text splitting is a more sophisticated approach that attempts to split text based on a hierarchy of separators, such as paragraphs, then sentences, then words. It iteratively tries to split the text until the chunks fall under the desired size. This method is preferred because it respects natural language boundaries, keeping semantically related information together. For instance, it will prioritize splitting at a double newline before falling back to a single newline. This preserves the logical flow of ideas, which is vital for Generative AI systems to retrieve coherent information rather than disjointed, meaningless snippets of data.
Compare semantic chunking with document-based structural chunking. When would you choose one over the other?
Semantic chunking uses embedding distance to determine where to split, creating chunks only when the thematic content of the text shifts, whereas document-based structural chunking relies on formatting markers like headers, markdown, or bullet points. You should choose semantic chunking when dealing with unstructured, prose-heavy data where topic shifts are subtle and occur mid-paragraph. Conversely, use structural chunking for technical documentation or manuals where headers provide clear, logical boundaries. Semantic chunking is computationally more expensive but offers superior retrieval performance for documents that lack a strict, predictable hierarchy.
What is the role of the 'chunk overlap' parameter, and how do you determine an optimal value?
Chunk overlap is the practice of having subsequent chunks share a portion of the text from the previous chunk. Its role is to maintain 'contextual continuity' across the cut points, ensuring that the model does not lose track of the subject matter at the end of one chunk and the start of another. To determine the optimal value, you must balance redundancy with token usage. A common starting point is 10-20% overlap. If your documents contain highly complex dependencies, increasing the overlap ensures that the retriever captures the necessary preceding context, preventing the loss of critical information that often happens at artificial splitting points.
How would you handle chunking for a multi-modal document containing both text and dense tables, ensuring the model retains the ability to reason over the tabular data?
Handling tables is notoriously difficult because standard character-based chunking destroys the row-column relationship. For these documents, I would use an agentic or multi-modal parsing approach that extracts tables into a structured format like Markdown or JSON before chunking. I would treat the table as a single logical 'chunk' rather than splitting it by line. If the table is too large, I would implement a summary-based retrieval, where the model first identifies the table's purpose and then retrieves the specific subset of data required for the query. This ensures the Generative AI preserves the relational context of the data.
Check yourself
1. When designing a RAG system, what is the primary drawback of using very small, fine-grained chunks?
- A.The embedding vectors become too large to store in a vector database.
- B.The model loses the broader context required to interpret the meaning of the snippet.
- C.The retrieval latency increases significantly compared to larger chunks.
- D.The system becomes incapable of handling overlapping tokens.
Show answer
B. The model loses the broader context required to interpret the meaning of the snippet.
Small chunks often lack the surrounding context needed for accurate embedding; the model cannot 'see' the big picture. Larger chunks don't necessarily increase latency or storage significantly, and overlap is a separate strategy, not a limitation of size.
2. How does implementing chunk overlap specifically benefit Generative AI document retrieval?
- A.It reduces the total number of chunks created, saving memory.
- B.It allows the model to summarize the entire document faster.
- C.It ensures that semantic information split across a boundary is preserved in two adjacent chunks.
- D.It automatically corrects grammatical errors in the source text.
Show answer
C. It ensures that semantic information split across a boundary is preserved in two adjacent chunks.
Overlap captures context that would otherwise be severed at a hard break, ensuring continuity. It does not reduce memory or speed up summarization, and it has no role in grammatical correction.
3. Which approach is most effective when chunking a codebase for a retrieval-augmented coding assistant?
- A.Splitting strictly by character count, regardless of indentation.
- B.Using a recursive character splitter that respects code structure like function and class definitions.
- C.Converting the entire code file into a single string to avoid fragmentation.
- D.Removing all white space and comments before chunking.
Show answer
B. Using a recursive character splitter that respects code structure like function and class definitions.
Code is hierarchical; preserving function and class blocks keeps related logic together. Fixed character splits break logic, full file processing exceeds limits, and removing white space destroys readability for the underlying transformer model.
4. Why might a 'RecursiveCharacterTextSplitter' be preferred over a simple split by newline characters?
- A.It avoids the need to calculate vector similarity scores.
- B.It attempts to keep paragraphs and sentences together by trying multiple separators in order.
- C.It automatically translates the document into different languages.
- D.It reduces the dimensionality of the generated embedding vectors.
Show answer
B. It attempts to keep paragraphs and sentences together by trying multiple separators in order.
It iterates through a list of separators (like paragraphs, then sentences) to keep related information in the same chunk. It doesn't affect embeddings, vector math, or translation.
5. What is the primary trade-off when selecting a larger chunk size for a document retrieval system?
- A.Increased noise in the retrieval process as chunks become less focused on specific topics.
- B.A decrease in the computational cost of embedding the document.
- C.The inability to use dense vector representations for the data.
- D.A decrease in the context window size of the underlying language model.
Show answer
A. Increased noise in the retrieval process as chunks become less focused on specific topics.
Larger chunks dilute the embedding, making it harder to find specific 'needles' in the document 'haystack' because they contain too many topics. It does not lower costs, prevent vector use, or change the LLM's architecture.