Chunking splits a document's text into smaller pieces before embedding — because embedding an entire long document as a single vector loses too much specificity for precise retrieval, and because LLM context windows can't hold arbitrarily large amounts of retrieved text anyway.
Why Not Embed the Whole Document?
Embedding one 50-page document as a single vector:
→ the resulting vector represents an "average" of everything
in the document — a very specific question about page 30
won't be strongly reflected in a vector dominated by the
other 49 pages' content
Embedding smaller, focused chunks:
→ each chunk's vector accurately represents its specific
content, so a query about page 30's topic can match that
chunk precisely, not get diluted by unrelated contentA Basic Chunking Example
def chunk_text(text, chunk_size=500, overlap=50):
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start = end - overlap # overlap prevents losing context
# right at chunk boundaries
return chunksThis simple fixed-size approach is a starting point — see Chunking Strategies for smarter approaches, and Chunk Size / Chunk Overlap for tuning these two specific parameters.
The Core Tradeoff
| Chunk Size | Effect |
|---|---|
| Too small | Loses surrounding context; a chunk may not make sense on its own, or may miss the answer entirely if it's split across chunk boundaries |
| Too large | Dilutes specificity (back to the "whole document" problem, just less severe); retrieves more irrelevant text alongside the relevant part |
Practical Use Case
Every RAG system's retrieval quality is fundamentally shaped by its chunking strategy — two systems using the identical LLM and vector database can have meaningfully different answer quality purely based on how documents were chunked.
Common Mistakes
- Using one fixed chunk size for structurally very different content (dense technical text vs conversational FAQ entries) without considering whether it suits each
- Chunking purely by character/token count with no regard for natural document structure (splitting mid-sentence or mid-table)
- No overlap between chunks, risking that relevant information spanning a chunk boundary gets split and neither chunk contains the complete answer
Interview Relevance
"Why does chunk size matter for RAG retrieval quality?" — the too-small-loses-context vs too-large-dilutes-specificity tradeoff above is exactly the expected answer.
Practice Question
A RAG system's answers are frequently missing information that spans across two adjacent chunks. Propose one specific fix.