Chunk size — how large each chunk is, typically measured in tokens or characters — is one of the most impactful, easiest-to-tune parameters in a RAG system, and there's no single universally correct value.
The Tradeoff, Concretely
Small chunks (e.g. ~100-200 tokens):
+ Highly specific — a match strongly indicates real relevance
- May lack surrounding context needed to fully answer a question
- More chunks needed to cover the same content → more storage,
potentially more retrieved chunks needed per query
Large chunks (e.g. ~800-1000+ tokens):
+ More self-contained context per chunk
- Less precise — a chunk might be "relevant enough" to match
but contain a lot of irrelevant surrounding text too
- Consumes more of the LLM's context window per retrieved chunkThere's No Universal "Correct" Size
The right chunk size depends on content type (dense technical text vs conversational FAQ), typical query complexity, and the embedding model's own effective range. Common starting points in practice range roughly from a few hundred to around a thousand tokens, but treat any specific number as a starting point to test against your own data — not a rule.
A Practical Way to Choose
1. Start with a reasonable default (e.g. ~500 tokens, informed
by your content type).
2. Build a small evaluation set of real questions with known
correct answers/source chunks.
3. Test retrieval quality at a few different chunk sizes.
4. Pick the size that empirically retrieves the correct
supporting content most reliably for YOUR content and
query patterns.See RAG Evaluation for the fuller measurement approach this fits into.
Practical Use Case
A FAQ knowledge base (short, self-contained Q&A pairs) often works well with small chunks matching each Q&A. A legal document knowledge base (context-dependent clauses referencing earlier sections) may need larger chunks, or structure-aware chunking, to preserve necessary context.
Common Mistakes
- Picking a chunk size once during initial setup and never revisiting it, even as retrieval quality issues surface
- Copying a chunk size from a tutorial or another project without considering whether it fits your specific content type
- Confusing chunk size (input to embedding/retrieval) with the LLM's context window (a completely separate constraint on final prompt size)
Interview Relevance
"Is there a 'correct' chunk size for RAG?" — no; a strong answer explains the tradeoff and describes an empirical, evaluation-driven approach to choosing one for a specific use case.
Practice Question
Explain why a chunk size that works well for a FAQ database might perform poorly for a knowledge base of dense academic research papers.