Beyond naive fixed-size splitting, several chunking strategies exist, each better suited to different content structures — choosing the right one is a real, measurable lever on retrieval quality.
Fixed-Size Chunking
Split text every N characters/tokens, with some overlap.
Simple, predictable, but ignores document structure entirely —
can split mid-sentence or mid-thought.Sentence/Paragraph-Based Chunking
Split at natural sentence or paragraph boundaries, grouping
them until reaching roughly the target chunk size.
→ Respects natural language structure, avoiding awkward
mid-sentence splits.Recursive/Hierarchical Chunking
Try splitting by larger structural units first (sections,
then paragraphs, then sentences), falling back to smaller
units only if a piece is still too large.
→ Better preserves document structure (headings, sections)
than naive fixed-size splitting.Semantic Chunking
Use embeddings to detect where the TOPIC actually shifts
within the text, and split there — rather than at a fixed
size or a purely structural boundary.
→ Chunks align with actual topic boundaries, but this
approach is more computationally expensive (requires
embedding calls during chunking itself) and adds complexity.Document-Structure-Aware Chunking
Use the document's own structure — headings, sections, table
boundaries, code blocks — as natural chunk boundaries.
→ Particularly effective for structured content like technical
documentation, Markdown files, or well-formatted reports.Choosing a Strategy
| Content Type | Reasonable Starting Point |
|---|---|
| Well-structured docs (Markdown, technical docs) | Structure-aware chunking |
| Prose-heavy content (articles, reports) | Sentence/paragraph-based or recursive chunking |
| Highly variable, mixed content | Semantic chunking, if the added complexity/cost is justified by measured quality gains |
| Quick prototype / unclear requirements yet | Fixed-size with reasonable overlap — simple, works as a baseline |
Practical Use Case
Technical documentation with headings and code examples benefits significantly from structure-aware chunking (keeping a code example with its explanation, not splitting them apart) — a chunking strategy mismatched to content type is a common, fixable source of mediocre RAG performance.
Common Mistakes
- Defaulting to fixed-size chunking for structured content where a structure-aware approach would clearly perform better
- Adopting semantic chunking's added complexity and cost without first measuring whether simpler approaches are actually insufficient
- Never comparing chunking strategies empirically against real queries — assuming one approach is "obviously" better without testing
Interview Relevance
"How would you chunk a technical documentation site with headings, code blocks, and prose?" — structure-aware chunking that respects headings and keeps code examples intact with their explanations is the expected strong answer.
Practice Question
Recommend a chunking strategy for a knowledge base of legal contracts with numbered clauses, and justify the choice.