After chunks are retrieved (and possibly reranked), there's still a real design step before generation: assembling the final context that actually gets sent to the LLM — ordering, formatting, and fitting it within the available context budget.
This Is a Distinct Step From Retrieval Itself
Retrieval: "which chunks are relevant?"
Context assembly: "how do I combine those chunks into the
actual text block sent to the model?"Ordering Matters
Some evidence suggests LLMs can pay less attention to information placed in the middle of a long context, compared to the beginning or end (a "lost in the middle" effect) — a reason to consider ordering retrieved chunks by relevance (most relevant first or last, not buried in the middle) rather than an arbitrary order, though the practical impact varies by model and should be evaluated rather than assumed.
Formatting for Clarity
Unclear (chunks just concatenated):
"Refunds within 30 days... Sale items final... Contact support
for exchanges... Shipping takes 3-5 days..."
Clearer (chunks separated and sourced):
"[Source: Return Policy, Section 2]
Refunds within 30 days of purchase are accepted.
[Source: Return Policy, Section 4]
Sale items are final sale.
---
Use only the information above to answer the question."Clear separation between chunks (and, where useful, source labeling) makes it easier for the model to reason about distinct pieces of information rather than treating retrieved content as one undifferentiated blob — and makes citation possible, since the model can reference which source supported which claim.
Fitting the Context Budget
The assembled context, combined with the system prompt, conversation history, and reserved output tokens, still has to fit within the model's context window — if retrieved (or reranked) content exceeds the available budget, a deliberate truncation or selection strategy is needed, not an arbitrary cutoff.
Practical Use Case
A RAG system supporting source citations in its answers relies entirely on well-structured context assembly — without clear source labeling in the context, there's no reliable way for the model to attribute its answer to a specific source document or section.
Common Mistakes
- Concatenating retrieved chunks with no separation or source labeling, making citation and clear reasoning harder
- Not considering chunk order, potentially burying the most relevant chunk in the middle of a long assembled context
- Not checking that assembled context plus everything else fits within the context window before sending the request
Interview Relevance
"Beyond retrieving the right chunks, what else matters for how you use them in a RAG prompt?" — ordering, clear formatting/separation, source labeling for citations, and context-budget management are all expected, distinct points beyond retrieval itself.
Practice Question
Design a context assembly format for 4 retrieved chunks that supports the model citing which specific chunk supported each part of its answer.