The context window is the maximum number of tokens (input and output combined) an LLM can process or generate within a single interaction — a hard architectural limit that shapes what tasks a model can practically handle.
Why a Limit Exists at All
Recall from Scaled Dot-Product Attention that self-attention computes a score between every pair of positions in a sequence — for a sequence of length \(n\), this means \(n^2\) score computations. Both the compute cost and the memory required to store this attention matrix grow quadratically with sequence length:
Doubling the context length roughly quadruples the attention computation's cost — this quadratic scaling is the fundamental reason context windows are limited rather than arbitrarily large, and it's also exactly why RoPE's extrapolation benefits (from Positional Embeddings) and specialized long-context techniques remain active areas of ongoing engineering effort.
What "Context Window" Practically Constrains
| Use Case | Constraint |
|---|---|
| Long document summarization | The full document plus the generated summary must fit within the window |
| Extended conversation | Earlier turns must either fit within the window or get truncated/summarized as the conversation grows |
| Retrieval-augmented generation | Retrieved context documents compete with the actual query and generated response for the same fixed budget |
Code — Checking Whether Text Fits
import tiktoken
encoding = tiktoken.encoding_for_model("gpt-4")
context_limit = 8192
document_tokens = len(encoding.encode(long_document_text))
prompt_overhead = 200 # instructions, formatting, etc.
max_response_tokens = context_limit - document_tokens - prompt_overhead
if max_response_tokens < 100:
print("Document too long for this context window -- truncation or chunking needed")
else:
print(f"Up to {max_response_tokens} tokens available for the response")Common Mistakes
- Forgetting that the context window is shared between input and output — a long input prompt directly reduces how much output the model can generate within the same interaction, for most model APIs.
- Assuming a larger context window always means better performance on long-document tasks — models often show degraded attention quality toward the middle of very long contexts (sometimes called the "lost in the middle" effect), so simply fitting text within the window doesn't guarantee the model will use all of it equally well.
Interview Relevance
Q: "Why can't LLM context windows simply be made arbitrarily large?" Self-attention computes a score between every pair of tokens in the sequence, so both its compute and memory cost scale quadratically with sequence length. Doubling context length roughly quadruples attention's cost — this fundamental scaling relationship is the core reason context windows are a hard, resource-driven limit rather than an arbitrary design choice, and why extending them efficiently remains an active area of research.
Practice Question
If a model's context window is 4096 tokens and a user's conversation history plus system instructions already total 3800 tokens, roughly how many tokens remain for the model's next response?
