This note builds attention's precise mathematical framework from first principles, starting with the three core objects — Query, Key, and Value — using a search-engine analogy to make each one's role concrete before any formulas appear.
The Search Engine Analogy
| Search Engine Concept | Attention Equivalent | Role |
|---|---|---|
| Your search query | Query (\(\mathbf{Q}\)) | "What am I currently looking for?" |
| Each webpage's title/tags | Key (\(\mathbf{K}\)) | "What does this piece of information advertise itself as being about?" |
| Each webpage's actual content | Value (\(\mathbf{V}\)) | "What information does this piece actually contain, once selected?" |
A search engine compares your query against every page's title/tags (keys) to find the best matches, then returns those matching pages' actual content (values). Attention does exactly this, but instead of a hard "best match" selection, it computes a soft, weighted blend — every value contributes something, weighted by how well its key matched the query.
Where Q, K, V Come From — Learned Projections
\(\mathbf{X}\) is the input sequence's representations (e.g. the encoder's hidden states, or in later, more general settings, token embeddings). \(\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V\) are three independently learned weight matrices — every attention layer learns its own way of projecting the same input into three different "views," specialized for querying, being matched against, and being retrieved, respectively.
Why Three Separate Projections, Not Just Reusing \(\mathbf{X}\) Directly
If queries, keys and values were all just the raw input \(\mathbf{X}\) itself, the network would have no flexibility to learn different, task-specialized representations for "how should I search," "how should I be matched," and "what should I actually retrieve" — these can genuinely be different useful representations of the same underlying information, and giving the network three independent learned projections lets it discover exactly what those representations should be, rather than being locked into one shared representation for all three roles.
Numerical Example — Tiny Case
A single token's embedding \(\mathbf{x}=[1, 0.5]\), with learned projection matrices \(\mathbf{W}_Q=\begin{bmatrix}1&0\\0&1\end{bmatrix}\) (identity, for simplicity), \(\mathbf{W}_K=\begin{bmatrix}0&1\\1&0\end{bmatrix}\) (swaps dimensions), \(\mathbf{W}_V=\begin{bmatrix}2&0\\0&2\end{bmatrix}\) (scales by 2):
Even from the same input token, the three projections produce three genuinely different vectors — exactly the specialization this design is intended to enable, learned automatically during training rather than fixed as in this illustrative example.
Code
import torch
import torch.nn as nn
d_model = 8 # input embedding dimension
d_k = 4 # projected query/key dimension
W_Q = nn.Linear(d_model, d_k, bias=False)
W_K = nn.Linear(d_model, d_k, bias=False)
W_V = nn.Linear(d_model, d_k, bias=False)
X = torch.randn(1, 5, d_model) # 5 tokens, each an 8-dim embedding
Q = W_Q(X) # (1, 5, 4)
K = W_K(X) # (1, 5, 4)
V = W_V(X) # (1, 5, 4)
print(Q.shape, K.shape, V.shape)Common Mistakes
- Assuming Q, K, V must always come from the same source sequence — this is true specifically for self-attention (covered later in this category); cross-attention deliberately draws Q from one sequence and K/V from another.
- Treating \(\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V\) as fixed, hand-designed transformations — they're ordinary learnable weight matrices, trained via backpropagation exactly like any other layer's weights.
Interview Relevance
Q: "Why does attention use three separate learned projections (Q, K, V) instead of using the raw input directly for all three roles?" Querying, matching, and retrieving are genuinely different functions, even when applied to the same underlying information — giving each role its own independently learned projection lets the network discover specialized representations optimized for each specific job, rather than being constrained to use one shared representation for all three, which would limit what the attention mechanism could learn to do.
Practice Question
In the search-engine analogy, what would it mean, practically, if a system used the exact same representation for both the search query and the page keys, with no separate "value" content ever returned?
