The attention score is the raw number measuring how well a given query matches a given key — before any normalization, this is what "relevance" looks like numerically in the attention mechanism.
Formula
The simplest and most common choice is the dot product between the query and key vectors — reusing exactly the geometric intuition from Dot Product: a large positive dot product means the query and key vectors point in a similar direction in the learned embedding space, interpreted here as "this key is highly relevant to this query."
Computing Scores Against Every Key
For one query \(\mathbf{q}\) and a sequence of \(n\) keys \(\mathbf{k}_1, \ldots, \mathbf{k}_n\), a full set of scores is computed — one per key:
For a full sequence of queries as well, this generalizes cleanly to a matrix multiplication: \(\mathbf{Q}\mathbf{K}^\top\) — the exact matrix-multiplication mechanics from Matrix Multiplication, producing a full grid of every query's score against every key.
Numerical Example
Query \(\mathbf{q}=[1, 0]\), three keys \(\mathbf{k}_1=[1,0]\), \(\mathbf{k}_2=[0,1]\), \(\mathbf{k}_3=[0.7,0.7]\):
Key 1 (pointing in the exact same direction as the query) scores highest; key 2 (perpendicular) scores zero — no relevance at all; key 3 (partially aligned) scores in between. These raw scores aren't yet usable as weights (they don't sum to 1, and can be negative) — that conversion is exactly what happens next.
Code
import torch
q = torch.tensor([1.0, 0.0])
keys = torch.tensor([[1.0, 0.0], [0.0, 1.0], [0.7, 0.7]])
scores = keys @ q # equivalently: torch.matmul(keys, q)
print(scores) # tensor([1.0000, 0.0000, 0.7000]) -- matches the hand-worked exampleFrom Scores to Usable Weights — A Preview
Raw scores need to be converted into a proper probability distribution over the keys — non-negative, summing to 1 — before they can be used as weights for combining values. This conversion, via softmax, is exactly the subject of Dot-Product Attention, the next note.
Common Mistakes
- Treating raw attention scores directly as weights without normalizing them — they can be negative, and don't sum to any particular value, so they can't be used directly to combine values via a weighted sum.
- Assuming the dot product is the only valid way to compute an attention score — additive/"Bahdanau-style" attention historically used a small feedforward network instead; the dot-product approach ("Luong-style") became the standard largely because it's more computationally efficient (pure matrix multiplication) at scale.
Interview Relevance
Q: "What does a large positive attention score between a query and a key represent, geometrically?" It means the query and key vectors point in a similar direction in the learned embedding space — the dot product being large and positive is exactly the geometric signature of directional alignment, as established in Dot Product. In attention's context, this is interpreted as "this key's associated value is highly relevant to what this query is looking for."
Practice Question
For query \(\mathbf{q}=[0, 1]\) and key \(\mathbf{k}=[0,-1]\), compute the attention score. What does the negative value suggest about the relevance of this key to this query?
