Dot Product

The dot product combines two vectors of equal length into a single scalar. It measures how much two vectors "point in the same direction" — and it is the exact operation behind cosine similarity and the attention mechanism you'll meet later in this hub.

Formula

The first form sums element-wise products directly. The second form connects it to geometry: \(\|\mathbf{a}\|\) and \(\|\mathbf{b}\|\) are the vectors' lengths (see Vector Norms), and \(\theta\) is the angle between them. When two vectors point in the same direction (\(\theta = 0\)), \(\cos\theta = 1\) and the dot product is maximal for their lengths; when they're perpendicular (\(\theta = 90°\)), the dot product is exactly 0.

Numerical Example

Geometric Intuition

Relationship to Matrix Multiplication

Matrix multiplication is literally a table of dot products: entry \(C_{ij}\) in Matrix Multiplication is the dot product of row \(i\) of \(\mathbf{A}\) and column \(j\) of \(\mathbf{B}\). Understanding the dot product is what makes matrix multiplication's mechanics click.

Code

import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.dot(a, b))   # 32
print(a @ b)            # 32 -- same result, matrix-multiply operator on 1D arrays

import torch
a = torch.tensor([1., 2., 3.])
b = torch.tensor([4., 5., 6.])
print(torch.dot(a, b))   # tensor(32.)

Where This Shows Up in Deep Learning

• Cosine similarity (comparing word/document embeddings) is a normalized dot product: \(\cos\theta = \dfrac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}\).
• Attention scores: the core of the Transformer computes \(\mathbf{Q}\mathbf{K}^\top\) — a matrix of dot products between every query vector and every key vector, measuring how relevant each token is to every other token (see Scaled Dot-Product Attention).
• A single neuron's weighted sum, \(\mathbf{w}\cdot\mathbf{x}\), is exactly a dot product between the weight vector and the input vector.

Common Mistakes

• Forgetting the dot product requires equal-length vectors — unlike matrix multiplication with its more flexible (but still strict) inner-dimension rule.
• Interpreting a large dot product as "similarity" without normalizing for vector length — two long vectors pointing in slightly different directions can have a larger raw dot product than two short, perfectly aligned vectors. Cosine similarity fixes this by dividing out the norms.

Interview Relevance

Q: "Why does the Transformer's attention mechanism use a dot product between queries and keys?" The dot product measures directional similarity — a query vector will have a large dot product with key vectors pointing in a similar direction in the learned embedding space, which the model uses as a relevance score for how much attention to pay to each token.

Practice Question

Compute the dot product of \([2, 0, -1]\) and \([3, 4, 2]\) by hand. Is the result positive, negative, or zero — and what does that tell you about the angle between them?

Related DL Notes

• Matrix Multiplication
• Vector Norms
• Scaled Dot-Product Attention