While variance measures how much a single variable spreads out, covariance measures how two variables move together — whether they tend to increase together, move in opposite directions, or show no consistent relationship at all.
Formula
| Sign of Covariance | Meaning |
|---|---|
| Positive | When \(X\) is above its mean, \(Y\) tends to be above its mean too — they move together |
| Negative | When \(X\) is above its mean, \(Y\) tends to be below its mean — they move oppositely |
| Near zero | No consistent linear relationship |
Numerical Example
House size (\(X\), in 100 sq ft): \([5, 7, 8, 10]\), mean \(=7.5\). Price (\(Y\), in lakhs): \([30, 45, 50, 65]\), mean \(=47.5\). Deviations: \(X\): \([-2.5,-0.5,0.5,2.5]\), \(Y\): \([-17.5,-2.5,2.5,17.5]\). Products: \([43.75, 1.25, 1.25, 43.75]\). \(\text{Cov}(X,Y) = \frac{43.75+1.25+1.25+43.75}{4}=22.5\) — positive, confirming size and price move together, as expected.
The Covariance Matrix
For a dataset with multiple features, the covariance matrix \(\boldsymbol\Sigma\) stores every pairwise covariance: entry \(\Sigma_{ij}\) is the covariance between feature \(i\) and feature \(j\), with the diagonal entries being each feature's own variance. This matrix is exactly what PCA operates on — its eigenvectors (see Eigenvectors) are the principal component directions, and its eigenvalues rank how much variance each direction captures.
Code
import numpy as np
X = np.array([5, 7, 8, 10])
Y = np.array([30, 45, 50, 65])
cov_matrix = np.cov(X, Y, bias=True) # bias=True uses the population formula (divide by n)
print(cov_matrix)
# [[ 3.125 22.5 ]
# [22.5 161.25 ]]
# cov_matrix[0,1] = Cov(X,Y) = 22.5, matching the manual calculationWhere This Shows Up in Deep Learning
- PCA / dimensionality reduction: built directly on the covariance matrix's eigendecomposition.
- Batch normalization: conceptually related — normalizing activations reduces problematic covariate shift between layers during training (the intuition behind why BatchNorm helps, though the precise mechanism is still actively debated in research).
- Feature redundancy detection: highly correlated (covarying) features often carry redundant information, a useful signal during feature engineering even for deep learning pipelines that otherwise learn features automatically.
Common Mistakes
- Interpreting covariance's raw magnitude as "strength" of relationship — its scale depends on the units of \(X\) and \(Y\), which is exactly why correlation (covariance normalized by both standard deviations) is used for comparing relationship strength across different variable pairs.
- Confusing covariance with causation — a strong positive covariance between two variables says nothing about whether one causes the other.
Interview Relevance
Q: "How does PCA use the covariance matrix?" PCA computes the covariance matrix of the (mean-centered) features, then finds its eigenvectors and eigenvalues. The eigenvectors give the directions of maximum variance (the principal components), and the eigenvalues rank how much variance each direction explains — letting you keep only the top few directions for dimensionality reduction while retaining most of the information.
Practice Question
If two features have a covariance close to zero, does that guarantee they're unrelated? (Hint: think about what kind of relationship covariance is specifically designed to detect.)
