An eigenvalue of a matrix answers a specific question: for which special vectors does this matrix act as a pure scaling operation — stretching or shrinking, but never changing direction? Those scaling factors are the eigenvalues.
Definition
\(\mathbf{A}\) is a square matrix, \(\mathbf{v}\) is a non-zero vector called an eigenvector (covered fully in the next note), and \(\lambda\) (a scalar) is the corresponding eigenvalue. The equation says: applying \(\mathbf{A}\) to \(\mathbf{v}\) gives back the same vector, just scaled by \(\lambda\) — no rotation, no change of direction.
Finding Eigenvalues — The Characteristic Equation
Rearranging \(\mathbf{A}\mathbf{v}=\lambda\mathbf{v}\) gives \((\mathbf{A}-\lambda\mathbf{I})\mathbf{v} = \mathbf{0}\). For a non-zero \(\mathbf{v}\) to satisfy this, the matrix \((\mathbf{A}-\lambda\mathbf{I})\) must be singular — meaning its determinant is zero (see Determinant). Solving that equation for \(\lambda\) gives the eigenvalues.
Numerical Example
Code
import numpy as np
A = np.array([[4., 1.], [2., 3.]])
eigenvalues, eigenvectors = np.linalg.eig(A)
print(eigenvalues) # [5. 2.]Where This Shows Up in Deep Learning
- PCA (Principal Component Analysis): the principal components are the eigenvectors of the data's covariance matrix, and the eigenvalues tell you how much variance each component explains — used for dimensionality reduction.
- Weight matrix conditioning: the spread of a weight matrix's eigenvalues affects how gradients scale as they pass through many layers — very large or very small eigenvalues are linked to exploding/vanishing gradients.
- Determinant shortcut: \(\det(\mathbf{A})\) equals the product of all its eigenvalues — a quick sanity link back to Determinant.
Common Mistakes
- Assuming every matrix has real eigenvalues — some (e.g. pure rotation matrices) have complex eigenvalues, since there's no real vector whose direction is preserved.
- Confusing eigenvalues with the matrix's individual entries — eigenvalues are a global property of how the matrix transforms space, not directly read off the matrix (except for triangular matrices, where they are the diagonal entries).
Interview Relevance
Q: "What's the intuitive meaning of an eigenvalue?" It's the scaling factor applied to a special direction (the eigenvector) that a matrix leaves unrotated. Most vectors get both rotated and scaled by a matrix; eigenvectors are the exceptions — they only get scaled, by exactly their eigenvalue.
Practice Question
Without solving fully, explain why a diagonal matrix's eigenvalues are simply its diagonal entries. (Hint: what does \(\mathbf{A}\mathbf{v}\) look like for a diagonal \(\mathbf{A}\) and \(\mathbf{v}\) a standard basis vector?)
