Eigenvectors

An eigenvector is the "special direction" paired with each eigenvalue — a vector that a matrix scales but never rotates off its own line. Together, eigenvalues and eigenvectors describe a matrix's fundamental behavior more compactly than its raw entries do.

Definition — Continuing from Eigenvalues

Once you know an eigenvalue \(\lambda\) (see Eigenvalues), you find its eigenvector by solving \((\mathbf{A}-\lambda\mathbf{I})\mathbf{v} = \mathbf{0}\) for \(\mathbf{v}\).

Numerical Example — Continuing the Previous Matrix

For \(\mathbf{A} = \begin{bmatrix}4 & 1\\2 & 3\end{bmatrix}\) with \(\lambda_1 = 5\):

Any vector of the form \([t, t]\) works — the standard convention is to report the normalized eigenvector \(\mathbf{v}_1 = \left[\frac{1}{\sqrt2}, \frac{1}{\sqrt2}\right]\). For \(\lambda_2 = 2\), the same process gives \(\mathbf{v}_2 = \left[\frac{1}{\sqrt2}, -\frac{1}{\sqrt2}\right]\).

Geometric Intuition

Code

import numpy as np
A = np.array([[4., 1.], [2., 3.]])
eigenvalues, eigenvectors = np.linalg.eig(A)
print(eigenvalues)      # [5. 2.]
print(eigenvectors)     # columns are the eigenvectors, normalized to unit length

# Verify: A @ v should equal lambda * v
v1 = eigenvectors[:, 0]
print(A @ v1, eigenvalues[0] * v1)   # should match

Where This Shows Up in Deep Learning

PCA's principal components are precisely the eigenvectors of a dataset's covariance matrix, ranked by their eigenvalues (largest eigenvalue = direction of most variance in the data). This is how PCA compresses high-dimensional features into a smaller number of the most informative directions — the mechanics of dimensionality reduction used before deep learning existed, and still used today for visualization and preprocessing.

Common Mistakes

• Treating an eigenvector as unique — any non-zero scalar multiple of an eigenvector is also a valid eigenvector for the same eigenvalue. Libraries return one, typically normalized to unit length, by convention.
• Assuming eigenvectors for different eigenvalues are always perpendicular — this is only guaranteed for symmetric matrices (like covariance matrices), not matrices in general.

Interview Relevance

Q: "How does PCA use eigenvectors?" PCA computes the covariance matrix of the (centered) data, then finds its eigenvectors and eigenvalues. The eigenvectors are the principal component directions; the eigenvalues rank how much variance each direction captures. Keeping only the top-\(k\) eigenvectors by eigenvalue gives a \(k\)-dimensional compression that preserves as much variance as possible.

Practice Question

If a matrix has eigenvalue \(\lambda = 1\) for some eigenvector \(\mathbf{v}\), what does that tell you about how the matrix transforms \(\mathbf{v}\) specifically?

Related DL Notes

• Eigenvalues
• Vector Spaces
• Linear Transformations