A linear transformation is a function that maps vectors to vectors while preserving addition and scalar multiplication — and every matrix multiplication you'll ever write, including every layer of a neural network before its activation function, is one.
Definition
Any function \(T\) satisfying both properties is a linear transformation, and every such transformation on finite-dimensional spaces can be represented as \(T(\mathbf{x}) = \mathbf{A}\mathbf{x}\) for some matrix \(\mathbf{A}\). This is why "matrix" and "linear transformation" are used almost interchangeably in this context.
Common Linear Transformations in 2-D
| Transformation | Matrix |
|---|---|
| Scale by factor \(k\) | \(\begin{bmatrix}k & 0\\0 & k\end{bmatrix}\) |
| Rotate by angle \(\theta\) | \(\begin{bmatrix}\cos\theta & -\sin\theta\\\sin\theta & \cos\theta\end{bmatrix}\) |
| Reflect across the x-axis | \(\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}\) |
| Project onto the x-axis | \(\begin{bmatrix}1 & 0\\0 & 0\end{bmatrix}\) |
Visualizing a Transformation
A linear transformation always maps the origin to itself and straight lines to straight lines — it can rotate, scale, shear or reflect, but never curve or bend space.
Code — Applying a Rotation
import numpy as np
theta = np.pi / 4 # 45 degrees
R = np.array([[np.cos(theta), -np.sin(theta)],
[np.sin(theta), np.cos(theta)]])
x = np.array([1, 0])
print(R @ x) # approximately [0.707, 0.707] -- rotated 45 degrees counterclockwiseWhy This Matters for Neural Networks — The Motivation for Activation Functions
A neural network layer without an activation function, \(\mathbf{y} = \mathbf{W}\mathbf{x}+\mathbf{b}\), is an affine transformation (a linear transformation plus a shift). Stack two such layers and you get \(\mathbf{W}_2(\mathbf{W}_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2\), which algebraically simplifies to a single linear transformation with a new combined weight matrix and bias. No matter how many purely linear layers you stack, the whole network collapses to one linear function — it can never learn a non-linear pattern. This is precisely why every layer in a real network is followed by a non-linear activation function (see the Activation Functions category next) — without it, depth adds no representational power at all.
Common Mistakes
- Assuming stacking more linear layers automatically increases a network's capacity — without non-linear activations between them, it doesn't; the layers collapse into one.
- Forgetting that a linear transformation always maps the zero vector to the zero vector — a transformation that doesn't (e.g. one with a constant offset) is affine, not purely linear, even though deep learning code casually calls both "linear layers."
Interview Relevance
Q: "Why do neural networks need activation functions between layers?" Without them, any stack of linear (matrix multiplication) layers mathematically collapses into a single linear transformation, regardless of depth. Non-linear activation functions are what let a deep network represent non-linear functions — they are the entire reason depth adds representational power.
Practice Question
Verify algebraically that stacking \(\mathbf{y} = \mathbf{W}_2(\mathbf{W}_1\mathbf{x})\) (no bias, no activation) is equivalent to a single linear transformation \(\mathbf{y}=\mathbf{W}'\mathbf{x}\). What is \(\mathbf{W}'\) in terms of \(\mathbf{W}_1\) and \(\mathbf{W}_2\)?
