The gradient of a function is a vector of its partial derivatives — one per input variable — that points in the direction where the function increases fastest. It's the compass every optimization algorithm in ML follows.
Formula
\(\nabla f\) ("nabla f", or "grad f") is a vector; each component is the partial derivative of \(f\) with respect to one variable, holding the others constant. It always points in the direction of steepest increase — which is exactly why ML moves in the opposite direction to minimize error.
Geometric Intuition — Contour Lines
Contour lines are curves of equal function value. The gradient always points perpendicular to the contour, straight uphill.
Numerical Example
For \(f(x,y) = x^2 + y^2\), the partial derivatives are \(\frac{\partial f}{\partial x}=2x\) and \(\frac{\partial f}{\partial y}=2y\), so \(\nabla f = [2x, 2y]\).
import numpy as np
def f(x, y):
return x**2 + y**2
def gradient(x, y):
return np.array([2*x, 2*y]) # analytical gradient for this function
g = gradient(3, 4)
print(g) # [6 8]
print(np.linalg.norm(g)) # 10.0Why This Matters for ML
Training a model means minimizing a loss function — and the gradient tells you exactly which direction makes the loss worse fastest. So every optimizer takes a step in the negative gradient direction, which is precisely what Gradient Descent does.
Common Mistakes
- Confusing the gradient's direction (steepest increase) with the direction you actually move in during training (steepest decrease — the negative gradient).
- Thinking the gradient is a single number — for a function of multiple variables, it's always a vector, one component per input variable.
Interview Relevance
Q: "What does it mean when the gradient is the zero vector?" It means you're at a flat point of the function — a local minimum, maximum, or saddle point. Optimizers stop making progress there, which is exactly the signal gradient descent uses to know it has (approximately) converged.
Practice Question
For \(f(x,y) = x^2 + 2y^2\), derive \(\nabla f\) and evaluate it at \((x,y) = (1, 2)\).
